{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"mixture-of-experts","title":"Mixture of experts, and the memory it still needs","url":"https://addaly.com/learn/open-models/mixture-of-experts","position":27,"of":71,"minutes":8,"answer":"A mixture-of-experts model computes like its active parameter count and occupies memory like its total, so it is outstanding on a machine with spare memory and useless on one without.","keyIdea":"A mixture-of-experts model computes like its active parameter count and occupies memory like its total, so it is outstanding on a machine with spare memory and useless on one without."},"body":{"format":"markdown","text":"## Two parameter counts, and only one of them is speed\n\nA mixture-of-experts model replaces the single feed-forward block in each layer with many of them (the experts) plus a small router that picks a few per token. A model described as 30B-A3B has around 30 billion total parameters and around 3 billion active for any given token.\n\nThe consequence people want is real: **compute scales with active parameters**. A 30B-A3B model generates at roughly the speed of a 3B dense model, which is a large multiple faster than a 30B dense model would be.\n\nThe consequence people forget is equally real: **memory scales with total parameters**. The router can send the next token to any expert, so every expert must be resident. That 30B-A3B model needs about 18 GB at 4-bit quantisation, the same as a 30B dense model. You get 3B speed only if you can hold 30B.\n\nThat single sentence decides whether a mixture-of-experts model is the best or the worst choice you could make. On a 32 GB workstation it is excellent. On an 8 GB laptop it is unusable, while a 4B dense model works fine.\n\n## How the routing works\n\nFor each token, at each layer, the router scores the experts and selects the top few: commonly two of eight, or eight of a hundred and twenty-eight in the larger designs. Only those run. Routing is learned during training, and it does not divide neatly by topic: experts do not correspond to physics and poetry, whatever the name suggests. They are learned specialisations of a kind nobody has cleanly interpreted.\n\nSome architectures add a shared expert that runs for every token, holding the general capability while the routed experts specialise.\n\nTraining these models is harder than training dense ones, because the router can collapse onto a few popular experts and leave the rest untrained. Load-balancing losses exist to prevent that, and they are a large part of what the DeepSeek papers describe in detail.\n\n## What you meet in practice\n\nThe open MoE models include Qwen3's 30B-A3B and larger variants, DeepSeek's V3 line at around 671B total with roughly 37B active, and Mixtral's earlier 8x7B and 8x22B. The very large ones are rental propositions, not self-hosting ones; the DeepSeek flagship needs several hundred gigabytes of memory before any context.\n\nTwo operational quirks are worth knowing.\n\n**Batching is less efficient.** In a batch, different tokens route to different experts, so the server ends up touching most of the weights anyway. Throughput gains at high concurrency are smaller than the single-stream speedup suggests.\n\n**Expert offloading works, slowly.** llama.cpp and others can keep some experts in CPU memory or on disk and fetch them as needed. It makes a model that would not fit run at all, at a speed that depends entirely on the transfer rate. Useful for batch work overnight, painful interactively.\n\n## The choice, stated plainly\n\nAsk one question: how much memory do I have?\n\n- **Plenty of memory, want speed.** MoE is the best value in the ecosystem. A 30B-A3B on a machine with 32 GB gives you close to 30B-class quality at close to 3B-class speed.\n- **Memory constrained.** Dense, every time. A 4B dense model that fits beats a 30B-A3B that does not, and the comparison is not close, because the alternative to fitting is not running.\n- **Renting.** It barely matters to you; the provider's price already reflects their side of this trade, and MoE models are often priced attractively for their quality precisely because they are cheap to serve.\n\n> The recurring mistake is reading 30B-A3B as if the model were 3B. Read it as a 30B model that runs quickly. The first number is the one your hardware has to satisfy."},"check":{"question":"Someone with a 16 GB laptop reads that a 30B-A3B model has only 3B active parameters and expects it to run like a 3B model. What will actually happen?","options":[{"text":"It will run at 3B speed but produce 30B-quality output, since only the active experts matter at inference.","correct":false,"tempting":"Describes the compute side correctly while ignoring that all experts must be resident before any of them can run."},{"text":"It will run correctly but use four times the memory of a 3B model, since only the routed experts are loaded.","correct":false,"tempting":"Assumes selective loading; the router can pick any expert for the next token, so partial residency means constant reloading."},{"text":"All experts must be resident because routing is per token, so around 18 GB at 4-bit will not fit and the model either fails or offloads to disk and crawls.","correct":true},{"text":"It will work, because mixture-of-experts models quantise better than dense models of the same total size.","correct":false,"tempting":"MoE models are in fact somewhat more sensitive to aggressive quantisation, and in any case the total size is what has to fit."}]},"prev":{"slug":"context-length-in-practice","title":"What a 128k context window really means","url":"https://addaly.com/learn/open-models/context-length-in-practice"},"next":{"slug":"multimodal-open-models","title":"Models that can see","url":"https://addaly.com/learn/open-models/multimodal-open-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Mixture of experts, and the memory it still needs', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/mixture-of-experts (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Mixture of experts, and the memory it still needs","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"A mixture-of-experts model computes like its active parameter count and occupies memory like its total, so it is outstanding on a machine with spare memory and useless on one without.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-parameter-counts-and-only-one-of-them-is-speed","runs":[{"text":"Two parameter counts, and only one of them is speed"}]},{"t":"p","runs":[{"text":"A mixture-of-experts model replaces the single feed-forward block in each layer with many of them (the experts) plus a small router that picks a few per token. A model described as 30B-A3B has around 30 billion total parameters and around 3 billion active for any given token."}]},{"t":"p","runs":[{"text":"The consequence people want is real: "},{"text":"compute scales with active parameters","b":true},{"text":". A 30B-A3B model generates at roughly the speed of a 3B dense model, which is a large multiple faster than a 30B dense model would be."}]},{"t":"p","runs":[{"text":"The consequence people forget is equally real: "},{"text":"memory scales with total parameters","b":true},{"text":". The router can send the next token to any expert, so every expert must be resident. That 30B-A3B model needs about 18 GB at 4-bit quantisation, the same as a 30B dense model. You get 3B speed only if you can hold 30B."}]},{"t":"p","runs":[{"text":"That single sentence decides whether a mixture-of-experts model is the best or the worst choice you could make. On a 32 GB workstation it is excellent. On an 8 GB laptop it is unusable, while a 4B dense model works fine."}]},{"t":"h2","id":"how-the-routing-works","runs":[{"text":"How the routing works"}]},{"t":"p","runs":[{"text":"For each token, at each layer, the router scores the experts and selects the top few: commonly two of eight, or eight of a hundred and twenty-eight in the larger designs. Only those run. Routing is learned during training, and it does not divide neatly by topic: experts do not correspond to physics and poetry, whatever the name suggests. They are learned specialisations of a kind nobody has cleanly interpreted."}]},{"t":"p","runs":[{"text":"Some architectures add a shared expert that runs for every token, holding the general capability while the routed experts specialise."}]},{"t":"p","runs":[{"text":"Training these models is harder than training dense ones, because the router can collapse onto a few popular experts and leave the rest untrained. Load-balancing losses exist to prevent that, and they are a large part of what the DeepSeek papers describe in detail."}]},{"t":"h2","id":"what-you-meet-in-practice","runs":[{"text":"What you meet in practice"}]},{"t":"p","runs":[{"text":"The open MoE models include Qwen3’s 30B-A3B and larger variants, DeepSeek’s V3 line at around 671B total with roughly 37B active, and Mixtral’s earlier 8x7B and 8x22B. The very large ones are rental propositions, not self-hosting ones; the DeepSeek flagship needs several hundred gigabytes of memory before any context."}]},{"t":"p","runs":[{"text":"Two operational quirks are worth knowing."}]},{"t":"p","runs":[{"text":"Batching is less efficient.","b":true},{"text":" In a batch, different tokens route to different experts, so the server ends up touching most of the weights anyway. Throughput gains at high concurrency are smaller than the single-stream speedup suggests."}]},{"t":"p","runs":[{"text":"Expert offloading works, slowly.","b":true},{"text":" llama.cpp and others can keep some experts in CPU memory or on disk and fetch them as needed. It makes a model that would not fit run at all, at a speed that depends entirely on the transfer rate. Useful for batch work overnight, painful interactively."}]},{"t":"h2","id":"the-choice-stated-plainly","runs":[{"text":"The choice, stated plainly"}]},{"t":"p","runs":[{"text":"Ask one question: how much memory do I have?"}]},{"t":"ul","items":[[{"text":"Plenty of memory, want speed.","b":true},{"text":" MoE is the best value in the ecosystem. A 30B-A3B on a machine with 32 GB gives you close to 30B-class quality at close to 3B-class speed."}],[{"text":"Memory constrained.","b":true},{"text":" Dense, every time. A 4B dense model that fits beats a 30B-A3B that does not, and the comparison is not close, because the alternative to fitting is not running."}],[{"text":"Renting.","b":true},{"text":" It barely matters to you; the provider’s price already reflects their side of this trade, and MoE models are often priced attractively for their quality precisely because they are cheap to serve."}]]},{"t":"quote","runs":[{"text":"The recurring mistake is reading 30B-A3B as if the model were 3B. Read it as a 30B model that runs quickly. The first number is the one your hardware has to satisfy."}]}],"check":{"question":[{"text":"Someone with a 16 GB laptop reads that a 30B-A3B model has only 3B active parameters and expects it to run like a 3B model. What will actually happen?"}],"options":[{"text":[{"text":"It will run at 3B speed but produce 30B-quality output, since only the active experts matter at inference."}],"correct":false,"tempting":[{"text":"Describes the compute side correctly while ignoring that all experts must be resident before any of them can run."}]},{"text":[{"text":"It will run correctly but use four times the memory of a 3B model, since only the routed experts are loaded."}],"correct":false,"tempting":[{"text":"Assumes selective loading; the router can pick any expert for the next token, so partial residency means constant reloading."}]},{"text":[{"text":"All experts must be resident because routing is per token, so around 18 GB at 4-bit will not fit and the model either fails or offloads to disk and crawls."}],"correct":true,"tempting":null},{"text":[{"text":"It will work, because mixture-of-experts models quantise better than dense models of the same total size."}],"correct":false,"tempting":[{"text":"MoE models are in fact somewhat more sensitive to aggressive quantisation, and in any case the total size is what has to fit."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"context-length-in-practice","title":"What a 128k context window really means","minutes":9},"next":{"slug":"multimodal-open-models","title":"Models that can see","minutes":9}}}