{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"mixture-of-experts-locally","title":"Mixture of experts, and why it changes the sum","url":"https://addaly.com/learn/running-models-yourself/mixture-of-experts-locally","position":5,"of":82,"minutes":9,"answer":"A mixture-of-experts model costs memory like its total parameter count and generates at the speed of its active parameter count.","keyIdea":"A mixture-of-experts model costs memory like its total parameter count and generates at the speed of its active parameter count, which makes it the right choice on a machine with abundant slow RAM and the wrong one on a small GPU."},"body":{"format":"markdown","text":"## Two parameter counts, not one\n\nA dense model uses every weight for every token. A mixture-of-experts model does not. Each feed-forward block is replaced by a set of parallel blocks called experts, plus a small router that picks a few of them per token. A model described as 30B-A3B has about 30 billion total parameters and activates about 3 billion of them for any given token.\n\nThat gap is the whole story, and it splits the two numbers you care about:\n\n- **Total parameters** decide how much memory the weights occupy. All experts must be resident, because the router may choose any of them at any token.\n- **Active parameters** decide how much arithmetic and how much memory traffic happens per token, and therefore how fast it generates.\n\nSo a 30B-A3B model needs the memory of a 30B and generates at roughly the speed of a 3B. That is the trade: capacity is cheap, memory is not.\n\n## What this does to the arithmetic\n\nUse the same two sums as for a dense model, with one substitution.\n\nWeights: total parameters times bytes per weight. A 30B-A3B at Q4_K_M is about 18 GB. That does not fit in 16 GB of VRAM and does fit comfortably in 32 GB of system RAM.\n\n> Figure: The same thirty billion parameters, two architectures. Bars (tokens per second, one stream, 30 GB/s of system memory): Dense 30B at Q4, reads 18 GB per token 1.7; Dense 8B at Q4, reads 4.9 GB per token 6; MoE 30B-A3B at Q4, reads about 2.5 GB 12. Memory holds the total parameter count; speed follows the active one. Both 30B files occupy about 18 GB of RAM and neither fits a 16 GB card. Only one of them generates at a speed anybody will sit through, and it is the one the small GPU cannot hold.\n\nSpeed: the memory-bandwidth ceiling applies to the *active* bytes per token, not the whole file. On a machine with about 30 GB/s of usable system bandwidth, a dense 30B at Q4 reads 18 GB per token and manages under 2 tokens per second: unusable. The same 30B-A3B reads roughly 2 GB of expert weights plus the shared attention weights, and lands somewhere around 10 to 15 tokens per second. On the same machine. Reading the same total file.\n\nThat is the single most useful fact in this lesson, and it is why MoE models have become the practical choice for people with plenty of RAM and no large GPU. A 128 GB machine with slow memory can run a very large MoE at a speed that a dense model of the same total size could never reach.\n\n## Where the speed goes anyway\n\nThree caveats, all mechanical.\n\n**Routing is per token, not per request.** The set of active experts changes at every token, so you cannot keep only the popular experts in fast memory and stream the rest. Attempts to do this exist and cost more than they save on most hardware, because a cache miss stalls the whole token.\n\n**Batching breaks the saving.** With one request, the router touches a few experts. With sixty concurrent requests, different requests choose different experts, and across the batch you end up touching most of them. Aggregate throughput on an MoE therefore scales worse than the active-parameter count suggests. MoE is a large win for one user and a smaller one for a busy server.\n\n**Partial offload behaves differently.** The usual advice (offload as many layers as fit) is not optimal for MoE. The better split is to keep attention and the shared weights on the GPU and leave the expert tensors in system RAM, because the experts are large and each one is touched rarely. llama.cpp exposes this:\n\n```bash\nllama-server -m Qwen3-30B-A3B-Q4_K_M.gguf -ngl 99 \\\n  --n-cpu-moe 24 -c 8192 --flash-attn\n```\n\nThat keeps everything on the GPU except the expert tensors of 24 layers. On a 12 GB card with a 20 GB model, this routinely doubles or triples the speed compared with an even layer split. It is worth measuring both ways on your own machine, because the crossover depends on your PCIe bandwidth.\n\n## Quality\n\nAn MoE with N active parameters is not as good as a dense model with N parameters: the router costs something and the experts are less general. It is also not as good as a dense model with the same *total* parameters. The usual rule of thumb, which is a rule of thumb and not a law, places it near the geometric mean: a 30B-A3B behaves roughly like a dense 9B or 10B, while costing 30B of memory and 3B of bandwidth.\n\nWhether that is a good deal depends entirely on which resource you are short of. If you have 8 GB of RAM, an MoE is irrelevant. If you have 64 or 128 GB of ordinary system RAM and no GPU worth the name, it is the best option available to you by a wide margin.\n\n## The check before downloading\n\nRead the card for two numbers: total and active. If the card only gives one, the config file has `num_experts` and `num_experts_per_tok`, and the arithmetic follows. Then ask which of your two constraints binds (capacity or bandwidth) and choose accordingly. People who skip this either download 40 GB onto a 16 GB machine, or dismiss MoE models as too big while sitting on exactly the hardware they were designed for."},"check":{"question":"Someone with a 12 GB GPU and 64 GB of system RAM runs a 30B-A3B model at Q4 (about 18 GB) with a standard even layer offload and gets 4 tokens per second. What change is most likely to help, and why?","options":[{"text":"Keep attention and shared weights on the GPU and put the expert tensors in system RAM, because experts are large but each is touched rarely, so they cost the least when they are slow.","correct":true},{"text":"Reduce the context window, since the KV cache is competing with the experts for VRAM.","correct":false,"tempting":"Targets the smallest term; at moderate context the cache is a fraction of a gigabyte against an 18 GB weight file, so it is not what is forcing the offload."},{"text":"Switch to a dense 14B, because MoE routing overhead dominates on consumer hardware.","correct":false,"tempting":"Blames the router, which is a tiny fraction of the compute; the actual bottleneck is where the expert weights live, and this abandons the model rather than placing it well."},{"text":"Quantise further to Q3 so the whole model fits in 12 GB of VRAM and no offload is needed.","correct":false,"tempting":"Gets the arithmetic wrong (a 30B at Q3 is still about 14 GB) and pays real quality for a fit that does not happen."}]},"prev":{"slug":"picking-a-size","title":"What each size class can actually do","url":"https://addaly.com/learn/running-models-yourself/picking-a-size"},"next":{"slug":"base-instruct-and-reasoning-variants","title":"Base, instruct and reasoning: three different products","url":"https://addaly.com/learn/running-models-yourself/base-instruct-and-reasoning-variants"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Mixture of experts, and why it changes the sum', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/mixture-of-experts-locally (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Mixture of experts, and why it changes the sum","courseTitle":"Running Models Yourself","module":{"index":1,"slug":"deciding-to-run-it-yourself","title":"Deciding to run it yourself"},"answer":null,"keyIdea":"A mixture-of-experts model costs memory like its total parameter count and generates at the speed of its active parameter count, which makes it the right choice on a machine with abundant slow RAM and the wrong one on a small GPU.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens per second, one stream, 30 GB/s of system memory","title":"The same thirty billion parameters, two architectures","series":[{"label":"Dense 30B at Q4, reads 18 GB per token","value":1.7},{"label":"Dense 8B at Q4, reads 4.9 GB per token","value":6},{"label":"MoE 30B-A3B at Q4, reads about 2.5 GB","value":12,"highlight":true}],"caption":"Memory holds the total parameter count; speed follows the active one. Both 30B files occupy about 18 GB of RAM and neither fits a 16 GB card. Only one of them generates at a speed anybody will sit through, and it is the one the small GPU cannot hold."},"alt":"Figure: The same thirty billion parameters, two architectures. Bars (tokens per second, one stream, 30 GB/s of system memory): Dense 30B at Q4, reads 18 GB per token 1.7; Dense 8B at Q4, reads 4.9 GB per token 6; MoE 30B-A3B at Q4, reads about 2.5 GB 12. Memory holds the total parameter count; speed follows the active one. Both 30B files occupy about 18 GB of RAM and neither fits a 16 GB card. Only one of them generates at a speed anybody will sit through, and it is the one the small GPU cannot hold."}},"blocks":[{"t":"h2","id":"two-parameter-counts-not-one","runs":[{"text":"Two parameter counts, not one"}]},{"t":"p","runs":[{"text":"A dense model uses every weight for every token. A mixture-of-experts model does not. Each feed-forward block is replaced by a set of parallel blocks called experts, plus a small router that picks a few of them per token. A model described as 30B-A3B has about 30 billion total parameters and activates about 3 billion of them for any given token."}]},{"t":"p","runs":[{"text":"That gap is the whole story, and it splits the two numbers you care about:"}]},{"t":"ul","items":[[{"text":"Total parameters","b":true},{"text":" decide how much memory the weights occupy. All experts must be resident, because the router may choose any of them at any token."}],[{"text":"Active parameters","b":true},{"text":" decide how much arithmetic and how much memory traffic happens per token, and therefore how fast it generates."}]]},{"t":"p","runs":[{"text":"So a 30B-A3B model needs the memory of a 30B and generates at roughly the speed of a 3B. That is the trade: capacity is cheap, memory is not."}]},{"t":"h2","id":"what-this-does-to-the-arithmetic","runs":[{"text":"What this does to the arithmetic"}]},{"t":"p","runs":[{"text":"Use the same two sums as for a dense model, with one substitution."}]},{"t":"p","runs":[{"text":"Weights: total parameters times bytes per weight. A 30B-A3B at Q4_K_M is about 18 GB. That does not fit in 16 GB of VRAM and does fit comfortably in 32 GB of system RAM."}]},{"t":"p","runs":[{"text":"Speed: the memory-bandwidth ceiling applies to the "},{"text":"active","i":true},{"text":" bytes per token, not the whole file. On a machine with about 30 GB/s of usable system bandwidth, a dense 30B at Q4 reads 18 GB per token and manages under 2 tokens per second: unusable. The same 30B-A3B reads roughly 2 GB of expert weights plus the shared attention weights, and lands somewhere around 10 to 15 tokens per second. On the same machine. Reading the same total file."}]},{"t":"p","runs":[{"text":"That is the single most useful fact in this lesson, and it is why MoE models have become the practical choice for people with plenty of RAM and no large GPU. A 128 GB machine with slow memory can run a very large MoE at a speed that a dense model of the same total size could never reach."}]},{"t":"h2","id":"where-the-speed-goes-anyway","runs":[{"text":"Where the speed goes anyway"}]},{"t":"p","runs":[{"text":"Three caveats, all mechanical."}]},{"t":"p","runs":[{"text":"Routing is per token, not per request.","b":true},{"text":" The set of active experts changes at every token, so you cannot keep only the popular experts in fast memory and stream the rest. Attempts to do this exist and cost more than they save on most hardware, because a cache miss stalls the whole token."}]},{"t":"p","runs":[{"text":"Batching breaks the saving.","b":true},{"text":" With one request, the router touches a few experts. With sixty concurrent requests, different requests choose different experts, and across the batch you end up touching most of them. Aggregate throughput on an MoE therefore scales worse than the active-parameter count suggests. MoE is a large win for one user and a smaller one for a busy server."}]},{"t":"p","runs":[{"text":"Partial offload behaves differently.","b":true},{"text":" The usual advice (offload as many layers as fit) is not optimal for MoE. The better split is to keep attention and the shared weights on the GPU and leave the expert tensors in system RAM, because the experts are large and each one is touched rarely. llama.cpp exposes this:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m Qwen3-30B-A3B-Q4_K_M.gguf -ngl 99 \\\n  --n-cpu-moe 24 -c 8192 --flash-attn"},{"t":"p","runs":[{"text":"That keeps everything on the GPU except the expert tensors of 24 layers. On a 12 GB card with a 20 GB model, this routinely doubles or triples the speed compared with an even layer split. It is worth measuring both ways on your own machine, because the crossover depends on your PCIe bandwidth."}]},{"t":"h2","id":"quality","runs":[{"text":"Quality"}]},{"t":"p","runs":[{"text":"An MoE with N active parameters is not as good as a dense model with N parameters: the router costs something and the experts are less general. It is also not as good as a dense model with the same "},{"text":"total","i":true},{"text":" parameters. The usual rule of thumb, which is a rule of thumb and not a law, places it near the geometric mean: a 30B-A3B behaves roughly like a dense 9B or 10B, while costing 30B of memory and 3B of bandwidth."}]},{"t":"p","runs":[{"text":"Whether that is a good deal depends entirely on which resource you are short of. If you have 8 GB of RAM, an MoE is irrelevant. If you have 64 or 128 GB of ordinary system RAM and no GPU worth the name, it is the best option available to you by a wide margin."}]},{"t":"h2","id":"the-check-before-downloading","runs":[{"text":"The check before downloading"}]},{"t":"p","runs":[{"text":"Read the card for two numbers: total and active. If the card only gives one, the config file has "},{"text":"num_experts","code":true},{"text":" and "},{"text":"num_experts_per_tok","code":true},{"text":", and the arithmetic follows. Then ask which of your two constraints binds (capacity or bandwidth) and choose accordingly. People who skip this either download 40 GB onto a 16 GB machine, or dismiss MoE models as too big while sitting on exactly the hardware they were designed for."}]}],"check":{"question":[{"text":"Someone with a 12 GB GPU and 64 GB of system RAM runs a 30B-A3B model at Q4 (about 18 GB) with a standard even layer offload and gets 4 tokens per second. What change is most likely to help, and why?"}],"options":[{"text":[{"text":"Keep attention and shared weights on the GPU and put the expert tensors in system RAM, because experts are large but each is touched rarely, so they cost the least when they are slow."}],"correct":true,"tempting":null},{"text":[{"text":"Reduce the context window, since the KV cache is competing with the experts for VRAM."}],"correct":false,"tempting":[{"text":"Targets the smallest term; at moderate context the cache is a fraction of a gigabyte against an 18 GB weight file, so it is not what is forcing the offload."}]},{"text":[{"text":"Switch to a dense 14B, because MoE routing overhead dominates on consumer hardware."}],"correct":false,"tempting":[{"text":"Blames the router, which is a tiny fraction of the compute; the actual bottleneck is where the expert weights live, and this abandons the model rather than placing it well."}]},{"text":[{"text":"Quantise further to Q3 so the whole model fits in 12 GB of VRAM and no offload is needed."}],"correct":false,"tempting":[{"text":"Gets the arithmetic wrong (a 30B at Q3 is still about 14 GB) and pays real quality for a fit that does not happen."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"picking-a-size","title":"What each size class can actually do","minutes":10},"next":{"slug":"base-instruct-and-reasoning-variants","title":"Base, instruct and reasoning: three different products","minutes":8}}}