{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"mixture-of-experts","title":"Mixture of experts: big model, small bill","url":"https://addaly.com/learn/how-llms-work/mixture-of-experts","position":18,"of":78,"minutes":9,"answer":"A sparse mixture-of-experts model has a large parameter count but activates only a fraction per token, which decouples what it knows from what it costs to run.","keyIdea":"A sparse mixture-of-experts model has a large parameter count but activates only a fraction per token, which decouples what it knows from what it costs to run."},"body":{"format":"markdown","text":"## Two different numbers, both called size\n\nYou will see a model described as \"671B total, 37B active\". That is not marketing sleight of hand; it is the defining property of a mixture-of-experts architecture, and it changes how you should reason about cost.\n\nIn a dense model, every parameter participates in every token. In a sparse mixture-of-experts model, the feed-forward half of each block is replaced by many parallel feed-forward networks (the experts) plus a small router that picks a couple of them per token. The rest sit idle for that token.\n\n## The mechanism\n\nAt each position, in each block:\n\n1. A router (a single small matrix) scores the incoming vector against every expert.\n2. The top *k* experts are selected, commonly two out of eight, or eight out of 256 in the larger designs.\n3. Only those experts run.\n4. Their outputs are combined, weighted by the router's scores, and added to the residual stream.\n\nAttention is unchanged and remains dense. Only the feed-forward half is made sparse, which is sensible given that is where most of the parameters were.\n\n## What it buys and what it costs\n\n**Buys:** far more parameters (more of the pattern-to-contribution pairs from the previous lesson) for roughly the compute of a much smaller dense model. Mixtral 8×7B holds about 47 billion parameters and activates around 13 billion per token. DeepSeek-V3 holds 671 billion and activates 37 billion.\n\n**Costs:** the memory to hold *all* of them. You cannot serve a 671B model on hardware sized for a 37B one. The weights must be resident even though most are unused per token, so mixture-of-experts trades arithmetic for memory capacity and bandwidth. This is why these models are common in large hosted deployments and rare on laptops.\n\n> Figure: Which of the two numbers answers which question. Total parameters: 671 billion: Decides whether you can run it at all; Every expert must stay resident; the next token may route anywhere; The better guide to quality; A sparse model still trails a dense one of the same total size. Active parameters: 37 billion: Decides speed and price per token; Only the routed experts are read for a given token; Attention is dense and counted in both figures; In a batch, different tokens route differently, so a busy server reads more. A parameter count has stopped being a proxy for price or speed. Ask which figure a vendor is quoting, and expect a sparse model to behave like a dense one somewhere between the two: nearer the geometric mean than either end.\n\n**Also costs:** training complexity. Experts can collapse: the router sends everything to a favourite few while the rest never learn. The standard fix is an auxiliary load-balancing loss that penalises uneven routing, with newer designs adjusting per-expert biases instead to avoid distorting the main objective. Capacity limits, where an overloaded expert drops tokens, are another live source of subtle degradation.\n\n## What the experts are not\n\nThe word \"expert\" invites a wrong picture: one for medicine, one for French, one for code. Measurements do not support that. Routing tends to correlate with surface features (token identity, syntactic role, formatting) far more than with topic, and it varies by layer. Some designs add a *shared* expert that every token passes through, on the theory that common knowledge should not be duplicated across specialists.\n\nTreat \"expert\" as a name for a subnetwork, not a description of its contents.\n\n## Reading a model's numbers correctly\n\nWhen you compare models, ask which number is quoted.\n\n- For **quality**, total parameters is the more relevant figure, though a sparse model generally underperforms a dense model of the same *total* size.\n- For **speed and price per token**, active parameters is what matters.\n- For **whether you can run it at all**, total parameters is what matters, since all of it must be in memory.\n\nA useful rough guide from the literature: a sparse model tends to behave roughly like a dense model somewhere between its active and total sizes: closer to the geometric mean than to either end. Treat that as a rule of thumb, not a law.\n\n## Why this shape took over\n\nThe economics are compelling. Serving costs scale with active parameters, and capability scales with total parameters, so a lab that can afford the memory gets a better model per unit of serving cost. That is a large part of why frontier models became cheaper per token over 2024 and 2025 without becoming less capable, alongside distillation and better hardware utilisation.\n\nFor you as a user it means one specific thing: a model's parameter count has stopped being a reliable proxy for its price or its speed. Check the active count if the vendor publishes it, and otherwise measure latency yourself."},"check":{"question":"A sparse model with 140 billion total and 20 billion active parameters is proposed for a self-hosted deployment on a server with 48 GB of GPU memory. What is the problem?","options":[{"text":"Sparse routing requires more memory bandwidth per token than a dense model of the same active size.","correct":false,"tempting":"Bandwidth is genuinely a constraint in serving, and routing does add irregular access patterns, so this sounds technically sharp."},{"text":"All 140 billion parameters must be resident in memory even though only 20 billion run per token, so at fp16 the weights alone need around 280 GB.","correct":true},{"text":"The router adds a sequential step that cannot be batched, making throughput unacceptable.","correct":false,"tempting":"Routing is an extra step, and sequential bottlenecks are a real serving concern, which makes this plausible if you have not seen the implementation."},{"text":"Sparse models cannot be quantised, so the usual memory savings are unavailable.","correct":false,"tempting":"Quantisation is the standard escape from memory limits, so assuming it is unavailable explains the difficulty neatly."}]},"prev":{"slug":"from-final-vector-to-logits","title":"From the final vector to a score for every word","url":"https://addaly.com/learn/how-llms-work/from-final-vector-to-logits"},"next":{"slug":"kv-cache-and-attention-variants","title":"The key-value cache, and the engineering built around it","url":"https://addaly.com/learn/how-llms-work/kv-cache-and-attention-variants"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Mixture of experts: big model, small bill', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/mixture-of-experts (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Mixture of experts: big model, small bill","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"A sparse mixture-of-experts model has a large parameter count but activates only a fraction per token, which decouples what it knows from what it costs to run.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Total parameters: 671 billion","items":["Decides whether you can run it at all","Every expert must stay resident; the next token may route anywhere","The better guide to quality","A sparse model still trails a dense one of the same total size"]},"type":"compare","right":{"title":"Active parameters: 37 billion","items":["Decides speed and price per token","Only the routed experts are read for a given token","Attention is dense and counted in both figures","In a batch, different tokens route differently, so a busy server reads more"]},"title":"Which of the two numbers answers which question","caption":"A parameter count has stopped being a proxy for price or speed. Ask which figure a vendor is quoting, and expect a sparse model to behave like a dense one somewhere between the two: nearer the geometric mean than either end."},"alt":"Figure: Which of the two numbers answers which question. Total parameters: 671 billion: Decides whether you can run it at all; Every expert must stay resident; the next token may route anywhere; The better guide to quality; A sparse model still trails a dense one of the same total size. Active parameters: 37 billion: Decides speed and price per token; Only the routed experts are read for a given token; Attention is dense and counted in both figures; In a batch, different tokens route differently, so a busy server reads more. A parameter count has stopped being a proxy for price or speed. Ask which figure a vendor is quoting, and expect a sparse model to behave like a dense one somewhere between the two: nearer the geometric mean than either end."}},"blocks":[{"t":"h2","id":"two-different-numbers-both-called-size","runs":[{"text":"Two different numbers, both called size"}]},{"t":"p","runs":[{"text":"You will see a model described as “671B total, 37B active”. That is not marketing sleight of hand; it is the defining property of a mixture-of-experts architecture, and it changes how you should reason about cost."}]},{"t":"p","runs":[{"text":"In a dense model, every parameter participates in every token. In a sparse mixture-of-experts model, the feed-forward half of each block is replaced by many parallel feed-forward networks (the experts) plus a small router that picks a couple of them per token. The rest sit idle for that token."}]},{"t":"h2","id":"the-mechanism","runs":[{"text":"The mechanism"}]},{"t":"p","runs":[{"text":"At each position, in each block:"}]},{"t":"ol","start":1,"items":[[{"text":"A router (a single small matrix) scores the incoming vector against every expert."}],[{"text":"The top "},{"text":"k","i":true},{"text":" experts are selected, commonly two out of eight, or eight out of 256 in the larger designs."}],[{"text":"Only those experts run."}],[{"text":"Their outputs are combined, weighted by the router’s scores, and added to the residual stream."}]]},{"t":"p","runs":[{"text":"Attention is unchanged and remains dense. Only the feed-forward half is made sparse, which is sensible given that is where most of the parameters were."}]},{"t":"h2","id":"what-it-buys-and-what-it-costs","runs":[{"text":"What it buys and what it costs"}]},{"t":"p","runs":[{"text":"Buys:","b":true},{"text":" far more parameters (more of the pattern-to-contribution pairs from the previous lesson) for roughly the compute of a much smaller dense model. Mixtral 8×7B holds about 47 billion parameters and activates around 13 billion per token. DeepSeek-V3 holds 671 billion and activates 37 billion."}]},{"t":"p","runs":[{"text":"Costs:","b":true},{"text":" the memory to hold "},{"text":"all","i":true},{"text":" of them. You cannot serve a 671B model on hardware sized for a 37B one. The weights must be resident even though most are unused per token, so mixture-of-experts trades arithmetic for memory capacity and bandwidth. This is why these models are common in large hosted deployments and rare on laptops."}]},{"t":"p","runs":[{"text":"Also costs:","b":true},{"text":" training complexity. Experts can collapse: the router sends everything to a favourite few while the rest never learn. The standard fix is an auxiliary load-balancing loss that penalises uneven routing, with newer designs adjusting per-expert biases instead to avoid distorting the main objective. Capacity limits, where an overloaded expert drops tokens, are another live source of subtle degradation."}]},{"t":"h2","id":"what-the-experts-are-not","runs":[{"text":"What the experts are not"}]},{"t":"p","runs":[{"text":"The word “expert” invites a wrong picture: one for medicine, one for French, one for code. Measurements do not support that. Routing tends to correlate with surface features (token identity, syntactic role, formatting) far more than with topic, and it varies by layer. Some designs add a "},{"text":"shared","i":true},{"text":" expert that every token passes through, on the theory that common knowledge should not be duplicated across specialists."}]},{"t":"p","runs":[{"text":"Treat “expert” as a name for a subnetwork, not a description of its contents."}]},{"t":"h2","id":"reading-a-model-s-numbers-correctly","runs":[{"text":"Reading a model’s numbers correctly"}]},{"t":"p","runs":[{"text":"When you compare models, ask which number is quoted."}]},{"t":"ul","items":[[{"text":"For "},{"text":"quality","b":true},{"text":", total parameters is the more relevant figure, though a sparse model generally underperforms a dense model of the same "},{"text":"total","i":true},{"text":" size."}],[{"text":"For "},{"text":"speed and price per token","b":true},{"text":", active parameters is what matters."}],[{"text":"For "},{"text":"whether you can run it at all","b":true},{"text":", total parameters is what matters, since all of it must be in memory."}]]},{"t":"p","runs":[{"text":"A useful rough guide from the literature: a sparse model tends to behave roughly like a dense model somewhere between its active and total sizes: closer to the geometric mean than to either end. Treat that as a rule of thumb, not a law."}]},{"t":"h2","id":"why-this-shape-took-over","runs":[{"text":"Why this shape took over"}]},{"t":"p","runs":[{"text":"The economics are compelling. Serving costs scale with active parameters, and capability scales with total parameters, so a lab that can afford the memory gets a better model per unit of serving cost. That is a large part of why frontier models became cheaper per token over 2024 and 2025 without becoming less capable, alongside distillation and better hardware utilisation."}]},{"t":"p","runs":[{"text":"For you as a user it means one specific thing: a model’s parameter count has stopped being a reliable proxy for its price or its speed. Check the active count if the vendor publishes it, and otherwise measure latency yourself."}]}],"check":{"question":[{"text":"A sparse model with 140 billion total and 20 billion active parameters is proposed for a self-hosted deployment on a server with 48 GB of GPU memory. What is the problem?"}],"options":[{"text":[{"text":"Sparse routing requires more memory bandwidth per token than a dense model of the same active size."}],"correct":false,"tempting":[{"text":"Bandwidth is genuinely a constraint in serving, and routing does add irregular access patterns, so this sounds technically sharp."}]},{"text":[{"text":"All 140 billion parameters must be resident in memory even though only 20 billion run per token, so at fp16 the weights alone need around 280 GB."}],"correct":true,"tempting":null},{"text":[{"text":"The router adds a sequential step that cannot be batched, making throughput unacceptable."}],"correct":false,"tempting":[{"text":"Routing is an extra step, and sequential bottlenecks are a real serving concern, which makes this plausible if you have not seen the implementation."}]},{"text":[{"text":"Sparse models cannot be quantised, so the usual memory savings are unavailable."}],"correct":false,"tempting":[{"text":"Quantisation is the standard escape from memory limits, so assuming it is unavailable explains the difficulty neatly."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"from-final-vector-to-logits","title":"From the final vector to a score for every word","minutes":8},"next":{"slug":"kv-cache-and-attention-variants","title":"The key-value cache, and the engineering built around it","minutes":9}}}