Mixture of experts: big model, small bill

Two different numbers, both called size

You will see a model described as “671B total, 37B active”. That is not marketing sleight of hand; it is the defining property of a mixture-of-experts architecture, and it changes how you should reason about cost.

In a dense model, every parameter participates in every token. In a sparse mixture-of-experts model, the feed-forward half of each block is replaced by many parallel feed-forward networks (the experts) plus a small router that picks a couple of them per token. The rest sit idle for that token.

The mechanism

At each position, in each block:

1. A router (a single small matrix) scores the incoming vector against every expert.
2. The top k experts are selected, commonly two out of eight, or eight out of 256 in the larger designs.
3. Only those experts run.
4. Their outputs are combined, weighted by the router’s scores, and added to the residual stream.

Attention is unchanged and remains dense. Only the feed-forward half is made sparse, which is sensible given that is where most of the parameters were.

What it buys and what it costs

Buys: far more parameters (more of the pattern-to-contribution pairs from the previous lesson) for roughly the compute of a much smaller dense model. Mixtral 8×7B holds about 47 billion parameters and activates around 13 billion per token. DeepSeek-V3 holds 671 billion and activates 37 billion.

Costs: the memory to hold all of them. You cannot serve a 671B model on hardware sized for a 37B one. The weights must be resident even though most are unused per token, so mixture-of-experts trades arithmetic for memory capacity and bandwidth. This is why these models are common in large hosted deployments and rare on laptops.

Also costs: training complexity. Experts can collapse: the router sends everything to a favourite few while the rest never learn. The standard fix is an auxiliary load-balancing loss that penalises uneven routing, with newer designs adjusting per-expert biases instead to avoid distorting the main objective. Capacity limits, where an overloaded expert drops tokens, are another live source of subtle degradation.

What the experts are not

The word “expert” invites a wrong picture: one for medicine, one for French, one for code. Measurements do not support that. Routing tends to correlate with surface features (token identity, syntactic role, formatting) far more than with topic, and it varies by layer. Some designs add a shared expert that every token passes through, on the theory that common knowledge should not be duplicated across specialists.

Treat “expert” as a name for a subnetwork, not a description of its contents.

Reading a model’s numbers correctly

When you compare models, ask which number is quoted.

• For quality, total parameters is the more relevant figure, though a sparse model generally underperforms a dense model of the same total size.
• For speed and price per token, active parameters is what matters.
• For whether you can run it at all, total parameters is what matters, since all of it must be in memory.

A useful rough guide from the literature: a sparse model tends to behave roughly like a dense model somewhere between its active and total sizes: closer to the geometric mean than to either end. Treat that as a rule of thumb, not a law.

Why this shape took over

The economics are compelling. Serving costs scale with active parameters, and capability scales with total parameters, so a lab that can afford the memory gets a better model per unit of serving cost. That is a large part of why frontier models became cheaper per token over 2024 and 2025 without becoming less capable, alongside distillation and better hardware utilisation.

For you as a user it means one specific thing: a model’s parameter count has stopped being a reliable proxy for its price or its speed. Check the active count if the vendor publishes it, and otherwise measure latency yourself.