Mixture of experts, and the memory it still needs

Two parameter counts, and only one of them is speed

A mixture-of-experts model replaces the single feed-forward block in each layer with many of them (the experts) plus a small router that picks a few per token. A model described as 30B-A3B has around 30 billion total parameters and around 3 billion active for any given token.

The consequence people want is real: compute scales with active parameters. A 30B-A3B model generates at roughly the speed of a 3B dense model, which is a large multiple faster than a 30B dense model would be.

The consequence people forget is equally real: memory scales with total parameters. The router can send the next token to any expert, so every expert must be resident. That 30B-A3B model needs about 18 GB at 4-bit quantisation, the same as a 30B dense model. You get 3B speed only if you can hold 30B.

That single sentence decides whether a mixture-of-experts model is the best or the worst choice you could make. On a 32 GB workstation it is excellent. On an 8 GB laptop it is unusable, while a 4B dense model works fine.

How the routing works

For each token, at each layer, the router scores the experts and selects the top few: commonly two of eight, or eight of a hundred and twenty-eight in the larger designs. Only those run. Routing is learned during training, and it does not divide neatly by topic: experts do not correspond to physics and poetry, whatever the name suggests. They are learned specialisations of a kind nobody has cleanly interpreted.

Some architectures add a shared expert that runs for every token, holding the general capability while the routed experts specialise.

Training these models is harder than training dense ones, because the router can collapse onto a few popular experts and leave the rest untrained. Load-balancing losses exist to prevent that, and they are a large part of what the DeepSeek papers describe in detail.

What you meet in practice

The open MoE models include Qwen3’s 30B-A3B and larger variants, DeepSeek’s V3 line at around 671B total with roughly 37B active, and Mixtral’s earlier 8x7B and 8x22B. The very large ones are rental propositions, not self-hosting ones; the DeepSeek flagship needs several hundred gigabytes of memory before any context.

Two operational quirks are worth knowing.

Batching is less efficient. In a batch, different tokens route to different experts, so the server ends up touching most of the weights anyway. Throughput gains at high concurrency are smaller than the single-stream speedup suggests.

Expert offloading works, slowly. llama.cpp and others can keep some experts in CPU memory or on disk and fetch them as needed. It makes a model that would not fit run at all, at a speed that depends entirely on the transfer rate. Useful for batch work overnight, painful interactively.

The choice, stated plainly

Ask one question: how much memory do I have?

• Plenty of memory, want speed. MoE is the best value in the ecosystem. A 30B-A3B on a machine with 32 GB gives you close to 30B-class quality at close to 3B-class speed.
• Memory constrained. Dense, every time. A 4B dense model that fits beats a 30B-A3B that does not, and the comparison is not close, because the alternative to fitting is not running.
• Renting. It barely matters to you; the provider’s price already reflects their side of this trade, and MoE models are often priced attractively for their quality precisely because they are cheap to serve.

The recurring mistake is reading 30B-A3B as if the model were 3B. Read it as a 30B model that runs quickly. The first number is the one your hardware has to satisfy.