Mixture of experts, and why it changes the sum

Two parameter counts, not one

A dense model uses every weight for every token. A mixture-of-experts model does not. Each feed-forward block is replaced by a set of parallel blocks called experts, plus a small router that picks a few of them per token. A model described as 30B-A3B has about 30 billion total parameters and activates about 3 billion of them for any given token.

That gap is the whole story, and it splits the two numbers you care about:

• Total parameters decide how much memory the weights occupy. All experts must be resident, because the router may choose any of them at any token.
• Active parameters decide how much arithmetic and how much memory traffic happens per token, and therefore how fast it generates.

So a 30B-A3B model needs the memory of a 30B and generates at roughly the speed of a 3B. That is the trade: capacity is cheap, memory is not.

What this does to the arithmetic

Use the same two sums as for a dense model, with one substitution.

Weights: total parameters times bytes per weight. A 30B-A3B at Q4_K_M is about 18 GB. That does not fit in 16 GB of VRAM and does fit comfortably in 32 GB of system RAM.

Speed: the memory-bandwidth ceiling applies to the active bytes per token, not the whole file. On a machine with about 30 GB/s of usable system bandwidth, a dense 30B at Q4 reads 18 GB per token and manages under 2 tokens per second: unusable. The same 30B-A3B reads roughly 2 GB of expert weights plus the shared attention weights, and lands somewhere around 10 to 15 tokens per second. On the same machine. Reading the same total file.

That is the single most useful fact in this lesson, and it is why MoE models have become the practical choice for people with plenty of RAM and no large GPU. A 128 GB machine with slow memory can run a very large MoE at a speed that a dense model of the same total size could never reach.

Where the speed goes anyway

Three caveats, all mechanical.

Routing is per token, not per request. The set of active experts changes at every token, so you cannot keep only the popular experts in fast memory and stream the rest. Attempts to do this exist and cost more than they save on most hardware, because a cache miss stalls the whole token.

Batching breaks the saving. With one request, the router touches a few experts. With sixty concurrent requests, different requests choose different experts, and across the batch you end up touching most of them. Aggregate throughput on an MoE therefore scales worse than the active-parameter count suggests. MoE is a large win for one user and a smaller one for a busy server.

Partial offload behaves differently. The usual advice (offload as many layers as fit) is not optimal for MoE. The better split is to keep attention and the shared weights on the GPU and leave the expert tensors in system RAM, because the experts are large and each one is touched rarely. llama.cpp exposes this:

llama-server -m Qwen3-30B-A3B-Q4_K_M.gguf -ngl 99 \
  --n-cpu-moe 24 -c 8192 --flash-attn

That keeps everything on the GPU except the expert tensors of 24 layers. On a 12 GB card with a 20 GB model, this routinely doubles or triples the speed compared with an even layer split. It is worth measuring both ways on your own machine, because the crossover depends on your PCIe bandwidth.

Quality

An MoE with N active parameters is not as good as a dense model with N parameters: the router costs something and the experts are less general. It is also not as good as a dense model with the same total parameters. The usual rule of thumb, which is a rule of thumb and not a law, places it near the geometric mean: a 30B-A3B behaves roughly like a dense 9B or 10B, while costing 30B of memory and 3B of bandwidth.

Whether that is a good deal depends entirely on which resource you are short of. If you have 8 GB of RAM, an MoE is irrelevant. If you have 64 or 128 GB of ordinary system RAM and no GPU worth the name, it is the best option available to you by a wide margin.

The check before downloading

Read the card for two numbers: total and active. If the card only gives one, the config file has num_experts and num_experts_per_tok, and the arithmetic follows. Then ask which of your two constraints binds (capacity or bandwidth) and choose accordingly. People who skip this either download 40 GB onto a 16 GB machine, or dismiss MoE models as too big while sitting on exactly the hardware they were designed for.