The bandwidth wall: why decode speed is a division

One division

Here is the most useful single fact in this module. When a dense model generates one token for one user, it must read every weight once. So the fastest it can possibly go is:

tokens per second  ≈  memory bandwidth  ÷  bytes of weights

That is it. Not the number of cores, not the FLOPS on the spec sheet. Bandwidth divided by size.

Put numbers in. A data-centre GPU of the H100 class moves about 3.35 terabytes a second from its memory. An 8-billion-parameter model in bf16 is 16 gigabytes. 3,350 ÷ 16 is about 209 tokens a second, and a real system reaches sixty to eighty per cent of that bound.

A consumer card with a terabyte a second and the same model: about 60 a second. A laptop with unified memory at 200 gigabytes a second, running the same model quantised to four bits (call it 5 gigabytes) has a bound of 40 a second, and delivers 25 to 35. A flagship phone at 60 gigabytes a second, running a 3-billion-parameter model at four bits, has a bound near 30 and delivers half that. A desktop CPU with two channels of DDR5, around 90 gigabytes a second, gets a bound of 18 on the 5-gigabyte model and delivers about ten.

Every one of those matches what people report, and none of them needed a benchmark to predict.

Why the compute sits idle

Generating one token is a matrix-vector multiplication for each weight matrix: a vector of a few thousand numbers against a matrix of millions. Each weight is loaded from memory, used in one multiply-add, and discarded. That is about one floating-point operation per byte moved.

A modern GPU can perform hundreds of operations for every byte it loads. When the work offers one per byte, the arithmetic units wait on memory almost all of the time. The technical phrase is that decode is memory-bandwidth bound, and it is why the spec-sheet number that matters for single-stream generation is the memory number.

Prefill is the other regime

Processing the prompt is different. All the prompt’s tokens go through at once, so each weight loaded is used against thousands of vectors, not one. That is a matrix-matrix multiplication with high arithmetic intensity, and it is bounded by compute instead.

The 8-billion model costs about 2 × 8 billion operations per token (module five’s inference formula). The H100-class card does roughly a thousand trillion bf16 operations a second, so the theoretical prefill rate is around 60,000 tokens a second, and real systems manage 10,000 to 20,000. A 10,000-token prompt therefore takes roughly half a second to a second before the first output token: module three’s time-to-first-token, now with a cause.

The crossover, and why servers batch

Divide compute by bandwidth for the card: a thousand trillion operations a second over 3.35 trillion bytes a second is about 300 operations per byte. Decode offers one. So you need about 300 tokens in flight (300 sequences generating at once) before the card is doing arithmetic as fast as it is reading memory.

That is the number behind module two’s batching lesson. A server holding a few hundred users’ sequences reads each weight once per step and uses it a few hundred times. Throughput per card rises by roughly the batch size, until the crossover, after which compute is the limit. Below the crossover, adding users is nearly free; that is the economic fact the whole serving industry is built on, and the next lessons develop it.

Three consequences you can now predict

Quantisation speeds decode. Halve the bytes and you double the bound. A four-bit model is not faster because the arithmetic is simpler; it is faster because there is less to read. The lesson after next explains what the halving costs.

Mixture-of-experts speeds decode. Only the active experts’ weights are read for a given token, mostly. In a batch, different tokens route to different experts, so a busy server reads more of the model per step than the per-token figure suggests.

Long contexts slow decode, not just prefill. The key-value cache is read at every step too. Module two’s arithmetic gave 131 kilobytes per token for the 8-billion model; at 32,000 tokens that is 4.2 gigabytes of cache read alongside 16 gigabytes of weights: a quarter more traffic, a quarter slower. At 128,000 tokens the cache exceeds the weights, and the model runs at less than half its short-context speed. People notice this and blame the model.

Measure it once

Run any model locally, note the tokens per second, and divide by the bound you computed. Fifty to eighty per cent is normal. Far below that means you are not bandwidth-bound: some layers are on the CPU, the weights are crossing a slow bus, or a kernel is mis-set. The ratio tells you whether to tune or to shrug.

What this is not about

It is not about FLOPS. A card with twice the compute and the same bandwidth generates single-stream tokens at the same rate as the old one, and teams that have bought the upgrade have been surprised. The H100 to H200 step kept the compute and raised the bandwidth to 4.8 terabytes a second, and it was a real decode speed-up for exactly that reason. It is also why Apple silicon, with its wide unified memory, is competitive per pound for local models despite far less raw compute. When you are shopping for generation speed, you are shopping for bandwidth.