Prefill and decode are two different machines

One request, two workloads

Every generation is two phases with opposite characteristics, and almost every confusing performance result comes from measuring one and reasoning about the other.

Prefill processes the prompt. All the input tokens go through the model at once, as a matrix multiply against a matrix rather than a vector. There is a great deal of arithmetic per byte of weights read. It is compute-bound.

Decode produces output, one token at a time. Each token reads every weight in the model to produce a single vector. There is very little arithmetic per byte read. It is memory-bandwidth-bound.

These two facts explain most of what follows.

The consequences you can predict

Decode speed has a ceiling you can compute. Tokens per second cannot exceed usable bandwidth divided by the bytes of weights read per token. A card with 900 GB/s running a 4.9 GB model cannot exceed about 180 tokens per second no matter how fast its cores are. If a vendor doubles compute and leaves bandwidth alone, single-user generation speed barely moves.

Prefill speed does scale with compute. This is why an integrated GPU, which shares the same slow system memory as the CPU, still speeds up prompt processing by two or three times while doing nothing for generation. Same bandwidth, far more parallel arithmetic.

A long prompt costs differently from a long output. Two thousand tokens of prompt on a laptop CPU at 40 tokens per second of prefill is fifty seconds before anything appears. Two thousand tokens of output at 6 tokens per second is five minutes but it streams, so it feels alive. Users report the first as broken and the second as slow.

Batching helps decode enormously and prefill barely at all. With one request, the weights are read once to produce one token. With thirty-two concurrent requests, the same read produces thirty-two tokens, because the matrix multiply against a batch of vectors reuses the loaded weights. That is why aggregate throughput on a server is many times single-user speed. Prefill is already compute-saturated, so batching adds little.

Arithmetic intensity, in one paragraph

The underlying quantity is operations performed per byte moved. A modern GPU can do hundreds of floating-point operations in the time it takes to fetch one byte from its own memory, so any kernel below that ratio is waiting on memory. Decode with a batch of one has an intensity of roughly 2 operations per byte, which is off the bottom of the chart. Prefill with a 2,000-token prompt has an intensity in the hundreds. Batching decode raises the intensity in proportion to batch size, which is exactly why throughput rises so steeply at first and then flattens once the ratio reaches the hardware’s balance point.

You do not need the full roofline model to use this. You need the single idea that batch size moves you from the memory-bound regime to the compute-bound one, and that the flattening point is where adding users stops being free.

Chunked prefill, and why servers do it

A long prompt arriving mid-stream would otherwise occupy the GPU for a full second, stalling everyone else’s token generation, visible as a stutter for every other user. Serving engines split the prompt into chunks and interleave them with decode steps, so a 20,000-token prompt does not freeze the server. It slightly slows that one prefill and greatly improves everybody’s latency variance.

vllm serve <model> --enable-chunked-prefill --max-num-batched-tokens 2048

If your users complain about occasional pauses rather than general slowness, this is usually the setting.

Measuring them separately

Any benchmark that reports one number for a model has averaged two different machines. The tool to use reports them apart:

llama-bench -m model.gguf -p 512 -n 128

pp512 is prompt processing at 512 tokens; tg128 is token generation of 128 tokens. On a mid-range laptop those might be 45 and 6. On a 24 GB desktop card, 3,000 and 70. Notice that the ratio between machines is completely different for the two numbers: that ratio is the compute gap versus the bandwidth gap, visible directly.

What to do with each

If prefill is your problem: shorten prompts, use prefix caching so a repeated system prompt is processed once, retrieve five passages instead of fifty, and get any GPU at all involved even a weak one.

If decode is your problem: a smaller or more heavily quantised model reduces bytes read per token, faster memory raises the ceiling, and speculative decoding produces several tokens per weight read. Adding cores does nothing.

Diagnosing which one you have takes one llama-bench run and saves a great deal of money spent on the wrong component.