Why training is parallel and generation is not

The same model, two completely different workloads

A transformer is trained and used with the same weights and the same arithmetic, but the shape of the computation could hardly be more different. Understanding why explains most of what you will ever need to know about AI infrastructure costs.

Training: one pass, thousands of predictions

Take a document of 4,096 tokens. Feed it in as one sequence. Because of the causal mask, the output at position 1 depends only on token 1, the output at position 2 only on tokens 1 and 2, and so on. So a single forward pass produces 4,096 separate next-token predictions, each one legitimately blind to its own answer.

Compare every prediction with the token that actually came next, average the errors, and update the weights once. One pass, 4,096 training examples, and the whole thing is a handful of large matrix multiplications that a GPU eats for breakfast.

This is called teacher forcing, and it is the reason training is expensive but efficient: hardware utilisation on a well-tuned pretraining run can exceed 40 per cent of a GPU’s theoretical peak, which is very high for any real workload.

Generation: one token at a time, no way around it

Now generate. The model produces a distribution for the next token; a sampler picks one; that token is appended and the whole thing runs again. Token 41 cannot be computed before token 40 exists, because token 40 is part of its input.

The consequence is stark. Producing 500 tokens means 500 sequential passes through all 32 or 80 layers. There is no batching your way out of it within a single request, only across requests.

Prefill and decode

This split gives serving its two phases, and they behave so differently that providers price them differently.

Prefill processes your prompt. Every token is present already, so all positions go through in parallel, exactly like training. A 10,000-token prompt is one big parallel operation, limited by raw arithmetic throughput.

Decode produces your response, one token per pass. Each pass reads all the model’s weights from memory to compute a single token, which is a terrible ratio of memory traffic to arithmetic: the machine is idle waiting for data most of the time.

That is the mechanism behind three things you have noticed:

• Time to first token depends mostly on prompt length; time per output token barely does.
• Output tokens are usually priced two to five times higher than input tokens.
• A long prompt with a short answer is cheap and fast; a short prompt with a long answer is neither.

Batching, and why your latency depends on other people

Since decode is starved for work, servers batch many users’ requests into one pass. Reading the weights once and using them for 64 users at once turns a memory-bound operation into an efficient one, which is why throughput per GPU can be dozens of times higher under load than for a single user.

Continuous batching (adding and removing sequences from the batch as they finish) is what modern serving stacks such as vLLM and TGI do, and both are free and open source. It also explains a fact that annoys people: your per-token latency can vary with someone else’s traffic.

Where the training curve comes from

One more consequence, this time for training. Because every position contributes a prediction, the number of training signals is the number of tokens, not the number of documents. That is why training corpora are quoted in tokens (15 trillion, 18 trillion) and why the compute formula from module one is 6 × parameters × tokens.

It also means a duplicated document is not free: it contributes its tokens again, and heavy duplication in a corpus is known to hurt quality and increase verbatim memorisation. Deduplication is one of the highest-value steps in data preparation, which module five returns to.

A useful mental test

When someone tells you a new architecture is faster, ask which phase they mean. Many “efficient transformer” results improve prefill on long inputs while leaving decode exactly as slow. Both matter, but they matter for different products: a document-summarising tool lives and dies on prefill, and a chat assistant on decode.