{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"the-causal-mask-and-parallel-training","title":"Why training is parallel and generation is not","url":"https://addaly.com/learn/how-llms-work/the-causal-mask-and-parallel-training","position":12,"of":78,"minutes":8,"answer":"The causal mask lets one pass over a document supply as many training examples as it has tokens, which is why training scales with hardware while generation is stuck at one token at a time.","keyIdea":"The causal mask lets one pass over a document supply as many training examples as it has tokens, which is why training scales with hardware while generation is stuck at one token at a time."},"body":{"format":"markdown","text":"## The same model, two completely different workloads\n\nA transformer is trained and used with the same weights and the same arithmetic, but the shape of the computation could hardly be more different. Understanding why explains most of what you will ever need to know about AI infrastructure costs.\n\n## Training: one pass, thousands of predictions\n\nTake a document of 4,096 tokens. Feed it in as one sequence. Because of the causal mask, the output at position 1 depends only on token 1, the output at position 2 only on tokens 1 and 2, and so on. So a single forward pass produces 4,096 separate next-token predictions, each one legitimately blind to its own answer.\n\nCompare every prediction with the token that actually came next, average the errors, and update the weights once. One pass, 4,096 training examples, and the whole thing is a handful of large matrix multiplications that a GPU eats for breakfast.\n\nThis is called teacher forcing, and it is the reason training is expensive but *efficient*: hardware utilisation on a well-tuned pretraining run can exceed 40 per cent of a GPU's theoretical peak, which is very high for any real workload.\n\n## Generation: one token at a time, no way around it\n\nNow generate. The model produces a distribution for the next token; a sampler picks one; that token is appended and the whole thing runs again. Token 41 cannot be computed before token 40 exists, because token 40 is part of its input.\n\nThe consequence is stark. Producing 500 tokens means 500 sequential passes through all 32 or 80 layers. There is no batching your way out of it *within a single request*, only across requests.\n\n## Prefill and decode\n\nThis split gives serving its two phases, and they behave so differently that providers price them differently.\n\n**Prefill** processes your prompt. Every token is present already, so all positions go through in parallel, exactly like training. A 10,000-token prompt is one big parallel operation, limited by raw arithmetic throughput.\n\n**Decode** produces your response, one token per pass. Each pass reads all the model's weights from memory to compute a single token, which is a terrible ratio of memory traffic to arithmetic: the machine is idle waiting for data most of the time.\n\nThat is the mechanism behind three things you have noticed:\n\n- **Time to first token** depends mostly on prompt length; **time per output token** barely does.\n- Output tokens are usually priced two to five times higher than input tokens.\n- A long prompt with a short answer is cheap and fast; a short prompt with a long answer is neither.\n\n## Batching, and why your latency depends on other people\n\nSince decode is starved for work, servers batch many users' requests into one pass. Reading the weights once and using them for 64 users at once turns a memory-bound operation into an efficient one, which is why throughput per GPU can be dozens of times higher under load than for a single user.\n\nContinuous batching (adding and removing sequences from the batch as they finish) is what modern serving stacks such as vLLM and TGI do, and both are free and open source. It also explains a fact that annoys people: your per-token latency can vary with someone else's traffic.\n\n## Where the training curve comes from\n\nOne more consequence, this time for training. Because every position contributes a prediction, the number of training signals is the number of tokens, not the number of documents. That is why training corpora are quoted in tokens (15 trillion, 18 trillion) and why the compute formula from module one is 6 × parameters × tokens.\n\nIt also means a duplicated document is not free: it contributes its tokens again, and heavy duplication in a corpus is known to hurt quality and increase verbatim memorisation. Deduplication is one of the highest-value steps in data preparation, which module five returns to.\n\n## A useful mental test\n\nWhen someone tells you a new architecture is faster, ask which phase they mean. Many \"efficient transformer\" results improve prefill on long inputs while leaving decode exactly as slow. Both matter, but they matter for different products: a document-summarising tool lives and dies on prefill, and a chat assistant on decode."},"check":{"question":"A provider charges $0.50 per million input tokens and $2.00 per million output tokens. Which mechanism best explains the difference?","options":[{"text":"Output tokens are more valuable to the customer, so they are priced higher.","correct":false,"tempting":"Value-based pricing is common in software, and providers do think about willingness to pay."},{"text":"Output tokens require an extra safety check that input tokens do not.","correct":false,"tempting":"Moderation passes are real and do add cost, which makes this operationally plausible."},{"text":"Input tokens are processed in one parallel pass, while each output token requires its own full pass over the model's weights, so output is far more expensive per token to produce.","correct":true},{"text":"Output tokens are longer on average and so consume more memory in the context window.","correct":false,"tempting":"Output does consume context, and memory costs money, so this chains two true facts into a wrong conclusion."}]},"prev":{"slug":"attention-in-forty-lines","title":"Attention in forty lines of NumPy","url":"https://addaly.com/learn/how-llms-work/attention-in-forty-lines"},"next":{"slug":"the-block","title":"What one transformer block actually does","url":"https://addaly.com/learn/how-llms-work/the-block"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why training is parallel and generation is not', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/the-causal-mask-and-parallel-training (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why training is parallel and generation is not","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"The causal mask lets one pass over a document supply as many training examples as it has tokens, which is why training scales with hardware while generation is stuck at one token at a time.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-same-model-two-completely-different-workloads","runs":[{"text":"The same model, two completely different workloads"}]},{"t":"p","runs":[{"text":"A transformer is trained and used with the same weights and the same arithmetic, but the shape of the computation could hardly be more different. Understanding why explains most of what you will ever need to know about AI infrastructure costs."}]},{"t":"h2","id":"training-one-pass-thousands-of-predictions","runs":[{"text":"Training: one pass, thousands of predictions"}]},{"t":"p","runs":[{"text":"Take a document of 4,096 tokens. Feed it in as one sequence. Because of the causal mask, the output at position 1 depends only on token 1, the output at position 2 only on tokens 1 and 2, and so on. So a single forward pass produces 4,096 separate next-token predictions, each one legitimately blind to its own answer."}]},{"t":"p","runs":[{"text":"Compare every prediction with the token that actually came next, average the errors, and update the weights once. One pass, 4,096 training examples, and the whole thing is a handful of large matrix multiplications that a GPU eats for breakfast."}]},{"t":"p","runs":[{"text":"This is called teacher forcing, and it is the reason training is expensive but "},{"text":"efficient","i":true},{"text":": hardware utilisation on a well-tuned pretraining run can exceed 40 per cent of a GPU’s theoretical peak, which is very high for any real workload."}]},{"t":"h2","id":"generation-one-token-at-a-time-no-way-around-it","runs":[{"text":"Generation: one token at a time, no way around it"}]},{"t":"p","runs":[{"text":"Now generate. The model produces a distribution for the next token; a sampler picks one; that token is appended and the whole thing runs again. Token 41 cannot be computed before token 40 exists, because token 40 is part of its input."}]},{"t":"p","runs":[{"text":"The consequence is stark. Producing 500 tokens means 500 sequential passes through all 32 or 80 layers. There is no batching your way out of it "},{"text":"within a single request","i":true},{"text":", only across requests."}]},{"t":"h2","id":"prefill-and-decode","runs":[{"text":"Prefill and decode"}]},{"t":"p","runs":[{"text":"This split gives serving its two phases, and they behave so differently that providers price them differently."}]},{"t":"p","runs":[{"text":"Prefill","b":true},{"text":" processes your prompt. Every token is present already, so all positions go through in parallel, exactly like training. A 10,000-token prompt is one big parallel operation, limited by raw arithmetic throughput."}]},{"t":"p","runs":[{"text":"Decode","b":true},{"text":" produces your response, one token per pass. Each pass reads all the model’s weights from memory to compute a single token, which is a terrible ratio of memory traffic to arithmetic: the machine is idle waiting for data most of the time."}]},{"t":"p","runs":[{"text":"That is the mechanism behind three things you have noticed:"}]},{"t":"ul","items":[[{"text":"Time to first token","b":true},{"text":" depends mostly on prompt length; "},{"text":"time per output token","b":true},{"text":" barely does."}],[{"text":"Output tokens are usually priced two to five times higher than input tokens."}],[{"text":"A long prompt with a short answer is cheap and fast; a short prompt with a long answer is neither."}]]},{"t":"h2","id":"batching-and-why-your-latency-depends-on-other-people","runs":[{"text":"Batching, and why your latency depends on other people"}]},{"t":"p","runs":[{"text":"Since decode is starved for work, servers batch many users’ requests into one pass. Reading the weights once and using them for 64 users at once turns a memory-bound operation into an efficient one, which is why throughput per GPU can be dozens of times higher under load than for a single user."}]},{"t":"p","runs":[{"text":"Continuous batching (adding and removing sequences from the batch as they finish) is what modern serving stacks such as vLLM and TGI do, and both are free and open source. It also explains a fact that annoys people: your per-token latency can vary with someone else’s traffic."}]},{"t":"h2","id":"where-the-training-curve-comes-from","runs":[{"text":"Where the training curve comes from"}]},{"t":"p","runs":[{"text":"One more consequence, this time for training. Because every position contributes a prediction, the number of training signals is the number of tokens, not the number of documents. That is why training corpora are quoted in tokens (15 trillion, 18 trillion) and why the compute formula from module one is 6 × parameters × tokens."}]},{"t":"p","runs":[{"text":"It also means a duplicated document is not free: it contributes its tokens again, and heavy duplication in a corpus is known to hurt quality and increase verbatim memorisation. Deduplication is one of the highest-value steps in data preparation, which module five returns to."}]},{"t":"h2","id":"a-useful-mental-test","runs":[{"text":"A useful mental test"}]},{"t":"p","runs":[{"text":"When someone tells you a new architecture is faster, ask which phase they mean. Many “efficient transformer” results improve prefill on long inputs while leaving decode exactly as slow. Both matter, but they matter for different products: a document-summarising tool lives and dies on prefill, and a chat assistant on decode."}]}],"check":{"question":[{"text":"A provider charges $0.50 per million input tokens and $2.00 per million output tokens. Which mechanism best explains the difference?"}],"options":[{"text":[{"text":"Output tokens are more valuable to the customer, so they are priced higher."}],"correct":false,"tempting":[{"text":"Value-based pricing is common in software, and providers do think about willingness to pay."}]},{"text":[{"text":"Output tokens require an extra safety check that input tokens do not."}],"correct":false,"tempting":[{"text":"Moderation passes are real and do add cost, which makes this operationally plausible."}]},{"text":[{"text":"Input tokens are processed in one parallel pass, while each output token requires its own full pass over the model’s weights, so output is far more expensive per token to produce."}],"correct":true,"tempting":null},{"text":[{"text":"Output tokens are longer on average and so consume more memory in the context window."}],"correct":false,"tempting":[{"text":"Output does consume context, and memory costs money, so this chains two true facts into a wrong conclusion."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"attention-in-forty-lines","title":"Attention in forty lines of NumPy","minutes":10},"next":{"slug":"the-block","title":"What one transformer block actually does","minutes":8}}}