{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"prefill-and-decode","title":"Prefill and decode are two different machines","url":"https://addaly.com/learn/running-models-yourself/prefill-and-decode","position":13,"of":82,"minutes":9,"answer":"Prefill is compute-bound and decode is memory-bandwidth-bound, so the same hardware change can transform one and do nothing for the other: measure and reason about them separately, always.","keyIdea":"Prefill is compute-bound and decode is memory-bandwidth-bound, so the same hardware change can transform one and do nothing for the other: measure and reason about them separately, always."},"body":{"format":"markdown","text":"## One request, two workloads\n\nEvery generation is two phases with opposite characteristics, and almost every confusing performance result comes from measuring one and reasoning about the other.\n\n**Prefill** processes the prompt. All the input tokens go through the model at once, as a matrix multiply against a matrix rather than a vector. There is a great deal of arithmetic per byte of weights read. It is **compute-bound**.\n\n**Decode** produces output, one token at a time. Each token reads every weight in the model to produce a single vector. There is very little arithmetic per byte read. It is **memory-bandwidth-bound**.\n\nThese two facts explain most of what follows.\n\n## The consequences you can predict\n\n**Decode speed has a ceiling you can compute.** Tokens per second cannot exceed usable bandwidth divided by the bytes of weights read per token. A card with 900 GB/s running a 4.9 GB model cannot exceed about 180 tokens per second no matter how fast its cores are. If a vendor doubles compute and leaves bandwidth alone, single-user generation speed barely moves.\n\n**Prefill speed does scale with compute.** This is why an integrated GPU, which shares the same slow system memory as the CPU, still speeds up prompt processing by two or three times while doing nothing for generation. Same bandwidth, far more parallel arithmetic.\n\n**A long prompt costs differently from a long output.** Two thousand tokens of prompt on a laptop CPU at 40 tokens per second of prefill is fifty seconds before anything appears. Two thousand tokens of output at 6 tokens per second is five minutes but it streams, so it feels alive. Users report the first as broken and the second as slow.\n\n**Batching helps decode enormously and prefill barely at all.** With one request, the weights are read once to produce one token. With thirty-two concurrent requests, the same read produces thirty-two tokens, because the matrix multiply against a batch of vectors reuses the loaded weights. That is why aggregate throughput on a server is many times single-user speed. Prefill is already compute-saturated, so batching adds little.\n\n## Arithmetic intensity, in one paragraph\n\nThe underlying quantity is operations performed per byte moved. A modern GPU can do hundreds of floating-point operations in the time it takes to fetch one byte from its own memory, so any kernel below that ratio is waiting on memory. Decode with a batch of one has an intensity of roughly 2 operations per byte, which is off the bottom of the chart. Prefill with a 2,000-token prompt has an intensity in the hundreds. Batching decode raises the intensity in proportion to batch size, which is exactly why throughput rises so steeply at first and then flattens once the ratio reaches the hardware's balance point.\n\nYou do not need the full roofline model to use this. You need the single idea that batch size moves you from the memory-bound regime to the compute-bound one, and that the flattening point is where adding users stops being free.\n\n## Chunked prefill, and why servers do it\n\nA long prompt arriving mid-stream would otherwise occupy the GPU for a full second, stalling everyone else's token generation, visible as a stutter for every other user. Serving engines split the prompt into chunks and interleave them with decode steps, so a 20,000-token prompt does not freeze the server. It slightly slows that one prefill and greatly improves everybody's latency variance.\n\n```bash\nvllm serve <model> --enable-chunked-prefill --max-num-batched-tokens 2048\n```\n\nIf your users complain about occasional pauses rather than general slowness, this is usually the setting.\n\n## Measuring them separately\n\nAny benchmark that reports one number for a model has averaged two different machines. The tool to use reports them apart:\n\n```bash\nllama-bench -m model.gguf -p 512 -n 128\n```\n\n`pp512` is prompt processing at 512 tokens; `tg128` is token generation of 128 tokens. On a mid-range laptop those might be 45 and 6. On a 24 GB desktop card, 3,000 and 70. Notice that the ratio between machines is completely different for the two numbers: that ratio is the compute gap versus the bandwidth gap, visible directly.\n\n## What to do with each\n\nIf **prefill** is your problem: shorten prompts, use prefix caching so a repeated system prompt is processed once, retrieve five passages instead of fifty, and get any GPU at all involved even a weak one.\n\nIf **decode** is your problem: a smaller or more heavily quantised model reduces bytes read per token, faster memory raises the ceiling, and speculative decoding produces several tokens per weight read. Adding cores does nothing.\n\nDiagnosing which one you have takes one `llama-bench` run and saves a great deal of money spent on the wrong component."},"check":{"question":"A laptop with integrated graphics runs an 8B model. Offloading to the iGPU with Vulkan takes prompt processing from 18 to 52 tokens per second, while generation stays at about 6 tokens per second. What explains both numbers at once?","options":[{"text":"The iGPU has far more parallel compute but shares the same system memory, so it lifts the compute-bound prefill and cannot lift the bandwidth-bound decode.","correct":true},{"text":"The Vulkan backend has less mature generation kernels than its prefill kernels, so decode is losing performance it should be gaining.","correct":false,"tempting":"Blames an implementation gap for a result that the memory hierarchy already predicts exactly, and would imply the ceiling is soft when it is arithmetic."},{"text":"Generation is limited by the KV cache write on each token, which happens on the CPU regardless of where the layers run.","correct":false,"tempting":"Invents a bottleneck; the cache write is kilobytes against gigabytes of weight reads per token."},{"text":"Only some layers were offloaded, so decode still waits on CPU layers while prefill was fully accelerated.","correct":false,"tempting":"Assumes an asymmetric offload that the flags do not produce; both phases run the same layer split."}]},"prev":{"slug":"the-kv-cache-in-depth","title":"The KV cache, in depth","url":"https://addaly.com/learn/running-models-yourself/the-kv-cache-in-depth"},"next":{"slug":"where-the-model-lives","title":"Where the model physically lives","url":"https://addaly.com/learn/running-models-yourself/where-the-model-lives"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Prefill and decode are two different machines', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/prefill-and-decode (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Prefill and decode are two different machines","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"Prefill is compute-bound and decode is memory-bandwidth-bound, so the same hardware change can transform one and do nothing for the other: measure and reason about them separately, always.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"one-request-two-workloads","runs":[{"text":"One request, two workloads"}]},{"t":"p","runs":[{"text":"Every generation is two phases with opposite characteristics, and almost every confusing performance result comes from measuring one and reasoning about the other."}]},{"t":"p","runs":[{"text":"Prefill","b":true},{"text":" processes the prompt. All the input tokens go through the model at once, as a matrix multiply against a matrix rather than a vector. There is a great deal of arithmetic per byte of weights read. It is "},{"text":"compute-bound","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"Decode","b":true},{"text":" produces output, one token at a time. Each token reads every weight in the model to produce a single vector. There is very little arithmetic per byte read. It is "},{"text":"memory-bandwidth-bound","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"These two facts explain most of what follows."}]},{"t":"h2","id":"the-consequences-you-can-predict","runs":[{"text":"The consequences you can predict"}]},{"t":"p","runs":[{"text":"Decode speed has a ceiling you can compute.","b":true},{"text":" Tokens per second cannot exceed usable bandwidth divided by the bytes of weights read per token. A card with 900 GB/s running a 4.9 GB model cannot exceed about 180 tokens per second no matter how fast its cores are. If a vendor doubles compute and leaves bandwidth alone, single-user generation speed barely moves."}]},{"t":"p","runs":[{"text":"Prefill speed does scale with compute.","b":true},{"text":" This is why an integrated GPU, which shares the same slow system memory as the CPU, still speeds up prompt processing by two or three times while doing nothing for generation. Same bandwidth, far more parallel arithmetic."}]},{"t":"p","runs":[{"text":"A long prompt costs differently from a long output.","b":true},{"text":" Two thousand tokens of prompt on a laptop CPU at 40 tokens per second of prefill is fifty seconds before anything appears. Two thousand tokens of output at 6 tokens per second is five minutes but it streams, so it feels alive. Users report the first as broken and the second as slow."}]},{"t":"p","runs":[{"text":"Batching helps decode enormously and prefill barely at all.","b":true},{"text":" With one request, the weights are read once to produce one token. With thirty-two concurrent requests, the same read produces thirty-two tokens, because the matrix multiply against a batch of vectors reuses the loaded weights. That is why aggregate throughput on a server is many times single-user speed. Prefill is already compute-saturated, so batching adds little."}]},{"t":"h2","id":"arithmetic-intensity-in-one-paragraph","runs":[{"text":"Arithmetic intensity, in one paragraph"}]},{"t":"p","runs":[{"text":"The underlying quantity is operations performed per byte moved. A modern GPU can do hundreds of floating-point operations in the time it takes to fetch one byte from its own memory, so any kernel below that ratio is waiting on memory. Decode with a batch of one has an intensity of roughly 2 operations per byte, which is off the bottom of the chart. Prefill with a 2,000-token prompt has an intensity in the hundreds. Batching decode raises the intensity in proportion to batch size, which is exactly why throughput rises so steeply at first and then flattens once the ratio reaches the hardware’s balance point."}]},{"t":"p","runs":[{"text":"You do not need the full roofline model to use this. You need the single idea that batch size moves you from the memory-bound regime to the compute-bound one, and that the flattening point is where adding users stops being free."}]},{"t":"h2","id":"chunked-prefill-and-why-servers-do-it","runs":[{"text":"Chunked prefill, and why servers do it"}]},{"t":"p","runs":[{"text":"A long prompt arriving mid-stream would otherwise occupy the GPU for a full second, stalling everyone else’s token generation, visible as a stutter for every other user. Serving engines split the prompt into chunks and interleave them with decode steps, so a 20,000-token prompt does not freeze the server. It slightly slows that one prefill and greatly improves everybody’s latency variance."}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve <model> --enable-chunked-prefill --max-num-batched-tokens 2048"},{"t":"p","runs":[{"text":"If your users complain about occasional pauses rather than general slowness, this is usually the setting."}]},{"t":"h2","id":"measuring-them-separately","runs":[{"text":"Measuring them separately"}]},{"t":"p","runs":[{"text":"Any benchmark that reports one number for a model has averaged two different machines. The tool to use reports them apart:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-bench -m model.gguf -p 512 -n 128"},{"t":"p","runs":[{"text":"pp512","code":true},{"text":" is prompt processing at 512 tokens; "},{"text":"tg128","code":true},{"text":" is token generation of 128 tokens. On a mid-range laptop those might be 45 and 6. On a 24 GB desktop card, 3,000 and 70. Notice that the ratio between machines is completely different for the two numbers: that ratio is the compute gap versus the bandwidth gap, visible directly."}]},{"t":"h2","id":"what-to-do-with-each","runs":[{"text":"What to do with each"}]},{"t":"p","runs":[{"text":"If "},{"text":"prefill","b":true},{"text":" is your problem: shorten prompts, use prefix caching so a repeated system prompt is processed once, retrieve five passages instead of fifty, and get any GPU at all involved even a weak one."}]},{"t":"p","runs":[{"text":"If "},{"text":"decode","b":true},{"text":" is your problem: a smaller or more heavily quantised model reduces bytes read per token, faster memory raises the ceiling, and speculative decoding produces several tokens per weight read. Adding cores does nothing."}]},{"t":"p","runs":[{"text":"Diagnosing which one you have takes one "},{"text":"llama-bench","code":true},{"text":" run and saves a great deal of money spent on the wrong component."}]}],"check":{"question":[{"text":"A laptop with integrated graphics runs an 8B model. Offloading to the iGPU with Vulkan takes prompt processing from 18 to 52 tokens per second, while generation stays at about 6 tokens per second. What explains both numbers at once?"}],"options":[{"text":[{"text":"The iGPU has far more parallel compute but shares the same system memory, so it lifts the compute-bound prefill and cannot lift the bandwidth-bound decode."}],"correct":true,"tempting":null},{"text":[{"text":"The Vulkan backend has less mature generation kernels than its prefill kernels, so decode is losing performance it should be gaining."}],"correct":false,"tempting":[{"text":"Blames an implementation gap for a result that the memory hierarchy already predicts exactly, and would imply the ceiling is soft when it is arithmetic."}]},{"text":[{"text":"Generation is limited by the KV cache write on each token, which happens on the CPU regardless of where the layers run."}],"correct":false,"tempting":[{"text":"Invents a bottleneck; the cache write is kilobytes against gigabytes of weight reads per token."}]},{"text":[{"text":"Only some layers were offloaded, so decode still waits on CPU layers while prefill was fully accelerated."}],"correct":false,"tempting":[{"text":"Assumes an asymmetric offload that the flags do not produce; both phases run the same layer split."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"the-kv-cache-in-depth","title":"The KV cache, in depth","minutes":10},"next":{"slug":"where-the-model-lives","title":"Where the model physically lives","minutes":9}}}