{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"where-inference-time-goes","title":"Where inference time actually goes","url":"https://addaly.com/learn/cloud-and-shipping/where-inference-time-goes","position":50,"of":74,"minutes":10,"answer":"Language-model latency splits into a parallel prefill that sets time-to-first-token and a memory-bandwidth-bound decode that sets tokens per second, which is why smaller and quantised models are faster, why batching raises throughput almost for free.","keyIdea":"Language-model latency splits into a parallel prefill that sets time-to-first-token and a memory-bandwidth-bound decode that sets tokens per second, which is why smaller and quantised models are faster, why batching raises throughput almost for free, and why output length is the lever that matters most."},"body":{"format":"markdown","text":"## Two very different clocks\n\nFor a small model (a tree, a logistic regression, an embedding) inference is arithmetic on a fixed-size input, and it takes the same time every request. Measure it once and you know it.\n\nFor a language model, generation is a loop, and the cost is in two distinct phases that behave differently:\n\n- **Prefill.** The whole prompt is processed at once. This phase is compute-bound and parallel, so a 2,000-token prompt is not twenty times slower than a 100-token one, but it does set your time to first token.\n- **Decode.** Tokens come out one at a time, each requiring a full pass over the weights. This phase is memory-bandwidth-bound: the limit is how fast the weights can be read from memory, not how fast the chip can multiply.\n\nThat second fact explains almost every surprising benchmark result. If decoding is limited by memory bandwidth, then a smaller model is faster for a simple mechanical reason (there is less to read per token), and quantising a model to 4 bits roughly halves the bytes moved, which roughly doubles the token rate. It also explains why decoding one sequence leaves an expensive GPU mostly idle.\n\n> Figure: Two clocks inside one language-model request. Prefill: the whole prompt at once: Compute-bound and parallel; A 2,000-token prompt is not twenty times a 100-token one; Sets time to first token; Shortening the prompt helps here. Decode: one token at a time: Memory-bandwidth-bound: a full pass over the weights per token; Fewer bytes to read means faster: 4-bit roughly doubles the rate; Sets tokens per second; Output length is the lever that matters most. Prefill sets the time to first token. Decode sets tokens per second, and because it is limited by reading the weights rather than by arithmetic, a smaller or quantised model is faster for a purely mechanical reason.\n\n## Batching, which is why hosted APIs are cheap\n\nBecause decode is bandwidth-bound, processing eight sequences at once costs barely more than processing one: the weights are read once and used for all eight. Throughput multiplies; per-request latency barely moves.\n\nThis is what a serving stack like vLLM or TGI does with continuous batching: new requests join the running batch between token steps rather than waiting for a batch to fill. It is also why a provider serving thousands of users per GPU can charge less per token than your idle GPU costs you.\n\nThe trade-off is real and worth stating: batching raises throughput and raises the latency of the individual request slightly. Under a strict latency budget with sparse traffic, small batches win.\n\n## The KV cache, and why long conversations get expensive\n\nDuring decode, the model keeps the intermediate attention state for every token so far (the key-value cache) so it does not recompute the whole conversation for each new token. It is the reason generation is not quadratic in practice.\n\nIt also lives in memory, and it grows with context length times batch size. On a GPU it competes with the weights for the same memory, which is why a server that comfortably handles ten short conversations falls over on three long ones. If you are self-hosting and see out-of-memory errors that correlate with long chats rather than with request volume, this is it.\n\n## Measure your own, in one command\n\nDo not trust a benchmark from a machine you do not have. For a hosted API:\n\n```bash\ncurl -s -w '\\ntotal=%{time_total}s ttfb=%{time_starttransfer}s\\n' \\\n  -o /dev/null -X POST https://api.example.com/v1/chat \\\n  -H \"Authorization: Bearer $KEY\" -H 'content-type: application/json' \\\n  -d '{\"model\":\"...\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"...\"}]}'\n```\n\n`time_starttransfer` with streaming on is your time to first token; the gap to `time_total` divided by the number of output tokens is your per-token rate. Record both for the p50 and the p95: model latency has a much longer tail than database latency, and the average will flatter you.\n\nFor a local model, `llama-bench` reports prefill and decode rates separately, which is exactly the split above.\n\n## The four levers, in order of effect\n\n1. **Generate fewer tokens.** Output length is the dominant term in both latency and cost. Ask for a shorter answer, set a max-tokens limit, and prefer structured output over prose.\n2. **Use a smaller model.** Often the largest single win, and often free of quality cost for classification and extraction. Test it rather than assuming.\n3. **Quantise, if self-hosting.** 8-bit is usually indistinguishable in quality; 4-bit is noticeably cheaper and sometimes noticeably worse. Measure on your own examples, not on a leaderboard.\n4. **Shorten the prompt.** Helps prefill and cost, but has less effect on total latency than people expect, because decode dominates for long outputs.\n\nAnd one that is not on the list: retrieving more context to improve quality makes prefill longer and the bill larger. That is a legitimate trade, but make it on purpose.\n\n## Stream, and the perceived time halves\n\nA four-second response delivered as a stream, with the first token in 400 ms, is experienced as fast. The same response delivered whole is experienced as broken. Server-sent events are the standard mechanism and are supported by every provider; the cost is that your proxy must not buffer, which is one configuration line and a common cause of \"streaming does not work in production but works locally\"."},"check":{"question":"Self-hosting a model, you cut the prompt from 1,500 tokens to 400 but keep the same 800-token answers. Total latency barely improves. Why?","options":[{"text":"The KV cache still holds the original context, so memory traffic is unchanged","correct":false,"tempting":"The cache is built from the tokens actually processed; a shorter prompt makes a smaller cache, so this is not what keeps the time constant."},{"text":"Prompt tokens are billed but not processed, so shortening them changes cost rather than time","correct":false,"tempting":"Prompt tokens are certainly processed (that is prefill) so the premise is wrong even though the conclusion about cost is partly right."},{"text":"Most of the time is decode, which depends on the number of output tokens, not on prompt length","correct":true},{"text":"Quantisation already removed the prefill cost, so prompt length no longer matters","correct":false,"tempting":"Quantisation reduces bytes read per pass across both phases; it does not eliminate prefill or make prompt length irrelevant."}]},"prev":{"slug":"a-model-is-a-process-with-weights","title":"A prediction endpoint is a web service with a large secret","url":"https://addaly.com/learn/cloud-and-shipping/a-model-is-a-process-with-weights"},"next":{"slug":"the-arithmetic-of-tokens-and-gpus","title":"What a prediction costs, worked out","url":"https://addaly.com/learn/cloud-and-shipping/the-arithmetic-of-tokens-and-gpus"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where inference time actually goes', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/where-inference-time-goes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where inference time actually goes","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"Language-model latency splits into a parallel prefill that sets time-to-first-token and a memory-bandwidth-bound decode that sets tokens per second, which is why smaller and quantised models are faster, why batching raises throughput almost for free, and why output length is the lever that matters most.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Prefill: the whole prompt at once","items":["Compute-bound and parallel","A 2,000-token prompt is not twenty times a 100-token one","Sets time to first token","Shortening the prompt helps here"]},"type":"compare","right":{"title":"Decode: one token at a time","items":["Memory-bandwidth-bound: a full pass over the weights per token","Fewer bytes to read means faster: 4-bit roughly doubles the rate","Sets tokens per second","Output length is the lever that matters most"]},"title":"Two clocks inside one language-model request","caption":"Prefill sets the time to first token. Decode sets tokens per second, and because it is limited by reading the weights rather than by arithmetic, a smaller or quantised model is faster for a purely mechanical reason."},"alt":"Figure: Two clocks inside one language-model request. Prefill: the whole prompt at once: Compute-bound and parallel; A 2,000-token prompt is not twenty times a 100-token one; Sets time to first token; Shortening the prompt helps here. Decode: one token at a time: Memory-bandwidth-bound: a full pass over the weights per token; Fewer bytes to read means faster: 4-bit roughly doubles the rate; Sets tokens per second; Output length is the lever that matters most. Prefill sets the time to first token. Decode sets tokens per second, and because it is limited by reading the weights rather than by arithmetic, a smaller or quantised model is faster for a purely mechanical reason."}},"blocks":[{"t":"h2","id":"two-very-different-clocks","runs":[{"text":"Two very different clocks"}]},{"t":"p","runs":[{"text":"For a small model (a tree, a logistic regression, an embedding) inference is arithmetic on a fixed-size input, and it takes the same time every request. Measure it once and you know it."}]},{"t":"p","runs":[{"text":"For a language model, generation is a loop, and the cost is in two distinct phases that behave differently:"}]},{"t":"ul","items":[[{"text":"Prefill.","b":true},{"text":" The whole prompt is processed at once. This phase is compute-bound and parallel, so a 2,000-token prompt is not twenty times slower than a 100-token one, but it does set your time to first token."}],[{"text":"Decode.","b":true},{"text":" Tokens come out one at a time, each requiring a full pass over the weights. This phase is memory-bandwidth-bound: the limit is how fast the weights can be read from memory, not how fast the chip can multiply."}]]},{"t":"p","runs":[{"text":"That second fact explains almost every surprising benchmark result. If decoding is limited by memory bandwidth, then a smaller model is faster for a simple mechanical reason (there is less to read per token), and quantising a model to 4 bits roughly halves the bytes moved, which roughly doubles the token rate. It also explains why decoding one sequence leaves an expensive GPU mostly idle."}]},{"t":"h2","id":"batching-which-is-why-hosted-apis-are-cheap","runs":[{"text":"Batching, which is why hosted APIs are cheap"}]},{"t":"p","runs":[{"text":"Because decode is bandwidth-bound, processing eight sequences at once costs barely more than processing one: the weights are read once and used for all eight. Throughput multiplies; per-request latency barely moves."}]},{"t":"p","runs":[{"text":"This is what a serving stack like vLLM or TGI does with continuous batching: new requests join the running batch between token steps rather than waiting for a batch to fill. It is also why a provider serving thousands of users per GPU can charge less per token than your idle GPU costs you."}]},{"t":"p","runs":[{"text":"The trade-off is real and worth stating: batching raises throughput and raises the latency of the individual request slightly. Under a strict latency budget with sparse traffic, small batches win."}]},{"t":"h2","id":"the-kv-cache-and-why-long-conversations-get-expensive","runs":[{"text":"The KV cache, and why long conversations get expensive"}]},{"t":"p","runs":[{"text":"During decode, the model keeps the intermediate attention state for every token so far (the key-value cache) so it does not recompute the whole conversation for each new token. It is the reason generation is not quadratic in practice."}]},{"t":"p","runs":[{"text":"It also lives in memory, and it grows with context length times batch size. On a GPU it competes with the weights for the same memory, which is why a server that comfortably handles ten short conversations falls over on three long ones. If you are self-hosting and see out-of-memory errors that correlate with long chats rather than with request volume, this is it."}]},{"t":"h2","id":"measure-your-own-in-one-command","runs":[{"text":"Measure your own, in one command"}]},{"t":"p","runs":[{"text":"Do not trust a benchmark from a machine you do not have. For a hosted API:"}]},{"t":"code","lang":"bash","label":"Bash","text":"curl -s -w '\\ntotal=%{time_total}s ttfb=%{time_starttransfer}s\\n' \\\n  -o /dev/null -X POST https://api.example.com/v1/chat \\\n  -H \"Authorization: Bearer $KEY\" -H 'content-type: application/json' \\\n  -d '{\"model\":\"...\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"...\"}]}'"},{"t":"p","runs":[{"text":"time_starttransfer","code":true},{"text":" with streaming on is your time to first token; the gap to "},{"text":"time_total","code":true},{"text":" divided by the number of output tokens is your per-token rate. Record both for the p50 and the p95: model latency has a much longer tail than database latency, and the average will flatter you."}]},{"t":"p","runs":[{"text":"For a local model, "},{"text":"llama-bench","code":true},{"text":" reports prefill and decode rates separately, which is exactly the split above."}]},{"t":"h2","id":"the-four-levers-in-order-of-effect","runs":[{"text":"The four levers, in order of effect"}]},{"t":"ol","start":1,"items":[[{"text":"Generate fewer tokens.","b":true},{"text":" Output length is the dominant term in both latency and cost. Ask for a shorter answer, set a max-tokens limit, and prefer structured output over prose."}],[{"text":"Use a smaller model.","b":true},{"text":" Often the largest single win, and often free of quality cost for classification and extraction. Test it rather than assuming."}],[{"text":"Quantise, if self-hosting.","b":true},{"text":" 8-bit is usually indistinguishable in quality; 4-bit is noticeably cheaper and sometimes noticeably worse. Measure on your own examples, not on a leaderboard."}],[{"text":"Shorten the prompt.","b":true},{"text":" Helps prefill and cost, but has less effect on total latency than people expect, because decode dominates for long outputs."}]]},{"t":"p","runs":[{"text":"And one that is not on the list: retrieving more context to improve quality makes prefill longer and the bill larger. That is a legitimate trade, but make it on purpose."}]},{"t":"h2","id":"stream-and-the-perceived-time-halves","runs":[{"text":"Stream, and the perceived time halves"}]},{"t":"p","runs":[{"text":"A four-second response delivered as a stream, with the first token in 400 ms, is experienced as fast. The same response delivered whole is experienced as broken. Server-sent events are the standard mechanism and are supported by every provider; the cost is that your proxy must not buffer, which is one configuration line and a common cause of “streaming does not work in production but works locally”."}]}],"check":{"question":[{"text":"Self-hosting a model, you cut the prompt from 1,500 tokens to 400 but keep the same 800-token answers. Total latency barely improves. Why?"}],"options":[{"text":[{"text":"The KV cache still holds the original context, so memory traffic is unchanged"}],"correct":false,"tempting":[{"text":"The cache is built from the tokens actually processed; a shorter prompt makes a smaller cache, so this is not what keeps the time constant."}]},{"text":[{"text":"Prompt tokens are billed but not processed, so shortening them changes cost rather than time"}],"correct":false,"tempting":[{"text":"Prompt tokens are certainly processed (that is prefill) so the premise is wrong even though the conclusion about cost is partly right."}]},{"text":[{"text":"Most of the time is decode, which depends on the number of output tokens, not on prompt length"}],"correct":true,"tempting":null},{"text":[{"text":"Quantisation already removed the prefill cost, so prompt length no longer matters"}],"correct":false,"tempting":[{"text":"Quantisation reduces bytes read per pass across both phases; it does not eliminate prefill or make prompt length irrelevant."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"a-model-is-a-process-with-weights","title":"A prediction endpoint is a web service with a large secret","minutes":9},"next":{"slug":"the-arithmetic-of-tokens-and-gpus","title":"What a prediction costs, worked out","minutes":10}}}