{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"prefill-decode-and-where-the-time-goes","title":"Where the milliseconds go","url":"https://addaly.com/learn/context-engineering/prefill-decode-and-where-the-time-goes","position":41,"of":87,"minutes":9,"answer":"Prefill is compute-bound and scales with input, decode is bandwidth-bound and scales with output, so input length sets time-to-first-token while output length sets the rest, and streaming only hides the second of the two.","keyIdea":"Prefill is compute-bound and scales with input, decode is bandwidth-bound and scales with output, so input length sets time-to-first-token while output length sets the rest, and streaming only hides the second of the two."},"body":{"format":"markdown","text":"## Two phases with different physics\n\nA model call has two stages and they are bottlenecked by different hardware.\n\n**Prefill** processes your entire input. Every token is handled in parallel, so this stage is compute-bound: it saturates the arithmetic units of the accelerator. Its duration scales with how many input tokens there are. Prefill is what you are waiting for during time-to-first-token.\n\n**Decode** produces the output one token at a time. Each token requires reading the whole model's weights from memory, so this stage is memory-bandwidth-bound and largely indifferent to how long your input was. Its duration scales with how many output tokens you asked for.\n\nThat split explains most latency behaviour people find confusing.\n\n```\ntime_to_first_token  ≈ queue + prefill(input_tokens)\ntotal_time           ≈ time_to_first_token + output_tokens × time_per_token\n```\n\n## What it implies\n\n**A long context is slow to start and then normal.** Going from 4,000 to 60,000 input tokens can add hundreds of milliseconds to several seconds before anything appears, and then the text streams at the same rate as always. Users describe this as \"it hangs and then works\", which is exactly right.\n\n**Streaming hides decode, not prefill.** Streaming is the standard latency fix and it only helps after the first token. If your problem is a five-second silence, streaming does not touch it. What touches it is a shorter input, a cache hit, or a smaller model.\n\n**Output length is the strongest lever on total time.** Each output token costs a full pass over the weights. Halving a verbose answer halves most of the wall-clock. \"Answer in at most three sentences\" is a latency optimisation as much as a style choice.\n\n> Figure: Input sets the wait, output sets the rest. Line chart of milliseconds against thousands of input tokens: Time to first token (4, 260) (16, 520) (32, 900) (60, 1700) (120, 3900); Decode of four hundred output tokens (4, 4200) (16, 4250) (32, 4300) (60, 4400) (120, 4600). One model, four hundred output tokens, ten runs a point, medians. Prefill is compute-bound and grows with input; decode is bandwidth-bound and barely notices how long the input was. Streaming hides the lower line and does nothing for the upper one, which is why a five-second silence is fixed by caching the prefix or cutting the input, never by streaming.\n\n**A cache hit removes prefill for the cached part.** This is why caching improves perceived speed as much as it improves cost, and often more noticeably.\n\n## The quadratic that mostly is not\n\nAttention is quadratic in sequence length in the arithmetic. In practice, serving stacks have spent years making that not dominate: flash-attention kernels avoid materialising the full matrix, chunked prefill breaks a long input into pieces, and paged caches keep memory usable.\n\nThe result is that measured prefill time is closer to linear in input length across the range most applications use, with the quadratic term becoming visible at very long contexts. So do not model prefill as quadratic and do not assume it is free: measure it at the lengths you actually send.\n\n## Measure it yourself\n\nEverything here is observable with a stopwatch and a streaming client.\n\n```python\nt0 = time.perf_counter()\nfirst = None\nfor i, chunk in enumerate(stream_call(messages)):\n    if first is None:\n        first = time.perf_counter() - t0        # TTFT\ntotal = time.perf_counter() - t0\nprint(f\"ttft={first:.2f}s total={total:.2f}s tps={n_out/(total-first):.1f}\")\n```\n\nRun it at three input lengths and two output lengths, ten times each, and record medians and 95th percentiles. Half an hour of work replaces every argument your team is going to have about latency.\n\nThe 95th percentile is the number that matters. Provider latency is noticeably variable (queueing, capacity, other tenants) and a median that looks fine can hide a tail where one request in twenty takes four times as long.\n\n## The rest of the latency budget\n\nThe model is often not the largest term. A realistic breakdown for a retrieval-augmented answer:\n\n- Query rewriting call: 200 to 500 ms, if you do it.\n- Embedding the query: 10 to 50 ms locally, 50 to 200 ms hosted.\n- Vector search: 10 to 100 ms.\n- Reranking 50 candidates: 50 to 400 ms depending on hardware.\n- Prefill on 8,000 tokens: 200 ms to 1 s.\n- Decode of 400 tokens: 2 to 8 s.\n\nTwo lessons. The pre-model stages add up to something comparable to prefill, and they are serial by default: run the ones that do not depend on each other in parallel. And decode dominates everything, which puts the answer's length at the centre of any latency work.\n\n## What to do when it is still too slow\n\nIn rough order of return:\n\n1. **Shorten the output.** Largest effect, no infrastructure change.\n2. **Cache the prefix.** Removes most of prefill on repeat traffic.\n3. **Stream, and show something immediately** (the retrieved sources, a skeleton) so the wait is occupied.\n4. **Parallelise the pre-model stages.**\n5. **Cut input length**, which the selection module was already arguing for.\n6. **Use a smaller model for the easy majority**, which the routing lesson covers.\n\nNotice that only the last two change what the model sees. Most latency work is not a context problem, but the two that are, are the ones you now know how to measure."},"check":{"question":"Users complain about a five-second pause before any text appears. The team enables streaming and the complaint persists. Why?","options":[{"text":"Streaming increases total time by adding per-chunk overhead, offsetting its benefit.","correct":false,"tempting":"Per-chunk overhead exists and is tiny; it cannot account for a five-second wait remaining unchanged."},{"text":"The pause is prefill over a long input, which happens before the first token exists to stream.","correct":true},{"text":"The output is too long, so the model is still generating when the user gives up.","correct":false,"tempting":"Long output delays completion, but with streaming the first tokens would appear quickly, which is not what is described."},{"text":"Streaming requires a cache hit to be effective, and the prefix was not cached.","correct":false,"tempting":"Invents a dependency; streaming and caching are independent, though caching would in fact shorten this particular wait."}]},"prev":{"slug":"what-a-token-costs","title":"The arithmetic of a request","url":"https://addaly.com/learn/context-engineering/what-a-token-costs"},"next":{"slug":"caching-the-prefix","title":"Caching a stable prefix","url":"https://addaly.com/learn/context-engineering/caching-the-prefix"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the milliseconds go', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/prefill-decode-and-where-the-time-goes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the milliseconds go","courseTitle":"Context Engineering","module":{"index":5,"slug":"cost-cache-and-latency","title":"Cost, caching and latency"},"answer":null,"keyIdea":"Prefill is compute-bound and scales with input, decode is bandwidth-bound and scales with output, so input length sets time-to-first-token while output length sets the rest, and streaming only hides the second of the two.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Input sets the wait, output sets the rest","series":[{"label":"Time to first token","points":[[4,260],[16,520],[32,900],[60,1700],[120,3900]]},{"label":"Decode of four hundred output tokens","points":[[4,4200],[16,4250],[32,4300],[60,4400],[120,4600]]}],"xLabel":"thousands of input tokens","yLabel":"milliseconds","caption":"One model, four hundred output tokens, ten runs a point, medians. Prefill is compute-bound and grows with input; decode is bandwidth-bound and barely notices how long the input was. Streaming hides the lower line and does nothing for the upper one, which is why a five-second silence is fixed by caching the prefix or cutting the input, never by streaming."},"alt":"Figure: Input sets the wait, output sets the rest. Line chart of milliseconds against thousands of input tokens: Time to first token (4, 260) (16, 520) (32, 900) (60, 1700) (120, 3900); Decode of four hundred output tokens (4, 4200) (16, 4250) (32, 4300) (60, 4400) (120, 4600). One model, four hundred output tokens, ten runs a point, medians. Prefill is compute-bound and grows with input; decode is bandwidth-bound and barely notices how long the input was. Streaming hides the lower line and does nothing for the upper one, which is why a five-second silence is fixed by caching the prefix or cutting the input, never by streaming."}},"blocks":[{"t":"h2","id":"two-phases-with-different-physics","runs":[{"text":"Two phases with different physics"}]},{"t":"p","runs":[{"text":"A model call has two stages and they are bottlenecked by different hardware."}]},{"t":"p","runs":[{"text":"Prefill","b":true},{"text":" processes your entire input. Every token is handled in parallel, so this stage is compute-bound: it saturates the arithmetic units of the accelerator. Its duration scales with how many input tokens there are. Prefill is what you are waiting for during time-to-first-token."}]},{"t":"p","runs":[{"text":"Decode","b":true},{"text":" produces the output one token at a time. Each token requires reading the whole model’s weights from memory, so this stage is memory-bandwidth-bound and largely indifferent to how long your input was. Its duration scales with how many output tokens you asked for."}]},{"t":"p","runs":[{"text":"That split explains most latency behaviour people find confusing."}]},{"t":"code","lang":"","label":"Text","text":"time_to_first_token  ≈ queue + prefill(input_tokens)\ntotal_time           ≈ time_to_first_token + output_tokens × time_per_token"},{"t":"h2","id":"what-it-implies","runs":[{"text":"What it implies"}]},{"t":"p","runs":[{"text":"A long context is slow to start and then normal.","b":true},{"text":" Going from 4,000 to 60,000 input tokens can add hundreds of milliseconds to several seconds before anything appears, and then the text streams at the same rate as always. Users describe this as “it hangs and then works”, which is exactly right."}]},{"t":"p","runs":[{"text":"Streaming hides decode, not prefill.","b":true},{"text":" Streaming is the standard latency fix and it only helps after the first token. If your problem is a five-second silence, streaming does not touch it. What touches it is a shorter input, a cache hit, or a smaller model."}]},{"t":"p","runs":[{"text":"Output length is the strongest lever on total time.","b":true},{"text":" Each output token costs a full pass over the weights. Halving a verbose answer halves most of the wall-clock. “Answer in at most three sentences” is a latency optimisation as much as a style choice."}]},{"t":"p","runs":[{"text":"A cache hit removes prefill for the cached part.","b":true},{"text":" This is why caching improves perceived speed as much as it improves cost, and often more noticeably."}]},{"t":"h2","id":"the-quadratic-that-mostly-is-not","runs":[{"text":"The quadratic that mostly is not"}]},{"t":"p","runs":[{"text":"Attention is quadratic in sequence length in the arithmetic. In practice, serving stacks have spent years making that not dominate: flash-attention kernels avoid materialising the full matrix, chunked prefill breaks a long input into pieces, and paged caches keep memory usable."}]},{"t":"p","runs":[{"text":"The result is that measured prefill time is closer to linear in input length across the range most applications use, with the quadratic term becoming visible at very long contexts. So do not model prefill as quadratic and do not assume it is free: measure it at the lengths you actually send."}]},{"t":"h2","id":"measure-it-yourself","runs":[{"text":"Measure it yourself"}]},{"t":"p","runs":[{"text":"Everything here is observable with a stopwatch and a streaming client."}]},{"t":"code","lang":"python","label":"Python","text":"t0 = time.perf_counter()\nfirst = None\nfor i, chunk in enumerate(stream_call(messages)):\n    if first is None:\n        first = time.perf_counter() - t0        # TTFT\ntotal = time.perf_counter() - t0\nprint(f\"ttft={first:.2f}s total={total:.2f}s tps={n_out/(total-first):.1f}\")"},{"t":"p","runs":[{"text":"Run it at three input lengths and two output lengths, ten times each, and record medians and 95th percentiles. Half an hour of work replaces every argument your team is going to have about latency."}]},{"t":"p","runs":[{"text":"The 95th percentile is the number that matters. Provider latency is noticeably variable (queueing, capacity, other tenants) and a median that looks fine can hide a tail where one request in twenty takes four times as long."}]},{"t":"h2","id":"the-rest-of-the-latency-budget","runs":[{"text":"The rest of the latency budget"}]},{"t":"p","runs":[{"text":"The model is often not the largest term. A realistic breakdown for a retrieval-augmented answer:"}]},{"t":"ul","items":[[{"text":"Query rewriting call: 200 to 500 ms, if you do it."}],[{"text":"Embedding the query: 10 to 50 ms locally, 50 to 200 ms hosted."}],[{"text":"Vector search: 10 to 100 ms."}],[{"text":"Reranking 50 candidates: 50 to 400 ms depending on hardware."}],[{"text":"Prefill on 8,000 tokens: 200 ms to 1 s."}],[{"text":"Decode of 400 tokens: 2 to 8 s."}]]},{"t":"p","runs":[{"text":"Two lessons. The pre-model stages add up to something comparable to prefill, and they are serial by default: run the ones that do not depend on each other in parallel. And decode dominates everything, which puts the answer’s length at the centre of any latency work."}]},{"t":"h2","id":"what-to-do-when-it-is-still-too-slow","runs":[{"text":"What to do when it is still too slow"}]},{"t":"p","runs":[{"text":"In rough order of return:"}]},{"t":"ol","start":1,"items":[[{"text":"Shorten the output.","b":true},{"text":" Largest effect, no infrastructure change."}],[{"text":"Cache the prefix.","b":true},{"text":" Removes most of prefill on repeat traffic."}],[{"text":"Stream, and show something immediately","b":true},{"text":" (the retrieved sources, a skeleton) so the wait is occupied."}],[{"text":"Parallelise the pre-model stages.","b":true}],[{"text":"Cut input length","b":true},{"text":", which the selection module was already arguing for."}],[{"text":"Use a smaller model for the easy majority","b":true},{"text":", which the routing lesson covers."}]]},{"t":"p","runs":[{"text":"Notice that only the last two change what the model sees. Most latency work is not a context problem, but the two that are, are the ones you now know how to measure."}]}],"check":{"question":[{"text":"Users complain about a five-second pause before any text appears. The team enables streaming and the complaint persists. Why?"}],"options":[{"text":[{"text":"Streaming increases total time by adding per-chunk overhead, offsetting its benefit."}],"correct":false,"tempting":[{"text":"Per-chunk overhead exists and is tiny; it cannot account for a five-second wait remaining unchanged."}]},{"text":[{"text":"The pause is prefill over a long input, which happens before the first token exists to stream."}],"correct":true,"tempting":null},{"text":[{"text":"The output is too long, so the model is still generating when the user gives up."}],"correct":false,"tempting":[{"text":"Long output delays completion, but with streaming the first tokens would appear quickly, which is not what is described."}]},{"text":[{"text":"Streaming requires a cache hit to be effective, and the prefix was not cached."}],"correct":false,"tempting":[{"text":"Invents a dependency; streaming and caching are independent, though caching would in fact shorten this particular wait."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"what-a-token-costs","title":"The arithmetic of a request","minutes":10},"next":{"slug":"caching-the-prefix","title":"Caching a stable prefix","minutes":8}}}