{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"streaming-and-latency","title":"Streaming, and the two latencies users feel","url":"https://addaly.com/learn/how-llms-work/streaming-and-latency","position":29,"of":78,"minutes":8,"answer":"Time to first token and time between tokens have different causes and different fixes, and conflating them produces optimisation work that users never notice.","keyIdea":"Time to first token and time between tokens have different causes and different fixes, and conflating them produces optimisation work that users never notice."},"body":{"format":"markdown","text":"## Two numbers, not one\n\n\"The model is slow\" is not a measurement. There are two independent latencies, they are governed by different parts of the system, and improving one does nothing for the other.\n\n**Time to first token (TTFT)** is how long before anything appears. It is dominated by the prefill phase (processing your whole prompt in parallel) plus network round trip and queueing. It grows with prompt length.\n\n**Inter-token latency (ITL)** is the gap between successive tokens once flowing. It is one decode step: read all the weights, produce one token. It barely depends on prompt length and is roughly constant for a given model and load.\n\nTotal time is approximately `TTFT + (output_tokens × ITL)`. Both terms matter, and which one is your problem determines what to fix.\n\n## Streaming changes what users perceive\n\nStreaming sends each token as it is produced, over server-sent events or a websocket. The total time is unchanged. The *perceived* wait collapses from the total to the TTFT, and that is a large difference in how a product feels.\n\nReading speed gives you the target. Comfortable adult reading is roughly 250 words per minute, and a word is about 1.3 tokens, so somewhere near 5 to 6 tokens per second keeps pace with a reader. Above about 20 tokens per second, further speed is invisible for text somebody is reading, though it still matters when output feeds code, when the user is skimming for a result, or when a long response is being generated before any action is taken.\n\n> Figure: Two latencies, two fixes, one request. Bars (seconds to the last token, at 40 tokens per second): As built: 3.0 s first token, 600 tokens out 18; Cache the prefix: 0.4 s first token 15.4; Ask for 200 tokens instead of 600 8; Both changes 5.4. Streaming changes none of these numbers; it changes which one the user feels. Caching the prefix moves the first token from three seconds to under half a second, and that is what a reader notices; but the output length governs the total, and forty tokens a second is already seven times faster than anybody reads.\n\nThis is a useful check on optimisation effort. If you are at 40 tokens per second and your users read the output, faster decoding buys you nothing they can perceive. Lowering TTFT from 3 seconds to 800 milliseconds buys you a great deal.\n\n## Fixing time to first token\n\nIn rough order of effect:\n\n- **Shorten the prompt.** Prefill is proportional to prompt length. Ten thousand tokens of instructions you never revised is the commonest cause of a slow first token.\n- **Cache the prefix.** If the first 8,000 tokens are identical across requests, the server can reuse the stored keys and values instead of recomputing them. Providers expose this as prompt caching, and it typically cuts both the latency and the price of the cached portion substantially. It requires a byte-identical prefix, which is why module one told you to put the variable material last.\n- **Reduce queueing.** Under load, TTFT is partly waiting for a slot. This is a capacity problem, not a model problem.\n- **Move the compute closer.** Network round trip is tens to hundreds of milliseconds and is not nothing when you are targeting under a second.\n\n## Fixing inter-token latency\n\n- **Use a smaller model**, or a sparse one with fewer active parameters. Decode time scales with the weights that must be read.\n- **Quantise.** Fewer bytes per weight means less memory traffic per token, which is the bottleneck. This is why a 4-bit model can be noticeably faster than the same model at 16-bit, not merely smaller.\n- **Speculative decoding**, from the previous lesson.\n- **Better batching** on the server, which raises throughput but can slightly worsen a single user's ITL. Throughput and latency are in tension; know which you are optimising.\n\n## Measure before you optimise\n\n```python\nimport time\nt0 = time.time(); first = None; n = 0\nfor chunk in client.chat.completions.create(..., stream=True):\n    if first is None: first = time.time() - t0\n    n += 1\ntotal = time.time() - t0\nprint(f\"TTFT {first:.2f}s  tokens {n}  ITL {(total-first)/max(n-1,1)*1000:.0f}ms\")\n```\n\nTen lines, and it turns \"it feels slow\" into two numbers you can act on. Run it at your real prompt length and at your real time of day, because both matter.\n\n## The design lever nobody uses\n\nThe largest latency improvement available to most applications is not technical. It is asking for less output. A response of 800 tokens takes four times as long to produce as one of 200, and users frequently prefer the shorter one. Before optimising the serving stack, check whether your prompt is asking the model to restate the question, explain its approach and offer three alternatives, and whether anybody reads that."},"check":{"question":"An application streams responses at 45 tokens per second with a 4-second time to first token, and users complain it is slow. Where should the work go?","options":[{"text":"Into the first token: shorten or cache the prompt prefix, because 45 tokens per second is already faster than anybody reads.","correct":true},{"text":"Into decode speed, since higher tokens per second is the standard measure of model performance.","correct":false,"tempting":"Tokens per second is the headline number in every benchmark, so optimising it feels like the obvious target."},{"text":"Into a smaller model, which improves both latencies at once.","correct":false,"tempting":"A smaller model does help both numbers, which makes it an appealing single change with broad effect."},{"text":"Into disabling streaming, since partial output makes the wait feel longer.","correct":false,"tempting":"Perceived-performance arguments can cut both ways, and there are interfaces where progressive rendering is distracting."}]},"prev":{"slug":"temperature-zero-is-not-deterministic","title":"Why temperature zero still gives you different answers","url":"https://addaly.com/learn/how-llms-work/temperature-zero-is-not-deterministic"},"next":{"slug":"context-is-not-memory","title":"The context window is not memory","url":"https://addaly.com/learn/how-llms-work/context-is-not-memory"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Streaming, and the two latencies users feel', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/streaming-and-latency (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Streaming, and the two latencies users feel","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Time to first token and time between tokens have different causes and different fixes, and conflating them produces optimisation work that users never notice.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"seconds to the last token, at 40 tokens per second","title":"Two latencies, two fixes, one request","series":[{"label":"As built: 3.0 s first token, 600 tokens out","value":18},{"label":"Cache the prefix: 0.4 s first token","value":15.4},{"label":"Ask for 200 tokens instead of 600","value":8},{"label":"Both changes","value":5.4,"highlight":true}],"caption":"Streaming changes none of these numbers; it changes which one the user feels. Caching the prefix moves the first token from three seconds to under half a second, and that is what a reader notices; but the output length governs the total, and forty tokens a second is already seven times faster than anybody reads."},"alt":"Figure: Two latencies, two fixes, one request. Bars (seconds to the last token, at 40 tokens per second): As built: 3.0 s first token, 600 tokens out 18; Cache the prefix: 0.4 s first token 15.4; Ask for 200 tokens instead of 600 8; Both changes 5.4. Streaming changes none of these numbers; it changes which one the user feels. Caching the prefix moves the first token from three seconds to under half a second, and that is what a reader notices; but the output length governs the total, and forty tokens a second is already seven times faster than anybody reads."}},"blocks":[{"t":"h2","id":"two-numbers-not-one","runs":[{"text":"Two numbers, not one"}]},{"t":"p","runs":[{"text":"“The model is slow” is not a measurement. There are two independent latencies, they are governed by different parts of the system, and improving one does nothing for the other."}]},{"t":"p","runs":[{"text":"Time to first token (TTFT)","b":true},{"text":" is how long before anything appears. It is dominated by the prefill phase (processing your whole prompt in parallel) plus network round trip and queueing. It grows with prompt length."}]},{"t":"p","runs":[{"text":"Inter-token latency (ITL)","b":true},{"text":" is the gap between successive tokens once flowing. It is one decode step: read all the weights, produce one token. It barely depends on prompt length and is roughly constant for a given model and load."}]},{"t":"p","runs":[{"text":"Total time is approximately "},{"text":"TTFT + (output_tokens × ITL)","code":true},{"text":". Both terms matter, and which one is your problem determines what to fix."}]},{"t":"h2","id":"streaming-changes-what-users-perceive","runs":[{"text":"Streaming changes what users perceive"}]},{"t":"p","runs":[{"text":"Streaming sends each token as it is produced, over server-sent events or a websocket. The total time is unchanged. The "},{"text":"perceived","i":true},{"text":" wait collapses from the total to the TTFT, and that is a large difference in how a product feels."}]},{"t":"p","runs":[{"text":"Reading speed gives you the target. Comfortable adult reading is roughly 250 words per minute, and a word is about 1.3 tokens, so somewhere near 5 to 6 tokens per second keeps pace with a reader. Above about 20 tokens per second, further speed is invisible for text somebody is reading, though it still matters when output feeds code, when the user is skimming for a result, or when a long response is being generated before any action is taken."}]},{"t":"p","runs":[{"text":"This is a useful check on optimisation effort. If you are at 40 tokens per second and your users read the output, faster decoding buys you nothing they can perceive. Lowering TTFT from 3 seconds to 800 milliseconds buys you a great deal."}]},{"t":"h2","id":"fixing-time-to-first-token","runs":[{"text":"Fixing time to first token"}]},{"t":"p","runs":[{"text":"In rough order of effect:"}]},{"t":"ul","items":[[{"text":"Shorten the prompt.","b":true},{"text":" Prefill is proportional to prompt length. Ten thousand tokens of instructions you never revised is the commonest cause of a slow first token."}],[{"text":"Cache the prefix.","b":true},{"text":" If the first 8,000 tokens are identical across requests, the server can reuse the stored keys and values instead of recomputing them. Providers expose this as prompt caching, and it typically cuts both the latency and the price of the cached portion substantially. It requires a byte-identical prefix, which is why module one told you to put the variable material last."}],[{"text":"Reduce queueing.","b":true},{"text":" Under load, TTFT is partly waiting for a slot. This is a capacity problem, not a model problem."}],[{"text":"Move the compute closer.","b":true},{"text":" Network round trip is tens to hundreds of milliseconds and is not nothing when you are targeting under a second."}]]},{"t":"h2","id":"fixing-inter-token-latency","runs":[{"text":"Fixing inter-token latency"}]},{"t":"ul","items":[[{"text":"Use a smaller model","b":true},{"text":", or a sparse one with fewer active parameters. Decode time scales with the weights that must be read."}],[{"text":"Quantise.","b":true},{"text":" Fewer bytes per weight means less memory traffic per token, which is the bottleneck. This is why a 4-bit model can be noticeably faster than the same model at 16-bit, not merely smaller."}],[{"text":"Speculative decoding","b":true},{"text":", from the previous lesson."}],[{"text":"Better batching","b":true},{"text":" on the server, which raises throughput but can slightly worsen a single user’s ITL. Throughput and latency are in tension; know which you are optimising."}]]},{"t":"h2","id":"measure-before-you-optimise","runs":[{"text":"Measure before you optimise"}]},{"t":"code","lang":"python","label":"Python","text":"import time\nt0 = time.time(); first = None; n = 0\nfor chunk in client.chat.completions.create(..., stream=True):\n    if first is None: first = time.time() - t0\n    n += 1\ntotal = time.time() - t0\nprint(f\"TTFT {first:.2f}s  tokens {n}  ITL {(total-first)/max(n-1,1)*1000:.0f}ms\")"},{"t":"p","runs":[{"text":"Ten lines, and it turns “it feels slow” into two numbers you can act on. Run it at your real prompt length and at your real time of day, because both matter."}]},{"t":"h2","id":"the-design-lever-nobody-uses","runs":[{"text":"The design lever nobody uses"}]},{"t":"p","runs":[{"text":"The largest latency improvement available to most applications is not technical. It is asking for less output. A response of 800 tokens takes four times as long to produce as one of 200, and users frequently prefer the shorter one. Before optimising the serving stack, check whether your prompt is asking the model to restate the question, explain its approach and offer three alternatives, and whether anybody reads that."}]}],"check":{"question":[{"text":"An application streams responses at 45 tokens per second with a 4-second time to first token, and users complain it is slow. Where should the work go?"}],"options":[{"text":[{"text":"Into the first token: shorten or cache the prompt prefix, because 45 tokens per second is already faster than anybody reads."}],"correct":true,"tempting":null},{"text":[{"text":"Into decode speed, since higher tokens per second is the standard measure of model performance."}],"correct":false,"tempting":[{"text":"Tokens per second is the headline number in every benchmark, so optimising it feels like the obvious target."}]},{"text":[{"text":"Into a smaller model, which improves both latencies at once."}],"correct":false,"tempting":[{"text":"A smaller model does help both numbers, which makes it an appealing single change with broad effect."}]},{"text":[{"text":"Into disabling streaming, since partial output makes the wait feel longer."}],"correct":false,"tempting":[{"text":"Perceived-performance arguments can cut both ways, and there are interfaces where progressive rendering is distracting."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"temperature-zero-is-not-deterministic","title":"Why temperature zero still gives you different answers","minutes":8},"next":{"slug":"context-is-not-memory","title":"The context window is not memory","minutes":7}}}