{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"where-the-milliseconds-go","title":"Where the milliseconds go","url":"https://addaly.com/learn/building-with-ai/where-the-milliseconds-go","position":72,"of":80,"minutes":9,"answer":"A user feels time to first token and total time, and each is a sum of stages you can measure from traces (network, retrieval, reranking, the model's prefill, its generation.","keyIdea":"A user feels time to first token and total time, and each is a sum of stages you can measure from traces (network, retrieval, reranking, the model's prefill, its generation, and a full extra round trip for every tool call) so the cheapest wins are prefix caching, a smaller model for the first token, parallel tools and streaming, not a faster provider."},"body":{"format":"markdown","text":"## Two numbers a user feels\n\nThe streaming lesson in the first module drew the distinction: time to first token is how long the screen stays empty, and total time is how long until the answer is complete. People forgive a long total if the first token is fast and the text keeps flowing. They do not forgive a long silence. So the budget has two lines, and the first is the one to protect.\n\n## The stages\n\nRead a trace from the evaluation module and the request breaks into pieces, each with a duration:\n\n- **Network to your server**: tens of milliseconds, more on a phone on a poor connection.\n- **Retrieval**: query rewrite, if any, at 200 to 500 milliseconds because it is a model call; embedding at 5 to 100; search at 5 to 50; reranking at 100 on a GPU and a second or more on a CPU. The retrieval module gave these.\n- **Network to the provider**: a round trip to the provider's region. From India to a US endpoint that can be 200 milliseconds before anything happens; a regional endpoint, where one exists, cuts it to tens.\n- **Prefill**: the model reads the whole input before producing anything. Roughly linear in input tokens; a 10,000-token prompt takes noticeably longer to first token than a 2,000-token one. A cached prefix is skipped, which is why caching cuts time to first token as well as cost.\n- **Generation**: tokens per second, from tens for a large model to well over a hundred for a small one. Total generation time is output length divided by that rate.\n- **Tool calls**: each one is a full extra round trip: the model stops, your code runs the tool, the model reads the result and starts again, paying prefill and time to first token a second time. Three sequential tool calls is four model calls.\n\nTime to first token is everything before generation. Total time is that plus generation plus every tool round trip.\n\n## Measure the tail, not the mean\n\nA mean of 800 milliseconds can hide a 95th percentile of four seconds, and the users at the 95th percentile are the ones who complain. Record every stage per request, and report the 50th, 95th and 99th percentiles per stage, per day. The evaluation course has a lesson on latency as a distribution; here the practical point is that the tail usually belongs to one stage (a reranker on a busy CPU, a tool that occasionally hangs, a provider under load) and the per-stage percentiles point at it.\n\n## The cheap wins, in order\n\n**Cache the prefix.** The prompt-caching lesson's ordering rule, applied here for speed: a stable system prompt and stable context at the front, variable text at the end. A cache hit removes most of the prefill.\n\n**Stream.** Nothing changes in the total, and everything changes in what the user feels.\n\n**Shrink the input.** Fewer retrieved chunks, a shorter system prompt, trimmed history. Prefill is linear; half the tokens is roughly half the wait before the first one.\n\n**Use a smaller model where the first token matters.** A small model's time to first token and generation rate are both several times better. For a conversational reply that must feel immediate, the small model with the retrieval done well often beats the large model with the same retrieval, on both speed and the evaluation set.\n\n**Run tools in parallel.** The loop lesson covered it: independent reads dispatched together cost one round trip instead of three.\n\n**Start the next stage before the last one finishes.** Begin embedding the query while the rewrite runs if the rewrite is optional; render the citation cards while the answer streams.\n\n## What a faster provider does not fix\n\nSwitching providers for speed usually moves one stage by a fraction while leaving the structure (retrieval, tool round trips, input length) untouched. Measure per stage first. A team that switches providers to fix a four-second p95 caused by a CPU reranker will find the p95 at 3.8 seconds afterwards.\n\n## A budget, written down\n\nFor an interactive assistant, a working target: first token under a second at the 95th percentile, and a total under four seconds for a typical answer. Assign the second to stages (say 300 milliseconds for retrieval, 200 for the provider round trip and prefill with a cache hit, the rest as headroom) and when a stage overspends its share, that is the stage to work on. A budget that exists only as a total is met by nobody in particular.\n\n## Latency on the phone\n\nMost of this readership is on a phone, sometimes on a slow connection. The network stage that is 30 milliseconds on office wifi is 300 on a weak mobile signal, and a streamed response that arrives token by token over a lossy connection can feel jerky. Test on a real phone with the connection throttled (every desktop browser can simulate a slow network) and make sure the first token still appears before the user wonders whether anything is happening."},"check":{"question":"An assistant's 95th-percentile time to first token is 3.5 seconds. Traces show retrieval at 300 ms, reranking at 2.4 s on a shared CPU, and the provider at 500 ms. The team proposes switching to a faster model provider. What will happen?","options":[{"text":"First-token time will roughly halve, since the provider stage is where model speed applies.","correct":false,"tempting":"The provider stage is 500 ms of 3.5 s; even eliminating it entirely leaves 3 s."},{"text":"Little change, because the reranker on a shared CPU dominates and a different provider does not touch it.","correct":true},{"text":"Total time will improve but first-token time will not, since providers only affect generation speed and not the stages that come before it.","correct":false,"tempting":"Providers affect prefill and first-token time as well as generation; the issue is proportion, not which stage they touch."},{"text":"First-token time will worsen, because switching providers loses the prompt cache.","correct":false,"tempting":"A cache reset is a one-time cost per prefix, not a lasting change to the tail."}]},"prev":{"slug":"the-cost-model-of-a-feature","title":"The cost model of a feature, line by line","url":"https://addaly.com/learn/building-with-ai/the-cost-model-of-a-feature"},"next":{"slug":"caching-at-three-levels","title":"Caching at three levels: exact, prefix and semantic","url":"https://addaly.com/learn/building-with-ai/caching-at-three-levels"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the milliseconds go', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/where-the-milliseconds-go (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the milliseconds go","courseTitle":"Building with AI","module":{"index":8,"slug":"cost-latency-and-running-it","title":"Cost, latency and running it in production"},"answer":null,"keyIdea":"A user feels time to first token and total time, and each is a sum of stages you can measure from traces (network, retrieval, reranking, the model’s prefill, its generation, and a full extra round trip for every tool call) so the cheapest wins are prefix caching, a smaller model for the first token, parallel tools and streaming, not a faster provider.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-numbers-a-user-feels","runs":[{"text":"Two numbers a user feels"}]},{"t":"p","runs":[{"text":"The streaming lesson in the first module drew the distinction: time to first token is how long the screen stays empty, and total time is how long until the answer is complete. People forgive a long total if the first token is fast and the text keeps flowing. They do not forgive a long silence. So the budget has two lines, and the first is the one to protect."}]},{"t":"h2","id":"the-stages","runs":[{"text":"The stages"}]},{"t":"p","runs":[{"text":"Read a trace from the evaluation module and the request breaks into pieces, each with a duration:"}]},{"t":"ul","items":[[{"text":"Network to your server","b":true},{"text":": tens of milliseconds, more on a phone on a poor connection."}],[{"text":"Retrieval","b":true},{"text":": query rewrite, if any, at 200 to 500 milliseconds because it is a model call; embedding at 5 to 100; search at 5 to 50; reranking at 100 on a GPU and a second or more on a CPU. The retrieval module gave these."}],[{"text":"Network to the provider","b":true},{"text":": a round trip to the provider’s region. From India to a US endpoint that can be 200 milliseconds before anything happens; a regional endpoint, where one exists, cuts it to tens."}],[{"text":"Prefill","b":true},{"text":": the model reads the whole input before producing anything. Roughly linear in input tokens; a 10,000-token prompt takes noticeably longer to first token than a 2,000-token one. A cached prefix is skipped, which is why caching cuts time to first token as well as cost."}],[{"text":"Generation","b":true},{"text":": tokens per second, from tens for a large model to well over a hundred for a small one. Total generation time is output length divided by that rate."}],[{"text":"Tool calls","b":true},{"text":": each one is a full extra round trip: the model stops, your code runs the tool, the model reads the result and starts again, paying prefill and time to first token a second time. Three sequential tool calls is four model calls."}]]},{"t":"p","runs":[{"text":"Time to first token is everything before generation. Total time is that plus generation plus every tool round trip."}]},{"t":"h2","id":"measure-the-tail-not-the-mean","runs":[{"text":"Measure the tail, not the mean"}]},{"t":"p","runs":[{"text":"A mean of 800 milliseconds can hide a 95th percentile of four seconds, and the users at the 95th percentile are the ones who complain. Record every stage per request, and report the 50th, 95th and 99th percentiles per stage, per day. The evaluation course has a lesson on latency as a distribution; here the practical point is that the tail usually belongs to one stage (a reranker on a busy CPU, a tool that occasionally hangs, a provider under load) and the per-stage percentiles point at it."}]},{"t":"h2","id":"the-cheap-wins-in-order","runs":[{"text":"The cheap wins, in order"}]},{"t":"p","runs":[{"text":"Cache the prefix.","b":true},{"text":" The prompt-caching lesson’s ordering rule, applied here for speed: a stable system prompt and stable context at the front, variable text at the end. A cache hit removes most of the prefill."}]},{"t":"p","runs":[{"text":"Stream.","b":true},{"text":" Nothing changes in the total, and everything changes in what the user feels."}]},{"t":"p","runs":[{"text":"Shrink the input.","b":true},{"text":" Fewer retrieved chunks, a shorter system prompt, trimmed history. Prefill is linear; half the tokens is roughly half the wait before the first one."}]},{"t":"p","runs":[{"text":"Use a smaller model where the first token matters.","b":true},{"text":" A small model’s time to first token and generation rate are both several times better. For a conversational reply that must feel immediate, the small model with the retrieval done well often beats the large model with the same retrieval, on both speed and the evaluation set."}]},{"t":"p","runs":[{"text":"Run tools in parallel.","b":true},{"text":" The loop lesson covered it: independent reads dispatched together cost one round trip instead of three."}]},{"t":"p","runs":[{"text":"Start the next stage before the last one finishes.","b":true},{"text":" Begin embedding the query while the rewrite runs if the rewrite is optional; render the citation cards while the answer streams."}]},{"t":"h2","id":"what-a-faster-provider-does-not-fix","runs":[{"text":"What a faster provider does not fix"}]},{"t":"p","runs":[{"text":"Switching providers for speed usually moves one stage by a fraction while leaving the structure (retrieval, tool round trips, input length) untouched. Measure per stage first. A team that switches providers to fix a four-second p95 caused by a CPU reranker will find the p95 at 3.8 seconds afterwards."}]},{"t":"h2","id":"a-budget-written-down","runs":[{"text":"A budget, written down"}]},{"t":"p","runs":[{"text":"For an interactive assistant, a working target: first token under a second at the 95th percentile, and a total under four seconds for a typical answer. Assign the second to stages (say 300 milliseconds for retrieval, 200 for the provider round trip and prefill with a cache hit, the rest as headroom) and when a stage overspends its share, that is the stage to work on. A budget that exists only as a total is met by nobody in particular."}]},{"t":"h2","id":"latency-on-the-phone","runs":[{"text":"Latency on the phone"}]},{"t":"p","runs":[{"text":"Most of this readership is on a phone, sometimes on a slow connection. The network stage that is 30 milliseconds on office wifi is 300 on a weak mobile signal, and a streamed response that arrives token by token over a lossy connection can feel jerky. Test on a real phone with the connection throttled (every desktop browser can simulate a slow network) and make sure the first token still appears before the user wonders whether anything is happening."}]}],"check":{"question":[{"text":"An assistant’s 95th-percentile time to first token is 3.5 seconds. Traces show retrieval at 300 ms, reranking at 2.4 s on a shared CPU, and the provider at 500 ms. The team proposes switching to a faster model provider. What will happen?"}],"options":[{"text":[{"text":"First-token time will roughly halve, since the provider stage is where model speed applies."}],"correct":false,"tempting":[{"text":"The provider stage is 500 ms of 3.5 s; even eliminating it entirely leaves 3 s."}]},{"text":[{"text":"Little change, because the reranker on a shared CPU dominates and a different provider does not touch it."}],"correct":true,"tempting":null},{"text":[{"text":"Total time will improve but first-token time will not, since providers only affect generation speed and not the stages that come before it."}],"correct":false,"tempting":[{"text":"Providers affect prefill and first-token time as well as generation; the issue is proportion, not which stage they touch."}]},{"text":[{"text":"First-token time will worsen, because switching providers loses the prompt cache."}],"correct":false,"tempting":[{"text":"A cache reset is a one-time cost per prefix, not a lasting change to the tail."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"the-cost-model-of-a-feature","title":"The cost model of a feature, line by line","minutes":9},"next":{"slug":"caching-at-three-levels","title":"Caching at three levels: exact, prefix and semantic","minutes":9}}}