{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"latency-is-a-distribution","title":"The tail is what your users feel","url":"https://addaly.com/learn/evaluating-ai/latency-is-a-distribution","position":69,"of":76,"minutes":9,"answer":"Report p50, p95 and p99 rather than a mean, because generation time is right-skewed and a session making ten calls meets the 95th percentile about forty per cent of the time.","keyIdea":"Report p50, p95 and p99 rather than a mean, because generation time is right-skewed and a session making ten calls meets the 95th percentile about forty per cent of the time."},"body":{"format":"markdown","text":"## Why the mean describes nobody\n\nLatency for a generative system is right-skewed by construction: total time scales with the number of output tokens, and output length has a long tail. So the distribution has a dense body and a stretched right side, and the mean sits somewhere between the median and the 90th percentile, describing neither.\n\nA real shape looks like this:\n\n- p50: 1.2 s\n- p90: 4.1 s\n- p95: 6.8 s\n- p99: 14 s\n- max: 62 s (a timeout that did not fire)\n\nThe mean might be 2.4 seconds, and quoting it hides that one request in twenty waits nearly seven seconds and one in a hundred waits fourteen.\n\n## The arithmetic that makes the tail matter\n\nPercentiles per request understate the experience, because users make more than one request. If a session involves ten calls, the chance of meeting at least one call above the 95th percentile is:\n\n```\n1 - 0.95 ** 10 = 0.40\n```\n\nForty per cent of sessions contain a slow call. An agent taking thirty steps meets it almost every time: `1 - 0.95 ** 30 = 0.79`. This is why tail latency dominates the felt quality of multi-step products, and why an improvement to p99 can matter more than an improvement to p50.\n\n> Figure: Sessions that meet at least one slow call. Line chart of Sessions hitting p95 at least once, % against Calls per session: 1 − 0.95 to the power n (1, 5) (5, 23) (10, 40) (20, 64) (30, 79). A ten-call session meets a 95th-percentile call 40% of the time; a thirty-step agent, 79%. That is why an improvement to p99 can matter more than one to p50.\n\nReport both the per-call percentiles and the **per-session** total, because the session is the unit of experience.\n\n## Time to first token is a different metric\n\nFor a streaming interface, the user's sense of speed is governed by time to first token, and then by the token rate. A response with a 400 ms first token and a steady stream feels fast at eight seconds total; one with three seconds of nothing feels broken at four.\n\nMeasure three numbers for streaming: time to first token, tokens per second, and total time. Optimising them pulls in different directions (prefill work and retrieval sit before the first token, while output length drives the rest) and a change that improves total time while delaying the first token will be reported by users as a slowdown.\n\n## What is actually in the tail\n\nLook before you optimise. The usual contributors, in rough order:\n\n- **Long outputs.** Check the correlation between output tokens and latency; it is usually most of the variance.\n- **Retries.** A timeout followed by a retry is two full attempts, so a 5-second timeout can produce an 11-second experience. Retries multiply the tail rather than trimming it.\n- **Long inputs**, since prefill time grows with input length: a retrieved context of 20 chunks costs before generation starts.\n- **Cold caches and cold starts** after a deploy or a scale-up.\n- **Provider queueing** at peak hours, which is invisible in your own metrics except as latency.\n\n## Timeouts change your quality measurement\n\nThis is the subtle one. If a request times out at 5 seconds and the slow requests are disproportionately the hard ones, then the answers you measured are the ones that finished, and hard, slow, more-likely-wrong requests are missing from the sample. Your quality metric describes the survivors.\n\nTwo consequences: count timed-out requests as failures in every quality metric, and report the timeout rate beside every quality number. A prompt change that improves quality by two points while doubling the timeout rate has made the product worse, and only the pair of numbers shows it.\n\n## Cost has a tail too\n\nThe same arithmetic applies to money. Mean cost per request hides that a small fraction of requests (long documents, deep agent loops, a retry storm) consume a disproportionate share of the bill. Report cost percentiles and the share of spend taken by the top 1% of requests. Then cap: a per-request token limit and a per-session step limit turn an unbounded tail into a bounded one, and the cap firing is a metric worth watching.\n\n## Measure under load, not in a notebook\n\nLatency measured one request at a time is a lower bound and nothing more. Run a simple load test (`k6`, `locust`, or a loop with `asyncio`) at the concurrency you expect, and plot latency against concurrency. The curve has a knee, and knowing where it is tells you both your capacity and what your users will experience at the busiest hour, which is the only hour anybody remembers."},"check":{"question":"An agent makes 30 model calls per task, each with a p95 latency of 6 seconds. What follows?","options":[{"text":"Roughly 5% of tasks will contain a slow call, matching the per-call percentile.","correct":false,"tempting":"Applies a per-call percentile directly to a task made of thirty calls; the chance compounds across calls rather than staying fixed."},{"text":"About 79% of tasks will contain at least one call above the 95th percentile, so tail latency dominates the task experience.","correct":true},{"text":"Task latency is 30 times the median, since the percentiles average out over many calls.","correct":false,"tempting":"Averaging is the intuition the arithmetic contradicts: with many draws the extremes are met routinely rather than cancelling."},{"text":"Nothing follows without knowing the mean, since percentiles cannot be combined across calls.","correct":false,"tempting":"Combining is exactly what the independence arithmetic allows, and the mean would tell you less about the tail than the percentile already does."}]},"prev":{"slug":"cost-latency-and-drift","title":"Cost, latency, and the model that changed under you","url":"https://addaly.com/learn/evaluating-ai/cost-latency-and-drift"},"next":{"slug":"monitoring-in-production","title":"Watching a system you can no longer test","url":"https://addaly.com/learn/evaluating-ai/monitoring-in-production"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The tail is what your users feel', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/latency-is-a-distribution (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The tail is what your users feel","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Report p50, p95 and p99 rather than a mean, because generation time is right-skewed and a session making ten calls meets the 95th percentile about forty per cent of the time.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Sessions that meet at least one slow call","series":[{"label":"1 − 0.95 to the power n","points":[[1,5],[5,23],[10,40],[20,64],[30,79]]}],"xLabel":"Calls per session","yLabel":"Sessions hitting p95 at least once, %","caption":"A ten-call session meets a 95th-percentile call 40% of the time; a thirty-step agent, 79%. That is why an improvement to p99 can matter more than one to p50."},"alt":"Figure: Sessions that meet at least one slow call. Line chart of Sessions hitting p95 at least once, % against Calls per session: 1 − 0.95 to the power n (1, 5) (5, 23) (10, 40) (20, 64) (30, 79). A ten-call session meets a 95th-percentile call 40% of the time; a thirty-step agent, 79%. That is why an improvement to p99 can matter more than one to p50."}},"blocks":[{"t":"h2","id":"why-the-mean-describes-nobody","runs":[{"text":"Why the mean describes nobody"}]},{"t":"p","runs":[{"text":"Latency for a generative system is right-skewed by construction: total time scales with the number of output tokens, and output length has a long tail. So the distribution has a dense body and a stretched right side, and the mean sits somewhere between the median and the 90th percentile, describing neither."}]},{"t":"p","runs":[{"text":"A real shape looks like this:"}]},{"t":"ul","items":[[{"text":"p50: 1.2 s"}],[{"text":"p90: 4.1 s"}],[{"text":"p95: 6.8 s"}],[{"text":"p99: 14 s"}],[{"text":"max: 62 s (a timeout that did not fire)"}]]},{"t":"p","runs":[{"text":"The mean might be 2.4 seconds, and quoting it hides that one request in twenty waits nearly seven seconds and one in a hundred waits fourteen."}]},{"t":"h2","id":"the-arithmetic-that-makes-the-tail-matter","runs":[{"text":"The arithmetic that makes the tail matter"}]},{"t":"p","runs":[{"text":"Percentiles per request understate the experience, because users make more than one request. If a session involves ten calls, the chance of meeting at least one call above the 95th percentile is:"}]},{"t":"code","lang":"","label":"Text","text":"1 - 0.95 ** 10 = 0.40"},{"t":"p","runs":[{"text":"Forty per cent of sessions contain a slow call. An agent taking thirty steps meets it almost every time: "},{"text":"1 - 0.95 ** 30 = 0.79","code":true},{"text":". This is why tail latency dominates the felt quality of multi-step products, and why an improvement to p99 can matter more than an improvement to p50."}]},{"t":"p","runs":[{"text":"Report both the per-call percentiles and the "},{"text":"per-session","b":true},{"text":" total, because the session is the unit of experience."}]},{"t":"h2","id":"time-to-first-token-is-a-different-metric","runs":[{"text":"Time to first token is a different metric"}]},{"t":"p","runs":[{"text":"For a streaming interface, the user’s sense of speed is governed by time to first token, and then by the token rate. A response with a 400 ms first token and a steady stream feels fast at eight seconds total; one with three seconds of nothing feels broken at four."}]},{"t":"p","runs":[{"text":"Measure three numbers for streaming: time to first token, tokens per second, and total time. Optimising them pulls in different directions (prefill work and retrieval sit before the first token, while output length drives the rest) and a change that improves total time while delaying the first token will be reported by users as a slowdown."}]},{"t":"h2","id":"what-is-actually-in-the-tail","runs":[{"text":"What is actually in the tail"}]},{"t":"p","runs":[{"text":"Look before you optimise. The usual contributors, in rough order:"}]},{"t":"ul","items":[[{"text":"Long outputs.","b":true},{"text":" Check the correlation between output tokens and latency; it is usually most of the variance."}],[{"text":"Retries.","b":true},{"text":" A timeout followed by a retry is two full attempts, so a 5-second timeout can produce an 11-second experience. Retries multiply the tail rather than trimming it."}],[{"text":"Long inputs","b":true},{"text":", since prefill time grows with input length: a retrieved context of 20 chunks costs before generation starts."}],[{"text":"Cold caches and cold starts","b":true},{"text":" after a deploy or a scale-up."}],[{"text":"Provider queueing","b":true},{"text":" at peak hours, which is invisible in your own metrics except as latency."}]]},{"t":"h2","id":"timeouts-change-your-quality-measurement","runs":[{"text":"Timeouts change your quality measurement"}]},{"t":"p","runs":[{"text":"This is the subtle one. If a request times out at 5 seconds and the slow requests are disproportionately the hard ones, then the answers you measured are the ones that finished, and hard, slow, more-likely-wrong requests are missing from the sample. Your quality metric describes the survivors."}]},{"t":"p","runs":[{"text":"Two consequences: count timed-out requests as failures in every quality metric, and report the timeout rate beside every quality number. A prompt change that improves quality by two points while doubling the timeout rate has made the product worse, and only the pair of numbers shows it."}]},{"t":"h2","id":"cost-has-a-tail-too","runs":[{"text":"Cost has a tail too"}]},{"t":"p","runs":[{"text":"The same arithmetic applies to money. Mean cost per request hides that a small fraction of requests (long documents, deep agent loops, a retry storm) consume a disproportionate share of the bill. Report cost percentiles and the share of spend taken by the top 1% of requests. Then cap: a per-request token limit and a per-session step limit turn an unbounded tail into a bounded one, and the cap firing is a metric worth watching."}]},{"t":"h2","id":"measure-under-load-not-in-a-notebook","runs":[{"text":"Measure under load, not in a notebook"}]},{"t":"p","runs":[{"text":"Latency measured one request at a time is a lower bound and nothing more. Run a simple load test ("},{"text":"k6","code":true},{"text":", "},{"text":"locust","code":true},{"text":", or a loop with "},{"text":"asyncio","code":true},{"text":") at the concurrency you expect, and plot latency against concurrency. The curve has a knee, and knowing where it is tells you both your capacity and what your users will experience at the busiest hour, which is the only hour anybody remembers."}]}],"check":{"question":[{"text":"An agent makes 30 model calls per task, each with a p95 latency of 6 seconds. What follows?"}],"options":[{"text":[{"text":"Roughly 5% of tasks will contain a slow call, matching the per-call percentile."}],"correct":false,"tempting":[{"text":"Applies a per-call percentile directly to a task made of thirty calls; the chance compounds across calls rather than staying fixed."}]},{"text":[{"text":"About 79% of tasks will contain at least one call above the 95th percentile, so tail latency dominates the task experience."}],"correct":true,"tempting":null},{"text":[{"text":"Task latency is 30 times the median, since the percentiles average out over many calls."}],"correct":false,"tempting":[{"text":"Averaging is the intuition the arithmetic contradicts: with many draws the extremes are met routinely rather than cancelling."}]},{"text":[{"text":"Nothing follows without knowing the mean, since percentiles cannot be combined across calls."}],"correct":false,"tempting":[{"text":"Combining is exactly what the independence arithmetic allows, and the mean would tell you less about the tail than the percentile already does."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"cost-latency-and-drift","title":"Cost, latency, and the model that changed under you","minutes":9},"next":{"slug":"monitoring-in-production","title":"Watching a system you can no longer test","minutes":9}}}