{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"measuring-latency-properly","title":"Measuring latency properly","url":"https://addaly.com/learn/open-models/measuring-latency-properly","position":68,"of":71,"minutes":8,"answer":"Report time to first token and per-token latency separately at p95 and at real concurrency after discarding warm-up requests, because prefill and decode are limited by different resources and the mean hides the tail.","keyIdea":"Report time to first token and per-token latency separately at p95 and at real concurrency after discarding warm-up requests, because prefill and decode are limited by different resources and the mean hides the tail."},"body":{"format":"markdown","text":"## Three numbers, not one\n\nTotal response time is a sum of two very different things, and reporting only the sum hides which one you can fix.\n\n**Time to first token (TTFT)** is prefill: reading the prompt. It scales with prompt length and, on a busy server, with queueing. This is what a user waiting for something to appear experiences.\n\n**Time per output token (TPOT)** is decode. It is roughly constant per token and set by memory bandwidth, as the earlier arithmetic showed.\n\n**Total** = TTFT + TPOT × output tokens.\n\nTwo systems with identical totals feel completely different: 200 ms to first token and then a steady stream reads as fast; 3 seconds of nothing followed by an instant block reads as broken. Measure and report both.\n\n## Five ways a latency measurement lies\n\n**No warm-up.** The first request loads weights, allocates the cache and compiles kernels. It can be ten times slower than steady state. Discard the first ten requests, always.\n\n**Measured at concurrency one.** A single-user measurement on a server that will carry thirty users describes a machine you will never operate. Queueing dominates real latency, and it does not appear at concurrency one.\n\n**Reporting the mean.** Latency distributions have long right tails. The mean sits below most of the pain. Report p50, p95 and p99, and make product promises on p95.\n\n**Varying prompt and output length between runs.** TTFT depends on input length and total depends on output length, so a comparison must fix both. Use a fixed synthetic prompt of a realistic length and a fixed `max_tokens` with the stop conditions disabled, or you are measuring how verbose each model is rather than how fast it is.\n\n**Measuring on the server.** Server-side timing excludes the network, the client's parsing, and any proxy. Measure where your user is.\n\n## A measurement you can trust\n\n```bash\n# vLLM ships a serving benchmark that does this properly\npython -m vllm.entrypoints.cli.benchmark serve   --backend openai-chat --model qwen3-8b   --base-url http://localhost:8000   --dataset-name random --random-input-len 1024 --random-output-len 256   --max-concurrency 16 --num-prompts 500\n```\n\nIt reports TTFT and per-token latency at percentiles, plus throughput, at a concurrency you choose. Run it at your real concurrency, and again at twice it, to see where the cliff is.\n\nFor anything else, `oha`, `hey` or a short asyncio script will do, provided you time the first streamed chunk separately from completion.\n\n## Reading the results\n\nLatency and throughput trade against each other, and the shape is a curve you should actually plot: concurrency on the horizontal axis, p95 total latency on the vertical, with throughput as a second series. What you are looking for is the knee: the concurrency beyond which latency rises steeply for little extra throughput. That point is your capacity, and it is usually well below the concurrency at which the server stops working.\n\n> Figure: Finding the knee: p95 latency against concurrency. Line chart of p95 total latency, seconds against Concurrent requests: 8k context limit, no prefix caching (1, 0.9) (4, 1.2) (8, 1.6) (16, 2.4) (24, 4.9) (32, 11.8); The same server with prefix caching (1, 0.7) (4, 0.9) (8, 1.1) (16, 1.5) (24, 2.3) (32, 4.6). Capacity is the concurrency beyond which latency rises steeply for little extra throughput, and it sits well below the point at which the server stops working. Both things that move the knee (a shorter context limit and prefix caching) are configuration rather than hardware.\n\nTwo things move the knee, and both were covered earlier: a shorter context limit and prefix caching. Both are configuration rather than hardware.\n\n## What to promise\n\nTake your p95 at your target concurrency, add the network, and add margin. If your p95 is 2.1 seconds at concurrency 16, do not promise two seconds.\n\nAnd measure the same thing after every change: a new quantisation, a new provider, a new prompt that is 400 tokens longer, a model swap. Latency regressions arrive silently and are noticed by users before they are noticed by dashboards.\n\n> Streaming deserves one line of its own. It does not make anything faster; it makes TTFT the number the user experiences instead of the total. For a long answer that is an enormous perceived improvement for no engineering cost, and it is the first thing to add when total latency is unavoidable."},"check":{"question":"A model swap leaves mean total latency unchanged, but users report the assistant now feels sluggish. What measurement would most likely explain it?","options":[{"text":"Throughput in aggregate tokens per second, which may have fallen while latency stayed flat.","correct":false,"tempting":"Aggregate throughput is a server-capacity number and does not describe what one user waits for."},{"text":"Time to first token, which may have risen substantially while faster decoding kept the total the same.","correct":true},{"text":"The p99 latency, which may have worsened even though the mean did not.","correct":false,"tempting":"Worth checking, but a tail regression affects a small share of requests rather than making the assistant feel sluggish generally."},{"text":"The number of output tokens, since a more verbose model takes longer overall.","correct":false,"tempting":"More output would raise total latency, which the premise says is unchanged."}]},"prev":{"slug":"evaluating-an-embedding-model","title":"Evaluating an embedding model on your corpus","url":"https://addaly.com/learn/open-models/evaluating-an-embedding-model"},"next":{"slug":"cost-per-task","title":"Cost per task, not cost per token","url":"https://addaly.com/learn/open-models/cost-per-task"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring latency properly', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/measuring-latency-properly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring latency properly","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Report time to first token and per-token latency separately at p95 and at real concurrency after discarding warm-up requests, because prefill and decode are limited by different resources and the mean hides the tail.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Finding the knee: p95 latency against concurrency","series":[{"label":"8k context limit, no prefix caching","points":[[1,0.9],[4,1.2],[8,1.6],[16,2.4],[24,4.9],[32,11.8]]},{"label":"The same server with prefix caching","points":[[1,0.7],[4,0.9],[8,1.1],[16,1.5],[24,2.3],[32,4.6]]}],"xLabel":"Concurrent requests","yLabel":"p95 total latency, seconds","caption":"Capacity is the concurrency beyond which latency rises steeply for little extra throughput, and it sits well below the point at which the server stops working. Both things that move the knee (a shorter context limit and prefix caching) are configuration rather than hardware."},"alt":"Figure: Finding the knee: p95 latency against concurrency. Line chart of p95 total latency, seconds against Concurrent requests: 8k context limit, no prefix caching (1, 0.9) (4, 1.2) (8, 1.6) (16, 2.4) (24, 4.9) (32, 11.8); The same server with prefix caching (1, 0.7) (4, 0.9) (8, 1.1) (16, 1.5) (24, 2.3) (32, 4.6). Capacity is the concurrency beyond which latency rises steeply for little extra throughput, and it sits well below the point at which the server stops working. Both things that move the knee (a shorter context limit and prefix caching) are configuration rather than hardware."}},"blocks":[{"t":"h2","id":"three-numbers-not-one","runs":[{"text":"Three numbers, not one"}]},{"t":"p","runs":[{"text":"Total response time is a sum of two very different things, and reporting only the sum hides which one you can fix."}]},{"t":"p","runs":[{"text":"Time to first token (TTFT)","b":true},{"text":" is prefill: reading the prompt. It scales with prompt length and, on a busy server, with queueing. This is what a user waiting for something to appear experiences."}]},{"t":"p","runs":[{"text":"Time per output token (TPOT)","b":true},{"text":" is decode. It is roughly constant per token and set by memory bandwidth, as the earlier arithmetic showed."}]},{"t":"p","runs":[{"text":"Total","b":true},{"text":" = TTFT + TPOT × output tokens."}]},{"t":"p","runs":[{"text":"Two systems with identical totals feel completely different: 200 ms to first token and then a steady stream reads as fast; 3 seconds of nothing followed by an instant block reads as broken. Measure and report both."}]},{"t":"h2","id":"five-ways-a-latency-measurement-lies","runs":[{"text":"Five ways a latency measurement lies"}]},{"t":"p","runs":[{"text":"No warm-up.","b":true},{"text":" The first request loads weights, allocates the cache and compiles kernels. It can be ten times slower than steady state. Discard the first ten requests, always."}]},{"t":"p","runs":[{"text":"Measured at concurrency one.","b":true},{"text":" A single-user measurement on a server that will carry thirty users describes a machine you will never operate. Queueing dominates real latency, and it does not appear at concurrency one."}]},{"t":"p","runs":[{"text":"Reporting the mean.","b":true},{"text":" Latency distributions have long right tails. The mean sits below most of the pain. Report p50, p95 and p99, and make product promises on p95."}]},{"t":"p","runs":[{"text":"Varying prompt and output length between runs.","b":true},{"text":" TTFT depends on input length and total depends on output length, so a comparison must fix both. Use a fixed synthetic prompt of a realistic length and a fixed "},{"text":"max_tokens","code":true},{"text":" with the stop conditions disabled, or you are measuring how verbose each model is rather than how fast it is."}]},{"t":"p","runs":[{"text":"Measuring on the server.","b":true},{"text":" Server-side timing excludes the network, the client’s parsing, and any proxy. Measure where your user is."}]},{"t":"h2","id":"a-measurement-you-can-trust","runs":[{"text":"A measurement you can trust"}]},{"t":"code","lang":"bash","label":"Bash","text":"# vLLM ships a serving benchmark that does this properly\npython -m vllm.entrypoints.cli.benchmark serve   --backend openai-chat --model qwen3-8b   --base-url http://localhost:8000   --dataset-name random --random-input-len 1024 --random-output-len 256   --max-concurrency 16 --num-prompts 500"},{"t":"p","runs":[{"text":"It reports TTFT and per-token latency at percentiles, plus throughput, at a concurrency you choose. Run it at your real concurrency, and again at twice it, to see where the cliff is."}]},{"t":"p","runs":[{"text":"For anything else, "},{"text":"oha","code":true},{"text":", "},{"text":"hey","code":true},{"text":" or a short asyncio script will do, provided you time the first streamed chunk separately from completion."}]},{"t":"h2","id":"reading-the-results","runs":[{"text":"Reading the results"}]},{"t":"p","runs":[{"text":"Latency and throughput trade against each other, and the shape is a curve you should actually plot: concurrency on the horizontal axis, p95 total latency on the vertical, with throughput as a second series. What you are looking for is the knee: the concurrency beyond which latency rises steeply for little extra throughput. That point is your capacity, and it is usually well below the concurrency at which the server stops working."}]},{"t":"p","runs":[{"text":"Two things move the knee, and both were covered earlier: a shorter context limit and prefix caching. Both are configuration rather than hardware."}]},{"t":"h2","id":"what-to-promise","runs":[{"text":"What to promise"}]},{"t":"p","runs":[{"text":"Take your p95 at your target concurrency, add the network, and add margin. If your p95 is 2.1 seconds at concurrency 16, do not promise two seconds."}]},{"t":"p","runs":[{"text":"And measure the same thing after every change: a new quantisation, a new provider, a new prompt that is 400 tokens longer, a model swap. Latency regressions arrive silently and are noticed by users before they are noticed by dashboards."}]},{"t":"quote","runs":[{"text":"Streaming deserves one line of its own. It does not make anything faster; it makes TTFT the number the user experiences instead of the total. For a long answer that is an enormous perceived improvement for no engineering cost, and it is the first thing to add when total latency is unavoidable."}]}],"check":{"question":[{"text":"A model swap leaves mean total latency unchanged, but users report the assistant now feels sluggish. What measurement would most likely explain it?"}],"options":[{"text":[{"text":"Throughput in aggregate tokens per second, which may have fallen while latency stayed flat."}],"correct":false,"tempting":[{"text":"Aggregate throughput is a server-capacity number and does not describe what one user waits for."}]},{"text":[{"text":"Time to first token, which may have risen substantially while faster decoding kept the total the same."}],"correct":true,"tempting":null},{"text":[{"text":"The p99 latency, which may have worsened even though the mean did not."}],"correct":false,"tempting":[{"text":"Worth checking, but a tail regression affects a small share of requests rather than making the assistant feel sluggish generally."}]},{"text":[{"text":"The number of output tokens, since a more verbose model takes longer overall."}],"correct":false,"tempting":[{"text":"More output would raise total latency, which the premise says is unchanged."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"evaluating-an-embedding-model","title":"Evaluating an embedding model on your corpus","minutes":9},"next":{"slug":"cost-per-task","title":"Cost per task, not cost per token","minutes":8}}}