Measuring latency properly

Three numbers, not one

Total response time is a sum of two very different things, and reporting only the sum hides which one you can fix.

Time to first token (TTFT) is prefill: reading the prompt. It scales with prompt length and, on a busy server, with queueing. This is what a user waiting for something to appear experiences.

Time per output token (TPOT) is decode. It is roughly constant per token and set by memory bandwidth, as the earlier arithmetic showed.

Total = TTFT + TPOT × output tokens.

Two systems with identical totals feel completely different: 200 ms to first token and then a steady stream reads as fast; 3 seconds of nothing followed by an instant block reads as broken. Measure and report both.

Five ways a latency measurement lies

No warm-up. The first request loads weights, allocates the cache and compiles kernels. It can be ten times slower than steady state. Discard the first ten requests, always.

Measured at concurrency one. A single-user measurement on a server that will carry thirty users describes a machine you will never operate. Queueing dominates real latency, and it does not appear at concurrency one.

Reporting the mean. Latency distributions have long right tails. The mean sits below most of the pain. Report p50, p95 and p99, and make product promises on p95.

Varying prompt and output length between runs. TTFT depends on input length and total depends on output length, so a comparison must fix both. Use a fixed synthetic prompt of a realistic length and a fixed max_tokens with the stop conditions disabled, or you are measuring how verbose each model is rather than how fast it is.

Measuring on the server. Server-side timing excludes the network, the client’s parsing, and any proxy. Measure where your user is.

A measurement you can trust

# vLLM ships a serving benchmark that does this properly
python -m vllm.entrypoints.cli.benchmark serve   --backend openai-chat --model qwen3-8b   --base-url http://localhost:8000   --dataset-name random --random-input-len 1024 --random-output-len 256   --max-concurrency 16 --num-prompts 500

It reports TTFT and per-token latency at percentiles, plus throughput, at a concurrency you choose. Run it at your real concurrency, and again at twice it, to see where the cliff is.

For anything else, oha, hey or a short asyncio script will do, provided you time the first streamed chunk separately from completion.

Reading the results

Latency and throughput trade against each other, and the shape is a curve you should actually plot: concurrency on the horizontal axis, p95 total latency on the vertical, with throughput as a second series. What you are looking for is the knee: the concurrency beyond which latency rises steeply for little extra throughput. That point is your capacity, and it is usually well below the concurrency at which the server stops working.

Two things move the knee, and both were covered earlier: a shorter context limit and prefix caching. Both are configuration rather than hardware.

What to promise

Take your p95 at your target concurrency, add the network, and add margin. If your p95 is 2.1 seconds at concurrency 16, do not promise two seconds.

And measure the same thing after every change: a new quantisation, a new provider, a new prompt that is 400 tokens longer, a model swap. Latency regressions arrive silently and are noticed by users before they are noticed by dashboards.

Streaming deserves one line of its own. It does not make anything faster; it makes TTFT the number the user experiences instead of the total. For a long answer that is an enormous perceived improvement for no engineering cost, and it is the first thing to add when total latency is unavoidable.