The tail is what your users feel

Why the mean describes nobody

Latency for a generative system is right-skewed by construction: total time scales with the number of output tokens, and output length has a long tail. So the distribution has a dense body and a stretched right side, and the mean sits somewhere between the median and the 90th percentile, describing neither.

A real shape looks like this:

• p50: 1.2 s
• p90: 4.1 s
• p95: 6.8 s
• p99: 14 s
• max: 62 s (a timeout that did not fire)

The mean might be 2.4 seconds, and quoting it hides that one request in twenty waits nearly seven seconds and one in a hundred waits fourteen.

The arithmetic that makes the tail matter

Percentiles per request understate the experience, because users make more than one request. If a session involves ten calls, the chance of meeting at least one call above the 95th percentile is:

1 - 0.95 ** 10 = 0.40

Forty per cent of sessions contain a slow call. An agent taking thirty steps meets it almost every time: 1 - 0.95 ** 30 = 0.79. This is why tail latency dominates the felt quality of multi-step products, and why an improvement to p99 can matter more than an improvement to p50.

Report both the per-call percentiles and the per-session total, because the session is the unit of experience.

Time to first token is a different metric

For a streaming interface, the user’s sense of speed is governed by time to first token, and then by the token rate. A response with a 400 ms first token and a steady stream feels fast at eight seconds total; one with three seconds of nothing feels broken at four.

Measure three numbers for streaming: time to first token, tokens per second, and total time. Optimising them pulls in different directions (prefill work and retrieval sit before the first token, while output length drives the rest) and a change that improves total time while delaying the first token will be reported by users as a slowdown.

What is actually in the tail

Look before you optimise. The usual contributors, in rough order:

• Long outputs. Check the correlation between output tokens and latency; it is usually most of the variance.
• Retries. A timeout followed by a retry is two full attempts, so a 5-second timeout can produce an 11-second experience. Retries multiply the tail rather than trimming it.
• Long inputs, since prefill time grows with input length: a retrieved context of 20 chunks costs before generation starts.
• Cold caches and cold starts after a deploy or a scale-up.
• Provider queueing at peak hours, which is invisible in your own metrics except as latency.

Timeouts change your quality measurement

This is the subtle one. If a request times out at 5 seconds and the slow requests are disproportionately the hard ones, then the answers you measured are the ones that finished, and hard, slow, more-likely-wrong requests are missing from the sample. Your quality metric describes the survivors.

Two consequences: count timed-out requests as failures in every quality metric, and report the timeout rate beside every quality number. A prompt change that improves quality by two points while doubling the timeout rate has made the product worse, and only the pair of numbers shows it.

Cost has a tail too

The same arithmetic applies to money. Mean cost per request hides that a small fraction of requests (long documents, deep agent loops, a retry storm) consume a disproportionate share of the bill. Report cost percentiles and the share of spend taken by the top 1% of requests. Then cap: a per-request token limit and a per-session step limit turn an unbounded tail into a bounded one, and the cap firing is a metric worth watching.

Measure under load, not in a notebook

Latency measured one request at a time is a lower bound and nothing more. Run a simple load test (k6, locust, or a loop with asyncio) at the concurrency you expect, and plot latency against concurrency. The curve has a knee, and knowing where it is tells you both your capacity and what your users will experience at the busiest hour, which is the only hour anybody remembers.