Where the milliseconds go

Two numbers a user feels

The streaming lesson in the first module drew the distinction: time to first token is how long the screen stays empty, and total time is how long until the answer is complete. People forgive a long total if the first token is fast and the text keeps flowing. They do not forgive a long silence. So the budget has two lines, and the first is the one to protect.

The stages

Read a trace from the evaluation module and the request breaks into pieces, each with a duration:

• Network to your server: tens of milliseconds, more on a phone on a poor connection.
• Retrieval: query rewrite, if any, at 200 to 500 milliseconds because it is a model call; embedding at 5 to 100; search at 5 to 50; reranking at 100 on a GPU and a second or more on a CPU. The retrieval module gave these.
• Network to the provider: a round trip to the provider’s region. From India to a US endpoint that can be 200 milliseconds before anything happens; a regional endpoint, where one exists, cuts it to tens.
• Prefill: the model reads the whole input before producing anything. Roughly linear in input tokens; a 10,000-token prompt takes noticeably longer to first token than a 2,000-token one. A cached prefix is skipped, which is why caching cuts time to first token as well as cost.
• Generation: tokens per second, from tens for a large model to well over a hundred for a small one. Total generation time is output length divided by that rate.
• Tool calls: each one is a full extra round trip: the model stops, your code runs the tool, the model reads the result and starts again, paying prefill and time to first token a second time. Three sequential tool calls is four model calls.

Time to first token is everything before generation. Total time is that plus generation plus every tool round trip.

Measure the tail, not the mean

A mean of 800 milliseconds can hide a 95th percentile of four seconds, and the users at the 95th percentile are the ones who complain. Record every stage per request, and report the 50th, 95th and 99th percentiles per stage, per day. The evaluation course has a lesson on latency as a distribution; here the practical point is that the tail usually belongs to one stage (a reranker on a busy CPU, a tool that occasionally hangs, a provider under load) and the per-stage percentiles point at it.

The cheap wins, in order

Cache the prefix. The prompt-caching lesson’s ordering rule, applied here for speed: a stable system prompt and stable context at the front, variable text at the end. A cache hit removes most of the prefill.

Stream. Nothing changes in the total, and everything changes in what the user feels.

Shrink the input. Fewer retrieved chunks, a shorter system prompt, trimmed history. Prefill is linear; half the tokens is roughly half the wait before the first one.

Use a smaller model where the first token matters. A small model’s time to first token and generation rate are both several times better. For a conversational reply that must feel immediate, the small model with the retrieval done well often beats the large model with the same retrieval, on both speed and the evaluation set.

Run tools in parallel. The loop lesson covered it: independent reads dispatched together cost one round trip instead of three.

Start the next stage before the last one finishes. Begin embedding the query while the rewrite runs if the rewrite is optional; render the citation cards while the answer streams.

What a faster provider does not fix

Switching providers for speed usually moves one stage by a fraction while leaving the structure (retrieval, tool round trips, input length) untouched. Measure per stage first. A team that switches providers to fix a four-second p95 caused by a CPU reranker will find the p95 at 3.8 seconds afterwards.

A budget, written down

For an interactive assistant, a working target: first token under a second at the 95th percentile, and a total under four seconds for a typical answer. Assign the second to stages (say 300 milliseconds for retrieval, 200 for the provider round trip and prefill with a cache hit, the rest as headroom) and when a stage overspends its share, that is the stage to work on. A budget that exists only as a total is met by nobody in particular.

Latency on the phone

Most of this readership is on a phone, sometimes on a slow connection. The network stage that is 30 milliseconds on office wifi is 300 on a weak mobile signal, and a streamed response that arrives token by token over a lossy connection can feel jerky. Test on a real phone with the connection throttled (every desktop browser can simulate a slow network) and make sure the first token still appears before the user wonders whether anything is happening.