Where the milliseconds go

Two phases with different physics

A model call has two stages and they are bottlenecked by different hardware.

Prefill processes your entire input. Every token is handled in parallel, so this stage is compute-bound: it saturates the arithmetic units of the accelerator. Its duration scales with how many input tokens there are. Prefill is what you are waiting for during time-to-first-token.

Decode produces the output one token at a time. Each token requires reading the whole model’s weights from memory, so this stage is memory-bandwidth-bound and largely indifferent to how long your input was. Its duration scales with how many output tokens you asked for.

That split explains most latency behaviour people find confusing.

time_to_first_token  ≈ queue + prefill(input_tokens)
total_time           ≈ time_to_first_token + output_tokens × time_per_token

What it implies

A long context is slow to start and then normal. Going from 4,000 to 60,000 input tokens can add hundreds of milliseconds to several seconds before anything appears, and then the text streams at the same rate as always. Users describe this as “it hangs and then works”, which is exactly right.

Streaming hides decode, not prefill. Streaming is the standard latency fix and it only helps after the first token. If your problem is a five-second silence, streaming does not touch it. What touches it is a shorter input, a cache hit, or a smaller model.

Output length is the strongest lever on total time. Each output token costs a full pass over the weights. Halving a verbose answer halves most of the wall-clock. “Answer in at most three sentences” is a latency optimisation as much as a style choice.

A cache hit removes prefill for the cached part. This is why caching improves perceived speed as much as it improves cost, and often more noticeably.

The quadratic that mostly is not

Attention is quadratic in sequence length in the arithmetic. In practice, serving stacks have spent years making that not dominate: flash-attention kernels avoid materialising the full matrix, chunked prefill breaks a long input into pieces, and paged caches keep memory usable.

The result is that measured prefill time is closer to linear in input length across the range most applications use, with the quadratic term becoming visible at very long contexts. So do not model prefill as quadratic and do not assume it is free: measure it at the lengths you actually send.

Measure it yourself

Everything here is observable with a stopwatch and a streaming client.

t0 = time.perf_counter()
first = None
for i, chunk in enumerate(stream_call(messages)):
    if first is None:
        first = time.perf_counter() - t0        # TTFT
total = time.perf_counter() - t0
print(f"ttft={first:.2f}s total={total:.2f}s tps={n_out/(total-first):.1f}")

Run it at three input lengths and two output lengths, ten times each, and record medians and 95th percentiles. Half an hour of work replaces every argument your team is going to have about latency.

The 95th percentile is the number that matters. Provider latency is noticeably variable (queueing, capacity, other tenants) and a median that looks fine can hide a tail where one request in twenty takes four times as long.

The rest of the latency budget

The model is often not the largest term. A realistic breakdown for a retrieval-augmented answer:

• Query rewriting call: 200 to 500 ms, if you do it.
• Embedding the query: 10 to 50 ms locally, 50 to 200 ms hosted.
• Vector search: 10 to 100 ms.
• Reranking 50 candidates: 50 to 400 ms depending on hardware.
• Prefill on 8,000 tokens: 200 ms to 1 s.
• Decode of 400 tokens: 2 to 8 s.

Two lessons. The pre-model stages add up to something comparable to prefill, and they are serial by default: run the ones that do not depend on each other in parallel. And decode dominates everything, which puts the answer’s length at the centre of any latency work.

What to do when it is still too slow

In rough order of return:

1. Shorten the output. Largest effect, no infrastructure change.
2. Cache the prefix. Removes most of prefill on repeat traffic.
3. Stream, and show something immediately (the retrieved sources, a skeleton) so the wait is occupied.
4. Parallelise the pre-model stages.
5. Cut input length, which the selection module was already arguing for.
6. Use a smaller model for the easy majority, which the routing lesson covers.

Notice that only the last two change what the model sees. Most latency work is not a context problem, but the two that are, are the ones you now know how to measure.