Where inference time actually goes

Two very different clocks

For a small model (a tree, a logistic regression, an embedding) inference is arithmetic on a fixed-size input, and it takes the same time every request. Measure it once and you know it.

For a language model, generation is a loop, and the cost is in two distinct phases that behave differently:

• Prefill. The whole prompt is processed at once. This phase is compute-bound and parallel, so a 2,000-token prompt is not twenty times slower than a 100-token one, but it does set your time to first token.
• Decode. Tokens come out one at a time, each requiring a full pass over the weights. This phase is memory-bandwidth-bound: the limit is how fast the weights can be read from memory, not how fast the chip can multiply.

That second fact explains almost every surprising benchmark result. If decoding is limited by memory bandwidth, then a smaller model is faster for a simple mechanical reason (there is less to read per token), and quantising a model to 4 bits roughly halves the bytes moved, which roughly doubles the token rate. It also explains why decoding one sequence leaves an expensive GPU mostly idle.

Batching, which is why hosted APIs are cheap

Because decode is bandwidth-bound, processing eight sequences at once costs barely more than processing one: the weights are read once and used for all eight. Throughput multiplies; per-request latency barely moves.

This is what a serving stack like vLLM or TGI does with continuous batching: new requests join the running batch between token steps rather than waiting for a batch to fill. It is also why a provider serving thousands of users per GPU can charge less per token than your idle GPU costs you.

The trade-off is real and worth stating: batching raises throughput and raises the latency of the individual request slightly. Under a strict latency budget with sparse traffic, small batches win.

The KV cache, and why long conversations get expensive

During decode, the model keeps the intermediate attention state for every token so far (the key-value cache) so it does not recompute the whole conversation for each new token. It is the reason generation is not quadratic in practice.

It also lives in memory, and it grows with context length times batch size. On a GPU it competes with the weights for the same memory, which is why a server that comfortably handles ten short conversations falls over on three long ones. If you are self-hosting and see out-of-memory errors that correlate with long chats rather than with request volume, this is it.

Measure your own, in one command

Do not trust a benchmark from a machine you do not have. For a hosted API:

curl -s -w '\ntotal=%{time_total}s ttfb=%{time_starttransfer}s\n' \
  -o /dev/null -X POST https://api.example.com/v1/chat \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{"model":"...","stream":true,"messages":[{"role":"user","content":"..."}]}'

time_starttransfer with streaming on is your time to first token; the gap to time_total divided by the number of output tokens is your per-token rate. Record both for the p50 and the p95: model latency has a much longer tail than database latency, and the average will flatter you.

For a local model, llama-bench reports prefill and decode rates separately, which is exactly the split above.

The four levers, in order of effect

1. Generate fewer tokens. Output length is the dominant term in both latency and cost. Ask for a shorter answer, set a max-tokens limit, and prefer structured output over prose.
2. Use a smaller model. Often the largest single win, and often free of quality cost for classification and extraction. Test it rather than assuming.
3. Quantise, if self-hosting. 8-bit is usually indistinguishable in quality; 4-bit is noticeably cheaper and sometimes noticeably worse. Measure on your own examples, not on a leaderboard.
4. Shorten the prompt. Helps prefill and cost, but has less effect on total latency than people expect, because decode dominates for long outputs.

And one that is not on the list: retrieving more context to improve quality makes prefill longer and the bill larger. That is a legitimate trade, but make it on purpose.

Stream, and the perceived time halves

A four-second response delivered as a stream, with the first token in 400 ms, is experienced as fast. The same response delivered whole is experienced as broken. Server-sent events are the standard mechanism and are supported by every provider; the cost is that your proxy must not buffer, which is one configuration line and a common cause of “streaming does not work in production but works locally”.