Streaming, and the two latencies users feel

Two numbers, not one

“The model is slow” is not a measurement. There are two independent latencies, they are governed by different parts of the system, and improving one does nothing for the other.

Time to first token (TTFT) is how long before anything appears. It is dominated by the prefill phase (processing your whole prompt in parallel) plus network round trip and queueing. It grows with prompt length.

Inter-token latency (ITL) is the gap between successive tokens once flowing. It is one decode step: read all the weights, produce one token. It barely depends on prompt length and is roughly constant for a given model and load.

Total time is approximately TTFT + (output_tokens × ITL). Both terms matter, and which one is your problem determines what to fix.

Streaming changes what users perceive

Streaming sends each token as it is produced, over server-sent events or a websocket. The total time is unchanged. The perceived wait collapses from the total to the TTFT, and that is a large difference in how a product feels.

Reading speed gives you the target. Comfortable adult reading is roughly 250 words per minute, and a word is about 1.3 tokens, so somewhere near 5 to 6 tokens per second keeps pace with a reader. Above about 20 tokens per second, further speed is invisible for text somebody is reading, though it still matters when output feeds code, when the user is skimming for a result, or when a long response is being generated before any action is taken.

This is a useful check on optimisation effort. If you are at 40 tokens per second and your users read the output, faster decoding buys you nothing they can perceive. Lowering TTFT from 3 seconds to 800 milliseconds buys you a great deal.

Fixing time to first token

In rough order of effect:

• Shorten the prompt. Prefill is proportional to prompt length. Ten thousand tokens of instructions you never revised is the commonest cause of a slow first token.
• Cache the prefix. If the first 8,000 tokens are identical across requests, the server can reuse the stored keys and values instead of recomputing them. Providers expose this as prompt caching, and it typically cuts both the latency and the price of the cached portion substantially. It requires a byte-identical prefix, which is why module one told you to put the variable material last.
• Reduce queueing. Under load, TTFT is partly waiting for a slot. This is a capacity problem, not a model problem.
• Move the compute closer. Network round trip is tens to hundreds of milliseconds and is not nothing when you are targeting under a second.

Fixing inter-token latency

• Use a smaller model, or a sparse one with fewer active parameters. Decode time scales with the weights that must be read.
• Quantise. Fewer bytes per weight means less memory traffic per token, which is the bottleneck. This is why a 4-bit model can be noticeably faster than the same model at 16-bit, not merely smaller.
• Speculative decoding, from the previous lesson.
• Better batching on the server, which raises throughput but can slightly worsen a single user’s ITL. Throughput and latency are in tension; know which you are optimising.

Measure before you optimise

import time
t0 = time.time(); first = None; n = 0
for chunk in client.chat.completions.create(..., stream=True):
    if first is None: first = time.time() - t0
    n += 1
total = time.time() - t0
print(f"TTFT {first:.2f}s  tokens {n}  ITL {(total-first)/max(n-1,1)*1000:.0f}ms")

Ten lines, and it turns “it feels slow” into two numbers you can act on. Run it at your real prompt length and at your real time of day, because both matter.

The design lever nobody uses

The largest latency improvement available to most applications is not technical. It is asking for less output. A response of 800 tokens takes four times as long to produce as one of 200, and users frequently prefer the shorter one. Before optimising the serving stack, check whether your prompt is asking the model to restate the question, explain its approach and offer three alternatives, and whether anybody reads that.