Streaming, and the difference between fast and feeling fast

Two numbers, not one

A non-streaming call has one latency: the time until the whole answer arrives. A streaming call has two, and they behave completely differently.

Time to first token (TTFT) is how long the user stares at nothing. It is dominated by network round trip, queueing at the provider, and the prefill pass over your input. Prefill is roughly proportional to input length, which means a 20,000-token prompt has a visibly worse TTFT than a 2,000-token one even though the answer is identical.

Tokens per second is how fast text then appears. Typical hosted rates run from about 20 to 120 tokens per second depending on model size and load.

Do the arithmetic on a 500-token answer at 60 tokens per second: 8.3 seconds of generation. Without streaming, the user sees a spinner for 8.6 seconds. With streaming, they see words at 400 milliseconds and read while the rest arrives. Same total time. Completely different product.

The reason it works is that people read at roughly 250 words per minute, which is about 5 tokens per second. Anything above about 15 tokens per second outruns the reader, so the perceived wait collapses to TTFT alone. This is why cutting your prompt in half improves the felt speed of your product more than switching to a faster model, and it costs nothing.

What streaming looks like on the wire

It is Server-Sent Events: one long-lived HTTP response, chunks separated by blank lines, each chunk a small JSON delta.

with client.messages.stream(
    model="...", max_tokens=800,
    messages=[{"role": "user", "content": q},
]) as stream:
    for text in stream.text_stream:
        emit(text)
    final = stream.get_final_message()

Two things people miss. The deltas are token fragments, not words: you will receive " refu" then "nd". Never split, trim or regex a delta in isolation; accumulate first. And the final message, not the concatenated text, is where usage and stop_reason live. If you throw the stream object away you have thrown away your cost data.

What streaming costs you

This is the part the tutorials skip, and it is where real systems break.

You cannot validate before you show. If the model is emitting JSON, you have no valid object until the last brace. Streaming raw JSON to a user is meaningless. Either do not stream structured output, or stream a prose field only and hold the structure back.

You cannot moderate before you show. Any check that runs on the complete text (a safety classifier, a policy filter, a groundedness check) happens after the user has already read the words. A design that promises nothing unsafe reaches a screen is incompatible with naive streaming. The honest options are: buffer to a sentence boundary and check each sentence, accept the risk with a fast retraction path, or do not stream that surface. Pick one deliberately.

Errors arrive mid-body. The HTTP status was 200 four seconds ago. A failure now shows up as an error event inside the stream, or as a socket that stops. Your client needs to handle a partial answer as a real state, and your UI has to say something honest about it.

Cancellation does not always stop the meter. The user closes the tab, your server aborts the fetch: whether the provider stops generating and stops charging depends on the provider. Test it, and assume you pay for what was generated.

Two infrastructure traps

Buffering proxies. Nginx, some CDNs and some serverless platforms buffer responses by default. Your stream works locally and arrives as one lump in production. The fixes are X-Accel-Buffering: no, disabling gzip on that route, and checking your platform’s docs for response streaming support. Test streaming on the deployed URL, never only on localhost.

Timeouts written for normal requests. A stream may be open for 90 seconds legitimately. A 30-second total timeout kills healthy answers. What you actually want is an inactivity timeout: fail if no chunk has arrived for 15 seconds, regardless of total duration.

When not to stream

Streaming is a user-interface decision, and about half of AI calls have no user watching. A background classifier, a nightly extraction job, a tool call inside an agent loop, anything whose output you parse: none of these benefit. Streaming adds connection handling, partial-failure states and lost validation for no gain.

The rule is simple. Stream when a human is reading the words as they appear. Do not stream when a machine is consuming the result.