Why temperature zero still gives you different answers

The complaint

Set temperature to 0. Set a seed. Send the same prompt twice. Get two different answers. This is reported constantly as a bug, and it is a genuine property of how these systems run, worth understanding because it changes what you can promise about a product.

Where the randomness is not

First, eliminate the obvious explanations, because they are usually wrong.

Temperature 0 does remove sampling randomness: the implementation takes the highest logit rather than drawing from a distribution. A seed does control any pseudo-random number generator. Neither is the source.

The source is that the logits themselves are not bit-identical between runs.

Floating-point addition is not associative

In exact arithmetic, (a + b) + c equals a + (b + c). In floating point it does not, because each addition rounds. With numbers of very different magnitudes the difference is easy to demonstrate:

>>> (1e16 + 1.0) - 1e16
0.0
>>> 1e16 + (1.0 - 1e16)
1.0

A GPU sums thousands of products in parallel, and the order in which partial results are combined depends on how the work was divided among the hardware. That division depends on the shape of the operation, which depends on how many sequences are in the batch and how long they are.

Batching is the culprit

Your request is batched with whatever other requests arrived at the same moment. A batch of 8 and a batch of 32 use different kernel configurations, different reduction orders, and therefore produce logits that differ in the last few decimal places.

Almost always this changes nothing: the top token wins by a wide margin either way. But when two tokens are nearly tied, and across a 500-token response there will be several such moments, a difference of 10⁻⁷ decides it. From that token on, the two responses diverge completely, because each subsequent token is conditioned on a different prefix.

That is the whole mechanism. Not hidden randomness, not model updates, not caching. Non-associative addition plus variable batch shapes plus occasional near-ties.

It also explains the shape of the divergence people report: responses that are word-for-word identical for two paragraphs and then differ entirely. A single flipped token, and everything after it follows.

What can and cannot be done

Locally, you can get close to determinism. Fix the batch size at one, pin library versions, disable non-deterministic kernels, and run on the same hardware. llama.cpp on a single CPU thread is reproducible. Expect a meaningful throughput cost: you are giving up the batching that makes serving efficient.

Recent work has made batch-invariant kernels practical, and some providers have begun offering a deterministic mode built on them. Check whether yours does; the cost is usually reduced throughput.

Through a shared hosted API, you generally cannot. Seeds and system_fingerprint fields help you detect when the backend changed, which is useful, but they do not make a batched service reproduce a previous run bit for bit.

What to do instead

Stop treating reproducibility as achievable and design for variation.

• Test on distributions, not single outputs. Run your evaluation set n times and compare score distributions. A test that asserts an exact string will fail eventually and teach you nothing when it does.
• Cache aggressively if you need the same answer for the same input. A cache gives you real determinism at the application layer, which is where you wanted it.
• Store the output, not the promise of reproducing it. For audit purposes, keep what the model actually said, with its inputs and version, rather than expecting to regenerate it later.

And separate this from the other cause of “the model changed”: providers do update models behind stable names. That produces a sustained shift in behaviour, where this produces run-to-run jitter. Distinguishing them is the first step in any investigation, and module six’s lesson on drift covers the other half.