Why the same seed gives different answers

Two sources of variation, and only one is the sampler

Set temperature to 0 and the sampler always picks the highest-probability token. That removes sampling randomness completely. Many people are then surprised to find the output still varies between runs, and conclude something is broken.

It is not. The logits themselves are not bit-identical between runs, because floating-point addition is not associative. Adding a set of numbers in a different order gives slightly different results, and a GPU reduction adds them in whatever order the scheduling happened to produce. Usually the differences are far below the gap between the top token and the second, so the argmax is unchanged. Occasionally two tokens are nearly tied, a difference in the twelfth decimal place flips which is larger, and from that token onward the two outputs diverge completely.

That is the whole mechanism. It is not a bug and it cannot be switched off by a seed.

Batching makes it worse, in a way worth knowing

On a server, your request is computed inside a batch with other people’s. The batch size and composition change which kernel configuration is chosen and in what order values are summed. So the same prompt sent to a busy server and to an idle one can produce different logits, and therefore, occasionally, different text.

This is why identical requests to a shared endpoint sometimes disagree while the same requests to a local single-user server agree. Nothing about your request changed; the company it was computed in did.

Some engines now offer batch-invariant kernels that produce identical results regardless of batch composition, at a cost in speed. If exact reproducibility matters more than throughput, look for that option; otherwise, plan around the variation rather than trying to eliminate it.

What a seed does and does not do

A seed fixes the random number stream the sampler draws from. With the same seed, the same prompt, the same model, the same engine version, the same quantisation and the same batch conditions, you get the same output.

Change any of those and the seed does not save you. In particular the seed does not survive: a different engine version, a different quantisation of the same model, a different context length, a different number of GPUs, or a different concurrent load. People treat a seed as a guarantee of reproducibility and it is a guarantee of one variable among six.

Getting as close to reproducible as you can

1. Temperature 0 for anything a program consumes. This removes the largest source of variation by far.
2. Pin everything: model revision, quantisation file, engine version, all sampler parameters, the context length.
3. Log what you used with the output: model, revision, engine version, parameters. A stored answer with no record of how it was produced cannot be investigated later.
4. Store the output, not just the prompt, for anything that matters. Regenerating is not reproducing.
5. Test at concurrency 1 when comparing two configurations, so batch composition is not a hidden variable.

Where this actually bites

Evaluations. A test suite that asserts on exact strings will fail intermittently. Assert on properties instead: does the JSON parse, does it contain the required fields, is the number within range, does the code compile. A local model is a stochastic component, and test design has to accept that.

Caching. If you cache responses by prompt hash, you are freezing one particular sample. That is often desirable (consistency for users, cost savings) but it is a decision, and it means your users see one draw from a distribution rather than the distribution.

Debugging. “It worked yesterday” needs a logged record to be checkable. Without one, you cannot distinguish a real regression from a different draw.

Comparing two models. Run both at temperature 0 and compare on the same items. Any difference is then the models, not the dice, which is why the audition set in the first block specified temperature 0, and why comparisons run at temperature 0.7 need many more items to say anything.

The position

Exact bit-level reproducibility across machines and versions is not achievable without deliberate effort and a speed penalty most people will not pay. What is achievable is practical reproducibility: temperature 0, everything pinned, everything logged, and tests that assert on properties rather than on exact text. That is enough to run a serious system, and pretending to more is where people get caught.