Temperature, sampling, and why the same input gives a different answer

What the model actually produces

A language model does not produce text. On each step it produces a logit for every token in its vocabulary: a raw score, maybe 128,000 of them. Turning those scores into one chosen token is done by your sampling settings, and that part is not the model at all. It is a few lines of ordinary code that you control.

The pipeline is short:

1. Divide every logit by the temperature.
2. Apply softmax to turn the scores into probabilities summing to 1.
3. Optionally cut the tail: top_k keeps the k highest, top_p (nucleus) keeps the smallest set whose probabilities sum to p.
4. Draw one token at random from what is left.

Temperature divides before the softmax, so it stretches or flattens the distribution. At temperature=2 the gaps between candidates shrink and unlikely tokens become reachable. At temperature=0.1 the top candidate dominates. At exactly 0, most implementations skip sampling and take the argmax.

That is the whole mechanism. Nothing about temperature makes a model more creative in any human sense: it makes the tail of the distribution more reachable, and some of that tail is nonsense.

Settings that match the job

• Extraction, classification, routing, structured output: 0. You want the same input to map to the same label. Any variance here is pure loss.
• Summarising, rewriting, answering from documents: 0.2 to 0.4. Enough to avoid stilted repetition, not enough to invent.
• Brainstorming, naming, first drafts, dialogue variety: 0.7 to 1.0.
• Above 1.2: use only when you want strangeness and are prepared to throw most outputs away.

One rule that saves hours: do not tune temperature and top_p at the same time. They interact, the interaction is not intuitive, and you will not be able to attribute any change you see. Pick one, leave the other at its default, write down which you chose.

Temperature 0 is not deterministic

This catches nearly everyone, including people who have shipped for years. You set temperature=0, run the same prompt twice, and get two different answers. Nothing is broken. There are three separate mechanical reasons.

Floating-point addition is not associative. Summing a batch of numbers on a GPU in a different order gives a very slightly different result. Two logits that are nearly tied can swap places. One swapped token at position 40 changes every token after it, because each token conditions the next. A rounding difference in the twelfth decimal place becomes a different paragraph.

Your request is batched with strangers. Hosted inference groups concurrent requests into a batch for throughput. The batch’s shape and composition change the reduction order in the kernels. You cannot control who else is calling the API at that moment, so you cannot control your own numerics.

Mixture-of-experts routing is batch-sensitive. Many current models route each token to a subset of experts. In some serving implementations, which expert a token lands on depends on capacity that is shared across the batch. Same token, different neighbours, different expert, different output.

Some providers expose a seed parameter and a system_fingerprint. The seed makes sampling reproducible; it does not fix the three causes above. Treat reproducibility as best-effort, never as guaranteed.

What to do about it

The conclusion is not “give up on testing”. It is assert properties, not strings.

# fragile: fails on a rewording that is entirely correct
assert out == "The refund window is 30 days."

# durable: fails only when the meaning is wrong
data = json.loads(out)
assert data["days"] == 30
assert data["currency"] in {"INR", "USD"}

Three habits follow from that:

• Where you can, make the model emit structured output and assert on the fields. A number in a field is checkable; a sentence is an argument.
• Where you cannot, assert on invariants: it cited at least one source, it did not mention a competitor, it stayed under 200 words, it refused an out-of-scope question.
• When you need one canonical answer for a repeated question, cache the first one by a hash of the inputs. This is the honest way to get determinism, and it also cuts your bill.

The uncomfortable consequence

If the same input can produce different output, then “it worked when I tried it” is not evidence, and neither is “it failed once”. Both are single draws from a distribution you have not characterised. A feature that succeeds 95% of the time will look perfect across four manual tries: the probability of seeing no failure in four attempts is about 81%. You will ship it, and then one user in twenty will meet the failure on their first contact with your product.

This is the reason the evaluation module later in this course exists, and it is why builders who skip it end up debugging by anecdote.