{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"temperature-zero-is-not-deterministic","title":"Why temperature zero still gives you different answers","url":"https://addaly.com/learn/how-llms-work/temperature-zero-is-not-deterministic","position":28,"of":78,"minutes":8,"answer":"Floating-point addition is not associative, so a request batched differently produces slightly different logits, and a near-tie between two tokens can flip.","keyIdea":"Floating-point addition is not associative, so a request batched differently produces slightly different logits, and a near-tie between two tokens can flip."},"body":{"format":"markdown","text":"## The complaint\n\nSet temperature to 0. Set a seed. Send the same prompt twice. Get two different answers. This is reported constantly as a bug, and it is a genuine property of how these systems run, worth understanding because it changes what you can promise about a product.\n\n## Where the randomness is not\n\nFirst, eliminate the obvious explanations, because they are usually wrong.\n\nTemperature 0 does remove sampling randomness: the implementation takes the highest logit rather than drawing from a distribution. A seed does control any pseudo-random number generator. Neither is the source.\n\nThe source is that **the logits themselves are not bit-identical between runs**.\n\n## Floating-point addition is not associative\n\nIn exact arithmetic, `(a + b) + c` equals `a + (b + c)`. In floating point it does not, because each addition rounds. With numbers of very different magnitudes the difference is easy to demonstrate:\n\n```python\n>>> (1e16 + 1.0) - 1e16\n0.0\n>>> 1e16 + (1.0 - 1e16)\n1.0\n```\n\nA GPU sums thousands of products in parallel, and the order in which partial results are combined depends on how the work was divided among the hardware. That division depends on the shape of the operation, which depends on how many sequences are in the batch and how long they are.\n\n## Batching is the culprit\n\nYour request is batched with whatever other requests arrived at the same moment. A batch of 8 and a batch of 32 use different kernel configurations, different reduction orders, and therefore produce logits that differ in the last few decimal places.\n\nAlmost always this changes nothing: the top token wins by a wide margin either way. But when two tokens are nearly tied, and across a 500-token response there will be several such moments, a difference of 10⁻⁷ decides it. From that token on, the two responses diverge completely, because each subsequent token is conditioned on a different prefix.\n\nThat is the whole mechanism. Not hidden randomness, not model updates, not caching. Non-associative addition plus variable batch shapes plus occasional near-ties.\n\nIt also explains the shape of the divergence people report: responses that are word-for-word identical for two paragraphs and then differ entirely. A single flipped token, and everything after it follows.\n\n## What can and cannot be done\n\n**Locally, you can get close to determinism.** Fix the batch size at one, pin library versions, disable non-deterministic kernels, and run on the same hardware. `llama.cpp` on a single CPU thread is reproducible. Expect a meaningful throughput cost: you are giving up the batching that makes serving efficient.\n\n**Recent work has made batch-invariant kernels practical**, and some providers have begun offering a deterministic mode built on them. Check whether yours does; the cost is usually reduced throughput.\n\n**Through a shared hosted API, you generally cannot.** Seeds and `system_fingerprint` fields help you detect *when* the backend changed, which is useful, but they do not make a batched service reproduce a previous run bit for bit.\n\n## What to do instead\n\nStop treating reproducibility as achievable and design for variation.\n\n- **Test on distributions, not single outputs.** Run your evaluation set *n* times and compare score distributions. A test that asserts an exact string will fail eventually and teach you nothing when it does.\n- **Cache aggressively** if you need the same answer for the same input. A cache gives you real determinism at the application layer, which is where you wanted it.\n- **Store the output, not the promise of reproducing it.** For audit purposes, keep what the model actually said, with its inputs and version, rather than expecting to regenerate it later.\n\nAnd separate this from the other cause of \"the model changed\": providers do update models behind stable names. That produces a *sustained* shift in behaviour, where this produces run-to-run jitter. Distinguishing them is the first step in any investigation, and module six's lesson on drift covers the other half."},"check":{"question":"A regression test asserts that a model returns an exact string at temperature 0 and passes for weeks, then fails intermittently while the model version is unchanged. What is the likely cause?","options":[{"text":"The provider silently updated the model behind the same version name.","correct":false,"tempting":"Silent updates are real and are the usual explanation for behaviour changes, which makes this the first thing most teams check."},{"text":"A cache expired and the request is now being computed rather than replayed.","correct":false,"tempting":"Response caching does exist in many deployments, and an expiring cache would explain a sudden change in behaviour."},{"text":"The prompt now exceeds a length threshold that triggers different handling.","correct":false,"tempting":"Length-triggered behaviour changes do occur in some stacks, and threshold effects fit an intermittent pattern well."},{"text":"Requests are batched with different neighbours each time, so floating-point reduction order differs and a near-tie between two tokens occasionally flips.","correct":true}]},"prev":{"slug":"speculative-decoding","title":"Speculative decoding: a small model guesses, a big model checks","url":"https://addaly.com/learn/how-llms-work/speculative-decoding"},"next":{"slug":"streaming-and-latency","title":"Streaming, and the two latencies users feel","url":"https://addaly.com/learn/how-llms-work/streaming-and-latency"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why temperature zero still gives you different answers', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/temperature-zero-is-not-deterministic (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why temperature zero still gives you different answers","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Floating-point addition is not associative, so a request batched differently produces slightly different logits, and a near-tie between two tokens can flip.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-complaint","runs":[{"text":"The complaint"}]},{"t":"p","runs":[{"text":"Set temperature to 0. Set a seed. Send the same prompt twice. Get two different answers. This is reported constantly as a bug, and it is a genuine property of how these systems run, worth understanding because it changes what you can promise about a product."}]},{"t":"h2","id":"where-the-randomness-is-not","runs":[{"text":"Where the randomness is not"}]},{"t":"p","runs":[{"text":"First, eliminate the obvious explanations, because they are usually wrong."}]},{"t":"p","runs":[{"text":"Temperature 0 does remove sampling randomness: the implementation takes the highest logit rather than drawing from a distribution. A seed does control any pseudo-random number generator. Neither is the source."}]},{"t":"p","runs":[{"text":"The source is that "},{"text":"the logits themselves are not bit-identical between runs","b":true},{"text":"."}]},{"t":"h2","id":"floating-point-addition-is-not-associative","runs":[{"text":"Floating-point addition is not associative"}]},{"t":"p","runs":[{"text":"In exact arithmetic, "},{"text":"(a + b) + c","code":true},{"text":" equals "},{"text":"a + (b + c)","code":true},{"text":". In floating point it does not, because each addition rounds. With numbers of very different magnitudes the difference is easy to demonstrate:"}]},{"t":"code","lang":"python","label":"Python","text":">>> (1e16 + 1.0) - 1e16\n0.0\n>>> 1e16 + (1.0 - 1e16)\n1.0"},{"t":"p","runs":[{"text":"A GPU sums thousands of products in parallel, and the order in which partial results are combined depends on how the work was divided among the hardware. That division depends on the shape of the operation, which depends on how many sequences are in the batch and how long they are."}]},{"t":"h2","id":"batching-is-the-culprit","runs":[{"text":"Batching is the culprit"}]},{"t":"p","runs":[{"text":"Your request is batched with whatever other requests arrived at the same moment. A batch of 8 and a batch of 32 use different kernel configurations, different reduction orders, and therefore produce logits that differ in the last few decimal places."}]},{"t":"p","runs":[{"text":"Almost always this changes nothing: the top token wins by a wide margin either way. But when two tokens are nearly tied, and across a 500-token response there will be several such moments, a difference of 10⁻⁷ decides it. From that token on, the two responses diverge completely, because each subsequent token is conditioned on a different prefix."}]},{"t":"p","runs":[{"text":"That is the whole mechanism. Not hidden randomness, not model updates, not caching. Non-associative addition plus variable batch shapes plus occasional near-ties."}]},{"t":"p","runs":[{"text":"It also explains the shape of the divergence people report: responses that are word-for-word identical for two paragraphs and then differ entirely. A single flipped token, and everything after it follows."}]},{"t":"h2","id":"what-can-and-cannot-be-done","runs":[{"text":"What can and cannot be done"}]},{"t":"p","runs":[{"text":"Locally, you can get close to determinism.","b":true},{"text":" Fix the batch size at one, pin library versions, disable non-deterministic kernels, and run on the same hardware. "},{"text":"llama.cpp","code":true},{"text":" on a single CPU thread is reproducible. Expect a meaningful throughput cost: you are giving up the batching that makes serving efficient."}]},{"t":"p","runs":[{"text":"Recent work has made batch-invariant kernels practical","b":true},{"text":", and some providers have begun offering a deterministic mode built on them. Check whether yours does; the cost is usually reduced throughput."}]},{"t":"p","runs":[{"text":"Through a shared hosted API, you generally cannot.","b":true},{"text":" Seeds and "},{"text":"system_fingerprint","code":true},{"text":" fields help you detect "},{"text":"when","i":true},{"text":" the backend changed, which is useful, but they do not make a batched service reproduce a previous run bit for bit."}]},{"t":"h2","id":"what-to-do-instead","runs":[{"text":"What to do instead"}]},{"t":"p","runs":[{"text":"Stop treating reproducibility as achievable and design for variation."}]},{"t":"ul","items":[[{"text":"Test on distributions, not single outputs.","b":true},{"text":" Run your evaluation set "},{"text":"n","i":true},{"text":" times and compare score distributions. A test that asserts an exact string will fail eventually and teach you nothing when it does."}],[{"text":"Cache aggressively","b":true},{"text":" if you need the same answer for the same input. A cache gives you real determinism at the application layer, which is where you wanted it."}],[{"text":"Store the output, not the promise of reproducing it.","b":true},{"text":" For audit purposes, keep what the model actually said, with its inputs and version, rather than expecting to regenerate it later."}]]},{"t":"p","runs":[{"text":"And separate this from the other cause of “the model changed”: providers do update models behind stable names. That produces a "},{"text":"sustained","i":true},{"text":" shift in behaviour, where this produces run-to-run jitter. Distinguishing them is the first step in any investigation, and module six’s lesson on drift covers the other half."}]}],"check":{"question":[{"text":"A regression test asserts that a model returns an exact string at temperature 0 and passes for weeks, then fails intermittently while the model version is unchanged. What is the likely cause?"}],"options":[{"text":[{"text":"The provider silently updated the model behind the same version name."}],"correct":false,"tempting":[{"text":"Silent updates are real and are the usual explanation for behaviour changes, which makes this the first thing most teams check."}]},{"text":[{"text":"A cache expired and the request is now being computed rather than replayed."}],"correct":false,"tempting":[{"text":"Response caching does exist in many deployments, and an expiring cache would explain a sudden change in behaviour."}]},{"text":[{"text":"The prompt now exceeds a length threshold that triggers different handling."}],"correct":false,"tempting":[{"text":"Length-triggered behaviour changes do occur in some stacks, and threshold effects fit an intermittent pattern well."}]},{"text":[{"text":"Requests are batched with different neighbours each time, so floating-point reduction order differs and a near-tie between two tokens occasionally flips."}],"correct":true,"tempting":null}],"order":[1,2,3,0],"why":null},"prev":{"slug":"speculative-decoding","title":"Speculative decoding: a small model guesses, a big model checks","minutes":8},"next":{"slug":"streaming-and-latency","title":"Streaming, and the two latencies users feel","minutes":8}}}