{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"determinism-and-reproducibility","title":"Why the same seed gives different answers","url":"https://addaly.com/learn/running-models-yourself/determinism-and-reproducibility","position":53,"of":82,"minutes":8,"answer":"Temperature 0 removes sampling randomness but not floating-point non-associativity, so identical requests can still diverge when two tokens are nearly tied: pin every version, log what produced each output.","keyIdea":"Temperature 0 removes sampling randomness but not floating-point non-associativity, so identical requests can still diverge when two tokens are nearly tied: pin every version, log what produced each output, and assert on properties rather than exact strings."},"body":{"format":"markdown","text":"## Two sources of variation, and only one is the sampler\n\nSet temperature to 0 and the sampler always picks the highest-probability token. That removes sampling randomness completely. Many people are then surprised to find the output still varies between runs, and conclude something is broken.\n\nIt is not. The logits themselves are not bit-identical between runs, because floating-point addition is not associative. Adding a set of numbers in a different order gives slightly different results, and a GPU reduction adds them in whatever order the scheduling happened to produce. Usually the differences are far below the gap between the top token and the second, so the argmax is unchanged. Occasionally two tokens are nearly tied, a difference in the twelfth decimal place flips which is larger, and from that token onward the two outputs diverge completely.\n\nThat is the whole mechanism. It is not a bug and it cannot be switched off by a seed.\n\n## Batching makes it worse, in a way worth knowing\n\nOn a server, your request is computed inside a batch with other people's. The batch size and composition change which kernel configuration is chosen and in what order values are summed. So the same prompt sent to a busy server and to an idle one can produce different logits, and therefore, occasionally, different text.\n\nThis is why identical requests to a shared endpoint sometimes disagree while the same requests to a local single-user server agree. Nothing about your request changed; the company it was computed in did.\n\nSome engines now offer batch-invariant kernels that produce identical results regardless of batch composition, at a cost in speed. If exact reproducibility matters more than throughput, look for that option; otherwise, plan around the variation rather than trying to eliminate it.\n\n## What a seed does and does not do\n\nA seed fixes the random number stream the sampler draws from. With the same seed, the same prompt, the same model, the same engine version, the same quantisation and the same batch conditions, you get the same output.\n\nChange any of those and the seed does not save you. In particular the seed does not survive: a different engine version, a different quantisation of the same model, a different context length, a different number of GPUs, or a different concurrent load. People treat a seed as a guarantee of reproducibility and it is a guarantee of one variable among six.\n\n## Getting as close to reproducible as you can\n\n1. **Temperature 0** for anything a program consumes. This removes the largest source of variation by far.\n2. **Pin everything**: model revision, quantisation file, engine version, all sampler parameters, the context length.\n3. **Log what you used** with the output: model, revision, engine version, parameters. A stored answer with no record of how it was produced cannot be investigated later.\n4. **Store the output**, not just the prompt, for anything that matters. Regenerating is not reproducing.\n5. **Test at concurrency 1** when comparing two configurations, so batch composition is not a hidden variable.\n\n## Where this actually bites\n\n**Evaluations.** A test suite that asserts on exact strings will fail intermittently. Assert on properties instead: does the JSON parse, does it contain the required fields, is the number within range, does the code compile. A local model is a stochastic component, and test design has to accept that.\n\n**Caching.** If you cache responses by prompt hash, you are freezing one particular sample. That is often desirable (consistency for users, cost savings) but it is a decision, and it means your users see one draw from a distribution rather than the distribution.\n\n**Debugging.** \"It worked yesterday\" needs a logged record to be checkable. Without one, you cannot distinguish a real regression from a different draw.\n\n**Comparing two models.** Run both at temperature 0 and compare on the same items. Any difference is then the models, not the dice, which is why the audition set in the first block specified temperature 0, and why comparisons run at temperature 0.7 need many more items to say anything.\n\n## The position\n\nExact bit-level reproducibility across machines and versions is not achievable without deliberate effort and a speed penalty most people will not pay. What is achievable is *practical* reproducibility: temperature 0, everything pinned, everything logged, and tests that assert on properties rather than on exact text. That is enough to run a serious system, and pretending to more is where people get caught."},"check":{"question":"A team runs a local model at temperature 0 with a fixed seed. Their evaluation suite passes on a quiet development server and fails intermittently against the shared staging server, with the same prompts and model file. What is the most likely explanation?","options":[{"text":"The staging server has prefix caching enabled, and cached keys and values from earlier requests are altering the computation.","correct":false,"tempting":"Cached blocks are the same values that would have been recomputed, so a cache hit does not change the result."},{"text":"The seed is not being propagated by the staging server's client library, so sampling randomness has returned.","correct":false,"tempting":"Plausible-sounding, but at temperature 0 there is no sampling randomness for a seed to control in the first place."},{"text":"Batch composition on the shared server changes the order of floating-point reductions, so near-tied tokens occasionally flip and outputs diverge.","correct":true},{"text":"The staging server is quantising the KV cache, which introduces rounding that a development server without it does not have.","correct":false,"tempting":"Would produce a consistent difference between the two servers rather than intermittent variation within staging."}]},"prev":{"slug":"long-context-in-practice","title":"Long context in practice","url":"https://addaly.com/learn/running-models-yourself/long-context-in-practice"},"next":{"slug":"debugging-a-bad-output","title":"The diagnostic ladder for a bad output","url":"https://addaly.com/learn/running-models-yourself/debugging-a-bad-output"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why the same seed gives different answers', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/determinism-and-reproducibility (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why the same seed gives different answers","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Temperature 0 removes sampling randomness but not floating-point non-associativity, so identical requests can still diverge when two tokens are nearly tied: pin every version, log what produced each output, and assert on properties rather than exact strings.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-sources-of-variation-and-only-one-is-the-sampler","runs":[{"text":"Two sources of variation, and only one is the sampler"}]},{"t":"p","runs":[{"text":"Set temperature to 0 and the sampler always picks the highest-probability token. That removes sampling randomness completely. Many people are then surprised to find the output still varies between runs, and conclude something is broken."}]},{"t":"p","runs":[{"text":"It is not. The logits themselves are not bit-identical between runs, because floating-point addition is not associative. Adding a set of numbers in a different order gives slightly different results, and a GPU reduction adds them in whatever order the scheduling happened to produce. Usually the differences are far below the gap between the top token and the second, so the argmax is unchanged. Occasionally two tokens are nearly tied, a difference in the twelfth decimal place flips which is larger, and from that token onward the two outputs diverge completely."}]},{"t":"p","runs":[{"text":"That is the whole mechanism. It is not a bug and it cannot be switched off by a seed."}]},{"t":"h2","id":"batching-makes-it-worse-in-a-way-worth-knowing","runs":[{"text":"Batching makes it worse, in a way worth knowing"}]},{"t":"p","runs":[{"text":"On a server, your request is computed inside a batch with other people’s. The batch size and composition change which kernel configuration is chosen and in what order values are summed. So the same prompt sent to a busy server and to an idle one can produce different logits, and therefore, occasionally, different text."}]},{"t":"p","runs":[{"text":"This is why identical requests to a shared endpoint sometimes disagree while the same requests to a local single-user server agree. Nothing about your request changed; the company it was computed in did."}]},{"t":"p","runs":[{"text":"Some engines now offer batch-invariant kernels that produce identical results regardless of batch composition, at a cost in speed. If exact reproducibility matters more than throughput, look for that option; otherwise, plan around the variation rather than trying to eliminate it."}]},{"t":"h2","id":"what-a-seed-does-and-does-not-do","runs":[{"text":"What a seed does and does not do"}]},{"t":"p","runs":[{"text":"A seed fixes the random number stream the sampler draws from. With the same seed, the same prompt, the same model, the same engine version, the same quantisation and the same batch conditions, you get the same output."}]},{"t":"p","runs":[{"text":"Change any of those and the seed does not save you. In particular the seed does not survive: a different engine version, a different quantisation of the same model, a different context length, a different number of GPUs, or a different concurrent load. People treat a seed as a guarantee of reproducibility and it is a guarantee of one variable among six."}]},{"t":"h2","id":"getting-as-close-to-reproducible-as-you-can","runs":[{"text":"Getting as close to reproducible as you can"}]},{"t":"ol","start":1,"items":[[{"text":"Temperature 0","b":true},{"text":" for anything a program consumes. This removes the largest source of variation by far."}],[{"text":"Pin everything","b":true},{"text":": model revision, quantisation file, engine version, all sampler parameters, the context length."}],[{"text":"Log what you used","b":true},{"text":" with the output: model, revision, engine version, parameters. A stored answer with no record of how it was produced cannot be investigated later."}],[{"text":"Store the output","b":true},{"text":", not just the prompt, for anything that matters. Regenerating is not reproducing."}],[{"text":"Test at concurrency 1","b":true},{"text":" when comparing two configurations, so batch composition is not a hidden variable."}]]},{"t":"h2","id":"where-this-actually-bites","runs":[{"text":"Where this actually bites"}]},{"t":"p","runs":[{"text":"Evaluations.","b":true},{"text":" A test suite that asserts on exact strings will fail intermittently. Assert on properties instead: does the JSON parse, does it contain the required fields, is the number within range, does the code compile. A local model is a stochastic component, and test design has to accept that."}]},{"t":"p","runs":[{"text":"Caching.","b":true},{"text":" If you cache responses by prompt hash, you are freezing one particular sample. That is often desirable (consistency for users, cost savings) but it is a decision, and it means your users see one draw from a distribution rather than the distribution."}]},{"t":"p","runs":[{"text":"Debugging.","b":true},{"text":" “It worked yesterday” needs a logged record to be checkable. Without one, you cannot distinguish a real regression from a different draw."}]},{"t":"p","runs":[{"text":"Comparing two models.","b":true},{"text":" Run both at temperature 0 and compare on the same items. Any difference is then the models, not the dice, which is why the audition set in the first block specified temperature 0, and why comparisons run at temperature 0.7 need many more items to say anything."}]},{"t":"h2","id":"the-position","runs":[{"text":"The position"}]},{"t":"p","runs":[{"text":"Exact bit-level reproducibility across machines and versions is not achievable without deliberate effort and a speed penalty most people will not pay. What is achievable is "},{"text":"practical","i":true},{"text":" reproducibility: temperature 0, everything pinned, everything logged, and tests that assert on properties rather than on exact text. That is enough to run a serious system, and pretending to more is where people get caught."}]}],"check":{"question":[{"text":"A team runs a local model at temperature 0 with a fixed seed. Their evaluation suite passes on a quiet development server and fails intermittently against the shared staging server, with the same prompts and model file. What is the most likely explanation?"}],"options":[{"text":[{"text":"The staging server has prefix caching enabled, and cached keys and values from earlier requests are altering the computation."}],"correct":false,"tempting":[{"text":"Cached blocks are the same values that would have been recomputed, so a cache hit does not change the result."}]},{"text":[{"text":"The seed is not being propagated by the staging server’s client library, so sampling randomness has returned."}],"correct":false,"tempting":[{"text":"Plausible-sounding, but at temperature 0 there is no sampling randomness for a seed to control in the first place."}]},{"text":[{"text":"Batch composition on the shared server changes the order of floating-point reductions, so near-tied tokens occasionally flip and outputs diverge."}],"correct":true,"tempting":null},{"text":[{"text":"The staging server is quantising the KV cache, which introduces rounding that a development server without it does not have."}],"correct":false,"tempting":[{"text":"Would produce a consistent difference between the two servers rather than intermittent variation within staging."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"long-context-in-practice","title":"Long context in practice","minutes":9},"next":{"slug":"debugging-a-bad-output","title":"The diagnostic ladder for a bad output","minutes":9}}}