{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"sampling-and-determinism","title":"Temperature, sampling, and why the same input gives a different answer","url":"https://addaly.com/learn/building-with-ai/sampling-and-determinism","position":4,"of":80,"minutes":9,"answer":"Sampling settings live in your code, not the model, and temperature 0 still varies because floating-point reductions and shared batching change which of two near-tied tokens wins.","keyIdea":"Sampling settings live in your code, not the model, and temperature 0 still varies because floating-point reductions and shared batching change which of two near-tied tokens wins."},"body":{"format":"markdown","text":"## What the model actually produces\n\nA language model does not produce text. On each step it produces a **logit** for every token in its vocabulary: a raw score, maybe 128,000 of them. Turning those scores into one chosen token is done by your sampling settings, and that part is not the model at all. It is a few lines of ordinary code that you control.\n\nThe pipeline is short:\n\n1. Divide every logit by the **temperature**.\n2. Apply softmax to turn the scores into probabilities summing to 1.\n3. Optionally cut the tail: **top_k** keeps the k highest, **top_p** (nucleus) keeps the smallest set whose probabilities sum to p.\n4. Draw one token at random from what is left.\n\nTemperature divides before the softmax, so it stretches or flattens the distribution. At `temperature=2` the gaps between candidates shrink and unlikely tokens become reachable. At `temperature=0.1` the top candidate dominates. At exactly 0, most implementations skip sampling and take the argmax.\n\nThat is the whole mechanism. Nothing about temperature makes a model more creative in any human sense: it makes the tail of the distribution more reachable, and some of that tail is nonsense.\n\n## Settings that match the job\n\n- **Extraction, classification, routing, structured output: 0.** You want the same input to map to the same label. Any variance here is pure loss.\n- **Summarising, rewriting, answering from documents: 0.2 to 0.4.** Enough to avoid stilted repetition, not enough to invent.\n- **Brainstorming, naming, first drafts, dialogue variety: 0.7 to 1.0.**\n- **Above 1.2:** use only when you want strangeness and are prepared to throw most outputs away.\n\nOne rule that saves hours: **do not tune temperature and top_p at the same time.** They interact, the interaction is not intuitive, and you will not be able to attribute any change you see. Pick one, leave the other at its default, write down which you chose.\n\n## Temperature 0 is not deterministic\n\nThis catches nearly everyone, including people who have shipped for years. You set `temperature=0`, run the same prompt twice, and get two different answers. Nothing is broken. There are three separate mechanical reasons.\n\n**Floating-point addition is not associative.** Summing a batch of numbers on a GPU in a different order gives a very slightly different result. Two logits that are nearly tied can swap places. One swapped token at position 40 changes every token after it, because each token conditions the next. A rounding difference in the twelfth decimal place becomes a different paragraph.\n\n**Your request is batched with strangers.** Hosted inference groups concurrent requests into a batch for throughput. The batch's shape and composition change the reduction order in the kernels. You cannot control who else is calling the API at that moment, so you cannot control your own numerics.\n\n**Mixture-of-experts routing is batch-sensitive.** Many current models route each token to a subset of experts. In some serving implementations, which expert a token lands on depends on capacity that is shared across the batch. Same token, different neighbours, different expert, different output.\n\nSome providers expose a `seed` parameter and a `system_fingerprint`. The seed makes sampling reproducible; it does not fix the three causes above. Treat reproducibility as best-effort, never as guaranteed.\n\n## What to do about it\n\nThe conclusion is not \"give up on testing\". It is **assert properties, not strings**.\n\n```python\n# fragile: fails on a rewording that is entirely correct\nassert out == \"The refund window is 30 days.\"\n\n# durable: fails only when the meaning is wrong\ndata = json.loads(out)\nassert data[\"days\"] == 30\nassert data[\"currency\"] in {\"INR\", \"USD\"}\n```\n\nThree habits follow from that:\n\n- Where you can, make the model emit **structured output** and assert on the fields. A number in a field is checkable; a sentence is an argument.\n- Where you cannot, assert on invariants: it cited at least one source, it did not mention a competitor, it stayed under 200 words, it refused an out-of-scope question.\n- When you need one canonical answer for a repeated question, **cache the first one** by a hash of the inputs. This is the honest way to get determinism, and it also cuts your bill.\n\n## The uncomfortable consequence\n\nIf the same input can produce different output, then \"it worked when I tried it\" is not evidence, and neither is \"it failed once\". Both are single draws from a distribution you have not characterised. A feature that succeeds 95% of the time will look perfect across four manual tries: the probability of seeing no failure in four attempts is about 81%. You will ship it, and then one user in twenty will meet the failure on their first contact with your product.\n\nThis is the reason the evaluation module later in this course exists, and it is why builders who skip it end up debugging by anecdote."},"check":{"question":"A developer sets temperature to 0, runs an identical prompt ten times, and gets two distinct answers. What is the most accurate explanation?","options":[{"text":"Some hidden randomness remains in the sampler that temperature 0 does not disable.","correct":false,"tempting":"Locates the cause in the sampler, when at temperature 0 most implementations take the argmax and do no sampling at all."},{"text":"The provider silently switched model versions between calls.","correct":false,"tempting":"A real risk worth logging for, but it would not explain variation within a few seconds on the same model string."},{"text":"The prompt contains ambiguity the model resolves differently each time.","correct":false,"tempting":"Treats the model as making a fresh interpretive choice, when the same weights and same argmax rule should give the same interpretation."},{"text":"Nearly tied logits can swap order because GPU floating-point reductions depend on batch composition, and one swapped token changes everything after it.","correct":true}]},"prev":{"slug":"tokens-and-the-context-window","title":"Tokens, and why a Hindi bot costs three times an English one","url":"https://addaly.com/learn/building-with-ai/tokens-and-the-context-window"},"next":{"slug":"streaming-and-perceived-speed","title":"Streaming, and the difference between fast and feeling fast","url":"https://addaly.com/learn/building-with-ai/streaming-and-perceived-speed"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Temperature, sampling, and why the same input gives a different answer', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/sampling-and-determinism (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Temperature, sampling, and why the same input gives a different answer","courseTitle":"Building with AI","module":{"index":1,"slug":"the-call-underneath-everything","title":"The call underneath everything"},"answer":null,"keyIdea":"Sampling settings live in your code, not the model, and temperature 0 still varies because floating-point reductions and shared batching change which of two near-tied tokens wins.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-the-model-actually-produces","runs":[{"text":"What the model actually produces"}]},{"t":"p","runs":[{"text":"A language model does not produce text. On each step it produces a "},{"text":"logit","b":true},{"text":" for every token in its vocabulary: a raw score, maybe 128,000 of them. Turning those scores into one chosen token is done by your sampling settings, and that part is not the model at all. It is a few lines of ordinary code that you control."}]},{"t":"p","runs":[{"text":"The pipeline is short:"}]},{"t":"ol","start":1,"items":[[{"text":"Divide every logit by the "},{"text":"temperature","b":true},{"text":"."}],[{"text":"Apply softmax to turn the scores into probabilities summing to 1."}],[{"text":"Optionally cut the tail: "},{"text":"top_k","b":true},{"text":" keeps the k highest, "},{"text":"top_p","b":true},{"text":" (nucleus) keeps the smallest set whose probabilities sum to p."}],[{"text":"Draw one token at random from what is left."}]]},{"t":"p","runs":[{"text":"Temperature divides before the softmax, so it stretches or flattens the distribution. At "},{"text":"temperature=2","code":true},{"text":" the gaps between candidates shrink and unlikely tokens become reachable. At "},{"text":"temperature=0.1","code":true},{"text":" the top candidate dominates. At exactly 0, most implementations skip sampling and take the argmax."}]},{"t":"p","runs":[{"text":"That is the whole mechanism. Nothing about temperature makes a model more creative in any human sense: it makes the tail of the distribution more reachable, and some of that tail is nonsense."}]},{"t":"h2","id":"settings-that-match-the-job","runs":[{"text":"Settings that match the job"}]},{"t":"ul","items":[[{"text":"Extraction, classification, routing, structured output: 0.","b":true},{"text":" You want the same input to map to the same label. Any variance here is pure loss."}],[{"text":"Summarising, rewriting, answering from documents: 0.2 to 0.4.","b":true},{"text":" Enough to avoid stilted repetition, not enough to invent."}],[{"text":"Brainstorming, naming, first drafts, dialogue variety: 0.7 to 1.0.","b":true}],[{"text":"Above 1.2:","b":true},{"text":" use only when you want strangeness and are prepared to throw most outputs away."}]]},{"t":"p","runs":[{"text":"One rule that saves hours: "},{"text":"do not tune temperature and top_p at the same time.","b":true},{"text":" They interact, the interaction is not intuitive, and you will not be able to attribute any change you see. Pick one, leave the other at its default, write down which you chose."}]},{"t":"h2","id":"temperature-0-is-not-deterministic","runs":[{"text":"Temperature 0 is not deterministic"}]},{"t":"p","runs":[{"text":"This catches nearly everyone, including people who have shipped for years. You set "},{"text":"temperature=0","code":true},{"text":", run the same prompt twice, and get two different answers. Nothing is broken. There are three separate mechanical reasons."}]},{"t":"p","runs":[{"text":"Floating-point addition is not associative.","b":true},{"text":" Summing a batch of numbers on a GPU in a different order gives a very slightly different result. Two logits that are nearly tied can swap places. One swapped token at position 40 changes every token after it, because each token conditions the next. A rounding difference in the twelfth decimal place becomes a different paragraph."}]},{"t":"p","runs":[{"text":"Your request is batched with strangers.","b":true},{"text":" Hosted inference groups concurrent requests into a batch for throughput. The batch’s shape and composition change the reduction order in the kernels. You cannot control who else is calling the API at that moment, so you cannot control your own numerics."}]},{"t":"p","runs":[{"text":"Mixture-of-experts routing is batch-sensitive.","b":true},{"text":" Many current models route each token to a subset of experts. In some serving implementations, which expert a token lands on depends on capacity that is shared across the batch. Same token, different neighbours, different expert, different output."}]},{"t":"p","runs":[{"text":"Some providers expose a "},{"text":"seed","code":true},{"text":" parameter and a "},{"text":"system_fingerprint","code":true},{"text":". The seed makes sampling reproducible; it does not fix the three causes above. Treat reproducibility as best-effort, never as guaranteed."}]},{"t":"h2","id":"what-to-do-about-it","runs":[{"text":"What to do about it"}]},{"t":"p","runs":[{"text":"The conclusion is not “give up on testing”. It is "},{"text":"assert properties, not strings","b":true},{"text":"."}]},{"t":"code","lang":"python","label":"Python","text":"# fragile: fails on a rewording that is entirely correct\nassert out == \"The refund window is 30 days.\"\n\n# durable: fails only when the meaning is wrong\ndata = json.loads(out)\nassert data[\"days\"] == 30\nassert data[\"currency\"] in {\"INR\", \"USD\"}"},{"t":"p","runs":[{"text":"Three habits follow from that:"}]},{"t":"ul","items":[[{"text":"Where you can, make the model emit "},{"text":"structured output","b":true},{"text":" and assert on the fields. A number in a field is checkable; a sentence is an argument."}],[{"text":"Where you cannot, assert on invariants: it cited at least one source, it did not mention a competitor, it stayed under 200 words, it refused an out-of-scope question."}],[{"text":"When you need one canonical answer for a repeated question, "},{"text":"cache the first one","b":true},{"text":" by a hash of the inputs. This is the honest way to get determinism, and it also cuts your bill."}]]},{"t":"h2","id":"the-uncomfortable-consequence","runs":[{"text":"The uncomfortable consequence"}]},{"t":"p","runs":[{"text":"If the same input can produce different output, then “it worked when I tried it” is not evidence, and neither is “it failed once”. Both are single draws from a distribution you have not characterised. A feature that succeeds 95% of the time will look perfect across four manual tries: the probability of seeing no failure in four attempts is about 81%. You will ship it, and then one user in twenty will meet the failure on their first contact with your product."}]},{"t":"p","runs":[{"text":"This is the reason the evaluation module later in this course exists, and it is why builders who skip it end up debugging by anecdote."}]}],"check":{"question":[{"text":"A developer sets temperature to 0, runs an identical prompt ten times, and gets two distinct answers. What is the most accurate explanation?"}],"options":[{"text":[{"text":"Some hidden randomness remains in the sampler that temperature 0 does not disable."}],"correct":false,"tempting":[{"text":"Locates the cause in the sampler, when at temperature 0 most implementations take the argmax and do no sampling at all."}]},{"text":[{"text":"The provider silently switched model versions between calls."}],"correct":false,"tempting":[{"text":"A real risk worth logging for, but it would not explain variation within a few seconds on the same model string."}]},{"text":[{"text":"The prompt contains ambiguity the model resolves differently each time."}],"correct":false,"tempting":[{"text":"Treats the model as making a fresh interpretive choice, when the same weights and same argmax rule should give the same interpretation."}]},{"text":[{"text":"Nearly tied logits can swap order because GPU floating-point reductions depend on batch composition, and one swapped token changes everything after it."}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":null},"prev":{"slug":"tokens-and-the-context-window","title":"Tokens, and why a Hindi bot costs three times an English one","minutes":10},"next":{"slug":"streaming-and-perceived-speed","title":"Streaming, and the difference between fast and feeling fast","minutes":9}}}