{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"why-answers-differ","title":"Why the same question gives different answers","url":"https://addaly.com/learn/ai-actually-explained/why-answers-differ","position":26,"of":92,"minutes":8,"answer":"A sampler, not the model, chooses each token, so answers vary by design, and because even temperature zero is not reproducible, never judge a prompt from a single run.","keyIdea":"A sampler, not the model, chooses each token, so answers vary by design, and because even temperature zero is not reproducible, never judge a prompt from a single run."},"body":{"format":"markdown","text":"## The model does not choose the word\n\nAt each step the model produces probabilities over its whole vocabulary. It does not pick one. A separate, small piece of code called the **sampler** does that, and how the sampler behaves is a setting.\n\nAlways taking the single most likely token is called greedy decoding. It sounds ideal and is not: greedy text is flat, repetitive, and prone to getting stuck in loops, because the most likely next word after a phrase is often a word that leads back into the same phrase.\n\nSo samplers introduce controlled randomness, and two dials control it.\n\n**Temperature** flattens or sharpens the probabilities before the draw. At 0, effectively always the top choice. Around 0.7, the default in most chat products, likely words are strongly favoured but alternatives get a look in. Above about 1.2, the distribution is flat enough that improbable words start appearing, and the text drifts towards nonsense.\n\n**Top-p**, or nucleus sampling, takes a different approach: keep only the smallest set of tokens whose probabilities add up to p, commonly 0.9, and draw from those. Where the model is confident, that set may be one token. Where it is unsure, it may be forty. It adapts, which is why it is now more common than a fixed top-k cut.\n\n## What this means for your work\n\nThe dial should match the task, and most people never touch it.\n\n- **Extraction, classification, data cleaning, code, anything checkable**: low temperature. You want the most likely answer, repeatable.\n- **Brainstorming, names, alternative phrasings, first drafts**: higher. Variety is the product.\n- **Anything you will run more than once and compare**: fix it low, or you are measuring the sampler.\n\nWhere is the dial? In APIs and local tools such as Ollama, LM Studio and llama.cpp it is a direct parameter. In consumer chat apps it is usually hidden, and asking for \"the most likely answer\" does not change it: the wording of your request cannot alter a setting outside the model. What you can do in a chat app is generate several times and look at the spread, which tells you how stable the answer is.\n\n## The uncomfortable detail\n\nSetting temperature to 0 does not guarantee identical output, and this catches out people trying to test carefully.\n\nOn the hardware these models run on, the arithmetic is done in parallel and in low precision, and the order in which many small numbers are added can vary between runs depending on batching: how many other people's requests were bundled with yours. Adding the same numbers in a different order produces very slightly different results, and where two tokens were nearly tied, a tiny difference flips the choice. From there the paths diverge completely.\n\nOn top of that, providers change which model actually serves a request. The same product name may route to different sizes depending on load, region or your plan, usually without announcement.\n\n## The habit this should create\n\n**Never conclude anything from one run.**\n\nThis single rule separates people who evaluate systems well from people who are repeatedly surprised in production. One good answer proves the good answer was possible. One bad answer proves the bad one was. Neither tells you the rate, and the rate is what you will live with.\n\nWhen you want to know whether a prompt works, run it five or ten times on the same input and count. If it succeeds nine times in ten, you have a system that fails weekly at a hundred uses a day. If it succeeds five times in ten, the demo you saw was a coin toss.\n\nThis also reframes a common complaint. \"It worked yesterday and not today\" is usually not a change in the model. It is the same variation, observed twice.\n\n## And a note on repetition\n\nIf output loops (repeating a phrase, restating the same point in slightly different words) the cause is usually a low temperature with no repetition penalty, and the cure is to raise the temperature slightly or, in a chat app, to start again with a more specific request. A model repeating itself is not confused. It has found a self-reinforcing groove in its own text, which is what greedy decoding does."},"check":{"question":"An engineer sets temperature to 0 to make an extraction pipeline reproducible, then finds occasional differences between runs on identical input. What is the most likely cause?","options":[{"text":"Temperature 0 is interpreted as a default value by most providers, so a small amount of randomness remains deliberately.","correct":false,"tempting":"Invents a silent override; providers do honour the setting, and the variation comes from elsewhere."},{"text":"The prompt contains ambiguity, and the model resolves ambiguity randomly regardless of the sampler.","correct":false,"tempting":"Locates the randomness inside the model, which produces a fixed distribution for a fixed input."},{"text":"Low-precision parallel arithmetic makes summation order depend on batching, so near-tied tokens can flip and the outputs then diverge.","correct":true},{"text":"The model updates its weights slightly after each request based on the input it has seen.","correct":false,"tempting":"Contradicts the read-only nature of the weights during use, which is a common and load-bearing misconception."}]},"prev":{"slug":"the-context-window","title":"The desk it works on, and what falls off it","url":"https://addaly.com/learn/ai-actually-explained/the-context-window"},"next":{"slug":"what-it-read","title":"What was in the pile","url":"https://addaly.com/learn/ai-actually-explained/what-it-read"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why the same question gives different answers', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/why-answers-differ (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why the same question gives different answers","courseTitle":"AI, Actually Explained","module":{"index":3,"slug":"how-a-chatbot-works","title":"What a chatbot is actually doing"},"answer":null,"keyIdea":"A sampler, not the model, chooses each token, so answers vary by design, and because even temperature zero is not reproducible, never judge a prompt from a single run.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-model-does-not-choose-the-word","runs":[{"text":"The model does not choose the word"}]},{"t":"p","runs":[{"text":"At each step the model produces probabilities over its whole vocabulary. It does not pick one. A separate, small piece of code called the "},{"text":"sampler","b":true},{"text":" does that, and how the sampler behaves is a setting."}]},{"t":"p","runs":[{"text":"Always taking the single most likely token is called greedy decoding. It sounds ideal and is not: greedy text is flat, repetitive, and prone to getting stuck in loops, because the most likely next word after a phrase is often a word that leads back into the same phrase."}]},{"t":"p","runs":[{"text":"So samplers introduce controlled randomness, and two dials control it."}]},{"t":"p","runs":[{"text":"Temperature","b":true},{"text":" flattens or sharpens the probabilities before the draw. At 0, effectively always the top choice. Around 0.7, the default in most chat products, likely words are strongly favoured but alternatives get a look in. Above about 1.2, the distribution is flat enough that improbable words start appearing, and the text drifts towards nonsense."}]},{"t":"p","runs":[{"text":"Top-p","b":true},{"text":", or nucleus sampling, takes a different approach: keep only the smallest set of tokens whose probabilities add up to p, commonly 0.9, and draw from those. Where the model is confident, that set may be one token. Where it is unsure, it may be forty. It adapts, which is why it is now more common than a fixed top-k cut."}]},{"t":"h2","id":"what-this-means-for-your-work","runs":[{"text":"What this means for your work"}]},{"t":"p","runs":[{"text":"The dial should match the task, and most people never touch it."}]},{"t":"ul","items":[[{"text":"Extraction, classification, data cleaning, code, anything checkable","b":true},{"text":": low temperature. You want the most likely answer, repeatable."}],[{"text":"Brainstorming, names, alternative phrasings, first drafts","b":true},{"text":": higher. Variety is the product."}],[{"text":"Anything you will run more than once and compare","b":true},{"text":": fix it low, or you are measuring the sampler."}]]},{"t":"p","runs":[{"text":"Where is the dial? In APIs and local tools such as Ollama, LM Studio and llama.cpp it is a direct parameter. In consumer chat apps it is usually hidden, and asking for “the most likely answer” does not change it: the wording of your request cannot alter a setting outside the model. What you can do in a chat app is generate several times and look at the spread, which tells you how stable the answer is."}]},{"t":"h2","id":"the-uncomfortable-detail","runs":[{"text":"The uncomfortable detail"}]},{"t":"p","runs":[{"text":"Setting temperature to 0 does not guarantee identical output, and this catches out people trying to test carefully."}]},{"t":"p","runs":[{"text":"On the hardware these models run on, the arithmetic is done in parallel and in low precision, and the order in which many small numbers are added can vary between runs depending on batching: how many other people’s requests were bundled with yours. Adding the same numbers in a different order produces very slightly different results, and where two tokens were nearly tied, a tiny difference flips the choice. From there the paths diverge completely."}]},{"t":"p","runs":[{"text":"On top of that, providers change which model actually serves a request. The same product name may route to different sizes depending on load, region or your plan, usually without announcement."}]},{"t":"h2","id":"the-habit-this-should-create","runs":[{"text":"The habit this should create"}]},{"t":"p","runs":[{"text":"Never conclude anything from one run.","b":true}]},{"t":"p","runs":[{"text":"This single rule separates people who evaluate systems well from people who are repeatedly surprised in production. One good answer proves the good answer was possible. One bad answer proves the bad one was. Neither tells you the rate, and the rate is what you will live with."}]},{"t":"p","runs":[{"text":"When you want to know whether a prompt works, run it five or ten times on the same input and count. If it succeeds nine times in ten, you have a system that fails weekly at a hundred uses a day. If it succeeds five times in ten, the demo you saw was a coin toss."}]},{"t":"p","runs":[{"text":"This also reframes a common complaint. “It worked yesterday and not today” is usually not a change in the model. It is the same variation, observed twice."}]},{"t":"h2","id":"and-a-note-on-repetition","runs":[{"text":"And a note on repetition"}]},{"t":"p","runs":[{"text":"If output loops (repeating a phrase, restating the same point in slightly different words) the cause is usually a low temperature with no repetition penalty, and the cure is to raise the temperature slightly or, in a chat app, to start again with a more specific request. A model repeating itself is not confused. It has found a self-reinforcing groove in its own text, which is what greedy decoding does."}]}],"check":{"question":[{"text":"An engineer sets temperature to 0 to make an extraction pipeline reproducible, then finds occasional differences between runs on identical input. What is the most likely cause?"}],"options":[{"text":[{"text":"Temperature 0 is interpreted as a default value by most providers, so a small amount of randomness remains deliberately."}],"correct":false,"tempting":[{"text":"Invents a silent override; providers do honour the setting, and the variation comes from elsewhere."}]},{"text":[{"text":"The prompt contains ambiguity, and the model resolves ambiguity randomly regardless of the sampler."}],"correct":false,"tempting":[{"text":"Locates the randomness inside the model, which produces a fixed distribution for a fixed input."}]},{"text":[{"text":"Low-precision parallel arithmetic makes summation order depend on batching, so near-tied tokens can flip and the outputs then diverge."}],"correct":true,"tempting":null},{"text":[{"text":"The model updates its weights slightly after each request based on the input it has seen."}],"correct":false,"tempting":[{"text":"Contradicts the read-only nature of the weights during use, which is a common and load-bearing misconception."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"the-context-window","title":"The desk it works on, and what falls off it","minutes":9},"next":{"slug":"what-it-read","title":"What was in the pile","minutes":9}}}