{"course":{"slug":"prompting","title":"Getting Real Work out of a Model","url":"https://addaly.com/learn/prompting","level":"beginner","track":"start"},"lesson":{"slug":"sampling-and-why-answers-differ","title":"Why the same prompt gives a different answer","url":"https://addaly.com/learn/prompting/sampling-and-why-answers-differ","position":3,"of":72,"minutes":8,"answer":"Variation between answers comes from the sampler, not from the model changing its mind, so a prompt is only tested when you have run it several times and seen what moved.","keyIdea":"Variation between answers comes from the sampler, not from the model changing its mind, so a prompt is only tested when you have run it several times and seen what moved."},"body":{"format":"markdown","text":"## The model does not choose a word\n\nAt each step the model produces a probability for every token in its vocabulary: tens of thousands of numbers, one per possible next chunk. Something outside the model then picks one. That picker is called the **sampler**, and its settings are why you can send an identical prompt twice and get two different answers.\n\nThe main setting is **temperature**. Low temperature sharpens the distribution towards whatever was already most likely; high temperature flattens it, giving unlikely tokens more of a chance. At temperature 0 the sampler takes the single most probable token every time, which is called greedy decoding. Typical chat products run somewhere around 0.7 to 1.0 by default, which is why they feel fluent and slightly unpredictable.\n\nThe second setting you will meet is **top-p**, or nucleus sampling. It restricts the choice to the smallest set of tokens whose probabilities add up to *p*, then samples inside that set. With `top_p = 0.9`, a token sitting in the bottom 10% of probability mass cannot be chosen at all, however high the temperature. Some interfaces also expose `top_k`, which keeps the *k* most likely tokens.\n\n## What this means for your work\n\n**Regenerating is a re-roll, not a repair.** Pressing regenerate draws a fresh sample from the same distribution produced by the same prompt. If the prompt is the problem, every draw comes from the same wrong pile. That is the whole argument of a later lesson, and now you know its mechanism.\n\n**A prompt that worked once has not been tested.** This is the practical lesson most people skip. If you are about to run a prompt over 200 customer messages, run it five times over the same three messages first. Variation you can see in five runs is variation you will get in two hundred. A prompt that gives the right format four times out of five will produce forty broken rows in your batch.\n\n**Match the temperature to the task, where you can.** Most consumer chat windows do not let you change it, but developer playgrounds and APIs do:\n\n```python\nclient.messages.create(\n    model=\"...\",\n    temperature=0,          # extraction, classification, formatting\n    messages=[...],\n)\n```\n\nUse a low temperature (0 to 0.3) when there is a right answer: pulling a date out of an invoice, labelling a message, converting a format. Use a higher one (0.8 to 1.2) when you want range: names, first drafts, ideas you will throw most of away. If you have ever complained that a model's brainstorms all sound the same, the temperature is the first thing to reach for, not the prompt.\n\n> Figure: Which temperature for which task. Low: 0 to 0.3: Pulling a date out of an invoice; Labelling a customer message; Converting one format to another; Extracting three fields from a document; Anything with one correct answer. High: 0.8 to 1.2: Names for a product; A first draft you will rewrite; Twenty ideas you will discard most of; Approaches that differ in their assumptions; Anything where sameness is the failure. Low when there is a right answer, high when you want range. If a model's brainstorms all sound the same, the temperature is the first thing to reach for, not the prompt.\n\n## The caveat about determinism\n\nSetting temperature to 0 makes the sampler deterministic. It does not reliably make the *system* deterministic.\n\nTwo runs at temperature 0 can still differ, and the reason is arithmetic rather than mystery. Providers run your request on GPUs alongside other people's requests, batched together. The batch size changes the order in which floating-point numbers are added, floating-point addition is not exactly associative, and a difference far below the fifth decimal place can flip which of two near-tied tokens comes out on top. Once one token differs, everything after it can differ.\n\nSo temperature 0 buys you much less variation, not a guarantee. If you need a fixed answer (an audit trail, a test that must not flake) store the output, do not expect to regenerate it. Some providers offer a `seed` parameter and describe it as \"best effort\", which is an unusually honest label.\n\n## A cheap experiment worth doing tonight\n\nTake a prompt you use often. Run it five times without changing a character. Put the five outputs side by side and mark what stayed the same and what moved.\n\nWhat stays is what your prompt actually determined. What moves is what you left to the sampler. If the thing that moves is the part you care about (the tone, the number of items, whether it included the caveat) you have found the next line to write, and no amount of regenerating would have found it for you.\n\nMost people never do this, and it is the single fastest way to see the difference between a prompt that works and a prompt that got lucky."},"check":{"question":"A shop owner has a prompt that pulls order number, date and total from pasted emails. It worked perfectly in his three trials. He runs it across 300 emails overnight and 40 rows come back with the date missing or reformatted. What did his trial miss?","options":[{"text":"Three successes at a default temperature do not show the spread of outputs the same prompt produces, and the failure rate was always there.","correct":true},{"text":"The batch was too large, and quality degrades once a model processes hundreds of items in a session.","correct":false,"tempting":"Each email is a separate request; there is no accumulating fatigue across a batch."},{"text":"The emails in the batch were more varied than his three trials, so the prompt met formats it had never seen.","correct":false,"tempting":"A real risk, but it does not explain why the *same* prompt on the *same* input can behave differently; the sampler is the mechanism being tested here."},{"text":"Long-running jobs hit rate limits, and truncated responses lose the last fields extracted.","correct":false,"tempting":"Attributes the pattern to infrastructure; a rate limit produces errors, not quietly reformatted dates."}]},"prev":{"slug":"what-the-model-sees","title":"What the model actually receives","url":"https://addaly.com/learn/prompting/what-the-model-sees"},"next":{"slug":"instruction-following-is-trained","title":"Why it obeys you at all","url":"https://addaly.com/learn/prompting/instruction-following-is-trained"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why the same prompt gives a different answer', in Getting Real Work out of a Model. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/prompting/sampling-and-why-answers-differ (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why the same prompt gives a different answer","courseTitle":"Getting Real Work out of a Model","module":{"index":1,"slug":"what-a-prompt-is","title":"What a prompt actually is"},"answer":null,"keyIdea":"Variation between answers comes from the sampler, not from the model changing its mind, so a prompt is only tested when you have run it several times and seen what moved.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Low: 0 to 0.3","items":["Pulling a date out of an invoice","Labelling a customer message","Converting one format to another","Extracting three fields from a document","Anything with one correct answer"]},"type":"compare","right":{"title":"High: 0.8 to 1.2","items":["Names for a product","A first draft you will rewrite","Twenty ideas you will discard most of","Approaches that differ in their assumptions","Anything where sameness is the failure"]},"title":"Which temperature for which task","caption":"Low when there is a right answer, high when you want range. If a model’s brainstorms all sound the same, the temperature is the first thing to reach for, not the prompt."},"alt":"Figure: Which temperature for which task. Low: 0 to 0.3: Pulling a date out of an invoice; Labelling a customer message; Converting one format to another; Extracting three fields from a document; Anything with one correct answer. High: 0.8 to 1.2: Names for a product; A first draft you will rewrite; Twenty ideas you will discard most of; Approaches that differ in their assumptions; Anything where sameness is the failure. Low when there is a right answer, high when you want range. If a model’s brainstorms all sound the same, the temperature is the first thing to reach for, not the prompt."}},"blocks":[{"t":"h2","id":"the-model-does-not-choose-a-word","runs":[{"text":"The model does not choose a word"}]},{"t":"p","runs":[{"text":"At each step the model produces a probability for every token in its vocabulary: tens of thousands of numbers, one per possible next chunk. Something outside the model then picks one. That picker is called the "},{"text":"sampler","b":true},{"text":", and its settings are why you can send an identical prompt twice and get two different answers."}]},{"t":"p","runs":[{"text":"The main setting is "},{"text":"temperature","b":true},{"text":". Low temperature sharpens the distribution towards whatever was already most likely; high temperature flattens it, giving unlikely tokens more of a chance. At temperature 0 the sampler takes the single most probable token every time, which is called greedy decoding. Typical chat products run somewhere around 0.7 to 1.0 by default, which is why they feel fluent and slightly unpredictable."}]},{"t":"p","runs":[{"text":"The second setting you will meet is "},{"text":"top-p","b":true},{"text":", or nucleus sampling. It restricts the choice to the smallest set of tokens whose probabilities add up to "},{"text":"p","i":true},{"text":", then samples inside that set. With "},{"text":"top_p = 0.9","code":true},{"text":", a token sitting in the bottom 10% of probability mass cannot be chosen at all, however high the temperature. Some interfaces also expose "},{"text":"top_k","code":true},{"text":", which keeps the "},{"text":"k","i":true},{"text":" most likely tokens."}]},{"t":"h2","id":"what-this-means-for-your-work","runs":[{"text":"What this means for your work"}]},{"t":"p","runs":[{"text":"Regenerating is a re-roll, not a repair.","b":true},{"text":" Pressing regenerate draws a fresh sample from the same distribution produced by the same prompt. If the prompt is the problem, every draw comes from the same wrong pile. That is the whole argument of a later lesson, and now you know its mechanism."}]},{"t":"p","runs":[{"text":"A prompt that worked once has not been tested.","b":true},{"text":" This is the practical lesson most people skip. If you are about to run a prompt over 200 customer messages, run it five times over the same three messages first. Variation you can see in five runs is variation you will get in two hundred. A prompt that gives the right format four times out of five will produce forty broken rows in your batch."}]},{"t":"p","runs":[{"text":"Match the temperature to the task, where you can.","b":true},{"text":" Most consumer chat windows do not let you change it, but developer playgrounds and APIs do:"}]},{"t":"code","lang":"python","label":"Python","text":"client.messages.create(\n    model=\"...\",\n    temperature=0,          # extraction, classification, formatting\n    messages=[...],\n)"},{"t":"p","runs":[{"text":"Use a low temperature (0 to 0.3) when there is a right answer: pulling a date out of an invoice, labelling a message, converting a format. Use a higher one (0.8 to 1.2) when you want range: names, first drafts, ideas you will throw most of away. If you have ever complained that a model’s brainstorms all sound the same, the temperature is the first thing to reach for, not the prompt."}]},{"t":"h2","id":"the-caveat-about-determinism","runs":[{"text":"The caveat about determinism"}]},{"t":"p","runs":[{"text":"Setting temperature to 0 makes the sampler deterministic. It does not reliably make the "},{"text":"system","i":true},{"text":" deterministic."}]},{"t":"p","runs":[{"text":"Two runs at temperature 0 can still differ, and the reason is arithmetic rather than mystery. Providers run your request on GPUs alongside other people’s requests, batched together. The batch size changes the order in which floating-point numbers are added, floating-point addition is not exactly associative, and a difference far below the fifth decimal place can flip which of two near-tied tokens comes out on top. Once one token differs, everything after it can differ."}]},{"t":"p","runs":[{"text":"So temperature 0 buys you much less variation, not a guarantee. If you need a fixed answer (an audit trail, a test that must not flake) store the output, do not expect to regenerate it. Some providers offer a "},{"text":"seed","code":true},{"text":" parameter and describe it as “best effort”, which is an unusually honest label."}]},{"t":"h2","id":"a-cheap-experiment-worth-doing-tonight","runs":[{"text":"A cheap experiment worth doing tonight"}]},{"t":"p","runs":[{"text":"Take a prompt you use often. Run it five times without changing a character. Put the five outputs side by side and mark what stayed the same and what moved."}]},{"t":"p","runs":[{"text":"What stays is what your prompt actually determined. What moves is what you left to the sampler. If the thing that moves is the part you care about (the tone, the number of items, whether it included the caveat) you have found the next line to write, and no amount of regenerating would have found it for you."}]},{"t":"p","runs":[{"text":"Most people never do this, and it is the single fastest way to see the difference between a prompt that works and a prompt that got lucky."}]}],"check":{"question":[{"text":"A shop owner has a prompt that pulls order number, date and total from pasted emails. It worked perfectly in his three trials. He runs it across 300 emails overnight and 40 rows come back with the date missing or reformatted. What did his trial miss?"}],"options":[{"text":[{"text":"Three successes at a default temperature do not show the spread of outputs the same prompt produces, and the failure rate was always there."}],"correct":true,"tempting":null},{"text":[{"text":"The batch was too large, and quality degrades once a model processes hundreds of items in a session."}],"correct":false,"tempting":[{"text":"Each email is a separate request; there is no accumulating fatigue across a batch."}]},{"text":[{"text":"The emails in the batch were more varied than his three trials, so the prompt met formats it had never seen."}],"correct":false,"tempting":[{"text":"A real risk, but it does not explain why the "},{"text":"same","i":true},{"text":" prompt on the "},{"text":"same","i":true},{"text":" input can behave differently; the sampler is the mechanism being tested here."}]},{"text":[{"text":"Long-running jobs hit rate limits, and truncated responses lose the last fields extracted."}],"correct":false,"tempting":[{"text":"Attributes the pattern to infrastructure; a rate limit produces errors, not quietly reformatted dates."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"what-the-model-sees","title":"What the model actually receives","minutes":10},"next":{"slug":"instruction-following-is-trained","title":"Why it obeys you at all","minutes":9}}}