{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"same-prompt-twice","title":"Run it five times","url":"https://addaly.com/learn/choosing-your-tools/same-prompt-twice","position":15,"of":76,"minutes":8,"answer":"Because products sample from a probability distribution rather than picking the single most likely reply.","keyIdea":"Because products sample from a probability distribution rather than picking the single most likely reply, a single run is one draw and any tool difference smaller than a tool's own run-to-run variation is not a difference."},"body":{"format":"markdown","text":"## Why the same question gives different answers\n\nSend an identical prompt twice and you usually get two different replies. This is not a fault. At each step the model produces a probability distribution over possible next tokens, and the product samples from it rather than always taking the most likely one. The setting that controls how adventurous that sampling is is called **temperature**: near zero it almost always takes the top choice and answers are nearly repeatable; higher and it wanders.\n\nConsumer chat products generally run at a moderate temperature and do not let you change it. APIs and local runners do, and it is one of the more useful dials you will ever be handed.\n\n## What this does to your testing\n\nIt means a single run is a sample, not a measurement. Three things follow, and ignoring them is how people reach confident conclusions from nothing.\n\n**One bad answer proves very little.** If a tool fails a prompt once, run it twice more before you write it down. A tool that fails a prompt three times out of three has a problem. A tool that fails once out of three may just have been unlucky, and you would not want to be judged on your third-best attempt either.\n\n**One brilliant answer proves even less.** This is the direction people forget. A launch demo, a screenshot in a thread, a friend's amazing result: all of these are one draw from a distribution, and the ones that get shared are the good tail by construction. Nobody screenshots the mediocre attempt.\n\n**Differences smaller than the variation are not differences.** If a tool's own answers to one prompt range across two points on your scale, then a one-point gap between two tools on that prompt tells you nothing at all.\n\n## The practical rule\n\nFor the two or three prompts that matter most to your decision, run each three times per tool. For the rest, once is fine. This is about ten extra minutes and it stops most of the wrong conclusions.\n\nWhen you do it, watch for one specific pattern: **does the tool vary in style, or in substance?** Three answers that say the same thing in different words is a healthy model. Three answers that give three different dates for the same event is a model that does not know the date, and the variation is the tell. You can use this deliberately as a confidence check: ask the same factual question in three separate conversations, and if the answers disagree, you have found something you must verify.\n\n> Figure: One prompt, five runs, two tools. Line chart of Score out of ten against the written criteria against Run number: Tool A (1, 7) (2, 5) (3, 8) (4, 6) (5, 7); Tool B (1, 6) (2, 8) (3, 5) (4, 7) (5, 6). Tool A averages 6.6 and tool B 6.4. Each tool's own answers range across three points, so a gap of two tenths is not a gap at all. A single run is one draw from a distribution, and the run that gets screenshotted and shared is the good tail by construction.\n\n## Turning it off where you can\n\nIf you have API access or a local model, set the temperature explicitly.\n\n```python\n# near-deterministic: extraction, classification, formatting, code\nclient.chat.completions.create(model=M, messages=msgs, temperature=0)\n\n# some variety: drafting, brainstorming, alternatives\nclient.chat.completions.create(model=M, messages=msgs, temperature=0.8)\n```\n\nFor anything where you want the same input to give the same output (pulling fields out of invoices, sorting tickets, generating code) run it near zero. It reduces the surprises without costing quality.\n\nOne honest caveat: temperature zero is not a guarantee of identical output. Providers run these models across large fleets of hardware where the order of floating-point operations varies between machines, so tiny numerical differences can occasionally change a token, and a single changed token changes everything after it. Near-deterministic is the accurate description. If you build something that assumes byte-identical replies, it will break eventually.\n\n## What the variation looks like across products\n\nProducts differ in how much they wander, and it is not only the temperature setting. A product that runs a router may serve you a different model on the second attempt, which produces variation far larger than sampling alone. If a tool's three answers differ in structure, length and depth rather than only in wording, suspect the router rather than the dice.\n\nYou can sometimes see this directly. Where the product names the model under each answer, check whether it is the same name all three times. Where it does not, note the pattern and treat that product's single answers with more caution than a product that is consistently serving one model.\n\n## The one place variation is the point\n\nWhen you want options rather than an answer (names, headlines, approaches, ways to phrase a difficult message) the variation is the feature. Ask for five and pick one. Raising the temperature produces more interesting alternatives, and asking the same question three times in three separate conversations produces more variety than asking for three options in one, because the model in a single reply tries to make its options differ from each other in shallow ways."},"check":{"question":"Asked the same factual question in three separate conversations, a tool gives three different dates. What should you take from that?","options":[{"text":"The model does not hold that date reliably, and the disagreement is a usable warning that the fact needs verifying elsewhere.","correct":true},{"text":"The temperature is set too high in that product, and the same question at temperature zero would give the correct date.","correct":false,"tempting":"Mistakes a knowledge gap for a sampling setting; temperature zero would return one consistent date, with nothing making it the right one."},{"text":"Separate conversations lack shared context, so the model was answering three subtly different questions.","correct":false,"tempting":"Explains away identical prompts as different ones; isolation is what makes the three draws independent and therefore informative."},{"text":"The tool is retrieving from the web each time and picking up different sources, so the variation is in the sources rather than the model.","correct":false,"tempting":"Assumes a retrieval step that may not have run at all, and cited sources would be visible and checkable rather than silently contradictory."}]},"prev":{"slug":"blind-comparison","title":"Taking the brand off the answers","url":"https://addaly.com/learn/choosing-your-tools/blind-comparison"},"next":{"slug":"scoring-without-fooling-yourself","title":"A rubric you can apply in ten seconds","url":"https://addaly.com/learn/choosing-your-tools/scoring-without-fooling-yourself"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Run it five times', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/same-prompt-twice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Run it five times","courseTitle":"Choosing and Using the Tools","module":{"index":2,"slug":"testing-it-yourself","title":"Testing a tool yourself"},"answer":null,"keyIdea":"Because products sample from a probability distribution rather than picking the single most likely reply, a single run is one draw and any tool difference smaller than a tool’s own run-to-run variation is not a difference.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"One prompt, five runs, two tools","series":[{"label":"Tool A","points":[[1,7],[2,5],[3,8],[4,6],[5,7]]},{"label":"Tool B","points":[[1,6],[2,8],[3,5],[4,7],[5,6]]}],"xLabel":"Run number","yLabel":"Score out of ten against the written criteria","caption":"Tool A averages 6.6 and tool B 6.4. Each tool’s own answers range across three points, so a gap of two tenths is not a gap at all. A single run is one draw from a distribution, and the run that gets screenshotted and shared is the good tail by construction."},"alt":"Figure: One prompt, five runs, two tools. Line chart of Score out of ten against the written criteria against Run number: Tool A (1, 7) (2, 5) (3, 8) (4, 6) (5, 7); Tool B (1, 6) (2, 8) (3, 5) (4, 7) (5, 6). Tool A averages 6.6 and tool B 6.4. Each tool’s own answers range across three points, so a gap of two tenths is not a gap at all. A single run is one draw from a distribution, and the run that gets screenshotted and shared is the good tail by construction."}},"blocks":[{"t":"h2","id":"why-the-same-question-gives-different-answers","runs":[{"text":"Why the same question gives different answers"}]},{"t":"p","runs":[{"text":"Send an identical prompt twice and you usually get two different replies. This is not a fault. At each step the model produces a probability distribution over possible next tokens, and the product samples from it rather than always taking the most likely one. The setting that controls how adventurous that sampling is is called "},{"text":"temperature","b":true},{"text":": near zero it almost always takes the top choice and answers are nearly repeatable; higher and it wanders."}]},{"t":"p","runs":[{"text":"Consumer chat products generally run at a moderate temperature and do not let you change it. APIs and local runners do, and it is one of the more useful dials you will ever be handed."}]},{"t":"h2","id":"what-this-does-to-your-testing","runs":[{"text":"What this does to your testing"}]},{"t":"p","runs":[{"text":"It means a single run is a sample, not a measurement. Three things follow, and ignoring them is how people reach confident conclusions from nothing."}]},{"t":"p","runs":[{"text":"One bad answer proves very little.","b":true},{"text":" If a tool fails a prompt once, run it twice more before you write it down. A tool that fails a prompt three times out of three has a problem. A tool that fails once out of three may just have been unlucky, and you would not want to be judged on your third-best attempt either."}]},{"t":"p","runs":[{"text":"One brilliant answer proves even less.","b":true},{"text":" This is the direction people forget. A launch demo, a screenshot in a thread, a friend’s amazing result: all of these are one draw from a distribution, and the ones that get shared are the good tail by construction. Nobody screenshots the mediocre attempt."}]},{"t":"p","runs":[{"text":"Differences smaller than the variation are not differences.","b":true},{"text":" If a tool’s own answers to one prompt range across two points on your scale, then a one-point gap between two tools on that prompt tells you nothing at all."}]},{"t":"h2","id":"the-practical-rule","runs":[{"text":"The practical rule"}]},{"t":"p","runs":[{"text":"For the two or three prompts that matter most to your decision, run each three times per tool. For the rest, once is fine. This is about ten extra minutes and it stops most of the wrong conclusions."}]},{"t":"p","runs":[{"text":"When you do it, watch for one specific pattern: "},{"text":"does the tool vary in style, or in substance?","b":true},{"text":" Three answers that say the same thing in different words is a healthy model. Three answers that give three different dates for the same event is a model that does not know the date, and the variation is the tell. You can use this deliberately as a confidence check: ask the same factual question in three separate conversations, and if the answers disagree, you have found something you must verify."}]},{"t":"h2","id":"turning-it-off-where-you-can","runs":[{"text":"Turning it off where you can"}]},{"t":"p","runs":[{"text":"If you have API access or a local model, set the temperature explicitly."}]},{"t":"code","lang":"python","label":"Python","text":"# near-deterministic: extraction, classification, formatting, code\nclient.chat.completions.create(model=M, messages=msgs, temperature=0)\n\n# some variety: drafting, brainstorming, alternatives\nclient.chat.completions.create(model=M, messages=msgs, temperature=0.8)"},{"t":"p","runs":[{"text":"For anything where you want the same input to give the same output (pulling fields out of invoices, sorting tickets, generating code) run it near zero. It reduces the surprises without costing quality."}]},{"t":"p","runs":[{"text":"One honest caveat: temperature zero is not a guarantee of identical output. Providers run these models across large fleets of hardware where the order of floating-point operations varies between machines, so tiny numerical differences can occasionally change a token, and a single changed token changes everything after it. Near-deterministic is the accurate description. If you build something that assumes byte-identical replies, it will break eventually."}]},{"t":"h2","id":"what-the-variation-looks-like-across-products","runs":[{"text":"What the variation looks like across products"}]},{"t":"p","runs":[{"text":"Products differ in how much they wander, and it is not only the temperature setting. A product that runs a router may serve you a different model on the second attempt, which produces variation far larger than sampling alone. If a tool’s three answers differ in structure, length and depth rather than only in wording, suspect the router rather than the dice."}]},{"t":"p","runs":[{"text":"You can sometimes see this directly. Where the product names the model under each answer, check whether it is the same name all three times. Where it does not, note the pattern and treat that product’s single answers with more caution than a product that is consistently serving one model."}]},{"t":"h2","id":"the-one-place-variation-is-the-point","runs":[{"text":"The one place variation is the point"}]},{"t":"p","runs":[{"text":"When you want options rather than an answer (names, headlines, approaches, ways to phrase a difficult message) the variation is the feature. Ask for five and pick one. Raising the temperature produces more interesting alternatives, and asking the same question three times in three separate conversations produces more variety than asking for three options in one, because the model in a single reply tries to make its options differ from each other in shallow ways."}]}],"check":{"question":[{"text":"Asked the same factual question in three separate conversations, a tool gives three different dates. What should you take from that?"}],"options":[{"text":[{"text":"The model does not hold that date reliably, and the disagreement is a usable warning that the fact needs verifying elsewhere."}],"correct":true,"tempting":null},{"text":[{"text":"The temperature is set too high in that product, and the same question at temperature zero would give the correct date."}],"correct":false,"tempting":[{"text":"Mistakes a knowledge gap for a sampling setting; temperature zero would return one consistent date, with nothing making it the right one."}]},{"text":[{"text":"Separate conversations lack shared context, so the model was answering three subtly different questions."}],"correct":false,"tempting":[{"text":"Explains away identical prompts as different ones; isolation is what makes the three draws independent and therefore informative."}]},{"text":[{"text":"The tool is retrieving from the web each time and picking up different sources, so the variation is in the sources rather than the model."}],"correct":false,"tempting":[{"text":"Assumes a retrieval step that may not have run at all, and cited sources would be visible and checkable rather than silently contradictory."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"blind-comparison","title":"Taking the brand off the answers","minutes":7},"next":{"slug":"scoring-without-fooling-yourself","title":"A rubric you can apply in ten seconds","minutes":7}}}