{"course":{"slug":"prompting","title":"Getting Real Work out of a Model","url":"https://addaly.com/learn/prompting","level":"beginner","track":"start"},"lesson":{"slug":"several-attempts","title":"Asking more than once, on purpose","url":"https://addaly.com/learn/prompting/several-attempts","position":45,"of":72,"minutes":8,"answer":"Use repeated runs as a measurement rather than a re-roll: agreement shows stability, disagreement shows exactly where the prompt is under-determined, and neither shows truth.","keyIdea":"Use repeated runs as a measurement rather than a re-roll: agreement shows stability, disagreement shows exactly where the prompt is under-determined, and neither shows truth."},"body":{"format":"markdown","text":"## Rerolling versus sampling deliberately\n\nPressing regenerate until something looks good is a slot machine, and you know why: same prompt, same distribution, different draw.\n\nRunning the same prompt several times *and comparing the results* is a different activity, and a useful one. The difference is what you do with the outputs. One picks a winner by feel. The other uses the spread as information.\n\n## What the spread tells you\n\nRun a prompt three times.\n\n**All three agree.** The answer is stable under this prompt. That is not proof it is right (a systematic misreading is perfectly stable) but instability has been ruled out.\n\n**Two agree, one differs.** Usually the odd one out is wrong, and occasionally the odd one out spotted something. Either way you now know there is something ambiguous, and looking at what differs points straight at it.\n\n**All three differ.** The prompt is under-determined. Do not pick a favourite; go and fix the prompt. This is the most useful of the three results and the one people ignore, because picking a favourite feels like progress.\n\nFor anything numerical or factual with a single right answer, this is close to a free reliability check. Three runs, look for agreement. Where they disagree is exactly where to check.\n\n## The formal version, briefly\n\nThis idea has a name in the research literature, self-consistency, where a model generates several reasoning paths at a non-zero temperature and the most common final answer is taken. It measurably improves accuracy on arithmetic and reasoning benchmarks, and it costs you several times as many tokens.\n\nThe plain version is the one you will use: **ask three times, take the answer that appears most often, and investigate any disagreement**.\n\nTwo limits, stated honestly. It only works where answers can be compared: a number, a label, a name. For prose there is no majority to take. And it cannot fix a systematic error: if the model consistently misreads the same clause, all three runs agree and are wrong together. Agreement measures stability, not truth.\n\n## Deliberate variation, which is different again\n\nInstead of the same prompt three times, change something on purpose:\n\n> Give me three approaches to this problem that differ in their basic assumption, not just in wording. Then say which you would choose and what would make you change your mind.\n\nOne call, three different answers, and a stated criterion for choosing. This is much better than three reruns for anything design-shaped (a plan, a structure, a strategy) where the value is in seeing options rather than in confirming one.\n\nYou can push it further by naming the axes: \"one that optimises for speed, one for cost, one for reversibility.\"\n\n## Comparing without fooling yourself\n\nIf you generate several candidates and pick, be aware that you are now the evaluator, with all that implies.\n\n- **Look at them side by side**, not in sequence. Sequential reading favours the last one.\n- **Decide the criterion before reading.** Otherwise the criterion becomes whatever the best-sounding candidate happens to do well.\n- **Hide which is which**, when it matters and you are comparing two prompt versions rather than two ideas.\n\n## The cost, and when it is worth paying\n\nThree runs cost three times as much and three times the wait. That is trivial for one important question and prohibitive for a batch of a thousand.\n\nThe sensible allocation is by consequence. One run for the routine. Three runs for a number that will go into a document, a claim you will repeat, or a decision you cannot easily undo.\n\nAnd the corollary, which is the honest end of this module: **if the answer matters enough to run three times, it matters enough to check against something that is not a model.**\n\n## Putting the module together\n\nSix techniques, one underlying move: give the work somewhere to happen and somewhere to be seen. Steps make the intermediate results visible. Code makes the arithmetic real. Splitting makes each stage checkable. A plan makes the assumptions visible before anything is built. A constraint check makes compliance evidence rather than assertion. Repeated runs make instability visible.\n\nNone of them makes the model more capable. All of them move the parts you would otherwise have to trust into places where you can look. That is the whole of what this module can offer, and it is worth more than it sounds, because the failures that hurt are precisely the ones that were never visible."},"check":{"question":"An accountant runs the same extraction prompt three times on a difficult invoice and gets the same total each time. What has she established?","options":[{"text":"That the total is correct, since three independent runs agree.","correct":false,"tempting":"Treats runs of the same prompt as independent evidence; a consistent misreading reproduces perfectly."},{"text":"Nothing, because repeated runs of one prompt are always uninformative.","correct":false,"tempting":"Overcorrects: stability is genuine information, and instability would have been a real warning."},{"text":"That the answer is stable under this prompt, which rules out sampling variation but not a systematic misreading of the invoice.","correct":true},{"text":"That the invoice is unambiguous, since ambiguity would produce different answers.","correct":false,"tempting":"Confuses ambiguity in the document with variation in the output; a model can resolve a genuine ambiguity the same way every time."}]},"prev":{"slug":"asking-it-to-interview-you","title":"Let it ask you the questions","url":"https://addaly.com/learn/prompting/asking-it-to-interview-you"},"next":{"slug":"iterate-dont-reroll","title":"Edit the prompt, do not spin the wheel","url":"https://addaly.com/learn/prompting/iterate-dont-reroll"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Asking more than once, on purpose', in Getting Real Work out of a Model. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/prompting/several-attempts (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Asking more than once, on purpose","courseTitle":"Getting Real Work out of a Model","module":{"index":5,"slug":"making-it-work-the-problem","title":"Making it work the problem"},"answer":null,"keyIdea":"Use repeated runs as a measurement rather than a re-roll: agreement shows stability, disagreement shows exactly where the prompt is under-determined, and neither shows truth.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"rerolling-versus-sampling-deliberately","runs":[{"text":"Rerolling versus sampling deliberately"}]},{"t":"p","runs":[{"text":"Pressing regenerate until something looks good is a slot machine, and you know why: same prompt, same distribution, different draw."}]},{"t":"p","runs":[{"text":"Running the same prompt several times "},{"text":"and comparing the results","i":true},{"text":" is a different activity, and a useful one. The difference is what you do with the outputs. One picks a winner by feel. The other uses the spread as information."}]},{"t":"h2","id":"what-the-spread-tells-you","runs":[{"text":"What the spread tells you"}]},{"t":"p","runs":[{"text":"Run a prompt three times."}]},{"t":"p","runs":[{"text":"All three agree.","b":true},{"text":" The answer is stable under this prompt. That is not proof it is right (a systematic misreading is perfectly stable) but instability has been ruled out."}]},{"t":"p","runs":[{"text":"Two agree, one differs.","b":true},{"text":" Usually the odd one out is wrong, and occasionally the odd one out spotted something. Either way you now know there is something ambiguous, and looking at what differs points straight at it."}]},{"t":"p","runs":[{"text":"All three differ.","b":true},{"text":" The prompt is under-determined. Do not pick a favourite; go and fix the prompt. This is the most useful of the three results and the one people ignore, because picking a favourite feels like progress."}]},{"t":"p","runs":[{"text":"For anything numerical or factual with a single right answer, this is close to a free reliability check. Three runs, look for agreement. Where they disagree is exactly where to check."}]},{"t":"h2","id":"the-formal-version-briefly","runs":[{"text":"The formal version, briefly"}]},{"t":"p","runs":[{"text":"This idea has a name in the research literature, self-consistency, where a model generates several reasoning paths at a non-zero temperature and the most common final answer is taken. It measurably improves accuracy on arithmetic and reasoning benchmarks, and it costs you several times as many tokens."}]},{"t":"p","runs":[{"text":"The plain version is the one you will use: "},{"text":"ask three times, take the answer that appears most often, and investigate any disagreement","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"Two limits, stated honestly. It only works where answers can be compared: a number, a label, a name. For prose there is no majority to take. And it cannot fix a systematic error: if the model consistently misreads the same clause, all three runs agree and are wrong together. Agreement measures stability, not truth."}]},{"t":"h2","id":"deliberate-variation-which-is-different-again","runs":[{"text":"Deliberate variation, which is different again"}]},{"t":"p","runs":[{"text":"Instead of the same prompt three times, change something on purpose:"}]},{"t":"quote","runs":[{"text":"Give me three approaches to this problem that differ in their basic assumption, not just in wording. Then say which you would choose and what would make you change your mind."}]},{"t":"p","runs":[{"text":"One call, three different answers, and a stated criterion for choosing. This is much better than three reruns for anything design-shaped (a plan, a structure, a strategy) where the value is in seeing options rather than in confirming one."}]},{"t":"p","runs":[{"text":"You can push it further by naming the axes: “one that optimises for speed, one for cost, one for reversibility.”"}]},{"t":"h2","id":"comparing-without-fooling-yourself","runs":[{"text":"Comparing without fooling yourself"}]},{"t":"p","runs":[{"text":"If you generate several candidates and pick, be aware that you are now the evaluator, with all that implies."}]},{"t":"ul","items":[[{"text":"Look at them side by side","b":true},{"text":", not in sequence. Sequential reading favours the last one."}],[{"text":"Decide the criterion before reading.","b":true},{"text":" Otherwise the criterion becomes whatever the best-sounding candidate happens to do well."}],[{"text":"Hide which is which","b":true},{"text":", when it matters and you are comparing two prompt versions rather than two ideas."}]]},{"t":"h2","id":"the-cost-and-when-it-is-worth-paying","runs":[{"text":"The cost, and when it is worth paying"}]},{"t":"p","runs":[{"text":"Three runs cost three times as much and three times the wait. That is trivial for one important question and prohibitive for a batch of a thousand."}]},{"t":"p","runs":[{"text":"The sensible allocation is by consequence. One run for the routine. Three runs for a number that will go into a document, a claim you will repeat, or a decision you cannot easily undo."}]},{"t":"p","runs":[{"text":"And the corollary, which is the honest end of this module: "},{"text":"if the answer matters enough to run three times, it matters enough to check against something that is not a model.","b":true}]},{"t":"h2","id":"putting-the-module-together","runs":[{"text":"Putting the module together"}]},{"t":"p","runs":[{"text":"Six techniques, one underlying move: give the work somewhere to happen and somewhere to be seen. Steps make the intermediate results visible. Code makes the arithmetic real. Splitting makes each stage checkable. A plan makes the assumptions visible before anything is built. A constraint check makes compliance evidence rather than assertion. Repeated runs make instability visible."}]},{"t":"p","runs":[{"text":"None of them makes the model more capable. All of them move the parts you would otherwise have to trust into places where you can look. That is the whole of what this module can offer, and it is worth more than it sounds, because the failures that hurt are precisely the ones that were never visible."}]}],"check":{"question":[{"text":"An accountant runs the same extraction prompt three times on a difficult invoice and gets the same total each time. What has she established?"}],"options":[{"text":[{"text":"That the total is correct, since three independent runs agree."}],"correct":false,"tempting":[{"text":"Treats runs of the same prompt as independent evidence; a consistent misreading reproduces perfectly."}]},{"text":[{"text":"Nothing, because repeated runs of one prompt are always uninformative."}],"correct":false,"tempting":[{"text":"Overcorrects: stability is genuine information, and instability would have been a real warning."}]},{"text":[{"text":"That the answer is stable under this prompt, which rules out sampling variation but not a systematic misreading of the invoice."}],"correct":true,"tempting":null},{"text":[{"text":"That the invoice is unambiguous, since ambiguity would produce different answers."}],"correct":false,"tempting":[{"text":"Confuses ambiguity in the document with variation in the output; a model can resolve a genuine ambiguity the same way every time."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"asking-it-to-interview-you","title":"Let it ask you the questions","minutes":8},"next":{"slug":"iterate-dont-reroll","title":"Edit the prompt, do not spin the wheel","minutes":6}}}