Asking more than once, on purpose

Rerolling versus sampling deliberately

Pressing regenerate until something looks good is a slot machine, and you know why: same prompt, same distribution, different draw.

Running the same prompt several times and comparing the results is a different activity, and a useful one. The difference is what you do with the outputs. One picks a winner by feel. The other uses the spread as information.

What the spread tells you

Run a prompt three times.

All three agree. The answer is stable under this prompt. That is not proof it is right (a systematic misreading is perfectly stable) but instability has been ruled out.

Two agree, one differs. Usually the odd one out is wrong, and occasionally the odd one out spotted something. Either way you now know there is something ambiguous, and looking at what differs points straight at it.

All three differ. The prompt is under-determined. Do not pick a favourite; go and fix the prompt. This is the most useful of the three results and the one people ignore, because picking a favourite feels like progress.

For anything numerical or factual with a single right answer, this is close to a free reliability check. Three runs, look for agreement. Where they disagree is exactly where to check.

The formal version, briefly

This idea has a name in the research literature, self-consistency, where a model generates several reasoning paths at a non-zero temperature and the most common final answer is taken. It measurably improves accuracy on arithmetic and reasoning benchmarks, and it costs you several times as many tokens.

The plain version is the one you will use: ask three times, take the answer that appears most often, and investigate any disagreement.

Two limits, stated honestly. It only works where answers can be compared: a number, a label, a name. For prose there is no majority to take. And it cannot fix a systematic error: if the model consistently misreads the same clause, all three runs agree and are wrong together. Agreement measures stability, not truth.

Deliberate variation, which is different again

Instead of the same prompt three times, change something on purpose:

Give me three approaches to this problem that differ in their basic assumption, not just in wording. Then say which you would choose and what would make you change your mind.

One call, three different answers, and a stated criterion for choosing. This is much better than three reruns for anything design-shaped (a plan, a structure, a strategy) where the value is in seeing options rather than in confirming one.

You can push it further by naming the axes: “one that optimises for speed, one for cost, one for reversibility.”

Comparing without fooling yourself

If you generate several candidates and pick, be aware that you are now the evaluator, with all that implies.

• Look at them side by side, not in sequence. Sequential reading favours the last one.
• Decide the criterion before reading. Otherwise the criterion becomes whatever the best-sounding candidate happens to do well.
• Hide which is which, when it matters and you are comparing two prompt versions rather than two ideas.

The cost, and when it is worth paying

Three runs cost three times as much and three times the wait. That is trivial for one important question and prohibitive for a batch of a thousand.

The sensible allocation is by consequence. One run for the routine. Three runs for a number that will go into a document, a claim you will repeat, or a decision you cannot easily undo.

And the corollary, which is the honest end of this module: if the answer matters enough to run three times, it matters enough to check against something that is not a model.

Putting the module together

Six techniques, one underlying move: give the work somewhere to happen and somewhere to be seen. Steps make the intermediate results visible. Code makes the arithmetic real. Splitting makes each stage checkable. A plan makes the assumptions visible before anything is built. A constraint check makes compliance evidence rather than assertion. Repeated runs make instability visible.

None of them makes the model more capable. All of them move the parts you would otherwise have to trust into places where you can look. That is the whole of what this module can offer, and it is worth more than it sounds, because the failures that hurt are precisely the ones that were never visible.