Asking the same question several times

One prompt, several draws

Because generation samples, running the same prompt twice gives you two independent attempts. That is not only a nuisance to be suppressed; it is a resource.

Three techniques use it, and they differ in what they need.

Self-consistency. Sample n answers at a moderate temperature, then take the majority answer. It needs a task with a checkable final answer (a number, a label, a chosen option) so that “majority” is well defined. On multi-step arithmetic and reasoning benchmarks this reliably beats a single sample, with the gain flattening somewhere around 10 to 40 samples depending on the task.

Best-of-n. Sample n answers, then score them with something (a reward model, a rubric-driven judge, or a test suite) and keep the best. Applicable to open-ended output where voting is meaningless, but only as good as the scorer.

Verification loops. Sample one answer, check it with code, and re-sample if the check fails. The strongest version by far, because the check is objective: a generated SQL query either runs or does not, generated code either passes the tests or does not.

Why it works, in mechanism terms

A model that is right 60 per cent of the time on a task is usually not making the same mistake each time. Errors scatter across many wrong answers while correct answers concentrate on one. Majority voting exploits exactly that asymmetry.

Which tells you when it fails: if the model is confidently and consistently wrong, every sample agrees and voting confirms the error with a large majority. The technique converts inconsistency into accuracy. It cannot repair a systematic misunderstanding, and a high agreement rate across samples is therefore not evidence of correctness.

What it costs

Linear in n. Five samples cost five times the output tokens and five times the decode time. Since output tokens are the expensive half, this is a real budget decision, and it is the honest framing of “test-time compute”: you are buying accuracy with money and latency at the moment of use rather than at training time.

The practical shape most teams land on:

• n = 1 for everything by default.
• n = 3 to 5 for the small fraction of requests where the stakes justify it, identified, ideally, by the log-probability signal from earlier in this module.
• A verification loop wherever an objective check exists, because it beats voting at the same cost.

Running it on every request is usually the wrong trade, and running it never means leaving a straightforward accuracy improvement unused.

Getting the sampling right

Two details that people get wrong.

Temperature must not be zero. At temperature zero all n samples are the same answer and you have paid n times for one. Self-consistency needs genuine variation; around 0.7 is the common starting point.

Aggregate the answer, not the text. Extract the final value from each sample and vote on that. Voting on whole responses almost never finds a majority, because the prose differs even when the conclusion agrees.

from collections import Counter
answers = [extract_final(sample(prompt, temperature=0.7)) for _ in range(5)]
best, votes = Counter(answers).most_common(1)[0]
print(best, votes, "/", len(answers))

That vote count is also a usable confidence signal, and often a better one than the model’s own stated confidence: 5 out of 5 is meaningfully different from 2 out of 5, and both are grounded in something the system actually did.

A worked case

A clinic wants to extract a diagnosis code from a doctor’s free-text note. A single sample is correct on 82 of 100 test notes. Sampling five at temperature 0.7 and taking the majority code raises it to 91. The eighteen original failures split into two groups: eleven where the five samples disagreed among themselves (voting recovered nine of those) and seven where all five gave the same wrong code, which voting could not touch.

That split is the useful output of the experiment, not the headline number. The seven are a different problem, and no amount of extra sampling will address them; they need better context, a clearer prompt, or a fine-tune. The eleven were noise, and noise is what voting is for. Running this analysis on your own set tells you whether test-time compute is the right spend before you commit to paying five times over on every request.

The connection to reasoning models

This is the manual version of what reasoning models do internally. They were trained to spend more tokens before answering, which buys the same kind of improvement without you orchestrating it. Sampling several answers and voting remains useful on top, and the two combine, at a cost that multiplies, so measure before you assume it is worth it.