{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"sampling-many-answers","title":"Asking the same question several times","url":"https://addaly.com/learn/how-llms-work/sampling-many-answers","position":26,"of":78,"minutes":8,"answer":"Sampling several answers and aggregating them buys accuracy with compute at inference time, and the gain is largest where the model is nearly right and inconsistent.","keyIdea":"Sampling several answers and aggregating them buys accuracy with compute at inference time, and the gain is largest where the model is nearly right and inconsistent."},"body":{"format":"markdown","text":"## One prompt, several draws\n\nBecause generation samples, running the same prompt twice gives you two independent attempts. That is not only a nuisance to be suppressed; it is a resource.\n\nThree techniques use it, and they differ in what they need.\n\n**Self-consistency.** Sample *n* answers at a moderate temperature, then take the majority answer. It needs a task with a checkable final answer (a number, a label, a chosen option) so that \"majority\" is well defined. On multi-step arithmetic and reasoning benchmarks this reliably beats a single sample, with the gain flattening somewhere around 10 to 40 samples depending on the task.\n\n**Best-of-n.** Sample *n* answers, then score them with something (a reward model, a rubric-driven judge, or a test suite) and keep the best. Applicable to open-ended output where voting is meaningless, but only as good as the scorer.\n\n**Verification loops.** Sample one answer, check it with code, and re-sample if the check fails. The strongest version by far, because the check is objective: a generated SQL query either runs or does not, generated code either passes the tests or does not.\n\n## Why it works, in mechanism terms\n\nA model that is right 60 per cent of the time on a task is usually not making the *same* mistake each time. Errors scatter across many wrong answers while correct answers concentrate on one. Majority voting exploits exactly that asymmetry.\n\nWhich tells you when it fails: if the model is confidently and consistently wrong, every sample agrees and voting confirms the error with a large majority. The technique converts inconsistency into accuracy. It cannot repair a systematic misunderstanding, and a high agreement rate across samples is therefore not evidence of correctness.\n\n## What it costs\n\nLinear in *n*. Five samples cost five times the output tokens and five times the decode time. Since output tokens are the expensive half, this is a real budget decision, and it is the honest framing of \"test-time compute\": you are buying accuracy with money and latency at the moment of use rather than at training time.\n\nThe practical shape most teams land on:\n\n- **n = 1** for everything by default.\n- **n = 3 to 5** for the small fraction of requests where the stakes justify it, identified, ideally, by the log-probability signal from earlier in this module.\n- **A verification loop** wherever an objective check exists, because it beats voting at the same cost.\n\nRunning it on every request is usually the wrong trade, and running it never means leaving a straightforward accuracy improvement unused.\n\n## Getting the sampling right\n\nTwo details that people get wrong.\n\n**Temperature must not be zero.** At temperature zero all *n* samples are the same answer and you have paid *n* times for one. Self-consistency needs genuine variation; around 0.7 is the common starting point.\n\n**Aggregate the answer, not the text.** Extract the final value from each sample and vote on that. Voting on whole responses almost never finds a majority, because the prose differs even when the conclusion agrees.\n\n```python\nfrom collections import Counter\nanswers = [extract_final(sample(prompt, temperature=0.7)) for _ in range(5)]\nbest, votes = Counter(answers).most_common(1)[0]\nprint(best, votes, \"/\", len(answers))\n```\n\nThat vote count is also a usable confidence signal, and often a better one than the model's own stated confidence: 5 out of 5 is meaningfully different from 2 out of 5, and both are grounded in something the system actually did.\n\n## A worked case\n\nA clinic wants to extract a diagnosis code from a doctor's free-text note. A single sample is correct on 82 of 100 test notes. Sampling five at temperature 0.7 and taking the majority code raises it to 91. The eighteen original failures split into two groups: eleven where the five samples disagreed among themselves (voting recovered nine of those) and seven where all five gave the same wrong code, which voting could not touch.\n\nThat split is the useful output of the experiment, not the headline number. The seven are a different problem, and no amount of extra sampling will address them; they need better context, a clearer prompt, or a fine-tune. The eleven were noise, and noise is what voting is for. Running this analysis on your own set tells you whether test-time compute is the right spend before you commit to paying five times over on every request.\n\n## The connection to reasoning models\n\nThis is the manual version of what reasoning models do internally. They were trained to spend more tokens before answering, which buys the same kind of improvement without you orchestrating it. Sampling several answers and voting remains useful on top, and the two combine, at a cost that multiplies, so measure before you assume it is worth it."},"check":{"question":"A team applies self-consistency with five samples to a factual lookup task and sees no accuracy improvement, with all five samples agreeing every time. What does this indicate?","options":[{"text":"The temperature is too high, so the samples are exploring irrelevant answers.","correct":false,"tempting":"High temperature is the usual suspect when sampling behaves oddly, though it would cause disagreement rather than agreement."},{"text":"Five samples is too few for voting to show a benefit on a hard task.","correct":false,"tempting":"Sample counts do matter and gains do increase with n, so scaling up is a reasonable next experiment to propose."},{"text":"The model is consistently producing the same answer, so voting has nothing to aggregate; the errors are systematic rather than scattered.","correct":true},{"text":"Factual tasks require a reward model rather than majority voting.","correct":false,"tempting":"Best-of-n with a scorer is genuinely the right tool for open-ended output, which makes this a sensible-sounding method mismatch."}]},"prev":{"slug":"constrained-decoding","title":"Making output structurally valid, without asking nicely","url":"https://addaly.com/learn/how-llms-work/constrained-decoding"},"next":{"slug":"speculative-decoding","title":"Speculative decoding: a small model guesses, a big model checks","url":"https://addaly.com/learn/how-llms-work/speculative-decoding"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Asking the same question several times', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/sampling-many-answers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Asking the same question several times","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Sampling several answers and aggregating them buys accuracy with compute at inference time, and the gain is largest where the model is nearly right and inconsistent.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"one-prompt-several-draws","runs":[{"text":"One prompt, several draws"}]},{"t":"p","runs":[{"text":"Because generation samples, running the same prompt twice gives you two independent attempts. That is not only a nuisance to be suppressed; it is a resource."}]},{"t":"p","runs":[{"text":"Three techniques use it, and they differ in what they need."}]},{"t":"p","runs":[{"text":"Self-consistency.","b":true},{"text":" Sample "},{"text":"n","i":true},{"text":" answers at a moderate temperature, then take the majority answer. It needs a task with a checkable final answer (a number, a label, a chosen option) so that “majority” is well defined. On multi-step arithmetic and reasoning benchmarks this reliably beats a single sample, with the gain flattening somewhere around 10 to 40 samples depending on the task."}]},{"t":"p","runs":[{"text":"Best-of-n.","b":true},{"text":" Sample "},{"text":"n","i":true},{"text":" answers, then score them with something (a reward model, a rubric-driven judge, or a test suite) and keep the best. Applicable to open-ended output where voting is meaningless, but only as good as the scorer."}]},{"t":"p","runs":[{"text":"Verification loops.","b":true},{"text":" Sample one answer, check it with code, and re-sample if the check fails. The strongest version by far, because the check is objective: a generated SQL query either runs or does not, generated code either passes the tests or does not."}]},{"t":"h2","id":"why-it-works-in-mechanism-terms","runs":[{"text":"Why it works, in mechanism terms"}]},{"t":"p","runs":[{"text":"A model that is right 60 per cent of the time on a task is usually not making the "},{"text":"same","i":true},{"text":" mistake each time. Errors scatter across many wrong answers while correct answers concentrate on one. Majority voting exploits exactly that asymmetry."}]},{"t":"p","runs":[{"text":"Which tells you when it fails: if the model is confidently and consistently wrong, every sample agrees and voting confirms the error with a large majority. The technique converts inconsistency into accuracy. It cannot repair a systematic misunderstanding, and a high agreement rate across samples is therefore not evidence of correctness."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"Linear in "},{"text":"n","i":true},{"text":". Five samples cost five times the output tokens and five times the decode time. Since output tokens are the expensive half, this is a real budget decision, and it is the honest framing of “test-time compute”: you are buying accuracy with money and latency at the moment of use rather than at training time."}]},{"t":"p","runs":[{"text":"The practical shape most teams land on:"}]},{"t":"ul","items":[[{"text":"n = 1","b":true},{"text":" for everything by default."}],[{"text":"n = 3 to 5","b":true},{"text":" for the small fraction of requests where the stakes justify it, identified, ideally, by the log-probability signal from earlier in this module."}],[{"text":"A verification loop","b":true},{"text":" wherever an objective check exists, because it beats voting at the same cost."}]]},{"t":"p","runs":[{"text":"Running it on every request is usually the wrong trade, and running it never means leaving a straightforward accuracy improvement unused."}]},{"t":"h2","id":"getting-the-sampling-right","runs":[{"text":"Getting the sampling right"}]},{"t":"p","runs":[{"text":"Two details that people get wrong."}]},{"t":"p","runs":[{"text":"Temperature must not be zero.","b":true},{"text":" At temperature zero all "},{"text":"n","i":true},{"text":" samples are the same answer and you have paid "},{"text":"n","i":true},{"text":" times for one. Self-consistency needs genuine variation; around 0.7 is the common starting point."}]},{"t":"p","runs":[{"text":"Aggregate the answer, not the text.","b":true},{"text":" Extract the final value from each sample and vote on that. Voting on whole responses almost never finds a majority, because the prose differs even when the conclusion agrees."}]},{"t":"code","lang":"python","label":"Python","text":"from collections import Counter\nanswers = [extract_final(sample(prompt, temperature=0.7)) for _ in range(5)]\nbest, votes = Counter(answers).most_common(1)[0]\nprint(best, votes, \"/\", len(answers))"},{"t":"p","runs":[{"text":"That vote count is also a usable confidence signal, and often a better one than the model’s own stated confidence: 5 out of 5 is meaningfully different from 2 out of 5, and both are grounded in something the system actually did."}]},{"t":"h2","id":"a-worked-case","runs":[{"text":"A worked case"}]},{"t":"p","runs":[{"text":"A clinic wants to extract a diagnosis code from a doctor’s free-text note. A single sample is correct on 82 of 100 test notes. Sampling five at temperature 0.7 and taking the majority code raises it to 91. The eighteen original failures split into two groups: eleven where the five samples disagreed among themselves (voting recovered nine of those) and seven where all five gave the same wrong code, which voting could not touch."}]},{"t":"p","runs":[{"text":"That split is the useful output of the experiment, not the headline number. The seven are a different problem, and no amount of extra sampling will address them; they need better context, a clearer prompt, or a fine-tune. The eleven were noise, and noise is what voting is for. Running this analysis on your own set tells you whether test-time compute is the right spend before you commit to paying five times over on every request."}]},{"t":"h2","id":"the-connection-to-reasoning-models","runs":[{"text":"The connection to reasoning models"}]},{"t":"p","runs":[{"text":"This is the manual version of what reasoning models do internally. They were trained to spend more tokens before answering, which buys the same kind of improvement without you orchestrating it. Sampling several answers and voting remains useful on top, and the two combine, at a cost that multiplies, so measure before you assume it is worth it."}]}],"check":{"question":[{"text":"A team applies self-consistency with five samples to a factual lookup task and sees no accuracy improvement, with all five samples agreeing every time. What does this indicate?"}],"options":[{"text":[{"text":"The temperature is too high, so the samples are exploring irrelevant answers."}],"correct":false,"tempting":[{"text":"High temperature is the usual suspect when sampling behaves oddly, though it would cause disagreement rather than agreement."}]},{"text":[{"text":"Five samples is too few for voting to show a benefit on a hard task."}],"correct":false,"tempting":[{"text":"Sample counts do matter and gains do increase with n, so scaling up is a reasonable next experiment to propose."}]},{"text":[{"text":"The model is consistently producing the same answer, so voting has nothing to aggregate; the errors are systematic rather than scattered."}],"correct":true,"tempting":null},{"text":[{"text":"Factual tasks require a reward model rather than majority voting."}],"correct":false,"tempting":[{"text":"Best-of-n with a scorer is genuinely the right tool for open-ended output, which makes this a sensible-sounding method mismatch."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"constrained-decoding","title":"Making output structurally valid, without asking nicely","minutes":9},"next":{"slug":"speculative-decoding","title":"Speculative decoding: a small model guesses, a big model checks","minutes":8}}}