{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"how-many-items-do-you-need","title":"Sizing the set before you run it","url":"https://addaly.com/learn/evaluating-ai/how-many-items-do-you-need","position":23,"of":76,"minutes":9,"answer":"Start from the smallest difference worth acting on, not the one you hope for.","keyIdea":"Start from the smallest difference worth acting on, not the one you hope for, and compute the item count it needs: for a five-point gap around 80% that is roughly a thousand items per side unless the comparison is paired."},"body":{"format":"markdown","text":"## Ask it in the other direction\n\nThe usual question is \"we have 200 items, what can we conclude\". The useful question is asked before the run: \"what is the smallest difference I would act on, and how many items does seeing it require\".\n\nThat first number is the minimum detectable effect, and it is a product decision, not a statistical one. Would you ship for 1 point? For 3? If a 2-point improvement would not change what you do, do not design an experiment to find one.\n\n## The arithmetic for two independent groups\n\nFor comparing two rates around `p`, with the conventional 80% power and 5% significance, the sample size per group is close to:\n\n```\nn ≈ 16 * p * (1 - p) / d²\n```\n\nwhere `d` is the difference you want to detect. Put in real numbers for a system passing about 80% of the time, hoping to detect 5 points:\n\n```\nn ≈ 16 * 0.8 * 0.2 / 0.05²  =  16 * 0.16 / 0.0025  =  1,024 per group\n```\n\nTwo thousand labelled items to resolve five points. That number surprises people and it is the honest arithmetic. Halve the effect you want to detect and it quadruples, because `d` is squared: detecting 2.5 points needs about 4,100 per group.\n\nThis is why most published comparisons on a hundred hand-labelled items cannot support the claims made from them, and why the next section matters so much.\n\n## Pairing changes everything\n\nRunning both systems on the *same* items removes the item-difficulty variation that dominates the noise above. What is left is the flips: items where the two systems disagree.\n\nThe count that matters is the number of discordant pairs, and it is much smaller than the total.\n\n- To detect that flips run 2 to 1 in your favour rather than evenly, you need roughly **68 discordant pairs**.\n- For a starker 3 to 1 split, about **29**.\n\nIf a change flips 15% of items, 68 discordant pairs means about 450 items. If it flips 30%, about 230. So a paired design turns a 2,000-item requirement into a few hundred: the single largest efficiency available in evaluation, and the reason the harness must record per-item results rather than totals.\n\n> Figure: Items needed to detect a gap around an 80% pass rate. Bars (labelled items): Unpaired, 10-point gap 256; Unpaired, 5-point gap 1024; Unpaired, 2.5-point gap 4096; Paired, 5-point gap, 15% of items flip 450. n ≈ 16 p(1 − p) / d² at 80% power and 5% significance, per group. Halving the gap quadruples the count. Pairing counts only the discordant items, so about 68 flips settle the same five-point question.\n\n## When you cannot remember any formula\n\nSimulate. This generalises to any metric and any design, and you can check it against your own data.\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\n\ndef power(n, p_a=0.80, effect=0.05, trials=2000):\n    wins = 0\n    for _ in range(trials):\n        a = rng.random(n) < p_a\n        b = rng.random(n) < p_a + effect\n        se = np.sqrt(p_a * (1 - p_a) * 2 / n)\n        wins += abs(b.mean() - a.mean()) > 1.96 * se\n    return wins / trials\n\nfor n in (200, 500, 1000, 2000):\n    print(n, round(power(n), 2))     # 0.22  0.47  0.77  0.97\n```\n\nTwo hundred items gives you a 22% chance of detecting a real five-point gain. Four times in five you would conclude nothing and move on, having thrown away a genuine improvement.\n\nThe simulation is also where you discover the second cost, which no formula shows: labelling. Two thousand items at two minutes each is over sixty hours of somebody's attention, and that is the real reason evaluation sets are small. Knowing the arithmetic lets you spend that budget where it buys the most: usually on the one comparison that decides an architecture, rather than spread thinly across every prompt tweak.\n\n## The effect you plug in is not the effect you hope for\n\nPower calculations are routinely run with the improvement somebody wants rather than the one that would change a decision, and the difference is not academic. Put in 10 points because that is the target, and the arithmetic returns a comfortable 260 items per group; the experiment then has almost no chance of detecting the 4-point gain that actually arrives, and reports nothing. Put in the smallest difference worth acting on, get a larger and more honest number, and decide from there whether to fund it.\n\n## What to do when you cannot afford it\n\nYou will often be unable to label 2,000 items. Four honest responses, in order of preference:\n\n1. **Pair the comparison** and count flips, as above.\n2. **Use a continuous per-item score** instead of pass or fail. A binary throws away magnitude; a 0-to-4 rubric total carries more information per item and needs fewer of them.\n3. **Reduce variance** by stratifying and by fixing seeds, the subject of a later lesson in this block.\n4. **Accept the limit and write it down.** \"This set can detect a 12-point difference. It cannot resolve the 4-point gap between these two prompts.\" That sentence is a result. It prevents a false claim, and it makes the case for the budget to label more.\n\nThe one thing not to do is run the small set, get a non-significant answer, and keep adding items until it turns significant. That converts a controlled error rate into an uncontrolled one, and it is the subject of the peeking lesson later in this block."},"check":{"question":"A team wants to detect a 2-point improvement on a system that currently passes 80% of items, using two independent sets. They have budget to label 600 items in total. What is the defensible plan?","options":[{"text":"Label the 600 items, run the comparison, and report whatever comes out with its confidence interval.","correct":false,"tempting":"Spends the budget on a test with roughly a 10% chance of detecting the effect, and the interval will be too wide to support any decision either way."},{"text":"Run both systems on the same 600 items and analyse the flips, accepting that only a difference producing a clear imbalance in discordant pairs will be detectable.","correct":true},{"text":"Split into three sets of 200 and take the majority verdict across the three comparisons.","correct":false,"tempting":"Splitting reduces power in each test and combining verdicts by majority has no defined error rate; three underpowered tests are worse than one."},{"text":"Use a 10% significance level instead of 5%, which roughly doubles the power at this sample size.","correct":false,"tempting":"Loosening alpha does raise power somewhat, but it does not approach what pairing achieves here, and it raises the false-alarm rate on every future comparison run the same way."}]},"prev":{"slug":"p-values-and-what-they-do-not-say","title":"What a p-value is, and the four things it is not","url":"https://addaly.com/learn/evaluating-ai/p-values-and-what-they-do-not-say"},"next":{"slug":"bootstrap-pitfalls","title":"The bootstrap, and the three places it lies","url":"https://addaly.com/learn/evaluating-ai/bootstrap-pitfalls"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Sizing the set before you run it', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/how-many-items-do-you-need (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Sizing the set before you run it","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"Start from the smallest difference worth acting on, not the one you hope for, and compute the item count it needs: for a five-point gap around 80% that is roughly a thousand items per side unless the comparison is paired.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"labelled items","title":"Items needed to detect a gap around an 80% pass rate","series":[{"label":"Unpaired, 10-point gap","value":256},{"label":"Unpaired, 5-point gap","value":1024},{"label":"Unpaired, 2.5-point gap","value":4096},{"label":"Paired, 5-point gap, 15% of items flip","value":450,"highlight":true}],"caption":"n ≈ 16 p(1 − p) / d² at 80% power and 5% significance, per group. Halving the gap quadruples the count. Pairing counts only the discordant items, so about 68 flips settle the same five-point question."},"alt":"Figure: Items needed to detect a gap around an 80% pass rate. Bars (labelled items): Unpaired, 10-point gap 256; Unpaired, 5-point gap 1024; Unpaired, 2.5-point gap 4096; Paired, 5-point gap, 15% of items flip 450. n ≈ 16 p(1 − p) / d² at 80% power and 5% significance, per group. Halving the gap quadruples the count. Pairing counts only the discordant items, so about 68 flips settle the same five-point question."}},"blocks":[{"t":"h2","id":"ask-it-in-the-other-direction","runs":[{"text":"Ask it in the other direction"}]},{"t":"p","runs":[{"text":"The usual question is “we have 200 items, what can we conclude”. The useful question is asked before the run: “what is the smallest difference I would act on, and how many items does seeing it require”."}]},{"t":"p","runs":[{"text":"That first number is the minimum detectable effect, and it is a product decision, not a statistical one. Would you ship for 1 point? For 3? If a 2-point improvement would not change what you do, do not design an experiment to find one."}]},{"t":"h2","id":"the-arithmetic-for-two-independent-groups","runs":[{"text":"The arithmetic for two independent groups"}]},{"t":"p","runs":[{"text":"For comparing two rates around "},{"text":"p","code":true},{"text":", with the conventional 80% power and 5% significance, the sample size per group is close to:"}]},{"t":"code","lang":"","label":"Text","text":"n ≈ 16 * p * (1 - p) / d²"},{"t":"p","runs":[{"text":"where "},{"text":"d","code":true},{"text":" is the difference you want to detect. Put in real numbers for a system passing about 80% of the time, hoping to detect 5 points:"}]},{"t":"code","lang":"","label":"Text","text":"n ≈ 16 * 0.8 * 0.2 / 0.05²  =  16 * 0.16 / 0.0025  =  1,024 per group"},{"t":"p","runs":[{"text":"Two thousand labelled items to resolve five points. That number surprises people and it is the honest arithmetic. Halve the effect you want to detect and it quadruples, because "},{"text":"d","code":true},{"text":" is squared: detecting 2.5 points needs about 4,100 per group."}]},{"t":"p","runs":[{"text":"This is why most published comparisons on a hundred hand-labelled items cannot support the claims made from them, and why the next section matters so much."}]},{"t":"h2","id":"pairing-changes-everything","runs":[{"text":"Pairing changes everything"}]},{"t":"p","runs":[{"text":"Running both systems on the "},{"text":"same","i":true},{"text":" items removes the item-difficulty variation that dominates the noise above. What is left is the flips: items where the two systems disagree."}]},{"t":"p","runs":[{"text":"The count that matters is the number of discordant pairs, and it is much smaller than the total."}]},{"t":"ul","items":[[{"text":"To detect that flips run 2 to 1 in your favour rather than evenly, you need roughly "},{"text":"68 discordant pairs","b":true},{"text":"."}],[{"text":"For a starker 3 to 1 split, about "},{"text":"29","b":true},{"text":"."}]]},{"t":"p","runs":[{"text":"If a change flips 15% of items, 68 discordant pairs means about 450 items. If it flips 30%, about 230. So a paired design turns a 2,000-item requirement into a few hundred: the single largest efficiency available in evaluation, and the reason the harness must record per-item results rather than totals."}]},{"t":"h2","id":"when-you-cannot-remember-any-formula","runs":[{"text":"When you cannot remember any formula"}]},{"t":"p","runs":[{"text":"Simulate. This generalises to any metric and any design, and you can check it against your own data."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nrng = np.random.default_rng(0)\n\ndef power(n, p_a=0.80, effect=0.05, trials=2000):\n    wins = 0\n    for _ in range(trials):\n        a = rng.random(n) < p_a\n        b = rng.random(n) < p_a + effect\n        se = np.sqrt(p_a * (1 - p_a) * 2 / n)\n        wins += abs(b.mean() - a.mean()) > 1.96 * se\n    return wins / trials\n\nfor n in (200, 500, 1000, 2000):\n    print(n, round(power(n), 2))     # 0.22  0.47  0.77  0.97"},{"t":"p","runs":[{"text":"Two hundred items gives you a 22% chance of detecting a real five-point gain. Four times in five you would conclude nothing and move on, having thrown away a genuine improvement."}]},{"t":"p","runs":[{"text":"The simulation is also where you discover the second cost, which no formula shows: labelling. Two thousand items at two minutes each is over sixty hours of somebody’s attention, and that is the real reason evaluation sets are small. Knowing the arithmetic lets you spend that budget where it buys the most: usually on the one comparison that decides an architecture, rather than spread thinly across every prompt tweak."}]},{"t":"h2","id":"the-effect-you-plug-in-is-not-the-effect-you-hope-for","runs":[{"text":"The effect you plug in is not the effect you hope for"}]},{"t":"p","runs":[{"text":"Power calculations are routinely run with the improvement somebody wants rather than the one that would change a decision, and the difference is not academic. Put in 10 points because that is the target, and the arithmetic returns a comfortable 260 items per group; the experiment then has almost no chance of detecting the 4-point gain that actually arrives, and reports nothing. Put in the smallest difference worth acting on, get a larger and more honest number, and decide from there whether to fund it."}]},{"t":"h2","id":"what-to-do-when-you-cannot-afford-it","runs":[{"text":"What to do when you cannot afford it"}]},{"t":"p","runs":[{"text":"You will often be unable to label 2,000 items. Four honest responses, in order of preference:"}]},{"t":"ol","start":1,"items":[[{"text":"Pair the comparison","b":true},{"text":" and count flips, as above."}],[{"text":"Use a continuous per-item score","b":true},{"text":" instead of pass or fail. A binary throws away magnitude; a 0-to-4 rubric total carries more information per item and needs fewer of them."}],[{"text":"Reduce variance","b":true},{"text":" by stratifying and by fixing seeds, the subject of a later lesson in this block."}],[{"text":"Accept the limit and write it down.","b":true},{"text":" “This set can detect a 12-point difference. It cannot resolve the 4-point gap between these two prompts.” That sentence is a result. It prevents a false claim, and it makes the case for the budget to label more."}]]},{"t":"p","runs":[{"text":"The one thing not to do is run the small set, get a non-significant answer, and keep adding items until it turns significant. That converts a controlled error rate into an uncontrolled one, and it is the subject of the peeking lesson later in this block."}]}],"check":{"question":[{"text":"A team wants to detect a 2-point improvement on a system that currently passes 80% of items, using two independent sets. They have budget to label 600 items in total. What is the defensible plan?"}],"options":[{"text":[{"text":"Label the 600 items, run the comparison, and report whatever comes out with its confidence interval."}],"correct":false,"tempting":[{"text":"Spends the budget on a test with roughly a 10% chance of detecting the effect, and the interval will be too wide to support any decision either way."}]},{"text":[{"text":"Run both systems on the same 600 items and analyse the flips, accepting that only a difference producing a clear imbalance in discordant pairs will be detectable."}],"correct":true,"tempting":null},{"text":[{"text":"Split into three sets of 200 and take the majority verdict across the three comparisons."}],"correct":false,"tempting":[{"text":"Splitting reduces power in each test and combining verdicts by majority has no defined error rate; three underpowered tests are worse than one."}]},{"text":[{"text":"Use a 10% significance level instead of 5%, which roughly doubles the power at this sample size."}],"correct":false,"tempting":[{"text":"Loosening alpha does raise power somewhat, but it does not approach what pairing achieves here, and it raises the false-alarm rate on every future comparison run the same way."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"p-values-and-what-they-do-not-say","title":"What a p-value is, and the four things it is not","minutes":9},"next":{"slug":"bootstrap-pitfalls","title":"The bootstrap, and the three places it lies","minutes":9}}}