Sizing the set before you run it

Ask it in the other direction

The usual question is “we have 200 items, what can we conclude”. The useful question is asked before the run: “what is the smallest difference I would act on, and how many items does seeing it require”.

That first number is the minimum detectable effect, and it is a product decision, not a statistical one. Would you ship for 1 point? For 3? If a 2-point improvement would not change what you do, do not design an experiment to find one.

The arithmetic for two independent groups

For comparing two rates around p, with the conventional 80% power and 5% significance, the sample size per group is close to:

n ≈ 16 * p * (1 - p) / d²

where d is the difference you want to detect. Put in real numbers for a system passing about 80% of the time, hoping to detect 5 points:

n ≈ 16 * 0.8 * 0.2 / 0.05²  =  16 * 0.16 / 0.0025  =  1,024 per group

Two thousand labelled items to resolve five points. That number surprises people and it is the honest arithmetic. Halve the effect you want to detect and it quadruples, because d is squared: detecting 2.5 points needs about 4,100 per group.

This is why most published comparisons on a hundred hand-labelled items cannot support the claims made from them, and why the next section matters so much.

Pairing changes everything

Running both systems on the same items removes the item-difficulty variation that dominates the noise above. What is left is the flips: items where the two systems disagree.

The count that matters is the number of discordant pairs, and it is much smaller than the total.

• To detect that flips run 2 to 1 in your favour rather than evenly, you need roughly 68 discordant pairs.
• For a starker 3 to 1 split, about 29.

If a change flips 15% of items, 68 discordant pairs means about 450 items. If it flips 30%, about 230. So a paired design turns a 2,000-item requirement into a few hundred: the single largest efficiency available in evaluation, and the reason the harness must record per-item results rather than totals.

When you cannot remember any formula

Simulate. This generalises to any metric and any design, and you can check it against your own data.

import numpy as np
rng = np.random.default_rng(0)

def power(n, p_a=0.80, effect=0.05, trials=2000):
    wins = 0
    for _ in range(trials):
        a = rng.random(n) < p_a
        b = rng.random(n) < p_a + effect
        se = np.sqrt(p_a * (1 - p_a) * 2 / n)
        wins += abs(b.mean() - a.mean()) > 1.96 * se
    return wins / trials

for n in (200, 500, 1000, 2000):
    print(n, round(power(n), 2))     # 0.22  0.47  0.77  0.97

Two hundred items gives you a 22% chance of detecting a real five-point gain. Four times in five you would conclude nothing and move on, having thrown away a genuine improvement.

The simulation is also where you discover the second cost, which no formula shows: labelling. Two thousand items at two minutes each is over sixty hours of somebody’s attention, and that is the real reason evaluation sets are small. Knowing the arithmetic lets you spend that budget where it buys the most: usually on the one comparison that decides an architecture, rather than spread thinly across every prompt tweak.

The effect you plug in is not the effect you hope for

Power calculations are routinely run with the improvement somebody wants rather than the one that would change a decision, and the difference is not academic. Put in 10 points because that is the target, and the arithmetic returns a comfortable 260 items per group; the experiment then has almost no chance of detecting the 4-point gain that actually arrives, and reports nothing. Put in the smallest difference worth acting on, get a larger and more honest number, and decide from there whether to fund it.

What to do when you cannot afford it

You will often be unable to label 2,000 items. Four honest responses, in order of preference:

1. Pair the comparison and count flips, as above.
2. Use a continuous per-item score instead of pass or fail. A binary throws away magnitude; a 0-to-4 rubric total carries more information per item and needs fewer of them.
3. Reduce variance by stratifying and by fixing seeds, the subject of a later lesson in this block.
4. Accept the limit and write it down. “This set can detect a 12-point difference. It cannot resolve the 4-point gap between these two prompts.” That sentence is a result. It prevents a false claim, and it makes the case for the budget to label more.

The one thing not to do is run the small set, get a non-significant answer, and keep adding items until it turns significant. That converts a controlled error rate into an uncontrolled one, and it is the subject of the peeking lesson later in this block.