How wrong your number probably is

A rate from a sample is an estimate

You ran 100 items and 82 passed. The system’s pass rate is not 82%. Eighty-two per cent is your best guess at it, drawn from one sample of a hundred, and if you drew a different hundred you would get a different number. How different is something you can calculate in one line, and almost nobody does.

For a pass rate p measured on n items, the standard error is:

se = sqrt(p * (1 - p) / n)

At p = 0.82 and n = 100 that is 0.038. A 95% interval is roughly plus or minus two standard errors, so about ±8 points: your true rate is plausibly anywhere from 74% to 90%.

Sit with that for a moment, because it reframes most of the conversations you have had about model quality. On a hundred items:

• 82% versus 79% is nothing. The intervals overlap almost entirely.
• 82% versus 91% is probably real.
• Announcing an improvement from 82% to 84% is announcing noise.

And the arithmetic runs the other way too. To halve the interval you need four times the data: ±8 points at 100 items becomes ±4 at 400 and ±2.5 at 1,000. Precision is expensive and you should buy only as much as your decision needs.

Paired comparison is far more sensitive

Here is the part that rescues small sets. When you compare two prompts, you are not comparing two independent samples. You are running the same items through both. That pairing carries information, and using it can detect differences the naive interval says are invisible.

Forget the two totals. Count the four cells:

• both passed: tells you nothing about the difference
• both failed: tells you nothing either
• old passed, new failed: call it b
• old failed, new passed: call it c

Only b and c carry the comparison. If the change did nothing, b and c should be about equal. McNemar’s test formalises this, and a usable rule of thumb is that the difference is worth believing when:

abs(b - c) > 2 * sqrt(b + c)

An example. Old prompt 82/100, new prompt 88/100. Six points, well inside the ±8 interval, so on the naive reading you cannot conclude anything. But suppose the flips are b = 2 and c = 8. Then abs(b - c) = 6 and 2 * sqrt(10) = 6.3, just short, so still not conclusive, but nearly. Whereas if the flips are b = 1, c = 7, you have 6 > 5.3 and a real result on a hundred items.

The same 100 items, the same 6-point gap, and a completely different conclusion depending on which items moved. This is why lesson five insists on tracking flips rather than totals, and it is the single highest-value statistical idea in the course.

The bootstrap, for everything that is not a simple rate

Mean latency, median cost, average judge score, recall@5: none of these have a neat formula you will remember. The bootstrap gives you an interval for any statistic without one, and it is five lines:

import numpy as np

scores = np.array(per_item_scores)   # one number per eval item
boot = [np.mean(np.random.choice(scores, len(scores), replace=True))
        for _ in range(2000)]
low, high = np.percentile(boot, [2.5, 97.5])
print(f"{scores.mean():.3f}  95% CI [{low:.3f}, {high:.3f}]")

Resample your own items with replacement, recompute the statistic, do it two thousand times, take the middle 95% of the answers. That is the whole idea. It works for any statistic you can compute from per-item results, which is why storing per-item results (not just the summary) matters so much.

For a paired comparison, bootstrap the difference: resample the item indices once, compute both systems’ scores on that resample, take the gap, repeat. If the resulting interval crosses zero, you cannot call it.

Non-determinism is a second source of spread

All of the above assumes one run. At temperature above zero, the same input gives different outputs, so there is variance in the system as well as in the sample. Run your set three times and you may see 82%, 79% and 84% with nothing changed at all.

The honest response is to run the set three or five times and report the mean and the spread across runs. If run-to-run spread is 5 points, then a 4-point improvement is not detectable no matter how many items you add. Setting temperature to 0 reduces this but does not eliminate it: batching, load and provider-side changes all leak in.

Write the number the honest way

One line, always in this shape:

Pass rate 82% (95% CI 74 to 90, n = 100, 3 runs, spread 3 pts), against 79% (74 to 86) for the previous prompt on the same items; 8 items improved, 5 regressed.

It is longer than “82%” and it is the difference between a measurement and a claim. It also has a social effect worth having: once a team writes intervals, it stops shipping on three-point movements, and the arguments about whose change helped get much shorter.