{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"error-bars-you-can-compute","title":"How wrong your number probably is","url":"https://addaly.com/learn/evaluating-ai/error-bars-you-can-compute","position":10,"of":76,"minutes":9,"answer":"A pass rate on 100 items carries roughly eight points of uncertainty, but comparing two systems on the same items (counting which ones flipped) can detect far smaller differences.","keyIdea":"A pass rate on 100 items carries roughly eight points of uncertainty, but comparing two systems on the same items (counting which ones flipped) can detect far smaller differences."},"body":{"format":"markdown","text":"## A rate from a sample is an estimate\n\nYou ran 100 items and 82 passed. The system's pass rate is not 82%. Eighty-two per cent is your best guess at it, drawn from one sample of a hundred, and if you drew a different hundred you would get a different number. How different is something you can calculate in one line, and almost nobody does.\n\nFor a pass rate `p` measured on `n` items, the standard error is:\n\n```\nse = sqrt(p * (1 - p) / n)\n```\n\nAt p = 0.82 and n = 100 that is 0.038. A 95% interval is roughly plus or minus two standard errors, so about **±8 points**: your true rate is plausibly anywhere from 74% to 90%.\n\nSit with that for a moment, because it reframes most of the conversations you have had about model quality. On a hundred items:\n\n- 82% versus 79% is nothing. The intervals overlap almost entirely.\n- 82% versus 91% is probably real.\n- Announcing an improvement from 82% to 84% is announcing noise.\n\nAnd the arithmetic runs the other way too. To halve the interval you need four times the data: ±8 points at 100 items becomes ±4 at 400 and ±2.5 at 1,000. Precision is expensive and you should buy only as much as your decision needs.\n\n## Paired comparison is far more sensitive\n\nHere is the part that rescues small sets. When you compare two prompts, you are not comparing two independent samples. You are running **the same items** through both. That pairing carries information, and using it can detect differences the naive interval says are invisible.\n\nForget the two totals. Count the four cells:\n\n- both passed: tells you nothing about the difference\n- both failed: tells you nothing either\n- old passed, new failed: call it `b`\n- old failed, new passed: call it `c`\n\nOnly `b` and `c` carry the comparison. If the change did nothing, `b` and `c` should be about equal. McNemar's test formalises this, and a usable rule of thumb is that the difference is worth believing when:\n\n```\nabs(b - c) > 2 * sqrt(b + c)\n```\n\nAn example. Old prompt 82/100, new prompt 88/100. Six points, well inside the ±8 interval, so on the naive reading you cannot conclude anything. But suppose the flips are `b = 2` and `c = 8`. Then `abs(b - c) = 6` and `2 * sqrt(10) = 6.3`, just short, so still not conclusive, but nearly. Whereas if the flips are `b = 1, c = 7`, you have `6 > 5.3` and a real result on a hundred items.\n\n> Figure: Same 100 items: old prompt 82 passed, new prompt 88. A two by two grid, rows Old: passed and Old: failed, columns New: passed and New: failed: Old: passed and New: passed: 81; Old: passed and New: failed: 1; Old: failed and New: passed: 7; Old: failed and New: failed: 11. Only b and c carry the comparison. Here |b − c| = 6 against 2√(b + c) = 2√8 ≈ 5.7, so this six-point gap is real on a hundred items, even though the naive ±8 interval would have called it noise.\n\nThe same 100 items, the same 6-point gap, and a completely different conclusion depending on *which* items moved. This is why lesson five insists on tracking flips rather than totals, and it is the single highest-value statistical idea in the course.\n\n## The bootstrap, for everything that is not a simple rate\n\nMean latency, median cost, average judge score, recall@5: none of these have a neat formula you will remember. The bootstrap gives you an interval for any statistic without one, and it is five lines:\n\n```python\nimport numpy as np\n\nscores = np.array(per_item_scores)   # one number per eval item\nboot = [np.mean(np.random.choice(scores, len(scores), replace=True))\n        for _ in range(2000)]\nlow, high = np.percentile(boot, [2.5, 97.5])\nprint(f\"{scores.mean():.3f}  95% CI [{low:.3f}, {high:.3f}]\")\n```\n\nResample your own items with replacement, recompute the statistic, do it two thousand times, take the middle 95% of the answers. That is the whole idea. It works for any statistic you can compute from per-item results, which is why storing per-item results (not just the summary) matters so much.\n\nFor a paired comparison, bootstrap the *difference*: resample the item indices once, compute both systems' scores on that resample, take the gap, repeat. If the resulting interval crosses zero, you cannot call it.\n\n## Non-determinism is a second source of spread\n\nAll of the above assumes one run. At temperature above zero, the same input gives different outputs, so there is variance in the system as well as in the sample. Run your set three times and you may see 82%, 79% and 84% with nothing changed at all.\n\nThe honest response is to run the set three or five times and report the mean and the spread across runs. If run-to-run spread is 5 points, then a 4-point improvement is not detectable no matter how many items you add. Setting temperature to 0 reduces this but does not eliminate it: batching, load and provider-side changes all leak in.\n\n## Write the number the honest way\n\nOne line, always in this shape:\n\n> Pass rate 82% (95% CI 74 to 90, n = 100, 3 runs, spread 3 pts), against 79% (74 to 86) for the previous prompt on the same items; 8 items improved, 5 regressed.\n\nIt is longer than \"82%\" and it is the difference between a measurement and a claim. It also has a social effect worth having: once a team writes intervals, it stops shipping on three-point movements, and the arguments about whose change helped get much shorter."},"check":{"question":"You compare two prompts on the same 100 items. Old scores 74, new scores 80. Of the items, 3 went from pass to fail and 9 from fail to pass. What is the soundest reading?","options":[{"text":"The flip counts support a real improvement: 9 versus 3 is a bigger gap than the paired rule of thumb tolerates by chance.","correct":true},{"text":"Six points is inside the ±9 point interval for a rate near 0.77 on 100 items, so the result is inconclusive.","correct":false,"tempting":"Applies the interval for a single independent rate to a paired comparison, discarding exactly the information that makes the comparison sensitive."},{"text":"The 3 regressions cancel the improvement, so the net gain is only 6 items and cannot be distinguished from noise.","correct":false,"tempting":"Nets the flips into a single count and then judges it by intuition, when the test is about the imbalance between the two flip directions."},{"text":"Nothing can be said until the set is grown to at least 400 items.","correct":false,"tempting":"Treats sample size as the only route to confidence and misses that pairing has already bought most of the precision more data would."}]},"prev":{"slug":"baselines-and-ceilings","title":"The number above it and the number below it","url":"https://addaly.com/learn/evaluating-ai/baselines-and-ceilings"},"next":{"slug":"measuring-text","title":"What to measure when the output is prose","url":"https://addaly.com/learn/evaluating-ai/measuring-text"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How wrong your number probably is', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/error-bars-you-can-compute (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How wrong your number probably is","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"A pass rate on 100 items carries roughly eight points of uncertainty, but comparing two systems on the same items (counting which ones flipped) can detect far smaller differences.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["New: passed","New: failed"],"rows":["Old: passed","Old: failed"],"type":"matrix","cells":[["81","1"],["7","11"]],"title":"Same 100 items: old prompt 82 passed, new prompt 88","caption":"Only b and c carry the comparison. Here |b − c| = 6 against 2√(b + c) = 2√8 ≈ 5.7, so this six-point gap is real on a hundred items, even though the naive ±8 interval would have called it noise.","cellLabels":[["no information","b: regressed"],["c: improved","no information"]]},"alt":"Figure: Same 100 items: old prompt 82 passed, new prompt 88. A two by two grid, rows Old: passed and Old: failed, columns New: passed and New: failed: Old: passed and New: passed: 81; Old: passed and New: failed: 1; Old: failed and New: passed: 7; Old: failed and New: failed: 11. Only b and c carry the comparison. Here |b − c| = 6 against 2√(b + c) = 2√8 ≈ 5.7, so this six-point gap is real on a hundred items, even though the naive ±8 interval would have called it noise."}},"blocks":[{"t":"h2","id":"a-rate-from-a-sample-is-an-estimate","runs":[{"text":"A rate from a sample is an estimate"}]},{"t":"p","runs":[{"text":"You ran 100 items and 82 passed. The system’s pass rate is not 82%. Eighty-two per cent is your best guess at it, drawn from one sample of a hundred, and if you drew a different hundred you would get a different number. How different is something you can calculate in one line, and almost nobody does."}]},{"t":"p","runs":[{"text":"For a pass rate "},{"text":"p","code":true},{"text":" measured on "},{"text":"n","code":true},{"text":" items, the standard error is:"}]},{"t":"code","lang":"","label":"Text","text":"se = sqrt(p * (1 - p) / n)"},{"t":"p","runs":[{"text":"At p = 0.82 and n = 100 that is 0.038. A 95% interval is roughly plus or minus two standard errors, so about "},{"text":"±8 points","b":true},{"text":": your true rate is plausibly anywhere from 74% to 90%."}]},{"t":"p","runs":[{"text":"Sit with that for a moment, because it reframes most of the conversations you have had about model quality. On a hundred items:"}]},{"t":"ul","items":[[{"text":"82% versus 79% is nothing. The intervals overlap almost entirely."}],[{"text":"82% versus 91% is probably real."}],[{"text":"Announcing an improvement from 82% to 84% is announcing noise."}]]},{"t":"p","runs":[{"text":"And the arithmetic runs the other way too. To halve the interval you need four times the data: ±8 points at 100 items becomes ±4 at 400 and ±2.5 at 1,000. Precision is expensive and you should buy only as much as your decision needs."}]},{"t":"h2","id":"paired-comparison-is-far-more-sensitive","runs":[{"text":"Paired comparison is far more sensitive"}]},{"t":"p","runs":[{"text":"Here is the part that rescues small sets. When you compare two prompts, you are not comparing two independent samples. You are running "},{"text":"the same items","b":true},{"text":" through both. That pairing carries information, and using it can detect differences the naive interval says are invisible."}]},{"t":"p","runs":[{"text":"Forget the two totals. Count the four cells:"}]},{"t":"ul","items":[[{"text":"both passed: tells you nothing about the difference"}],[{"text":"both failed: tells you nothing either"}],[{"text":"old passed, new failed: call it "},{"text":"b","code":true}],[{"text":"old failed, new passed: call it "},{"text":"c","code":true}]]},{"t":"p","runs":[{"text":"Only "},{"text":"b","code":true},{"text":" and "},{"text":"c","code":true},{"text":" carry the comparison. If the change did nothing, "},{"text":"b","code":true},{"text":" and "},{"text":"c","code":true},{"text":" should be about equal. McNemar’s test formalises this, and a usable rule of thumb is that the difference is worth believing when:"}]},{"t":"code","lang":"","label":"Text","text":"abs(b - c) > 2 * sqrt(b + c)"},{"t":"p","runs":[{"text":"An example. Old prompt 82/100, new prompt 88/100. Six points, well inside the ±8 interval, so on the naive reading you cannot conclude anything. But suppose the flips are "},{"text":"b = 2","code":true},{"text":" and "},{"text":"c = 8","code":true},{"text":". Then "},{"text":"abs(b - c) = 6","code":true},{"text":" and "},{"text":"2 * sqrt(10) = 6.3","code":true},{"text":", just short, so still not conclusive, but nearly. Whereas if the flips are "},{"text":"b = 1, c = 7","code":true},{"text":", you have "},{"text":"6 > 5.3","code":true},{"text":" and a real result on a hundred items."}]},{"t":"p","runs":[{"text":"The same 100 items, the same 6-point gap, and a completely different conclusion depending on "},{"text":"which","i":true},{"text":" items moved. This is why lesson five insists on tracking flips rather than totals, and it is the single highest-value statistical idea in the course."}]},{"t":"h2","id":"the-bootstrap-for-everything-that-is-not-a-simple-rate","runs":[{"text":"The bootstrap, for everything that is not a simple rate"}]},{"t":"p","runs":[{"text":"Mean latency, median cost, average judge score, recall@5: none of these have a neat formula you will remember. The bootstrap gives you an interval for any statistic without one, and it is five lines:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\n\nscores = np.array(per_item_scores)   # one number per eval item\nboot = [np.mean(np.random.choice(scores, len(scores), replace=True))\n        for _ in range(2000)]\nlow, high = np.percentile(boot, [2.5, 97.5])\nprint(f\"{scores.mean():.3f}  95% CI [{low:.3f}, {high:.3f}]\")"},{"t":"p","runs":[{"text":"Resample your own items with replacement, recompute the statistic, do it two thousand times, take the middle 95% of the answers. That is the whole idea. It works for any statistic you can compute from per-item results, which is why storing per-item results (not just the summary) matters so much."}]},{"t":"p","runs":[{"text":"For a paired comparison, bootstrap the "},{"text":"difference","i":true},{"text":": resample the item indices once, compute both systems’ scores on that resample, take the gap, repeat. If the resulting interval crosses zero, you cannot call it."}]},{"t":"h2","id":"non-determinism-is-a-second-source-of-spread","runs":[{"text":"Non-determinism is a second source of spread"}]},{"t":"p","runs":[{"text":"All of the above assumes one run. At temperature above zero, the same input gives different outputs, so there is variance in the system as well as in the sample. Run your set three times and you may see 82%, 79% and 84% with nothing changed at all."}]},{"t":"p","runs":[{"text":"The honest response is to run the set three or five times and report the mean and the spread across runs. If run-to-run spread is 5 points, then a 4-point improvement is not detectable no matter how many items you add. Setting temperature to 0 reduces this but does not eliminate it: batching, load and provider-side changes all leak in."}]},{"t":"h2","id":"write-the-number-the-honest-way","runs":[{"text":"Write the number the honest way"}]},{"t":"p","runs":[{"text":"One line, always in this shape:"}]},{"t":"quote","runs":[{"text":"Pass rate 82% (95% CI 74 to 90, n = 100, 3 runs, spread 3 pts), against 79% (74 to 86) for the previous prompt on the same items; 8 items improved, 5 regressed."}]},{"t":"p","runs":[{"text":"It is longer than “82%” and it is the difference between a measurement and a claim. It also has a social effect worth having: once a team writes intervals, it stops shipping on three-point movements, and the arguments about whose change helped get much shorter."}]}],"check":{"question":[{"text":"You compare two prompts on the same 100 items. Old scores 74, new scores 80. Of the items, 3 went from pass to fail and 9 from fail to pass. What is the soundest reading?"}],"options":[{"text":[{"text":"The flip counts support a real improvement: 9 versus 3 is a bigger gap than the paired rule of thumb tolerates by chance."}],"correct":true,"tempting":null},{"text":[{"text":"Six points is inside the ±9 point interval for a rate near 0.77 on 100 items, so the result is inconclusive."}],"correct":false,"tempting":[{"text":"Applies the interval for a single independent rate to a paired comparison, discarding exactly the information that makes the comparison sensitive."}]},{"text":[{"text":"The 3 regressions cancel the improvement, so the net gain is only 6 items and cannot be distinguished from noise."}],"correct":false,"tempting":[{"text":"Nets the flips into a single count and then judges it by intuition, when the test is about the imbalance between the two flip directions."}]},{"text":[{"text":"Nothing can be said until the set is grown to at least 400 items."}],"correct":false,"tempting":[{"text":"Treats sample size as the only route to confidence and misses that pairing has already bought most of the precision more data would."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"baselines-and-ceilings","title":"The number above it and the number below it","minutes":9},"next":{"slug":"measuring-text","title":"What to measure when the output is prose","minutes":8}}}