{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"many-comparisons","title":"Twenty slices and one guaranteed surprise","url":"https://addaly.com/learn/evaluating-ai/many-comparisons","position":26,"of":76,"minutes":8,"answer":"Testing twenty slices at the five per cent level makes a false alarm more likely than not.","keyIdea":"Testing twenty slices at the five per cent level makes a false alarm more likely than not, so name one primary metric in advance and treat every slice as a lead to replicate rather than a result to ship."},"body":{"format":"markdown","text":"## The arithmetic of looking everywhere\n\nA change ships. You slice the result by language, device, tenure, plan, region, channel, input length, and intent: twenty slices. One of them shows a significant regression. The team spends two days on it.\n\nCompute the chance of that happening when the change did nothing at all:\n\n```\nP(at least one false alarm) = 1 - 0.95 ** 20 = 0.64\n```\n\nNearly two times in three you will find a \"significant\" slice in a change with no effect whatsoever. Not because anybody cheated; because twenty independent chances at a one-in-twenty event is what that expression says.\n\nThe same arithmetic applies to comparing eight prompt variants against a control, to tracking fifteen metrics on one experiment, and to running the same test weekly for a quarter.\n\n## Three corrections, and when each fits\n\n**Bonferroni.** Divide the threshold by the number of tests: with 20 slices, require p below 0.0025. It is correct, it is one line, and it is severe: power collapses, and you will miss real regressions in small slices. Use it when a false alarm is expensive and the tests are few.\n\n**Holm.** Sort the p-values, compare the smallest against `alpha/m`, the next against `alpha/(m-1)`, and stop at the first failure. It controls the same error rate as Bonferroni and is uniformly more powerful, so there is no reason to prefer plain Bonferroni once you know it exists.\n\n**Benjamini–Hochberg.** Controls the false discovery rate (the expected share of your claims that are wrong) rather than the chance of any error at all. Sort the p-values ascending and find the largest `i` with:\n\n```\np(i) <= (i / m) * q          # q = 0.10 is a common choice\n```\n\nReject everything up to that rank. This is the right tool for exploratory slicing, where you want a list of leads with a known contamination rate rather than a guarantee of purity.\n\n```python\nfrom statsmodels.stats.multitest import multipletests\nreject, p_adj, _, _ = multipletests(pvals, alpha=0.10, method=\"fdr_bh\")\n```\n\nWorked through: with 20 slices and q = 0.10, a p-value of 0.004 sitting at rank 1 passes because `1/20 * 0.10 = 0.005`, while a p-value of 0.03 at rank 5 fails because `5/20 * 0.10 = 0.025`. Bonferroni would have rejected both. The gain is real and it is bought by accepting that roughly one in ten of your accepted findings is noise, which is fine for a list of leads and not fine for a shipping decision.\n\n## The correction is not the real fix\n\nCorrections manage the arithmetic; they do not manage the behaviour that creates it. Two habits matter more.\n\n**Name the primary metric before the run.** One metric, one slice definition, one threshold, written down. Everything else in the report is labelled exploratory. This single convention removes most multiplicity problems, because the decision now rests on a test that was specified before any data existed.\n\n**Replicate anything surprising on fresh data.** A real regression in Hindi shows up again next week. A fluke does not. One replication is worth more than any correction, and it costs a rerun.\n\n## You do not need twenty tests to have the problem\n\nThe subtler version has no visible count. You look at the results, notice that long inputs seem worse, and test that. You ran one test, but you chose it after seeing the data, from among the dozens of slices you might have noticed. The error rate belongs to the set of things you *could* have chosen, not the one you did. This is why exploratory analysis is a source of hypotheses, and hypotheses need their own data.\n\nThe practical version: when a slice finding is found by looking, collect fresh items for that slice and re-measure. Frequently the effect halves, which is itself informative.\n\n## Guardrails go the other way\n\nThere is one deliberate exception. For metrics protecting against harm (safety violations, refusals, severe latency, error rates) you *want* a sensitive tripwire, and you accept more false alarms to get it. Do not apply a multiplicity correction to a guardrail. Set it loose, investigate every alert, and accept that most investigations find nothing. The asymmetry is intentional: an unnecessary investigation costs an afternoon, and a missed safety regression costs something you cannot buy back.\n\nWrite both policies into the report template, so nobody has to argue about it while looking at a fresh number: primary metric with its pre-registered threshold, exploratory slices with FDR control, guardrails uncorrected and always investigated."},"check":{"question":"A team reports that a new prompt significantly improved one of eighteen slices they examined at the 5% level, and proposes shipping on that basis. What is the strongest objection?","options":[{"text":"The slice was probably too small to support a significant result, so the finding is driven by a handful of items.","correct":false,"tempting":"Small slices do produce unstable estimates, but a significant result on a small slice is not automatically driven by few items, and the multiplicity issue holds even for large slices."},{"text":"Eighteen tests at 5% give roughly a 60% chance of at least one false positive, so the finding is a lead that needs replication on fresh data rather than a result.","correct":true},{"text":"Slice-level results should never be used for decisions, only whole-set results.","correct":false,"tempting":"Overreaches: slice results are essential for fairness and for diagnosis, and the discipline needed is pre-registration and replication rather than a ban."},{"text":"They should have applied a Bonferroni correction, which would have made the result significant at the corrected level.","correct":false,"tempting":"Gets the direction of the correction backwards; Bonferroni makes the threshold stricter, so a result significant at 0.05 across eighteen tests would almost certainly not survive it."}]},"prev":{"slug":"clustered-and-repeated-data","title":"Five hundred rows, forty users","url":"https://addaly.com/learn/evaluating-ai/clustered-and-repeated-data"},"next":{"slug":"peeking-and-stopping-rules","title":"Watching the dashboard until it says yes","url":"https://addaly.com/learn/evaluating-ai/peeking-and-stopping-rules"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Twenty slices and one guaranteed surprise', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/many-comparisons (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Twenty slices and one guaranteed surprise","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"Testing twenty slices at the five per cent level makes a false alarm more likely than not, so name one primary metric in advance and treat every slice as a lead to replicate rather than a result to ship.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-arithmetic-of-looking-everywhere","runs":[{"text":"The arithmetic of looking everywhere"}]},{"t":"p","runs":[{"text":"A change ships. You slice the result by language, device, tenure, plan, region, channel, input length, and intent: twenty slices. One of them shows a significant regression. The team spends two days on it."}]},{"t":"p","runs":[{"text":"Compute the chance of that happening when the change did nothing at all:"}]},{"t":"code","lang":"","label":"Text","text":"P(at least one false alarm) = 1 - 0.95 ** 20 = 0.64"},{"t":"p","runs":[{"text":"Nearly two times in three you will find a “significant” slice in a change with no effect whatsoever. Not because anybody cheated; because twenty independent chances at a one-in-twenty event is what that expression says."}]},{"t":"p","runs":[{"text":"The same arithmetic applies to comparing eight prompt variants against a control, to tracking fifteen metrics on one experiment, and to running the same test weekly for a quarter."}]},{"t":"h2","id":"three-corrections-and-when-each-fits","runs":[{"text":"Three corrections, and when each fits"}]},{"t":"p","runs":[{"text":"Bonferroni.","b":true},{"text":" Divide the threshold by the number of tests: with 20 slices, require p below 0.0025. It is correct, it is one line, and it is severe: power collapses, and you will miss real regressions in small slices. Use it when a false alarm is expensive and the tests are few."}]},{"t":"p","runs":[{"text":"Holm.","b":true},{"text":" Sort the p-values, compare the smallest against "},{"text":"alpha/m","code":true},{"text":", the next against "},{"text":"alpha/(m-1)","code":true},{"text":", and stop at the first failure. It controls the same error rate as Bonferroni and is uniformly more powerful, so there is no reason to prefer plain Bonferroni once you know it exists."}]},{"t":"p","runs":[{"text":"Benjamini–Hochberg.","b":true},{"text":" Controls the false discovery rate (the expected share of your claims that are wrong) rather than the chance of any error at all. Sort the p-values ascending and find the largest "},{"text":"i","code":true},{"text":" with:"}]},{"t":"code","lang":"","label":"Text","text":"p(i) <= (i / m) * q          # q = 0.10 is a common choice"},{"t":"p","runs":[{"text":"Reject everything up to that rank. This is the right tool for exploratory slicing, where you want a list of leads with a known contamination rate rather than a guarantee of purity."}]},{"t":"code","lang":"python","label":"Python","text":"from statsmodels.stats.multitest import multipletests\nreject, p_adj, _, _ = multipletests(pvals, alpha=0.10, method=\"fdr_bh\")"},{"t":"p","runs":[{"text":"Worked through: with 20 slices and q = 0.10, a p-value of 0.004 sitting at rank 1 passes because "},{"text":"1/20 * 0.10 = 0.005","code":true},{"text":", while a p-value of 0.03 at rank 5 fails because "},{"text":"5/20 * 0.10 = 0.025","code":true},{"text":". Bonferroni would have rejected both. The gain is real and it is bought by accepting that roughly one in ten of your accepted findings is noise, which is fine for a list of leads and not fine for a shipping decision."}]},{"t":"h2","id":"the-correction-is-not-the-real-fix","runs":[{"text":"The correction is not the real fix"}]},{"t":"p","runs":[{"text":"Corrections manage the arithmetic; they do not manage the behaviour that creates it. Two habits matter more."}]},{"t":"p","runs":[{"text":"Name the primary metric before the run.","b":true},{"text":" One metric, one slice definition, one threshold, written down. Everything else in the report is labelled exploratory. This single convention removes most multiplicity problems, because the decision now rests on a test that was specified before any data existed."}]},{"t":"p","runs":[{"text":"Replicate anything surprising on fresh data.","b":true},{"text":" A real regression in Hindi shows up again next week. A fluke does not. One replication is worth more than any correction, and it costs a rerun."}]},{"t":"h2","id":"you-do-not-need-twenty-tests-to-have-the-problem","runs":[{"text":"You do not need twenty tests to have the problem"}]},{"t":"p","runs":[{"text":"The subtler version has no visible count. You look at the results, notice that long inputs seem worse, and test that. You ran one test, but you chose it after seeing the data, from among the dozens of slices you might have noticed. The error rate belongs to the set of things you "},{"text":"could","i":true},{"text":" have chosen, not the one you did. This is why exploratory analysis is a source of hypotheses, and hypotheses need their own data."}]},{"t":"p","runs":[{"text":"The practical version: when a slice finding is found by looking, collect fresh items for that slice and re-measure. Frequently the effect halves, which is itself informative."}]},{"t":"h2","id":"guardrails-go-the-other-way","runs":[{"text":"Guardrails go the other way"}]},{"t":"p","runs":[{"text":"There is one deliberate exception. For metrics protecting against harm (safety violations, refusals, severe latency, error rates) you "},{"text":"want","i":true},{"text":" a sensitive tripwire, and you accept more false alarms to get it. Do not apply a multiplicity correction to a guardrail. Set it loose, investigate every alert, and accept that most investigations find nothing. The asymmetry is intentional: an unnecessary investigation costs an afternoon, and a missed safety regression costs something you cannot buy back."}]},{"t":"p","runs":[{"text":"Write both policies into the report template, so nobody has to argue about it while looking at a fresh number: primary metric with its pre-registered threshold, exploratory slices with FDR control, guardrails uncorrected and always investigated."}]}],"check":{"question":[{"text":"A team reports that a new prompt significantly improved one of eighteen slices they examined at the 5% level, and proposes shipping on that basis. What is the strongest objection?"}],"options":[{"text":[{"text":"The slice was probably too small to support a significant result, so the finding is driven by a handful of items."}],"correct":false,"tempting":[{"text":"Small slices do produce unstable estimates, but a significant result on a small slice is not automatically driven by few items, and the multiplicity issue holds even for large slices."}]},{"text":[{"text":"Eighteen tests at 5% give roughly a 60% chance of at least one false positive, so the finding is a lead that needs replication on fresh data rather than a result."}],"correct":true,"tempting":null},{"text":[{"text":"Slice-level results should never be used for decisions, only whole-set results."}],"correct":false,"tempting":[{"text":"Overreaches: slice results are essential for fairness and for diagnosis, and the discipline needed is pre-registration and replication rather than a ban."}]},{"text":[{"text":"They should have applied a Bonferroni correction, which would have made the result significant at the corrected level."}],"correct":false,"tempting":[{"text":"Gets the direction of the correction backwards; Bonferroni makes the threshold stricter, so a result significant at 0.05 across eighteen tests would almost certainly not survive it."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"clustered-and-repeated-data","title":"Five hundred rows, forty users","minutes":9},"next":{"slug":"peeking-and-stopping-rules","title":"Watching the dashboard until it says yes","minutes":9}}}