Twenty slices and one guaranteed surprise

The arithmetic of looking everywhere

A change ships. You slice the result by language, device, tenure, plan, region, channel, input length, and intent: twenty slices. One of them shows a significant regression. The team spends two days on it.

Compute the chance of that happening when the change did nothing at all:

P(at least one false alarm) = 1 - 0.95 ** 20 = 0.64

Nearly two times in three you will find a “significant” slice in a change with no effect whatsoever. Not because anybody cheated; because twenty independent chances at a one-in-twenty event is what that expression says.

The same arithmetic applies to comparing eight prompt variants against a control, to tracking fifteen metrics on one experiment, and to running the same test weekly for a quarter.

Three corrections, and when each fits

Bonferroni. Divide the threshold by the number of tests: with 20 slices, require p below 0.0025. It is correct, it is one line, and it is severe: power collapses, and you will miss real regressions in small slices. Use it when a false alarm is expensive and the tests are few.

Holm. Sort the p-values, compare the smallest against alpha/m, the next against alpha/(m-1), and stop at the first failure. It controls the same error rate as Bonferroni and is uniformly more powerful, so there is no reason to prefer plain Bonferroni once you know it exists.

Benjamini–Hochberg. Controls the false discovery rate (the expected share of your claims that are wrong) rather than the chance of any error at all. Sort the p-values ascending and find the largest i with:

p(i) <= (i / m) * q          # q = 0.10 is a common choice

Reject everything up to that rank. This is the right tool for exploratory slicing, where you want a list of leads with a known contamination rate rather than a guarantee of purity.

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(pvals, alpha=0.10, method="fdr_bh")

Worked through: with 20 slices and q = 0.10, a p-value of 0.004 sitting at rank 1 passes because 1/20 * 0.10 = 0.005, while a p-value of 0.03 at rank 5 fails because 5/20 * 0.10 = 0.025. Bonferroni would have rejected both. The gain is real and it is bought by accepting that roughly one in ten of your accepted findings is noise, which is fine for a list of leads and not fine for a shipping decision.

The correction is not the real fix

Corrections manage the arithmetic; they do not manage the behaviour that creates it. Two habits matter more.

Name the primary metric before the run. One metric, one slice definition, one threshold, written down. Everything else in the report is labelled exploratory. This single convention removes most multiplicity problems, because the decision now rests on a test that was specified before any data existed.

Replicate anything surprising on fresh data. A real regression in Hindi shows up again next week. A fluke does not. One replication is worth more than any correction, and it costs a rerun.

You do not need twenty tests to have the problem

The subtler version has no visible count. You look at the results, notice that long inputs seem worse, and test that. You ran one test, but you chose it after seeing the data, from among the dozens of slices you might have noticed. The error rate belongs to the set of things you could have chosen, not the one you did. This is why exploratory analysis is a source of hypotheses, and hypotheses need their own data.

The practical version: when a slice finding is found by looking, collect fresh items for that slice and re-measure. Frequently the effect halves, which is itself informative.

Guardrails go the other way

There is one deliberate exception. For metrics protecting against harm (safety violations, refusals, severe latency, error rates) you want a sensitive tripwire, and you accept more false alarms to get it. Do not apply a multiplicity correction to a guardrail. Set it loose, investigate every alert, and accept that most investigations find nothing. The asymmetry is intentional: an unnecessary investigation costs an afternoon, and a missed safety regression costs something you cannot buy back.

Write both policies into the report template, so nobody has to argue about it while looking at a fresh number: primary metric with its pre-registered threshold, exploratory slices with FDR control, guardrails uncorrected and always investigated.