{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"bootstrap-pitfalls","title":"The bootstrap, and the three places it lies","url":"https://addaly.com/learn/evaluating-ai/bootstrap-pitfalls","position":24,"of":76,"minutes":9,"answer":"Resampling can only ever redraw values you already observed, so it gives honest intervals for means and rates and badly overconfident ones for maxima, extreme percentiles and clustered data.","keyIdea":"Resampling can only ever redraw values you already observed, so it gives honest intervals for means and rates and badly overconfident ones for maxima, extreme percentiles and clustered data."},"body":{"format":"markdown","text":"## What resampling actually assumes\n\nThe bootstrap treats your sample as a miniature of the population: draw from it with replacement, recompute the statistic, and the spread of those recomputations approximates the spread you would see across new samples.\n\nThat assumption is reasonable for a mean or a rate on a decent-sized independent sample, which is why the five-line bootstrap covers most of the intervals you will ever need. It fails in three specific ways, each with a mechanism you can state.\n\n## One: it cannot see past the largest value you have\n\nResampling draws from the values you already hold. The largest possible bootstrap maximum is your observed maximum, so an interval for a maximum is bounded by data rather than by the world.\n\nThe practical version of this is tail latency. Bootstrap p99 from 200 requests and the estimate is computed from the two slowest requests you happen to have. The interval will look tight and it is meaningless: the next 200 requests will contain a 9-second outlier your sample never saw.\n\nRules that follow:\n\n- Do not bootstrap a maximum, a minimum, or a percentile beyond roughly `1 - 5/n`. With 200 items, p95 is already shaky and p99 is not measurable.\n- For tails, get more data or model the tail's shape explicitly. Ten thousand requests give a p99 computed from a hundred observations, which is a real estimate.\n\n## Two: it inherits any dependence in your rows\n\nBootstrap rows drawn independently from a set whose rows are not independent produces intervals that are too narrow, sometimes by a factor of two. Five responses per user, or five runs of the same item, are not five independent observations. The next lesson deals with this properly; the rule here is that whatever the unit of independence is, that is the unit you resample.\n\n```python\n# resample customers, take all of each customer's rows\ngroups = df.groupby(\"customer_id\").indices\nkeys = list(groups)\ndraw = rng.choice(keys, size=len(keys), replace=True)\nsample = df.iloc[np.concatenate([groups[k] for k in draw])]\n```\n\n## Three: small samples and lumpy statistics\n\nBelow roughly 30 items the bootstrap's coverage degrades: the 95% interval contains the truth noticeably less than 95% of the time, because a sample that small is a poor miniature of anything. And for statistics that move in jumps, such as a median over integer rubric scores, the bootstrap distribution becomes a few spikes rather than a smooth spread, so the percentiles land on arbitrary points.\n\nFor those cases prefer the mean of a per-item score over the median, or use an exact method, or state a range rather than an interval.\n\n## Percentile intervals and the better default\n\nThe simple approach (take the 2.5th and 97.5th percentiles of the bootstrap distribution) is fine when the statistic's distribution is roughly symmetric. When it is skewed, which is normal for ratios and for metrics near 0 or 1, the interval is shifted. The bias-corrected and accelerated (BCa) interval corrects for that, and `scipy` does it for you:\n\n```python\nfrom scipy.stats import bootstrap\nres = bootstrap((scores,), np.mean, n_resamples=10_000,\n                confidence_level=0.95, method=\"BCa\", random_state=0)\nprint(res.confidence_interval)\n```\n\nUse 1,000 resamples while exploring and 10,000 for a number you will publish; the Monte Carlo noise in the interval's endpoints shrinks with the square root of that count, and 1,000 leaves visible jitter in the third digit.\n\n## For comparisons, resample once\n\nThe most common mistake in a paired comparison is bootstrapping each system separately and comparing the two intervals. Overlapping intervals do not mean no difference, because the two systems' errors are correlated: they were measured on the same items.\n\nResample the item indices once, score both systems on that resample, and record the difference. The interval you want is the interval of the difference. It is routinely half the width of what the two separate intervals suggest, and it is the correct one.\n\n## The habit worth keeping\n\nStore per-item results, always. Every technique in this module (permutation, bootstrap, paired analysis, slicing) needs the vector of per-item outcomes and can do nothing with a summary. A harness that logs \"84%\" and discards the rows has thrown away every question you will want to ask next week."},"check":{"question":"A team bootstraps the 99th percentile of response latency from 300 logged requests and reports a 95% interval of 4.1 to 4.4 seconds. Why is that interval untrustworthy?","options":[{"text":"Latency is skewed, so a percentile interval is invalid and BCa should have been used instead.","correct":false,"tempting":"BCa corrects for skew in the bootstrap distribution, but it cannot conjure information about a tail that only three observations describe."},{"text":"p99 from 300 requests is computed from about three observations, and resampling can only redraw values already present, so the interval reflects those three points rather than the tail.","correct":true},{"text":"Three hundred requests is below the threshold where the bootstrap has adequate coverage.","correct":false,"tempting":"The general small-sample caution starts around 30 items; 300 is ample for a mean, and the specific problem here is the extremity of the statistic rather than the overall count."},{"text":"The bootstrap assumes the metric is normally distributed and latency is not.","correct":false,"tempting":"Attributes an assumption the bootstrap was designed to avoid; freedom from a distributional assumption is its main attraction."}]},"prev":{"slug":"how-many-items-do-you-need","title":"Sizing the set before you run it","url":"https://addaly.com/learn/evaluating-ai/how-many-items-do-you-need"},"next":{"slug":"clustered-and-repeated-data","title":"Five hundred rows, forty users","url":"https://addaly.com/learn/evaluating-ai/clustered-and-repeated-data"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The bootstrap, and the three places it lies', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/bootstrap-pitfalls (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The bootstrap, and the three places it lies","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"Resampling can only ever redraw values you already observed, so it gives honest intervals for means and rates and badly overconfident ones for maxima, extreme percentiles and clustered data.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-resampling-actually-assumes","runs":[{"text":"What resampling actually assumes"}]},{"t":"p","runs":[{"text":"The bootstrap treats your sample as a miniature of the population: draw from it with replacement, recompute the statistic, and the spread of those recomputations approximates the spread you would see across new samples."}]},{"t":"p","runs":[{"text":"That assumption is reasonable for a mean or a rate on a decent-sized independent sample, which is why the five-line bootstrap covers most of the intervals you will ever need. It fails in three specific ways, each with a mechanism you can state."}]},{"t":"h2","id":"one-it-cannot-see-past-the-largest-value-you-have","runs":[{"text":"One: it cannot see past the largest value you have"}]},{"t":"p","runs":[{"text":"Resampling draws from the values you already hold. The largest possible bootstrap maximum is your observed maximum, so an interval for a maximum is bounded by data rather than by the world."}]},{"t":"p","runs":[{"text":"The practical version of this is tail latency. Bootstrap p99 from 200 requests and the estimate is computed from the two slowest requests you happen to have. The interval will look tight and it is meaningless: the next 200 requests will contain a 9-second outlier your sample never saw."}]},{"t":"p","runs":[{"text":"Rules that follow:"}]},{"t":"ul","items":[[{"text":"Do not bootstrap a maximum, a minimum, or a percentile beyond roughly "},{"text":"1 - 5/n","code":true},{"text":". With 200 items, p95 is already shaky and p99 is not measurable."}],[{"text":"For tails, get more data or model the tail’s shape explicitly. Ten thousand requests give a p99 computed from a hundred observations, which is a real estimate."}]]},{"t":"h2","id":"two-it-inherits-any-dependence-in-your-rows","runs":[{"text":"Two: it inherits any dependence in your rows"}]},{"t":"p","runs":[{"text":"Bootstrap rows drawn independently from a set whose rows are not independent produces intervals that are too narrow, sometimes by a factor of two. Five responses per user, or five runs of the same item, are not five independent observations. The next lesson deals with this properly; the rule here is that whatever the unit of independence is, that is the unit you resample."}]},{"t":"code","lang":"python","label":"Python","text":"# resample customers, take all of each customer's rows\ngroups = df.groupby(\"customer_id\").indices\nkeys = list(groups)\ndraw = rng.choice(keys, size=len(keys), replace=True)\nsample = df.iloc[np.concatenate([groups[k] for k in draw])]"},{"t":"h2","id":"three-small-samples-and-lumpy-statistics","runs":[{"text":"Three: small samples and lumpy statistics"}]},{"t":"p","runs":[{"text":"Below roughly 30 items the bootstrap’s coverage degrades: the 95% interval contains the truth noticeably less than 95% of the time, because a sample that small is a poor miniature of anything. And for statistics that move in jumps, such as a median over integer rubric scores, the bootstrap distribution becomes a few spikes rather than a smooth spread, so the percentiles land on arbitrary points."}]},{"t":"p","runs":[{"text":"For those cases prefer the mean of a per-item score over the median, or use an exact method, or state a range rather than an interval."}]},{"t":"h2","id":"percentile-intervals-and-the-better-default","runs":[{"text":"Percentile intervals and the better default"}]},{"t":"p","runs":[{"text":"The simple approach (take the 2.5th and 97.5th percentiles of the bootstrap distribution) is fine when the statistic’s distribution is roughly symmetric. When it is skewed, which is normal for ratios and for metrics near 0 or 1, the interval is shifted. The bias-corrected and accelerated (BCa) interval corrects for that, and "},{"text":"scipy","code":true},{"text":" does it for you:"}]},{"t":"code","lang":"python","label":"Python","text":"from scipy.stats import bootstrap\nres = bootstrap((scores,), np.mean, n_resamples=10_000,\n                confidence_level=0.95, method=\"BCa\", random_state=0)\nprint(res.confidence_interval)"},{"t":"p","runs":[{"text":"Use 1,000 resamples while exploring and 10,000 for a number you will publish; the Monte Carlo noise in the interval’s endpoints shrinks with the square root of that count, and 1,000 leaves visible jitter in the third digit."}]},{"t":"h2","id":"for-comparisons-resample-once","runs":[{"text":"For comparisons, resample once"}]},{"t":"p","runs":[{"text":"The most common mistake in a paired comparison is bootstrapping each system separately and comparing the two intervals. Overlapping intervals do not mean no difference, because the two systems’ errors are correlated: they were measured on the same items."}]},{"t":"p","runs":[{"text":"Resample the item indices once, score both systems on that resample, and record the difference. The interval you want is the interval of the difference. It is routinely half the width of what the two separate intervals suggest, and it is the correct one."}]},{"t":"h2","id":"the-habit-worth-keeping","runs":[{"text":"The habit worth keeping"}]},{"t":"p","runs":[{"text":"Store per-item results, always. Every technique in this module (permutation, bootstrap, paired analysis, slicing) needs the vector of per-item outcomes and can do nothing with a summary. A harness that logs “84%” and discards the rows has thrown away every question you will want to ask next week."}]}],"check":{"question":[{"text":"A team bootstraps the 99th percentile of response latency from 300 logged requests and reports a 95% interval of 4.1 to 4.4 seconds. Why is that interval untrustworthy?"}],"options":[{"text":[{"text":"Latency is skewed, so a percentile interval is invalid and BCa should have been used instead."}],"correct":false,"tempting":[{"text":"BCa corrects for skew in the bootstrap distribution, but it cannot conjure information about a tail that only three observations describe."}]},{"text":[{"text":"p99 from 300 requests is computed from about three observations, and resampling can only redraw values already present, so the interval reflects those three points rather than the tail."}],"correct":true,"tempting":null},{"text":[{"text":"Three hundred requests is below the threshold where the bootstrap has adequate coverage."}],"correct":false,"tempting":[{"text":"The general small-sample caution starts around 30 items; 300 is ample for a mean, and the specific problem here is the extremity of the statistic rather than the overall count."}]},{"text":[{"text":"The bootstrap assumes the metric is normally distributed and latency is not."}],"correct":false,"tempting":[{"text":"Attributes an assumption the bootstrap was designed to avoid; freedom from a distributional assumption is its main attraction."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"how-many-items-do-you-need","title":"Sizing the set before you run it","minutes":9},"next":{"slug":"clustered-and-repeated-data","title":"Five hundred rows, forty users","minutes":9}}}