The bootstrap, and the three places it lies

What resampling actually assumes

The bootstrap treats your sample as a miniature of the population: draw from it with replacement, recompute the statistic, and the spread of those recomputations approximates the spread you would see across new samples.

That assumption is reasonable for a mean or a rate on a decent-sized independent sample, which is why the five-line bootstrap covers most of the intervals you will ever need. It fails in three specific ways, each with a mechanism you can state.

One: it cannot see past the largest value you have

Resampling draws from the values you already hold. The largest possible bootstrap maximum is your observed maximum, so an interval for a maximum is bounded by data rather than by the world.

The practical version of this is tail latency. Bootstrap p99 from 200 requests and the estimate is computed from the two slowest requests you happen to have. The interval will look tight and it is meaningless: the next 200 requests will contain a 9-second outlier your sample never saw.

Rules that follow:

• Do not bootstrap a maximum, a minimum, or a percentile beyond roughly 1 - 5/n. With 200 items, p95 is already shaky and p99 is not measurable.
• For tails, get more data or model the tail’s shape explicitly. Ten thousand requests give a p99 computed from a hundred observations, which is a real estimate.

Two: it inherits any dependence in your rows

Bootstrap rows drawn independently from a set whose rows are not independent produces intervals that are too narrow, sometimes by a factor of two. Five responses per user, or five runs of the same item, are not five independent observations. The next lesson deals with this properly; the rule here is that whatever the unit of independence is, that is the unit you resample.

# resample customers, take all of each customer's rows
groups = df.groupby("customer_id").indices
keys = list(groups)
draw = rng.choice(keys, size=len(keys), replace=True)
sample = df.iloc[np.concatenate([groups[k] for k in draw])]

Three: small samples and lumpy statistics

Below roughly 30 items the bootstrap’s coverage degrades: the 95% interval contains the truth noticeably less than 95% of the time, because a sample that small is a poor miniature of anything. And for statistics that move in jumps, such as a median over integer rubric scores, the bootstrap distribution becomes a few spikes rather than a smooth spread, so the percentiles land on arbitrary points.

For those cases prefer the mean of a per-item score over the median, or use an exact method, or state a range rather than an interval.

Percentile intervals and the better default

The simple approach (take the 2.5th and 97.5th percentiles of the bootstrap distribution) is fine when the statistic’s distribution is roughly symmetric. When it is skewed, which is normal for ratios and for metrics near 0 or 1, the interval is shifted. The bias-corrected and accelerated (BCa) interval corrects for that, and scipy does it for you:

from scipy.stats import bootstrap
res = bootstrap((scores,), np.mean, n_resamples=10_000,
                confidence_level=0.95, method="BCa", random_state=0)
print(res.confidence_interval)

Use 1,000 resamples while exploring and 10,000 for a number you will publish; the Monte Carlo noise in the interval’s endpoints shrinks with the square root of that count, and 1,000 leaves visible jitter in the third digit.

For comparisons, resample once

The most common mistake in a paired comparison is bootstrapping each system separately and comparing the two intervals. Overlapping intervals do not mean no difference, because the two systems’ errors are correlated: they were measured on the same items.

Resample the item indices once, score both systems on that resample, and record the difference. The interval you want is the interval of the difference. It is routinely half the width of what the two separate intervals suggest, and it is the correct one.

The habit worth keeping

Store per-item results, always. Every technique in this module (permutation, bootstrap, paired analysis, slicing) needs the vector of per-item outcomes and can do nothing with a summary. A harness that logs “84%” and discards the rows has thrown away every question you will want to ask next week.