Is that improvement real?

The situation

Model A scores 0.847. Model B scores 0.852. Do you ship B?

The wrong answer is “yes, it is higher”. Both numbers were measured on a finite sample, both carry uncertainty, and 0.005 may be well inside it.

The bootstrap, which is all you need

Resample your evaluation set with replacement, recompute the metric, repeat a thousand times. The spread of those thousand values is the sampling distribution of your score.

import numpy as np
rng = np.random.default_rng(0)
n = len(y_test)
diffs = []
for _ in range(2000):
    idx = rng.integers(0, n, n)
    diffs.append(roc_auc_score(y_test[idx], pB[idx])
                 - roc_auc_score(y_test[idx], pA[idx]))
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"difference {np.mean(diffs):.4f}  95% CI [{lo:.4f}, {hi:.4f}]")

One detail carries most of the value: resample the rows once and score both models on the same resampled rows. This is a paired comparison, and it removes the variance caused by some rows being harder. An unpaired comparison of two intervals is far less sensitive and will tell you “inconclusive” when the paired version would have given you a clear answer.

If the interval on the difference contains zero, you do not have evidence that B is better. That is not the same as evidence they are equal; it means your test set cannot resolve a difference this small.

McNemar’s test, for two classifiers on the same data

When comparing predicted labels rather than scores, the relevant information is only in the rows where the two models disagree.

• b = rows A got right and B got wrong.
• c = rows A got wrong and B got right.

Rows where both were right or both were wrong tell you nothing about which is better. Under the hypothesis that the models are equally good, b and c should be about equal, and McNemar’s test asks how surprising the observed imbalance is.

from statsmodels.stats.contingency_tables import mcnemar
mcnemar([[both_right, b], [c, both_wrong]], exact=True)

If B beat A on 60 rows and lost on 45, that is a 60-45 split out of 105 disagreements: nowhere near significant. If it is 60-20, it is. The absolute accuracy difference can look identical in both cases, which is why the disagreement counts are the thing to look at.

Cross-validation is not a free significance test

A tempting move: run 10-fold cross-validation on both models and run a t-test on the ten pairs of scores.

Do not. The folds share training data, so the ten differences are correlated, and the standard t-test assumes independence. Dietterich showed in 1998 that this procedure has a badly inflated false positive rate: it declares differences significant far more often than it should. Repeated cross-validation makes it worse, not better, because the extra repeats are even more correlated.

What to do instead: use a held-out test set and bootstrap it, which is what the first section does. Or report the fold-to-fold spread as a description of stability without attaching a p-value to it.

How large a difference can you even see?

Work it out before running the comparison. On a 2,000-row test set with accuracy near 85%, the standard error is about 0.8 points, and the standard error on a paired difference is smaller: often around 0.3 to 0.5 points depending on how correlated the models are.

So a 0.2-point improvement is not measurable on that set, no matter how many times you rerun it. Knowing this in advance saves you from a week of tuning aimed at a difference your evaluation cannot detect.

Practical significance

The last step is the one that gets skipped. Suppose the improvement is real: 0.847 to 0.852, statistically solid on a large test set.

Is it worth shipping? Model B is a stacked ensemble that takes four hours to train, 300ms to serve instead of 12ms, and cannot be explained to a regulator. The gain is 0.5% of AUC, which might translate to a handful of extra fraud cases caught per month.

Statistical significance is a statement about whether an effect exists. Whether it matters is a separate judgement involving latency, cost, maintenance and explainability, and the person who can hold both is considerably more useful than the one who can only compute the first.