Regression to the mean, and the winner's curse in model selection

The best of twenty is not the best

Train twenty models that are, in truth, identical: each has an accuracy of exactly 80 per cent on the population. Evaluate each on its own 200 items. The standard error is 2.8 points, so the twenty scores scatter around 80 with that spread. Pick the highest. Its expected value is not 80. The expected maximum of twenty draws from a bell curve is about 1.9 standard deviations above the mean, so the winner scores around

80 + 1.9 × 2.8 ≈ 85.3

Now evaluate that winner on a fresh 200 items. Expected score: 80. It "lost" five points. Nothing happened to the model. It was chosen because its noise was favourable, and fresh items come with fresh noise.

This is regression to the mean, and the mechanism is only this: any measurement is truth plus noise; selecting on the measurement selects partly on the noise; the next measurement draws new noise, which is on average zero. No force pulls the winner down. The luck simply does not repeat.

How much falls back

The size of the effect depends on how much of the measurement was noise. If the correlation between one measurement and a repeat is r, then something observed z standard deviations above the mean is expected to come back at r × z. With r = 0.5, a model at +2 comes back at +1. With r = 0.9, at +1.8. With r = 0, when the measurement was all noise, at exactly the mean. You can estimate r by measuring twice; it is the test-retest reliability, and it is the number that says how much of a leaderboard is real.

The inflation of a chosen maximum also depends on how many candidates there were. The expected maximum of k standard normal draws:

k =    10  →  +1.5 SD
k =    20  →  +1.9 SD
k =   100  →  +2.5 SD
k = 1,000  →  +3.2 SD

A hundred hyperparameter configurations, scored on a validation set with a standard error of 1 point, will produce a winner about 2.5 points better than it is. This is the winner's curse: the act of choosing the best guarantees that its score overstates it. The course machine-learning-foundations describes the validation set wearing out; this is the arithmetic of how fast.

Where it shows up

• Hyperparameter search. The best configuration's validation score is optimistic by roughly the table above. Report the fresh test score, not the validation score you selected on.
• Leaderboards. The top entry on any public benchmark is, on average, worse than its rank suggests, and the gap between first and fifth is often less than the noise. Re-evaluation on a private set nearly always compresses the top.
• "Our best red-teaming result." The attack that worked best on this model was the luckiest of many tried; expect it to work less well on the next.
• Interventions on the worst. Retrain on the examples the model got most wrong and the next measurement improves on them regardless of whether the retraining helped, because the worst examples were partly the unluckiest. Attribute the improvement carefully.
• Everywhere else. The sophomore slump, the sports team that regresses after a record season, the patient who improves after a treatment started at their worst point. Same mechanism.

Three fixes, all cheap

1. Measure the winner again on data it was not chosen on. A held-out test set exists for this reason and no other. Its number is unbiased because nothing was selected on it.
2. Shrink the estimate toward the mean. If you cannot re-measure, multiply the winner's excess over the average by r, or subtract the expected maximum from the table. A reported 91 from the best of 100 configurations, with a 1-point standard error, is honestly about 88.5.
3. Report how many were tried. "Best of 3" and "best of 300" are different claims, and a score without that count cannot be read.

Watch it, once

import numpy as np
rng = np.random.default_rng(0)
true_acc, n, k = 0.80, 200, 20
val  = rng.binomial(n, true_acc, size=k) / n     # 20 identical models, one val set each
best = val.argmax()
test = rng.binomial(n, true_acc) / n             # winner on fresh items
print(val[best], test)                           # about 0.85, then about 0.80

Run it a few times. The first number is reliably above the second, and no model was ever better than another.

What it is not

Regression to the mean is not overfitting in the model; the model never saw the validation items. It is not a hard test set; the fresh items were drawn the same way. It is not fraud, and it is not fixed by training longer. It is selection on noise, and it operates whenever anyone picks the best of anything by a measured score. The only defence is fresh data, and the only honest report includes the count of things you chose from.