{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"the-validation-set-wears-out","title":"A validation set you have used two hundred times","url":"https://addaly.com/learn/machine-learning-foundations/the-validation-set-wears-out","position":39,"of":92,"minutes":8,"answer":"The best of many noisy validation estimates is biased upward by roughly the standard error times the square root of twice the log of the number of comparisons.","keyIdea":"The best of many noisy validation estimates is biased upward by roughly the standard error times the square root of twice the log of the number of comparisons, so the count of experiments belongs in the report alongside the score."},"body":{"format":"markdown","text":"## Overfitting the validation set\n\nYou did everything right. You split off a validation set, you never trained on it, you used it only to compare models.\n\nAnd then you compared 340 models against it over six weeks.\n\nEach comparison used the validation set to make a decision. The final model was chosen because it scored highest, and part of why it scored highest is that its particular errors happened not to land on those particular rows. The validation set has been slowly incorporated into the model through your decisions, and its score is no longer an unbiased estimate of anything.\n\nThis is sometimes called the garden of forking paths, and it has a mathematical shape: **the maximum of many noisy estimates is biased upward**, and the bias grows with the number of estimates and shrinks with the size of the set.\n\n## Roughly how much\n\nA rule of thumb that is worth carrying. If the standard error of your validation score is `s` and you compare `k` equivalent models, the best of them will beat the true value by roughly `s` times the square root of `2 ln k`.\n\nWith a 2,000-row validation set, accuracy around 85%, the standard error is about 0.8 percentage points. Compare 100 models and the winner is inflated by roughly 2.4 points: enough to turn \"no real improvement\" into \"a clear win\". Compare 1,000 and it is about 3 points.\n\n> Figure: How far the winning score overstates the truth. Bars (points of inflation in the best score): 2 models compared 0.9; 10 models compared 1.7; 100 models compared 2.4; 1,000 models compared 3. A 2,000-row validation set, accuracy near 85 per cent, standard error about 0.8 points, models genuinely equivalent. The best of a hundred beats its true value by roughly 2.4 points: enough to turn no improvement into a clear win. Nothing was faked; noise was selected for.\n\nThat calculation explains a phenomenon everybody has seen: a long tuning session that produces a two-point gain which then fails to appear in production. Nothing was faked. The gain was measurement noise, selected for.\n\n## Four defences\n\n**1. Keep a test set that is untouched.** The test set is your protection against exactly this. It works only if you use it once, at the end. A test set consulted three times has become a slow validation set.\n\n**2. Make the validation set large enough that its noise is small.** If the standard error is 0.2 points, a hundred comparisons cost you 0.6 points instead of 2.4. Data spent on validation is not wasted; it buys resolution.\n\n**3. Rotate.** Refresh the validation set periodically from newly collected data, so that decisions made against the old one do not accumulate against the new one. This is normal practice on long-running production systems and it is worth setting up early.\n\n**4. Compare fewer things.** The bias grows with the number of comparisons, so a smaller, better-reasoned search is not just cheaper: it is more honest. This is the practical case against searching 5,000 configurations because the compute was available.\n\n## The competition evidence\n\nKaggle provides an unusually clean demonstration. Competitors tune against a public leaderboard computed on part of the test data, and the final ranking uses a separate private portion. Teams that tuned heavily against the public score routinely drop dozens of places when the private score is revealed, and the effect is largest in competitions with small test sets.\n\nA useful piece of context on the other side, though. Several careful studies have rebuilt fresh test sets for well-known benchmarks (a new CIFAR-10 test set, a new ImageNet test set) and found that while every model scored lower on the new data, the *ranking* of models was largely preserved. So years of community-wide tuning against a fixed benchmark inflated absolute numbers considerably and distorted relative comparisons much less than many expected. Both halves of that finding are worth carrying: the absolute number decays fast, the ordering is sturdier.\n\n## Writing it down\n\nThe practice that actually helps is unglamorous. Keep a log with one line per experiment: what you changed, the validation score, the date. At the end, count the lines.\n\nIf you ran 400 experiments and your final model beats your first by 1.5 points, you should say so in exactly those terms, because that sentence lets a reader apply the correction themselves. \"We improved AUC from 0.831 to 0.846 over 400 experiments on a 3,000-row validation set\" is an honest claim. \"We achieved 0.846 AUC\" is technically true and tells the reader nothing they need.\n\nThe number of times you looked is part of the result."},"check":{"question":"Two teams each report 0.846 validation AUC on the same 3,000-row set. Team A tried 6 configurations; Team B tried 900. What should you conclude?","options":[{"text":"Team B's model is more likely to be genuinely better, since it explored the space far more thoroughly.","correct":false,"tempting":"Wider search does raise the chance of finding a genuinely good configuration, but it raises the selection bias in the reported score at the same time, so the identical number does not favour B."},{"text":"Team A's number is more likely to hold up, because the same score selected from 900 candidates carries substantially more upward selection bias than one selected from 6.","correct":true},{"text":"The two are equivalent, since both used the same validation set and the same metric.","correct":false,"tempting":"Identical set and metric make the numbers comparable in form, but the number of selections made against that set changes what each number means."},{"text":"Neither can be assessed without knowing the training set sizes, which determine the variance of the estimates.","correct":false,"tempting":"Training size affects model quality, but the bias described here is governed by the validation set size and the number of comparisons, both of which are given."}]},"prev":{"slug":"hyperparameter-search","title":"Searching hyperparameters without fooling yourself","url":"https://addaly.com/learn/machine-learning-foundations/hyperparameter-search"},"next":{"slug":"how-much-data-is-enough","title":"How much data do I need?","url":"https://addaly.com/learn/machine-learning-foundations/how-much-data-is-enough"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A validation set you have used two hundred times', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/the-validation-set-wears-out (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A validation set you have used two hundred times","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"The best of many noisy validation estimates is biased upward by roughly the standard error times the square root of twice the log of the number of comparisons, so the count of experiments belongs in the report alongside the score.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"points of inflation in the best score","title":"How far the winning score overstates the truth","series":[{"label":"2 models compared","value":0.9},{"label":"10 models compared","value":1.7},{"label":"100 models compared","value":2.4,"highlight":true},{"label":"1,000 models compared","value":3}],"caption":"A 2,000-row validation set, accuracy near 85 per cent, standard error about 0.8 points, models genuinely equivalent. The best of a hundred beats its true value by roughly 2.4 points: enough to turn no improvement into a clear win. Nothing was faked; noise was selected for."},"alt":"Figure: How far the winning score overstates the truth. Bars (points of inflation in the best score): 2 models compared 0.9; 10 models compared 1.7; 100 models compared 2.4; 1,000 models compared 3. A 2,000-row validation set, accuracy near 85 per cent, standard error about 0.8 points, models genuinely equivalent. The best of a hundred beats its true value by roughly 2.4 points: enough to turn no improvement into a clear win. Nothing was faked; noise was selected for."}},"blocks":[{"t":"h2","id":"overfitting-the-validation-set","runs":[{"text":"Overfitting the validation set"}]},{"t":"p","runs":[{"text":"You did everything right. You split off a validation set, you never trained on it, you used it only to compare models."}]},{"t":"p","runs":[{"text":"And then you compared 340 models against it over six weeks."}]},{"t":"p","runs":[{"text":"Each comparison used the validation set to make a decision. The final model was chosen because it scored highest, and part of why it scored highest is that its particular errors happened not to land on those particular rows. The validation set has been slowly incorporated into the model through your decisions, and its score is no longer an unbiased estimate of anything."}]},{"t":"p","runs":[{"text":"This is sometimes called the garden of forking paths, and it has a mathematical shape: "},{"text":"the maximum of many noisy estimates is biased upward","b":true},{"text":", and the bias grows with the number of estimates and shrinks with the size of the set."}]},{"t":"h2","id":"roughly-how-much","runs":[{"text":"Roughly how much"}]},{"t":"p","runs":[{"text":"A rule of thumb that is worth carrying. If the standard error of your validation score is "},{"text":"s","code":true},{"text":" and you compare "},{"text":"k","code":true},{"text":" equivalent models, the best of them will beat the true value by roughly "},{"text":"s","code":true},{"text":" times the square root of "},{"text":"2 ln k","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"With a 2,000-row validation set, accuracy around 85%, the standard error is about 0.8 percentage points. Compare 100 models and the winner is inflated by roughly 2.4 points: enough to turn “no real improvement” into “a clear win”. Compare 1,000 and it is about 3 points."}]},{"t":"p","runs":[{"text":"That calculation explains a phenomenon everybody has seen: a long tuning session that produces a two-point gain which then fails to appear in production. Nothing was faked. The gain was measurement noise, selected for."}]},{"t":"h2","id":"four-defences","runs":[{"text":"Four defences"}]},{"t":"p","runs":[{"text":"1. Keep a test set that is untouched.","b":true},{"text":" The test set is your protection against exactly this. It works only if you use it once, at the end. A test set consulted three times has become a slow validation set."}]},{"t":"p","runs":[{"text":"2. Make the validation set large enough that its noise is small.","b":true},{"text":" If the standard error is 0.2 points, a hundred comparisons cost you 0.6 points instead of 2.4. Data spent on validation is not wasted; it buys resolution."}]},{"t":"p","runs":[{"text":"3. Rotate.","b":true},{"text":" Refresh the validation set periodically from newly collected data, so that decisions made against the old one do not accumulate against the new one. This is normal practice on long-running production systems and it is worth setting up early."}]},{"t":"p","runs":[{"text":"4. Compare fewer things.","b":true},{"text":" The bias grows with the number of comparisons, so a smaller, better-reasoned search is not just cheaper: it is more honest. This is the practical case against searching 5,000 configurations because the compute was available."}]},{"t":"h2","id":"the-competition-evidence","runs":[{"text":"The competition evidence"}]},{"t":"p","runs":[{"text":"Kaggle provides an unusually clean demonstration. Competitors tune against a public leaderboard computed on part of the test data, and the final ranking uses a separate private portion. Teams that tuned heavily against the public score routinely drop dozens of places when the private score is revealed, and the effect is largest in competitions with small test sets."}]},{"t":"p","runs":[{"text":"A useful piece of context on the other side, though. Several careful studies have rebuilt fresh test sets for well-known benchmarks (a new CIFAR-10 test set, a new ImageNet test set) and found that while every model scored lower on the new data, the "},{"text":"ranking","i":true},{"text":" of models was largely preserved. So years of community-wide tuning against a fixed benchmark inflated absolute numbers considerably and distorted relative comparisons much less than many expected. Both halves of that finding are worth carrying: the absolute number decays fast, the ordering is sturdier."}]},{"t":"h2","id":"writing-it-down","runs":[{"text":"Writing it down"}]},{"t":"p","runs":[{"text":"The practice that actually helps is unglamorous. Keep a log with one line per experiment: what you changed, the validation score, the date. At the end, count the lines."}]},{"t":"p","runs":[{"text":"If you ran 400 experiments and your final model beats your first by 1.5 points, you should say so in exactly those terms, because that sentence lets a reader apply the correction themselves. “We improved AUC from 0.831 to 0.846 over 400 experiments on a 3,000-row validation set” is an honest claim. “We achieved 0.846 AUC” is technically true and tells the reader nothing they need."}]},{"t":"p","runs":[{"text":"The number of times you looked is part of the result."}]}],"check":{"question":[{"text":"Two teams each report 0.846 validation AUC on the same 3,000-row set. Team A tried 6 configurations; Team B tried 900. What should you conclude?"}],"options":[{"text":[{"text":"Team B’s model is more likely to be genuinely better, since it explored the space far more thoroughly."}],"correct":false,"tempting":[{"text":"Wider search does raise the chance of finding a genuinely good configuration, but it raises the selection bias in the reported score at the same time, so the identical number does not favour B."}]},{"text":[{"text":"Team A’s number is more likely to hold up, because the same score selected from 900 candidates carries substantially more upward selection bias than one selected from 6."}],"correct":true,"tempting":null},{"text":[{"text":"The two are equivalent, since both used the same validation set and the same metric."}],"correct":false,"tempting":[{"text":"Identical set and metric make the numbers comparable in form, but the number of selections made against that set changes what each number means."}]},{"text":[{"text":"Neither can be assessed without knowing the training set sizes, which determine the variance of the estimates."}],"correct":false,"tempting":[{"text":"Training size affects model quality, but the bias described here is governed by the validation set size and the number of comparisons, both of which are given."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"hyperparameter-search","title":"Searching hyperparameters without fooling yourself","minutes":9},"next":{"slug":"how-much-data-is-enough","title":"How much data do I need?","minutes":8}}}