{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"cross-validation","title":"Cross-validation: using every row twice, legally","url":"https://addaly.com/learn/machine-learning-foundations/cross-validation","position":34,"of":92,"minutes":9,"answer":"Cross-validation averages several held-out estimates so a score depends less on where the split fell.","keyIdea":"Cross-validation averages several held-out estimates so a score depends less on where the split fell, but the best score from a hyperparameter search is biased upward and needs an outer loop or an untouched test set to correct."},"body":{"format":"markdown","text":"## The problem with one split\n\nHold out 20% and score on it. Now do it again with a different random seed. The two scores differ: sometimes by half a point, sometimes by four, depending on how much data you have.\n\nWith 500 rows, a 20% test set is 100 rows. If the model gets 83 right you report 83%, and the 95% interval around that runs from roughly 74% to 90%. That range is wide enough to cover \"ship it\" and \"abandon it\". A single split is a measurement with a large and usually unreported error bar.\n\nCross-validation reduces that error by measuring more than once.\n\n## The mechanism\n\nSplit the data into k parts. Train on k-1 of them, score on the one left out. Repeat until every part has served as the held-out fold once. You get k scores; report their mean and their standard deviation.\n\n```python\nfrom sklearn.model_selection import cross_val_score\nimport numpy as np\ns = cross_val_score(pipe, X_train, y_train, cv=5, scoring=\"roc_auc\")\nprint(f\"{s.mean():.3f} +/- {s.std():.3f}\")\n```\n\nEvery row is used for training in k-1 of the runs and for evaluation in exactly one. Nothing is wasted, and the mean of five scores has less variance than any one of them.\n\nThe cost is that you fit the model k times. For a linear model on 50,000 rows that is seconds. For a large network it is a real decision, which is why deep learning usually uses a single large validation set instead.\n\n## Choosing k\n\n**k = 5** is the default and the right answer most of the time. Each fold trains on 80% of the data, which is close enough to the full set that the estimate is not badly pessimistic.\n\n**k = 10** gives a slightly less biased estimate at twice the cost. Worth it on small datasets where 80% is meaningfully less than 100%.\n\n**Leave-one-out** (k = n) trains on everything but a single row, n times. The estimate has very low bias and famously high variance, and it costs n fits. It is occasionally right for very small datasets and is usually a sign that somebody has more compute than data.\n\nThere is a trade-off in k that is worth stating rather than absorbing: larger k means each model trains on more data, so the estimate is less pessimistic, but the k training sets overlap almost entirely, so the k scores are highly correlated and averaging them removes less variance than it appears to.\n\n## Stratify, always, for classification\n\n`StratifiedKFold` makes each fold contain the same proportion of each class as the whole set. This is the default for classifiers in scikit-learn's `cross_val_score`, and you should keep it.\n\nWithout it, on a 2% positive rate with 5 folds and 1,000 rows, a fold might get 4 positives and another 12. The scores swing wildly for reasons that have nothing to do with the model, and at extreme imbalance a fold can end up with zero positives, at which point some metrics are undefined and others silently return zero.\n\n## Repeated cross-validation\n\nIf you have the compute and the dataset is small, run the whole k-fold procedure several times with different random partitions:\n\n```python\nfrom sklearn.model_selection import RepeatedStratifiedKFold\ncv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)\n```\n\nFifteen fits, and the variability across repeats tells you how much of your score depends on where the fold boundaries happened to fall. If that variability is comparable to the difference between two models you are comparing, you do not have evidence to prefer one.\n\n## What the standard deviation is telling you\n\nReport it. \"0.847 +/- 0.004\" and \"0.847 +/- 0.061\" are completely different situations, and quoting only the mean hides which one you are in.\n\nOne warning about that number: the standard deviation across folds is not a proper confidence interval on the score. The folds share most of their training data, so the scores are correlated, and the usual formula for a standard error understates the true uncertainty, sometimes badly. Treat it as a rough indication of stability rather than a statistic you can put in a hypothesis test. Module 5 covers comparing two models properly.\n\n## The nested problem\n\nHere is the trap that catches people who have understood everything so far.\n\nUse cross-validation to choose hyperparameters, then report the best cross-validated score as your estimate of performance. That number is optimistic, because you selected the maximum of many noisy estimates, and the maximum of noisy things is biased upward. Try 200 configurations and the best one is partly good and partly lucky.\n\nThe fix is nested cross-validation: an inner loop that chooses hyperparameters and an outer loop that measures the whole procedure including the choosing.\n\n```python\nfrom sklearn.model_selection import GridSearchCV, cross_val_score\ninner = GridSearchCV(pipe, grid, cv=5)\nscores = cross_val_score(inner, X, y, cv=5)   # 5 x 5 = 25 model fits\n```\n\nExpensive, and honest. In practice most teams use a simpler equivalent: tune with cross-validation on the training set, and keep a untouched test set for the final number. That is cheaper and gives you the same protection, as long as the test set is touched once."},"check":{"question":"A team runs GridSearchCV over 300 configurations with cv=5 and reports the best mean cross-validated AUC of 0.882 as their expected production performance. Why is that number optimistic?","options":[{"text":"Selecting the maximum over 300 noisy estimates favours configurations that happened to suit these particular folds, so the winner's score includes some luck that will not repeat.","correct":true},{"text":"Five folds is too few to estimate AUC reliably, and ten folds would have given the true value.","correct":false,"tempting":"More folds reduce variance slightly but do nothing about the selection bias introduced by taking a maximum over many candidates."},{"text":"GridSearchCV refits on the full training set at the end, and the refitted model differs from the ones that were scored.","correct":false,"tempting":"Refitting on all training data usually helps rather than hurts, and it is not the source of the upward bias in the reported score."},{"text":"Cross-validated AUC always overstates production AUC because production data is drawn from a later period.","correct":false,"tempting":"Temporal shift is a real separate hazard, but the bias described here would exist even if production data were drawn from exactly the same distribution."}]},"prev":{"slug":"overfitting-and-underfitting","title":"Overfitting, underfitting, and how to tell","url":"https://addaly.com/learn/machine-learning-foundations/overfitting-and-underfitting"},"next":{"slug":"splits-that-respect-structure","title":"Splits that respect the structure of the data","url":"https://addaly.com/learn/machine-learning-foundations/splits-that-respect-structure"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Cross-validation: using every row twice, legally', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/cross-validation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Cross-validation: using every row twice, legally","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"Cross-validation averages several held-out estimates so a score depends less on where the split fell, but the best score from a hyperparameter search is biased upward and needs an outer loop or an untouched test set to correct.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-problem-with-one-split","runs":[{"text":"The problem with one split"}]},{"t":"p","runs":[{"text":"Hold out 20% and score on it. Now do it again with a different random seed. The two scores differ: sometimes by half a point, sometimes by four, depending on how much data you have."}]},{"t":"p","runs":[{"text":"With 500 rows, a 20% test set is 100 rows. If the model gets 83 right you report 83%, and the 95% interval around that runs from roughly 74% to 90%. That range is wide enough to cover “ship it” and “abandon it”. A single split is a measurement with a large and usually unreported error bar."}]},{"t":"p","runs":[{"text":"Cross-validation reduces that error by measuring more than once."}]},{"t":"h2","id":"the-mechanism","runs":[{"text":"The mechanism"}]},{"t":"p","runs":[{"text":"Split the data into k parts. Train on k-1 of them, score on the one left out. Repeat until every part has served as the held-out fold once. You get k scores; report their mean and their standard deviation."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import cross_val_score\nimport numpy as np\ns = cross_val_score(pipe, X_train, y_train, cv=5, scoring=\"roc_auc\")\nprint(f\"{s.mean():.3f} +/- {s.std():.3f}\")"},{"t":"p","runs":[{"text":"Every row is used for training in k-1 of the runs and for evaluation in exactly one. Nothing is wasted, and the mean of five scores has less variance than any one of them."}]},{"t":"p","runs":[{"text":"The cost is that you fit the model k times. For a linear model on 50,000 rows that is seconds. For a large network it is a real decision, which is why deep learning usually uses a single large validation set instead."}]},{"t":"h2","id":"choosing-k","runs":[{"text":"Choosing k"}]},{"t":"p","runs":[{"text":"k = 5","b":true},{"text":" is the default and the right answer most of the time. Each fold trains on 80% of the data, which is close enough to the full set that the estimate is not badly pessimistic."}]},{"t":"p","runs":[{"text":"k = 10","b":true},{"text":" gives a slightly less biased estimate at twice the cost. Worth it on small datasets where 80% is meaningfully less than 100%."}]},{"t":"p","runs":[{"text":"Leave-one-out","b":true},{"text":" (k = n) trains on everything but a single row, n times. The estimate has very low bias and famously high variance, and it costs n fits. It is occasionally right for very small datasets and is usually a sign that somebody has more compute than data."}]},{"t":"p","runs":[{"text":"There is a trade-off in k that is worth stating rather than absorbing: larger k means each model trains on more data, so the estimate is less pessimistic, but the k training sets overlap almost entirely, so the k scores are highly correlated and averaging them removes less variance than it appears to."}]},{"t":"h2","id":"stratify-always-for-classification","runs":[{"text":"Stratify, always, for classification"}]},{"t":"p","runs":[{"text":"StratifiedKFold","code":true},{"text":" makes each fold contain the same proportion of each class as the whole set. This is the default for classifiers in scikit-learn’s "},{"text":"cross_val_score","code":true},{"text":", and you should keep it."}]},{"t":"p","runs":[{"text":"Without it, on a 2% positive rate with 5 folds and 1,000 rows, a fold might get 4 positives and another 12. The scores swing wildly for reasons that have nothing to do with the model, and at extreme imbalance a fold can end up with zero positives, at which point some metrics are undefined and others silently return zero."}]},{"t":"h2","id":"repeated-cross-validation","runs":[{"text":"Repeated cross-validation"}]},{"t":"p","runs":[{"text":"If you have the compute and the dataset is small, run the whole k-fold procedure several times with different random partitions:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import RepeatedStratifiedKFold\ncv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)"},{"t":"p","runs":[{"text":"Fifteen fits, and the variability across repeats tells you how much of your score depends on where the fold boundaries happened to fall. If that variability is comparable to the difference between two models you are comparing, you do not have evidence to prefer one."}]},{"t":"h2","id":"what-the-standard-deviation-is-telling-you","runs":[{"text":"What the standard deviation is telling you"}]},{"t":"p","runs":[{"text":"Report it. “0.847 +/- 0.004” and “0.847 +/- 0.061” are completely different situations, and quoting only the mean hides which one you are in."}]},{"t":"p","runs":[{"text":"One warning about that number: the standard deviation across folds is not a proper confidence interval on the score. The folds share most of their training data, so the scores are correlated, and the usual formula for a standard error understates the true uncertainty, sometimes badly. Treat it as a rough indication of stability rather than a statistic you can put in a hypothesis test. Module 5 covers comparing two models properly."}]},{"t":"h2","id":"the-nested-problem","runs":[{"text":"The nested problem"}]},{"t":"p","runs":[{"text":"Here is the trap that catches people who have understood everything so far."}]},{"t":"p","runs":[{"text":"Use cross-validation to choose hyperparameters, then report the best cross-validated score as your estimate of performance. That number is optimistic, because you selected the maximum of many noisy estimates, and the maximum of noisy things is biased upward. Try 200 configurations and the best one is partly good and partly lucky."}]},{"t":"p","runs":[{"text":"The fix is nested cross-validation: an inner loop that chooses hyperparameters and an outer loop that measures the whole procedure including the choosing."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import GridSearchCV, cross_val_score\ninner = GridSearchCV(pipe, grid, cv=5)\nscores = cross_val_score(inner, X, y, cv=5)   # 5 x 5 = 25 model fits"},{"t":"p","runs":[{"text":"Expensive, and honest. In practice most teams use a simpler equivalent: tune with cross-validation on the training set, and keep a untouched test set for the final number. That is cheaper and gives you the same protection, as long as the test set is touched once."}]}],"check":{"question":[{"text":"A team runs GridSearchCV over 300 configurations with cv=5 and reports the best mean cross-validated AUC of 0.882 as their expected production performance. Why is that number optimistic?"}],"options":[{"text":[{"text":"Selecting the maximum over 300 noisy estimates favours configurations that happened to suit these particular folds, so the winner’s score includes some luck that will not repeat."}],"correct":true,"tempting":null},{"text":[{"text":"Five folds is too few to estimate AUC reliably, and ten folds would have given the true value."}],"correct":false,"tempting":[{"text":"More folds reduce variance slightly but do nothing about the selection bias introduced by taking a maximum over many candidates."}]},{"text":[{"text":"GridSearchCV refits on the full training set at the end, and the refitted model differs from the ones that were scored."}],"correct":false,"tempting":[{"text":"Refitting on all training data usually helps rather than hurts, and it is not the source of the upward bias in the reported score."}]},{"text":[{"text":"Cross-validated AUC always overstates production AUC because production data is drawn from a later period."}],"correct":false,"tempting":[{"text":"Temporal shift is a real separate hazard, but the bias described here would exist even if production data were drawn from exactly the same distribution."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"overfitting-and-underfitting","title":"Overfitting, underfitting, and how to tell","minutes":8},"next":{"slug":"splits-that-respect-structure","title":"Splits that respect the structure of the data","minutes":9}}}