{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"not-looking","title":"Train, validation, test, and the discipline of not looking","url":"https://addaly.com/learn/machine-learning-foundations/not-looking","position":32,"of":92,"minutes":9,"answer":"The test set is a single-use measuring device; every peek turns it into another validation set.","keyIdea":"The test set is a single-use measuring device; every peek turns it into another validation set."},"body":{"format":"markdown","text":"## Three piles, three jobs\n\nSplit your data into three piles before you do anything else.\n\n- **Train**: the model's parameters are fitted here. Typically 60 to 80%.\n- **Validation**: everything *you* choose is chosen here: which model, how deep, which features, what threshold. Typically 10 to 20%.\n- **Test**: used once, at the end, to estimate how the thing will perform. Typically 10 to 20%.\n\nMost people understand train and test. The pile that gets misused is validation, and the misuse has a specific mechanism.\n\n## Why a second pile exists\n\nSuppose you try 40 configurations and score each on the same held-out data. You pick the best. That number is now optimistic, and not by a little.\n\nHere is the arithmetic. Take 1,000 held-out rows and 40 models that are all 80% accurate. Each measured score is 80% give or take about 1.3 points of sampling noise. Take the maximum of 40 such draws and you land around 82.5% on average. Two and a half points of pure luck, and it looks exactly like progress.\n\nSo the moment you use a set to *choose*, you have spent it. It can no longer *measure*. Validation is the pile you are allowed to spend. Test is the pile you are not.\n\nTreat the test set as a single-use instrument. Every extra look converts it into another validation set. If you evaluate on test, adjust something, and evaluate again, you no longer have a test set; you have a second validation set and no honest estimate.\n\n## Splitting wrongly is the common failure\n\nA random split assumes every row is an independent draw. Very often it is not.\n\n**Rows that share a person.** A no-show model has one row per appointment, and patients appear five or ten times. Split rows at random and the same patient sits on both sides. The model learns \"patient 40218 misses appointments\", which is easy and useless: patient 40218 is not who you will be predicting for. Split by patient instead: every appointment of a given patient goes wholly into train or wholly into validation.\n\n**Rows that share time.** With anything that evolves (prices, demand, fraud tactics) a random split lets the model see next month while predicting this month. Split by date: train on January to September, validate on October, test on November and December.\n\n**Rows that are near-duplicates.** Scraped datasets are full of the same item twice. Deduplicate before splitting, or the split is a formality.\n\nAsk what you are actually going to predict for. A new patient? Split by patient. A future week? Split by time. A new city? Hold out a city.\n\n## Preprocessing belongs inside the split\n\nA quiet leak: scaling a column by the mean and standard deviation of the whole dataset, then splitting. Validation rows contributed to that mean, so information crossed the line. The same applies to imputing missing values, fitting an encoder for categories, or selecting the top 20 features by correlation.\n\nCompute all of it on train only, then apply the stored values to validation and test. In scikit-learn that is what a `Pipeline` is for, and it is why fitting a scaler outside one is a recurring bug.\n\n## When data is scarce\n\nWith 800 rows, a 15% validation slice is 120 rows, which is too noisy to choose between models. Use **k-fold cross-validation**: split the training data into 5 parts, train five times, each time holding out a different part, and average the five scores. Every row gets used for validation exactly once and the estimate is far more stable.\n\nTwo cautions. Cross-validation replaces the validation pile, not the test pile: keep the test set out of the folds entirely. And it inherits whatever splitting mistake you made: if the folds are random rows and you needed groups, you have now made the same error five times and averaged it into something that looks solid.\n\n## Write the number down\n\nBefore the final test-set evaluation, write down what you expect and what result would make you ship. Then run it once.\n\nIf the number disappoints and you go back and tune, that is allowed, but say so out loud, and understand that your test estimate is now optimistic. This is not bureaucracy. Roughly every over-promised model comes from a team that quietly iterated against their test set and then reported the best number as if it were the first."},"check":{"question":"A clinic predicts whether a patient will miss an appointment. The data has one row per appointment, and most patients appear several times. They split rows at random, 80/20. Validation accuracy is 84%. After launch it behaves like 71%. What went wrong?","options":[{"text":"The same patients sit on both sides of the split, so the model was rewarded for recognising individuals rather than generalising to new ones.","correct":true},{"text":"The validation set was too small to give a stable estimate.","correct":false,"tempting":"Small samples really do produce unreliable scores, but sampling noise is symmetric: it would sometimes flatter and sometimes penalise, not deliver a consistent thirteen-point gift."},{"text":"Patient behaviour changed after launch, so the data drifted.","correct":false,"tempting":"Drift is real and common, and it is the first explanation most teams reach for, but this gap was built into the split before a single live prediction was made."},{"text":"They should have used cross-validation instead of a single split.","correct":false,"tempting":"Cross-validation is the more rigorous-sounding option, yet folds built from the same random row assignment repeat the identical mistake five times and average it."}]},"prev":{"slug":"imbalanced-classes","title":"When one class is 0.3% of the data","url":"https://addaly.com/learn/machine-learning-foundations/imbalanced-classes"},"next":{"slug":"overfitting-and-underfitting","title":"Overfitting, underfitting, and how to tell","url":"https://addaly.com/learn/machine-learning-foundations/overfitting-and-underfitting"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Train, validation, test, and the discipline of not looking', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/not-looking (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Train, validation, test, and the discipline of not looking","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"The test set is a single-use measuring device; every peek turns it into another validation set.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-piles-three-jobs","runs":[{"text":"Three piles, three jobs"}]},{"t":"p","runs":[{"text":"Split your data into three piles before you do anything else."}]},{"t":"ul","items":[[{"text":"Train","b":true},{"text":": the model’s parameters are fitted here. Typically 60 to 80%."}],[{"text":"Validation","b":true},{"text":": everything "},{"text":"you","i":true},{"text":" choose is chosen here: which model, how deep, which features, what threshold. Typically 10 to 20%."}],[{"text":"Test","b":true},{"text":": used once, at the end, to estimate how the thing will perform. Typically 10 to 20%."}]]},{"t":"p","runs":[{"text":"Most people understand train and test. The pile that gets misused is validation, and the misuse has a specific mechanism."}]},{"t":"h2","id":"why-a-second-pile-exists","runs":[{"text":"Why a second pile exists"}]},{"t":"p","runs":[{"text":"Suppose you try 40 configurations and score each on the same held-out data. You pick the best. That number is now optimistic, and not by a little."}]},{"t":"p","runs":[{"text":"Here is the arithmetic. Take 1,000 held-out rows and 40 models that are all 80% accurate. Each measured score is 80% give or take about 1.3 points of sampling noise. Take the maximum of 40 such draws and you land around 82.5% on average. Two and a half points of pure luck, and it looks exactly like progress."}]},{"t":"p","runs":[{"text":"So the moment you use a set to "},{"text":"choose","i":true},{"text":", you have spent it. It can no longer "},{"text":"measure","i":true},{"text":". Validation is the pile you are allowed to spend. Test is the pile you are not."}]},{"t":"p","runs":[{"text":"Treat the test set as a single-use instrument. Every extra look converts it into another validation set. If you evaluate on test, adjust something, and evaluate again, you no longer have a test set; you have a second validation set and no honest estimate."}]},{"t":"h2","id":"splitting-wrongly-is-the-common-failure","runs":[{"text":"Splitting wrongly is the common failure"}]},{"t":"p","runs":[{"text":"A random split assumes every row is an independent draw. Very often it is not."}]},{"t":"p","runs":[{"text":"Rows that share a person.","b":true},{"text":" A no-show model has one row per appointment, and patients appear five or ten times. Split rows at random and the same patient sits on both sides. The model learns “patient 40218 misses appointments”, which is easy and useless: patient 40218 is not who you will be predicting for. Split by patient instead: every appointment of a given patient goes wholly into train or wholly into validation."}]},{"t":"p","runs":[{"text":"Rows that share time.","b":true},{"text":" With anything that evolves (prices, demand, fraud tactics) a random split lets the model see next month while predicting this month. Split by date: train on January to September, validate on October, test on November and December."}]},{"t":"p","runs":[{"text":"Rows that are near-duplicates.","b":true},{"text":" Scraped datasets are full of the same item twice. Deduplicate before splitting, or the split is a formality."}]},{"t":"p","runs":[{"text":"Ask what you are actually going to predict for. A new patient? Split by patient. A future week? Split by time. A new city? Hold out a city."}]},{"t":"h2","id":"preprocessing-belongs-inside-the-split","runs":[{"text":"Preprocessing belongs inside the split"}]},{"t":"p","runs":[{"text":"A quiet leak: scaling a column by the mean and standard deviation of the whole dataset, then splitting. Validation rows contributed to that mean, so information crossed the line. The same applies to imputing missing values, fitting an encoder for categories, or selecting the top 20 features by correlation."}]},{"t":"p","runs":[{"text":"Compute all of it on train only, then apply the stored values to validation and test. In scikit-learn that is what a "},{"text":"Pipeline","code":true},{"text":" is for, and it is why fitting a scaler outside one is a recurring bug."}]},{"t":"h2","id":"when-data-is-scarce","runs":[{"text":"When data is scarce"}]},{"t":"p","runs":[{"text":"With 800 rows, a 15% validation slice is 120 rows, which is too noisy to choose between models. Use "},{"text":"k-fold cross-validation","b":true},{"text":": split the training data into 5 parts, train five times, each time holding out a different part, and average the five scores. Every row gets used for validation exactly once and the estimate is far more stable."}]},{"t":"p","runs":[{"text":"Two cautions. Cross-validation replaces the validation pile, not the test pile: keep the test set out of the folds entirely. And it inherits whatever splitting mistake you made: if the folds are random rows and you needed groups, you have now made the same error five times and averaged it into something that looks solid."}]},{"t":"h2","id":"write-the-number-down","runs":[{"text":"Write the number down"}]},{"t":"p","runs":[{"text":"Before the final test-set evaluation, write down what you expect and what result would make you ship. Then run it once."}]},{"t":"p","runs":[{"text":"If the number disappoints and you go back and tune, that is allowed, but say so out loud, and understand that your test estimate is now optimistic. This is not bureaucracy. Roughly every over-promised model comes from a team that quietly iterated against their test set and then reported the best number as if it were the first."}]}],"check":{"question":[{"text":"A clinic predicts whether a patient will miss an appointment. The data has one row per appointment, and most patients appear several times. They split rows at random, 80/20. Validation accuracy is 84%. After launch it behaves like 71%. What went wrong?"}],"options":[{"text":[{"text":"The same patients sit on both sides of the split, so the model was rewarded for recognising individuals rather than generalising to new ones."}],"correct":true,"tempting":null},{"text":[{"text":"The validation set was too small to give a stable estimate."}],"correct":false,"tempting":[{"text":"Small samples really do produce unreliable scores, but sampling noise is symmetric: it would sometimes flatter and sometimes penalise, not deliver a consistent thirteen-point gift."}]},{"text":[{"text":"Patient behaviour changed after launch, so the data drifted."}],"correct":false,"tempting":[{"text":"Drift is real and common, and it is the first explanation most teams reach for, but this gap was built into the split before a single live prediction was made."}]},{"text":[{"text":"They should have used cross-validation instead of a single split."}],"correct":false,"tempting":[{"text":"Cross-validation is the more rigorous-sounding option, yet folds built from the same random row assignment repeat the identical mistake five times and average it."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"imbalanced-classes","title":"When one class is 0.3% of the data","minutes":9},"next":{"slug":"overfitting-and-underfitting","title":"Overfitting, underfitting, and how to tell","minutes":8}}}