{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"leakage-in-detail","title":"Leakage: a catalogue of the ways the answer gets in","url":"https://addaly.com/learn/machine-learning-foundations/leakage-in-detail","position":28,"of":92,"minutes":10,"answer":"Leakage is any feature whose training value would not be available in that form at prediction time, and the reliable test is to reconstruct a row using only information timestamped before the moment of prediction.","keyIdea":"Leakage is any feature whose training value would not be available in that form at prediction time, and the reliable test is to reconstruct a row using only information timestamped before the moment of prediction."},"body":{"format":"markdown","text":"## The definition that covers every case\n\nLeakage is any information in your training features that will not be available, in that form, at the moment you have to make a real prediction.\n\nIt is the most expensive error in applied machine learning because it does not look like an error. It looks like success. Your score goes up, your cross-validation agrees, your colleagues are impressed, and the model fails the day it meets production.\n\nThe signature is worth memorising: **a result substantially better than you expected, arriving with no effort.** Every experienced practitioner has learned to distrust that feeling, and the check takes fifteen minutes.\n\n## Target leakage: the column that knows\n\nA feature that is a consequence of the target rather than a cause.\n\n- Predicting churn with `cancellation_reason`. Only cancelled customers have one.\n- Predicting hospital readmission with `discharge_destination`, when \"transferred to another hospital\" is recorded as part of the readmission process.\n- Predicting fraud with `investigation_opened`. Investigations open after fraud is suspected.\n- Predicting default with `current_balance`, refreshed nightly and therefore reflecting today rather than the application date.\n\nThe last one is the dangerous shape, because the column is legitimate: balance at application time would be a fine feature. What leaks is the *timestamp* of the value, not the concept. Any feature stored in a table that gets overwritten is suspect for exactly this reason, and it is why event-sourced data with valid-from timestamps is worth so much to a data science team.\n\n> Figure: Where each column sits against the moment of prediction. Timeline: Day 0, Application submitted. The moment of prediction: every feature must exist now, with the value it has now.; Day 0, later, Loan disbursed. The clock on the target starts.; Every night, current_balance is overwritten. The training table holds today's balance, not day zero's: the concept is fine, the timestamp leaks.; Day 90+, Ninety days past due is recorded. This is the target.; Afterwards, investigation_opened, cancellation_reason. These are consequences of the outcome, not causes, and only rows with the outcome have them.. Only values fixed at day zero, as they were on day zero, are features. current_balance is a legitimate concept with a leaking timestamp; investigation_opened is a consequence of the target. The test for every column is the same question: what is its value at the moment of prediction, for a row whose outcome is not yet known?\n\n**The test:** for each feature, ask *what is its value at the moment of prediction, for a row whose outcome is not yet known?* Say it out loud for the ten most important features. It takes fifteen minutes and it is the highest-return quarter hour in this course.\n\n## Train-test contamination: fitting on everything\n\nAny transformation that learns from the data must learn from training rows only. The list of things that learn:\n\n- Scalers (mean, standard deviation)\n- Imputers (median, mode)\n- Target encoders (per-category means)\n- Feature selection (which columns to keep)\n- PCA (the components)\n- Outlier clipping (the percentiles)\n- Resampling such as SMOTE (which synthesises rows from neighbours)\n\nRun any of these on the full dataset before splitting and information from the held-out rows has entered the training process. The effect is usually small (one or two points) which is worse than a large effect, because it survives review and quietly makes every comparison you run slightly wrong.\n\nThe cleanest defence is structural: put every one of them inside a `Pipeline`, and let cross-validation refit the whole pipeline on each fold. The next lesson is entirely about that.\n\n## Group leakage: the same entity on both sides\n\nOne patient with 40 visits. One customer with 11 loans. One user with 300 sessions. One document photographed 6 times.\n\nA random row split scatters these across train and test, and the model recognises the entity rather than learning the pattern. The score is inflated by an amount that depends on how much of the signal is entity-specific, which can be most of it.\n\nSplit by group, never by row, whenever rows share an entity:\n\n```python\nfrom sklearn.model_selection import GroupKFold\ncv = GroupKFold(n_splits=5)\ncv.split(X, y, groups=df[\"patient_id\"])\n```\n\n## Temporal leakage: training on the future\n\nA random split lets the model train on November and December while being tested on the intervening period. In production it will only ever have the past.\n\nThis matters even without explicit date features, because data has structure in time that a model absorbs implicitly: a product launch, a policy change, a season. Use a time-ordered split for anything with a time axis, which is nearly everything transactional.\n\n## Duplicate rows: the quietest one\n\nExact or near-duplicate rows split across train and test give the model the answer directly. Scraped datasets, merged exports and image sets are full of them, often at 5 to 30% of rows.\n\nCheck before splitting:\n\n```python\ndf.duplicated().sum()\ndf.duplicated(subset=key_columns).sum()\n```\n\nFor images and text, near-duplicates need fuzzy matching: perceptual hashes for images, shingling or MinHash for text. Several famous published benchmarks have had their results revised downward after duplicate overlap between splits was found, so this is not a beginner's problem.\n\n## How to find leakage you have not thought of\n\nThree practical moves, in the order they cost you time:\n\n1. **Look at the top features.** If one feature dominates a model of a hard problem, investigate it specifically. This finds most cases.\n2. **Ablate.** Drop the suspect feature and refit. If the score falls from 0.97 to 0.79, you have learned that essentially all the performance came from it, which is worth knowing either way.\n3. **Rebuild the row as of the prediction moment.** For one or two rows, reconstruct every feature value using only data with a timestamp before the prediction time. Compare against the values in your training table. Any difference is leakage, and this is the only method that finds the ones you did not think to suspect.\n\nThe third is tedious and is what separates a model that works from one that demos."},"check":{"question":"A loan default model uses `current_account_balance`, pulled from a table that is overwritten nightly. The feature is legitimate in principle. Why is it leaking?","options":[{"text":"Account balance is correlated with the target, and highly correlated features always indicate leakage.","correct":false,"tempting":"Correlation with the target is what a good feature has; it is the timing of the value, not its predictiveness, that constitutes leakage."},{"text":"The table is overwritten, so the stored value reflects the balance today, after the default, rather than the balance at application time, which is what production would supply.","correct":true},{"text":"Balance is a continuous feature and needs scaling before it can be used with a penalised model.","correct":false,"tempting":"A preprocessing requirement unrelated to whether the value is knowable at prediction time."},{"text":"Balances vary seasonally, so training on them introduces temporal bias that a random split cannot detect.","correct":false,"tempting":"Seasonality is a genuine modelling concern, but the defect here would remain under any split because the recorded value is from the wrong point in time."}]},"prev":{"slug":"text-as-numbers","title":"Text as columns: counting words before you embed them","url":"https://addaly.com/learn/machine-learning-foundations/text-as-numbers"},"next":{"slug":"the-fit-transform-contract","title":"Fit on train, transform everywhere: the contract that keeps you honest","url":"https://addaly.com/learn/machine-learning-foundations/the-fit-transform-contract"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Leakage: a catalogue of the ways the answer gets in', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/leakage-in-detail (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Leakage: a catalogue of the ways the answer gets in","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"Leakage is any feature whose training value would not be available in that form at prediction time, and the reliable test is to reconstruct a row using only information timestamped before the moment of prediction.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"Where each column sits against the moment of prediction","events":[{"when":"Day 0","what":"Application submitted. The moment of prediction: every feature must exist now, with the value it has now."},{"when":"Day 0, later","what":"Loan disbursed. The clock on the target starts."},{"when":"Every night","what":"current_balance is overwritten. The training table holds today’s balance, not day zero’s: the concept is fine, the timestamp leaks."},{"when":"Day 90+","what":"Ninety days past due is recorded. This is the target."},{"when":"Afterwards","what":"investigation_opened, cancellation_reason. These are consequences of the outcome, not causes, and only rows with the outcome have them."}],"caption":"Only values fixed at day zero, as they were on day zero, are features. current_balance is a legitimate concept with a leaking timestamp; investigation_opened is a consequence of the target. The test for every column is the same question: what is its value at the moment of prediction, for a row whose outcome is not yet known?"},"alt":"Figure: Where each column sits against the moment of prediction. Timeline: Day 0, Application submitted. The moment of prediction: every feature must exist now, with the value it has now.; Day 0, later, Loan disbursed. The clock on the target starts.; Every night, current_balance is overwritten. The training table holds today’s balance, not day zero’s: the concept is fine, the timestamp leaks.; Day 90+, Ninety days past due is recorded. This is the target.; Afterwards, investigation_opened, cancellation_reason. These are consequences of the outcome, not causes, and only rows with the outcome have them.. Only values fixed at day zero, as they were on day zero, are features. current_balance is a legitimate concept with a leaking timestamp; investigation_opened is a consequence of the target. The test for every column is the same question: what is its value at the moment of prediction, for a row whose outcome is not yet known?"}},"blocks":[{"t":"h2","id":"the-definition-that-covers-every-case","runs":[{"text":"The definition that covers every case"}]},{"t":"p","runs":[{"text":"Leakage is any information in your training features that will not be available, in that form, at the moment you have to make a real prediction."}]},{"t":"p","runs":[{"text":"It is the most expensive error in applied machine learning because it does not look like an error. It looks like success. Your score goes up, your cross-validation agrees, your colleagues are impressed, and the model fails the day it meets production."}]},{"t":"p","runs":[{"text":"The signature is worth memorising: "},{"text":"a result substantially better than you expected, arriving with no effort.","b":true},{"text":" Every experienced practitioner has learned to distrust that feeling, and the check takes fifteen minutes."}]},{"t":"h2","id":"target-leakage-the-column-that-knows","runs":[{"text":"Target leakage: the column that knows"}]},{"t":"p","runs":[{"text":"A feature that is a consequence of the target rather than a cause."}]},{"t":"ul","items":[[{"text":"Predicting churn with "},{"text":"cancellation_reason","code":true},{"text":". Only cancelled customers have one."}],[{"text":"Predicting hospital readmission with "},{"text":"discharge_destination","code":true},{"text":", when “transferred to another hospital” is recorded as part of the readmission process."}],[{"text":"Predicting fraud with "},{"text":"investigation_opened","code":true},{"text":". Investigations open after fraud is suspected."}],[{"text":"Predicting default with "},{"text":"current_balance","code":true},{"text":", refreshed nightly and therefore reflecting today rather than the application date."}]]},{"t":"p","runs":[{"text":"The last one is the dangerous shape, because the column is legitimate: balance at application time would be a fine feature. What leaks is the "},{"text":"timestamp","i":true},{"text":" of the value, not the concept. Any feature stored in a table that gets overwritten is suspect for exactly this reason, and it is why event-sourced data with valid-from timestamps is worth so much to a data science team."}]},{"t":"p","runs":[{"text":"The test:","b":true},{"text":" for each feature, ask "},{"text":"what is its value at the moment of prediction, for a row whose outcome is not yet known?","i":true},{"text":" Say it out loud for the ten most important features. It takes fifteen minutes and it is the highest-return quarter hour in this course."}]},{"t":"h2","id":"train-test-contamination-fitting-on-everything","runs":[{"text":"Train-test contamination: fitting on everything"}]},{"t":"p","runs":[{"text":"Any transformation that learns from the data must learn from training rows only. The list of things that learn:"}]},{"t":"ul","items":[[{"text":"Scalers (mean, standard deviation)"}],[{"text":"Imputers (median, mode)"}],[{"text":"Target encoders (per-category means)"}],[{"text":"Feature selection (which columns to keep)"}],[{"text":"PCA (the components)"}],[{"text":"Outlier clipping (the percentiles)"}],[{"text":"Resampling such as SMOTE (which synthesises rows from neighbours)"}]]},{"t":"p","runs":[{"text":"Run any of these on the full dataset before splitting and information from the held-out rows has entered the training process. The effect is usually small (one or two points) which is worse than a large effect, because it survives review and quietly makes every comparison you run slightly wrong."}]},{"t":"p","runs":[{"text":"The cleanest defence is structural: put every one of them inside a "},{"text":"Pipeline","code":true},{"text":", and let cross-validation refit the whole pipeline on each fold. The next lesson is entirely about that."}]},{"t":"h2","id":"group-leakage-the-same-entity-on-both-sides","runs":[{"text":"Group leakage: the same entity on both sides"}]},{"t":"p","runs":[{"text":"One patient with 40 visits. One customer with 11 loans. One user with 300 sessions. One document photographed 6 times."}]},{"t":"p","runs":[{"text":"A random row split scatters these across train and test, and the model recognises the entity rather than learning the pattern. The score is inflated by an amount that depends on how much of the signal is entity-specific, which can be most of it."}]},{"t":"p","runs":[{"text":"Split by group, never by row, whenever rows share an entity:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import GroupKFold\ncv = GroupKFold(n_splits=5)\ncv.split(X, y, groups=df[\"patient_id\"])"},{"t":"h2","id":"temporal-leakage-training-on-the-future","runs":[{"text":"Temporal leakage: training on the future"}]},{"t":"p","runs":[{"text":"A random split lets the model train on November and December while being tested on the intervening period. In production it will only ever have the past."}]},{"t":"p","runs":[{"text":"This matters even without explicit date features, because data has structure in time that a model absorbs implicitly: a product launch, a policy change, a season. Use a time-ordered split for anything with a time axis, which is nearly everything transactional."}]},{"t":"h2","id":"duplicate-rows-the-quietest-one","runs":[{"text":"Duplicate rows: the quietest one"}]},{"t":"p","runs":[{"text":"Exact or near-duplicate rows split across train and test give the model the answer directly. Scraped datasets, merged exports and image sets are full of them, often at 5 to 30% of rows."}]},{"t":"p","runs":[{"text":"Check before splitting:"}]},{"t":"code","lang":"python","label":"Python","text":"df.duplicated().sum()\ndf.duplicated(subset=key_columns).sum()"},{"t":"p","runs":[{"text":"For images and text, near-duplicates need fuzzy matching: perceptual hashes for images, shingling or MinHash for text. Several famous published benchmarks have had their results revised downward after duplicate overlap between splits was found, so this is not a beginner’s problem."}]},{"t":"h2","id":"how-to-find-leakage-you-have-not-thought-of","runs":[{"text":"How to find leakage you have not thought of"}]},{"t":"p","runs":[{"text":"Three practical moves, in the order they cost you time:"}]},{"t":"ol","start":1,"items":[[{"text":"Look at the top features.","b":true},{"text":" If one feature dominates a model of a hard problem, investigate it specifically. This finds most cases."}],[{"text":"Ablate.","b":true},{"text":" Drop the suspect feature and refit. If the score falls from 0.97 to 0.79, you have learned that essentially all the performance came from it, which is worth knowing either way."}],[{"text":"Rebuild the row as of the prediction moment.","b":true},{"text":" For one or two rows, reconstruct every feature value using only data with a timestamp before the prediction time. Compare against the values in your training table. Any difference is leakage, and this is the only method that finds the ones you did not think to suspect."}]]},{"t":"p","runs":[{"text":"The third is tedious and is what separates a model that works from one that demos."}]}],"check":{"question":[{"text":"A loan default model uses "},{"text":"current_account_balance","code":true},{"text":", pulled from a table that is overwritten nightly. The feature is legitimate in principle. Why is it leaking?"}],"options":[{"text":[{"text":"Account balance is correlated with the target, and highly correlated features always indicate leakage."}],"correct":false,"tempting":[{"text":"Correlation with the target is what a good feature has; it is the timing of the value, not its predictiveness, that constitutes leakage."}]},{"text":[{"text":"The table is overwritten, so the stored value reflects the balance today, after the default, rather than the balance at application time, which is what production would supply."}],"correct":true,"tempting":null},{"text":[{"text":"Balance is a continuous feature and needs scaling before it can be used with a penalised model."}],"correct":false,"tempting":[{"text":"A preprocessing requirement unrelated to whether the value is knowable at prediction time."}]},{"text":[{"text":"Balances vary seasonally, so training on them introduces temporal bias that a random split cannot detect."}],"correct":false,"tempting":[{"text":"Seasonality is a genuine modelling concern, but the defect here would remain under any split because the recorded value is from the wrong point in time."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"text-as-numbers","title":"Text as columns: counting words before you embed them","minutes":9},"next":{"slug":"the-fit-transform-contract","title":"Fit on train, transform everywhere: the contract that keeps you honest","minutes":8}}}