Leakage: a catalogue of the ways the answer gets in

The definition that covers every case

Leakage is any information in your training features that will not be available, in that form, at the moment you have to make a real prediction.

It is the most expensive error in applied machine learning because it does not look like an error. It looks like success. Your score goes up, your cross-validation agrees, your colleagues are impressed, and the model fails the day it meets production.

The signature is worth memorising: a result substantially better than you expected, arriving with no effort. Every experienced practitioner has learned to distrust that feeling, and the check takes fifteen minutes.

Target leakage: the column that knows

A feature that is a consequence of the target rather than a cause.

• Predicting churn with cancellation_reason. Only cancelled customers have one.
• Predicting hospital readmission with discharge_destination, when “transferred to another hospital” is recorded as part of the readmission process.
• Predicting fraud with investigation_opened. Investigations open after fraud is suspected.
• Predicting default with current_balance, refreshed nightly and therefore reflecting today rather than the application date.

The last one is the dangerous shape, because the column is legitimate: balance at application time would be a fine feature. What leaks is the timestamp of the value, not the concept. Any feature stored in a table that gets overwritten is suspect for exactly this reason, and it is why event-sourced data with valid-from timestamps is worth so much to a data science team.

The test: for each feature, ask what is its value at the moment of prediction, for a row whose outcome is not yet known? Say it out loud for the ten most important features. It takes fifteen minutes and it is the highest-return quarter hour in this course.

Train-test contamination: fitting on everything

Any transformation that learns from the data must learn from training rows only. The list of things that learn:

• Scalers (mean, standard deviation)
• Imputers (median, mode)
• Target encoders (per-category means)
• Feature selection (which columns to keep)
• PCA (the components)
• Outlier clipping (the percentiles)
• Resampling such as SMOTE (which synthesises rows from neighbours)

Run any of these on the full dataset before splitting and information from the held-out rows has entered the training process. The effect is usually small (one or two points) which is worse than a large effect, because it survives review and quietly makes every comparison you run slightly wrong.

The cleanest defence is structural: put every one of them inside a Pipeline, and let cross-validation refit the whole pipeline on each fold. The next lesson is entirely about that.

Group leakage: the same entity on both sides

One patient with 40 visits. One customer with 11 loans. One user with 300 sessions. One document photographed 6 times.

A random row split scatters these across train and test, and the model recognises the entity rather than learning the pattern. The score is inflated by an amount that depends on how much of the signal is entity-specific, which can be most of it.

Split by group, never by row, whenever rows share an entity:

from sklearn.model_selection import GroupKFold
cv = GroupKFold(n_splits=5)
cv.split(X, y, groups=df["patient_id"])

Temporal leakage: training on the future

A random split lets the model train on November and December while being tested on the intervening period. In production it will only ever have the past.

This matters even without explicit date features, because data has structure in time that a model absorbs implicitly: a product launch, a policy change, a season. Use a time-ordered split for anything with a time axis, which is nearly everything transactional.

Duplicate rows: the quietest one

Exact or near-duplicate rows split across train and test give the model the answer directly. Scraped datasets, merged exports and image sets are full of them, often at 5 to 30% of rows.

Check before splitting:

df.duplicated().sum()
df.duplicated(subset=key_columns).sum()

For images and text, near-duplicates need fuzzy matching: perceptual hashes for images, shingling or MinHash for text. Several famous published benchmarks have had their results revised downward after duplicate overlap between splits was found, so this is not a beginner’s problem.

How to find leakage you have not thought of

Three practical moves, in the order they cost you time:

1. Look at the top features. If one feature dominates a model of a hard problem, investigate it specifically. This finds most cases.
2. Ablate. Drop the suspect feature and refit. If the score falls from 0.97 to 0.79, you have learned that essentially all the performance came from it, which is worth knowing either way.
3. Rebuild the row as of the prediction moment. For one or two rows, reconstruct every feature value using only data with a timestamp before the prediction time. Compare against the values in your training table. Any difference is leakage, and this is the only method that finds the ones you did not think to suspect.

The third is tedious and is what separates a model that works from one that demos.