Three piles, three jobs
Split your data into three piles before you do anything else.
- Train — the model's parameters are fitted here. Typically 60–80%.
- Validation — everything *you* choose is chosen here: which model, how deep, which features, what threshold. Typically 10–20%.
- Test — used once, at the end, to estimate how the thing will perform. Typically 10–20%.
Most people understand train and test. The pile that gets misused is validation, and the misuse has a specific mechanism.
Why a second pile exists
Suppose you try 40 configurations and score each on the same held-out data. You pick the best. That number is now optimistic, and not by a little.
Here is the arithmetic. Take 1,000 held-out rows and 40 models that are all genuinely 80% accurate. Each measured score is 80% give or take about 1.3 points of sampling noise. Take the maximum of 40 such draws and you land around 82.5% on average. Two and a half points of pure luck, and it looks exactly like progress.
So the moment you use a set to *choose*, you have spent it. It can no longer *measure*. Validation is the pile you are allowed to spend. Test is the pile you are not.
Treat the test set as a single-use instrument. Every extra look converts it into another validation set. If you evaluate on test, adjust something, and evaluate again, you no longer have a test set — you have a second validation set and no honest estimate.
Splitting wrongly is the common failure
A random split assumes every row is an independent draw. Very often it is not.
Rows that share a person. A no-show model has one row per appointment, and patients appear five or ten times. Split rows at random and the same patient sits on both sides. The model learns "patient 40218 misses appointments", which is easy and useless — patient 40218 is not who you will be predicting for. Split by patient instead: every appointment of a given patient goes wholly into train or wholly into validation.
Rows that share time. With anything that evolves — prices, demand, fraud tactics — a random split lets the model see next month while predicting this month. Split by date: train on January to September, validate on October, test on November and December.
Rows that are near-duplicates. Scraped datasets are full of the same item twice. Deduplicate before splitting, or the split is a formality.
Ask what you are actually going to predict for. A new patient? Split by patient. A future week? Split by time. A new city? Hold out a city.
Preprocessing belongs inside the split
A quiet leak: scaling a column by the mean and standard deviation of the whole dataset, then splitting. Validation rows contributed to that mean, so information crossed the line. The same applies to imputing missing values, fitting an encoder for categories, or selecting the top 20 features by correlation.
Compute all of it on train only, then apply the stored values to validation and test. In scikit-learn that is what a Pipeline is for, and it is why fitting a scaler outside one is a recurring bug.
When data is scarce
With 800 rows, a 15% validation slice is 120 rows, which is too noisy to choose between models. Use k-fold cross-validation: split the training data into 5 parts, train five times, each time holding out a different part, and average the five scores. Every row gets used for validation exactly once and the estimate is far more stable.
Two cautions. Cross-validation replaces the validation pile, not the test pile — keep the test set out of the folds entirely. And it inherits whatever splitting mistake you made: if the folds are random rows and you needed groups, you have now made the same error five times and averaged it into something that looks solid.
Write the number down
Before the final test-set evaluation, write down what you expect and what result would make you ship. Then run it once.
If the number disappoints and you go back and tune, that is allowed — but say so out loud, and understand that your test estimate is now optimistic. This is not bureaucracy. Roughly every over-promised model comes from a team that quietly iterated against their test set and then reported the best number as if it were the first.
Before you move on