Splits that respect the structure of the data

Random is a choice, not a default

train_test_split shuffles rows and cuts. That is correct only when rows are independent, and in real data they usually are not. Three structures break it, and each has a matching splitter.

Grouped data: split the entity, not the row

Whenever several rows belong to the same underlying thing (patient, customer, device, document, household) those rows share information. A random split puts some of a patient’s visits in training and the rest in test, and the model can identify the patient rather than the condition.

from sklearn.model_selection import GroupKFold, StratifiedGroupKFold
cv = StratifiedGroupKFold(n_splits=5)
for tr, te in cv.split(X, y, groups=df["patient_id"]):
    ...

StratifiedGroupKFold keeps whole groups together and balances the classes, which is what you almost always want and which did not exist in scikit-learn until relatively recently.

The question to ask: at prediction time, will I have other rows from this same entity in my training data? For a hospital predicting readmission for an existing patient, yes, and a random split may be defensible. For a model that will run on new patients, no, and grouped splitting is mandatory.

Time series: the future does not leak backwards

For anything with a time axis, train on the past and test on the future. Always. Even when there are no explicit date features, because data carries temporal structure implicitly.

from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5, test_size=30)

This produces expanding windows: train on days 1-100 test on 101-130, train on 1-130 test on 131-160, and so on. Each evaluation only ever uses the past.

Two refinements matter in practice.

A gap. If your label takes 30 days to become known (did this customer default within 30 days) then at the moment of prediction you cannot know the labels for the last 30 days of “past” data. Leave a gap between the training window and the test window equal to the label delay, or you are training on labels that would not have existed yet. TimeSeriesSplit(gap=30) does it.

Rolling versus expanding. Expanding uses all history, which is right when old patterns still hold. Rolling uses a fixed recent window, which is right when they do not. Which one is better is an empirical question about your data, and comparing them tells you something real about how fast your problem changes.

Stratification: keeping the folds comparable

For classification, stratify on the label. For regression with a skewed target, you can stratify on binned quantiles of the target, not built in, but three lines:

import pandas as pd
bins = pd.qcut(y, q=10, labels=False, duplicates="drop")
train_test_split(X, y, stratify=bins, test_size=0.2, random_state=0)

This matters most on small datasets, where one fold can otherwise get all the expensive houses.

When several structures apply at once

Real problems stack them. Predicting equipment failure: rows are grouped by machine and ordered in time and the failure class is rare. You need all three constraints, and no single scikit-learn splitter does it.

The usual answer is to write the split yourself: choose a cut date, assign whole machines to train or test, and check the class balance on both sides afterwards. Twenty lines, and it is the honest version.

When you cannot satisfy every constraint (you have 40 machines and need both a time split and a group split) say which one you relaxed and why. A stated compromise is engineering; an unstated one is a result nobody can interpret.

Two sanity checks worth running

The adversarial validation trick. Label your training rows 0 and your test rows 1, and try to build a classifier that tells them apart. If it can, AUC well above 0.5, then the two sets differ systematically, and your test score is measuring performance on a different population. This finds split bugs, hidden time ordering, and genuine distribution shift, and it takes ten lines.

Shuffle the labels. Randomly permute y and rerun the whole pipeline. Every score should collapse to chance. If it does not, something in your pipeline is reaching the answer by a route you did not intend, and you have found leakage that no amount of reading the code would have revealed.

That second check is one of the highest-value five-minute tests in this course. Run it on any result that surprises you.