{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"splits-that-respect-structure","title":"Splits that respect the structure of the data","url":"https://addaly.com/learn/machine-learning-foundations/splits-that-respect-structure","position":35,"of":92,"minutes":9,"answer":"Match the split to the structure (group by entity, order by time with a gap equal to the label delay.","keyIdea":"Match the split to the structure (group by entity, order by time with a gap equal to the label delay, stratify small or imbalanced sets) and verify with a label-shuffle test that a correct pipeline collapses to chance."},"body":{"format":"markdown","text":"## Random is a choice, not a default\n\n`train_test_split` shuffles rows and cuts. That is correct only when rows are independent, and in real data they usually are not. Three structures break it, and each has a matching splitter.\n\n## Grouped data: split the entity, not the row\n\nWhenever several rows belong to the same underlying thing (patient, customer, device, document, household) those rows share information. A random split puts some of a patient's visits in training and the rest in test, and the model can identify the patient rather than the condition.\n\n```python\nfrom sklearn.model_selection import GroupKFold, StratifiedGroupKFold\ncv = StratifiedGroupKFold(n_splits=5)\nfor tr, te in cv.split(X, y, groups=df[\"patient_id\"]):\n    ...\n```\n\n`StratifiedGroupKFold` keeps whole groups together *and* balances the classes, which is what you almost always want and which did not exist in scikit-learn until relatively recently.\n\nThe question to ask: *at prediction time, will I have other rows from this same entity in my training data?* For a hospital predicting readmission for an existing patient, yes, and a random split may be defensible. For a model that will run on new patients, no, and grouped splitting is mandatory.\n\n## Time series: the future does not leak backwards\n\nFor anything with a time axis, train on the past and test on the future. Always. Even when there are no explicit date features, because data carries temporal structure implicitly.\n\n```python\nfrom sklearn.model_selection import TimeSeriesSplit\ncv = TimeSeriesSplit(n_splits=5, test_size=30)\n```\n\nThis produces expanding windows: train on days 1-100 test on 101-130, train on 1-130 test on 131-160, and so on. Each evaluation only ever uses the past.\n\nTwo refinements matter in practice.\n\n**A gap.** If your label takes 30 days to become known (did this customer default within 30 days) then at the moment of prediction you cannot know the labels for the last 30 days of \"past\" data. Leave a gap between the training window and the test window equal to the label delay, or you are training on labels that would not have existed yet. `TimeSeriesSplit(gap=30)` does it.\n\n**Rolling versus expanding.** Expanding uses all history, which is right when old patterns still hold. Rolling uses a fixed recent window, which is right when they do not. Which one is better is an empirical question about your data, and comparing them tells you something real about how fast your problem changes.\n\n## Stratification: keeping the folds comparable\n\nFor classification, stratify on the label. For regression with a skewed target, you can stratify on binned quantiles of the target, not built in, but three lines:\n\n```python\nimport pandas as pd\nbins = pd.qcut(y, q=10, labels=False, duplicates=\"drop\")\ntrain_test_split(X, y, stratify=bins, test_size=0.2, random_state=0)\n```\n\nThis matters most on small datasets, where one fold can otherwise get all the expensive houses.\n\n## When several structures apply at once\n\nReal problems stack them. Predicting equipment failure: rows are grouped by machine *and* ordered in time *and* the failure class is rare. You need all three constraints, and no single scikit-learn splitter does it.\n\nThe usual answer is to write the split yourself: choose a cut date, assign whole machines to train or test, and check the class balance on both sides afterwards. Twenty lines, and it is the honest version.\n\nWhen you cannot satisfy every constraint (you have 40 machines and need both a time split and a group split) say which one you relaxed and why. A stated compromise is engineering; an unstated one is a result nobody can interpret.\n\n## Two sanity checks worth running\n\n**The adversarial validation trick.** Label your training rows 0 and your test rows 1, and try to build a classifier that tells them apart. If it can, AUC well above 0.5, then the two sets differ systematically, and your test score is measuring performance on a different population. This finds split bugs, hidden time ordering, and genuine distribution shift, and it takes ten lines.\n\n**Shuffle the labels.** Randomly permute `y` and rerun the whole pipeline. Every score should collapse to chance. If it does not, something in your pipeline is reaching the answer by a route you did not intend, and you have found leakage that no amount of reading the code would have revealed.\n\nThat second check is one of the highest-value five-minute tests in this course. Run it on any result that surprises you."},"check":{"question":"A model predicts whether a customer will default within 30 days. The team uses TimeSeriesSplit without a gap, training up to day 200 and testing from day 201. What is wrong?","options":[{"text":"The test window should be randomly sampled from the whole period, so it represents the full range of conditions.","correct":false,"tempting":"Random sampling would reintroduce future information into training, which is the failure TimeSeriesSplit exists to prevent."},{"text":"The training set should be a rolling window rather than an expanding one, since old customer behaviour is no longer relevant.","correct":false,"tempting":"Rolling versus expanding is a real empirical choice, but it does not address information that would not have been available at prediction time."},{"text":"Test rows from day 201 onward have labels that are only known 30 days later, so the test set includes outcomes that cannot yet be observed.","correct":false,"tempting":"Correctly notes label delay, but the test set is allowed to use labels resolved after the fact: the problem lies on the training side of the boundary."},{"text":"Labels for training rows near day 200 are only known by day 230, so a model built at day 200 could not have had them; the split trains on information that would not have existed.","correct":true}]},"prev":{"slug":"cross-validation","title":"Cross-validation: using every row twice, legally","url":"https://addaly.com/learn/machine-learning-foundations/cross-validation"},"next":{"slug":"bias-and-variance","title":"Bias and variance, with numbers attached","url":"https://addaly.com/learn/machine-learning-foundations/bias-and-variance"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Splits that respect the structure of the data', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/splits-that-respect-structure (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Splits that respect the structure of the data","courseTitle":"Machine Learning, Foundations","module":{"index":4,"slug":"generalisation","title":"Generalisation, and the discipline of held-out data"},"answer":null,"keyIdea":"Match the split to the structure (group by entity, order by time with a gap equal to the label delay, stratify small or imbalanced sets) and verify with a label-shuffle test that a correct pipeline collapses to chance.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"random-is-a-choice-not-a-default","runs":[{"text":"Random is a choice, not a default"}]},{"t":"p","runs":[{"text":"train_test_split","code":true},{"text":" shuffles rows and cuts. That is correct only when rows are independent, and in real data they usually are not. Three structures break it, and each has a matching splitter."}]},{"t":"h2","id":"grouped-data-split-the-entity-not-the-row","runs":[{"text":"Grouped data: split the entity, not the row"}]},{"t":"p","runs":[{"text":"Whenever several rows belong to the same underlying thing (patient, customer, device, document, household) those rows share information. A random split puts some of a patient’s visits in training and the rest in test, and the model can identify the patient rather than the condition."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import GroupKFold, StratifiedGroupKFold\ncv = StratifiedGroupKFold(n_splits=5)\nfor tr, te in cv.split(X, y, groups=df[\"patient_id\"]):\n    ..."},{"t":"p","runs":[{"text":"StratifiedGroupKFold","code":true},{"text":" keeps whole groups together "},{"text":"and","i":true},{"text":" balances the classes, which is what you almost always want and which did not exist in scikit-learn until relatively recently."}]},{"t":"p","runs":[{"text":"The question to ask: "},{"text":"at prediction time, will I have other rows from this same entity in my training data?","i":true},{"text":" For a hospital predicting readmission for an existing patient, yes, and a random split may be defensible. For a model that will run on new patients, no, and grouped splitting is mandatory."}]},{"t":"h2","id":"time-series-the-future-does-not-leak-backwards","runs":[{"text":"Time series: the future does not leak backwards"}]},{"t":"p","runs":[{"text":"For anything with a time axis, train on the past and test on the future. Always. Even when there are no explicit date features, because data carries temporal structure implicitly."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import TimeSeriesSplit\ncv = TimeSeriesSplit(n_splits=5, test_size=30)"},{"t":"p","runs":[{"text":"This produces expanding windows: train on days 1-100 test on 101-130, train on 1-130 test on 131-160, and so on. Each evaluation only ever uses the past."}]},{"t":"p","runs":[{"text":"Two refinements matter in practice."}]},{"t":"p","runs":[{"text":"A gap.","b":true},{"text":" If your label takes 30 days to become known (did this customer default within 30 days) then at the moment of prediction you cannot know the labels for the last 30 days of “past” data. Leave a gap between the training window and the test window equal to the label delay, or you are training on labels that would not have existed yet. "},{"text":"TimeSeriesSplit(gap=30)","code":true},{"text":" does it."}]},{"t":"p","runs":[{"text":"Rolling versus expanding.","b":true},{"text":" Expanding uses all history, which is right when old patterns still hold. Rolling uses a fixed recent window, which is right when they do not. Which one is better is an empirical question about your data, and comparing them tells you something real about how fast your problem changes."}]},{"t":"h2","id":"stratification-keeping-the-folds-comparable","runs":[{"text":"Stratification: keeping the folds comparable"}]},{"t":"p","runs":[{"text":"For classification, stratify on the label. For regression with a skewed target, you can stratify on binned quantiles of the target, not built in, but three lines:"}]},{"t":"code","lang":"python","label":"Python","text":"import pandas as pd\nbins = pd.qcut(y, q=10, labels=False, duplicates=\"drop\")\ntrain_test_split(X, y, stratify=bins, test_size=0.2, random_state=0)"},{"t":"p","runs":[{"text":"This matters most on small datasets, where one fold can otherwise get all the expensive houses."}]},{"t":"h2","id":"when-several-structures-apply-at-once","runs":[{"text":"When several structures apply at once"}]},{"t":"p","runs":[{"text":"Real problems stack them. Predicting equipment failure: rows are grouped by machine "},{"text":"and","i":true},{"text":" ordered in time "},{"text":"and","i":true},{"text":" the failure class is rare. You need all three constraints, and no single scikit-learn splitter does it."}]},{"t":"p","runs":[{"text":"The usual answer is to write the split yourself: choose a cut date, assign whole machines to train or test, and check the class balance on both sides afterwards. Twenty lines, and it is the honest version."}]},{"t":"p","runs":[{"text":"When you cannot satisfy every constraint (you have 40 machines and need both a time split and a group split) say which one you relaxed and why. A stated compromise is engineering; an unstated one is a result nobody can interpret."}]},{"t":"h2","id":"two-sanity-checks-worth-running","runs":[{"text":"Two sanity checks worth running"}]},{"t":"p","runs":[{"text":"The adversarial validation trick.","b":true},{"text":" Label your training rows 0 and your test rows 1, and try to build a classifier that tells them apart. If it can, AUC well above 0.5, then the two sets differ systematically, and your test score is measuring performance on a different population. This finds split bugs, hidden time ordering, and genuine distribution shift, and it takes ten lines."}]},{"t":"p","runs":[{"text":"Shuffle the labels.","b":true},{"text":" Randomly permute "},{"text":"y","code":true},{"text":" and rerun the whole pipeline. Every score should collapse to chance. If it does not, something in your pipeline is reaching the answer by a route you did not intend, and you have found leakage that no amount of reading the code would have revealed."}]},{"t":"p","runs":[{"text":"That second check is one of the highest-value five-minute tests in this course. Run it on any result that surprises you."}]}],"check":{"question":[{"text":"A model predicts whether a customer will default within 30 days. The team uses TimeSeriesSplit without a gap, training up to day 200 and testing from day 201. What is wrong?"}],"options":[{"text":[{"text":"The test window should be randomly sampled from the whole period, so it represents the full range of conditions."}],"correct":false,"tempting":[{"text":"Random sampling would reintroduce future information into training, which is the failure TimeSeriesSplit exists to prevent."}]},{"text":[{"text":"The training set should be a rolling window rather than an expanding one, since old customer behaviour is no longer relevant."}],"correct":false,"tempting":[{"text":"Rolling versus expanding is a real empirical choice, but it does not address information that would not have been available at prediction time."}]},{"text":[{"text":"Test rows from day 201 onward have labels that are only known 30 days later, so the test set includes outcomes that cannot yet be observed."}],"correct":false,"tempting":[{"text":"Correctly notes label delay, but the test set is allowed to use labels resolved after the fact: the problem lies on the training side of the boundary."}]},{"text":[{"text":"Labels for training rows near day 200 are only known by day 230, so a model built at day 200 could not have had them; the split trains on information that would not have existed."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"cross-validation","title":"Cross-validation: using every row twice, legally","minutes":9},"next":{"slug":"bias-and-variance","title":"Bias and variance, with numbers attached","minutes":9}}}