{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-tabular-models","title":"Tables, cross-validation, and the leak that gives you 0.99","url":"https://addaly.com/learn/evaluating-ai/evaluating-tabular-models","position":39,"of":76,"minutes":9,"answer":"Fit every preprocessing step inside the fold, split by group and by time rather than at random.","keyIdea":"Fit every preprocessing step inside the fold, split by group and by time rather than at random, and treat an implausibly high score on a business problem as a leak to be found rather than a result to be announced."},"body":{"format":"markdown","text":"## The oldest kind of model, still most of the work\n\nChurn, default, fraud, demand, conversion, triage: a great deal of applied machine learning is a table with rows and columns and a target. The metrics are the ones from earlier in this course. What differs is how the data splits and how it leaks.\n\n## One split is not enough on small data\n\nA single train-test split on 2,000 rows gives an estimate with visible sampling noise; rerun with a different seed and the AUC moves. **K-fold cross-validation** (five folds, each row held out exactly once) uses everything and gives you five estimates whose spread is itself informative. If your folds range from 0.72 to 0.86, no single number should be quoted without that range.\n\nTwo variants matter more than the choice of k:\n\n- **Grouped folds** whenever rows share an entity: several loans per customer, several visits per patient. Same argument as the clustering lesson: without grouping, the model has seen the customer.\n- **Time-based folds** for anything with a time dimension. Train on the past, test on the future, walking forward. A random split lets the model learn from March to predict February, which no deployed model will ever be able to do.\n\n```python\nfrom sklearn.model_selection import GroupKFold, TimeSeriesSplit\n```\n\n## Leakage, in the four forms you will actually meet\n\n**A column that could not exist yet.** `days_to_payment` in a model predicting default. `claim_closed_reason` in a model predicting claim approval. The rule: for every feature, ask when its value becomes known, and delete anything known only after the target.\n\n**Preprocessing fitted outside the fold.** Scalers, imputers, and especially target encoding fitted on the whole dataset leak the test rows' information into training. The symptom is a cross-validation score better than any honest one, and the fix is to put every step inside a pipeline so it is refitted per fold:\n\n```python\nfrom sklearn.pipeline import make_pipeline\npipe = make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression())\ncross_val_score(pipe, X, y, cv=GroupKFold(5), groups=g)   # fitted inside each fold\n```\n\n**Duplicates across splits.** Re-submitted forms, re-imported rows, the same customer under two IDs. Deduplicate on business keys before splitting.\n\n**An identifier that encodes the answer.** Customer numbers issued in order encode signup date; branch codes encode region; a row index correlates with when the label was collected. Drop identifiers, or hash them and check the model does not improve when you do.\n\nThe tell for all four is the same: **an implausible score**. AUC 0.99 on churn, or 0.97 on fraud, is a bug until proven otherwise. Print feature importances the moment you see one: the leaking column is almost always at the top, and the whole investigation takes fifteen minutes.\n\n## Learning curves answer \"should we collect more data\"\n\nPlot the score against training-set size, from 10% to 100%, with the same folds. Two readings:\n\n- The curve is still climbing at 100%: more rows will help, and collecting them is the cheapest available improvement.\n- The curve flattened at 40%: more rows will not help. You need better features, a better target definition, or to accept the ceiling.\n\nThis one plot has redirected more projects usefully than any model comparison, and it costs a loop over five sample sizes.\n\n## Baselines and the honest state of the field\n\nRun `DummyClassifier`, then logistic regression, then a gradient-boosted tree: `LightGBM` or `XGBoost`, both free and both fine on a CPU. On tabular data, boosted trees remain very hard to beat, and the published comparisons with deep-learning approaches are still contested rather than settled; treat any claim in either direction as a claim about a particular set of datasets. What is not contested is that the boosted tree takes an hour, so it belongs in your comparison regardless.\n\n## The gap between the notebook and the service\n\nThe last failure is not statistical. Features computed in a notebook from a warehouse table are rarely computed identically by the serving code from live data: a different time zone, a different default for missing values, an aggregation window that is 30 days in one place and a calendar month in the other.\n\nCheck it directly. Take 100 rows, compute the features both ways, and compare column by column. Any mismatch is a silent accuracy loss that no offline evaluation will ever show you, and it is the most common reason a model that measured well arrives in production worse."},"check":{"question":"A team fits a `StandardScaler` and a target encoder on the full dataset, then runs 5-fold cross-validation on the transformed data, and reports AUC 0.94. What is wrong?","options":[{"text":"Nothing, provided the model itself is refitted inside each fold, which cross-validation does automatically.","correct":false,"tempting":"Refitting the estimator is not enough; the transformed features already carry information from the held-out rows before the model sees them."},{"text":"Target encoding is unsuitable for cross-validation and should be replaced with one-hot encoding.","correct":false,"tempting":"Blames the technique rather than where it was fitted; target encoding is fine when computed inside each fold."},{"text":"The preprocessing saw the held-out rows, so each fold's test data influenced its own features and the 0.94 is optimistic by an unknown amount.","correct":true},{"text":"Five folds is too few for a reliable estimate, and ten would have exposed the discrepancy.","correct":false,"tempting":"More folds change the variance of the estimate slightly and do nothing about a leak that is present in every fold."}]},"prev":{"slug":"evaluating-documents-and-ocr","title":"Documents: stages that multiply, and checks that need no labels","url":"https://addaly.com/learn/evaluating-ai/evaluating-documents-and-ocr"},"next":{"slug":"the-judge-needs-judging","title":"LLM-as-judge, and where it misleads you","url":"https://addaly.com/learn/evaluating-ai/the-judge-needs-judging"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Tables, cross-validation, and the leak that gives you 0.99', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-tabular-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Tables, cross-validation, and the leak that gives you 0.99","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"Fit every preprocessing step inside the fold, split by group and by time rather than at random, and treat an implausibly high score on a business problem as a leak to be found rather than a result to be announced.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-oldest-kind-of-model-still-most-of-the-work","runs":[{"text":"The oldest kind of model, still most of the work"}]},{"t":"p","runs":[{"text":"Churn, default, fraud, demand, conversion, triage: a great deal of applied machine learning is a table with rows and columns and a target. The metrics are the ones from earlier in this course. What differs is how the data splits and how it leaks."}]},{"t":"h2","id":"one-split-is-not-enough-on-small-data","runs":[{"text":"One split is not enough on small data"}]},{"t":"p","runs":[{"text":"A single train-test split on 2,000 rows gives an estimate with visible sampling noise; rerun with a different seed and the AUC moves. "},{"text":"K-fold cross-validation","b":true},{"text":" (five folds, each row held out exactly once) uses everything and gives you five estimates whose spread is itself informative. If your folds range from 0.72 to 0.86, no single number should be quoted without that range."}]},{"t":"p","runs":[{"text":"Two variants matter more than the choice of k:"}]},{"t":"ul","items":[[{"text":"Grouped folds","b":true},{"text":" whenever rows share an entity: several loans per customer, several visits per patient. Same argument as the clustering lesson: without grouping, the model has seen the customer."}],[{"text":"Time-based folds","b":true},{"text":" for anything with a time dimension. Train on the past, test on the future, walking forward. A random split lets the model learn from March to predict February, which no deployed model will ever be able to do."}]]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import GroupKFold, TimeSeriesSplit"},{"t":"h2","id":"leakage-in-the-four-forms-you-will-actually-meet","runs":[{"text":"Leakage, in the four forms you will actually meet"}]},{"t":"p","runs":[{"text":"A column that could not exist yet.","b":true},{"text":" "},{"text":"days_to_payment","code":true},{"text":" in a model predicting default. "},{"text":"claim_closed_reason","code":true},{"text":" in a model predicting claim approval. The rule: for every feature, ask when its value becomes known, and delete anything known only after the target."}]},{"t":"p","runs":[{"text":"Preprocessing fitted outside the fold.","b":true},{"text":" Scalers, imputers, and especially target encoding fitted on the whole dataset leak the test rows’ information into training. The symptom is a cross-validation score better than any honest one, and the fix is to put every step inside a pipeline so it is refitted per fold:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.pipeline import make_pipeline\npipe = make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression())\ncross_val_score(pipe, X, y, cv=GroupKFold(5), groups=g)   # fitted inside each fold"},{"t":"p","runs":[{"text":"Duplicates across splits.","b":true},{"text":" Re-submitted forms, re-imported rows, the same customer under two IDs. Deduplicate on business keys before splitting."}]},{"t":"p","runs":[{"text":"An identifier that encodes the answer.","b":true},{"text":" Customer numbers issued in order encode signup date; branch codes encode region; a row index correlates with when the label was collected. Drop identifiers, or hash them and check the model does not improve when you do."}]},{"t":"p","runs":[{"text":"The tell for all four is the same: "},{"text":"an implausible score","b":true},{"text":". AUC 0.99 on churn, or 0.97 on fraud, is a bug until proven otherwise. Print feature importances the moment you see one: the leaking column is almost always at the top, and the whole investigation takes fifteen minutes."}]},{"t":"h2","id":"learning-curves-answer-should-we-collect-more-data","runs":[{"text":"Learning curves answer “should we collect more data”"}]},{"t":"p","runs":[{"text":"Plot the score against training-set size, from 10% to 100%, with the same folds. Two readings:"}]},{"t":"ul","items":[[{"text":"The curve is still climbing at 100%: more rows will help, and collecting them is the cheapest available improvement."}],[{"text":"The curve flattened at 40%: more rows will not help. You need better features, a better target definition, or to accept the ceiling."}]]},{"t":"p","runs":[{"text":"This one plot has redirected more projects usefully than any model comparison, and it costs a loop over five sample sizes."}]},{"t":"h2","id":"baselines-and-the-honest-state-of-the-field","runs":[{"text":"Baselines and the honest state of the field"}]},{"t":"p","runs":[{"text":"Run "},{"text":"DummyClassifier","code":true},{"text":", then logistic regression, then a gradient-boosted tree: "},{"text":"LightGBM","code":true},{"text":" or "},{"text":"XGBoost","code":true},{"text":", both free and both fine on a CPU. On tabular data, boosted trees remain very hard to beat, and the published comparisons with deep-learning approaches are still contested rather than settled; treat any claim in either direction as a claim about a particular set of datasets. What is not contested is that the boosted tree takes an hour, so it belongs in your comparison regardless."}]},{"t":"h2","id":"the-gap-between-the-notebook-and-the-service","runs":[{"text":"The gap between the notebook and the service"}]},{"t":"p","runs":[{"text":"The last failure is not statistical. Features computed in a notebook from a warehouse table are rarely computed identically by the serving code from live data: a different time zone, a different default for missing values, an aggregation window that is 30 days in one place and a calendar month in the other."}]},{"t":"p","runs":[{"text":"Check it directly. Take 100 rows, compute the features both ways, and compare column by column. Any mismatch is a silent accuracy loss that no offline evaluation will ever show you, and it is the most common reason a model that measured well arrives in production worse."}]}],"check":{"question":[{"text":"A team fits a "},{"text":"StandardScaler","code":true},{"text":" and a target encoder on the full dataset, then runs 5-fold cross-validation on the transformed data, and reports AUC 0.94. What is wrong?"}],"options":[{"text":[{"text":"Nothing, provided the model itself is refitted inside each fold, which cross-validation does automatically."}],"correct":false,"tempting":[{"text":"Refitting the estimator is not enough; the transformed features already carry information from the held-out rows before the model sees them."}]},{"text":[{"text":"Target encoding is unsuitable for cross-validation and should be replaced with one-hot encoding."}],"correct":false,"tempting":[{"text":"Blames the technique rather than where it was fitted; target encoding is fine when computed inside each fold."}]},{"text":[{"text":"The preprocessing saw the held-out rows, so each fold’s test data influenced its own features and the 0.94 is optimistic by an unknown amount."}],"correct":true,"tempting":null},{"text":[{"text":"Five folds is too few for a reliable estimate, and ten would have exposed the discrepancy."}],"correct":false,"tempting":[{"text":"More folds change the variance of the estimate slightly and do nothing about a leak that is present in every fold."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"evaluating-documents-and-ocr","title":"Documents: stages that multiply, and checks that need no labels","minutes":9},"next":{"slug":"the-judge-needs-judging","title":"LLM-as-judge, and where it misleads you","minutes":9}}}