Tables, cross-validation, and the leak that gives you 0.99

The oldest kind of model, still most of the work

Churn, default, fraud, demand, conversion, triage: a great deal of applied machine learning is a table with rows and columns and a target. The metrics are the ones from earlier in this course. What differs is how the data splits and how it leaks.

One split is not enough on small data

A single train-test split on 2,000 rows gives an estimate with visible sampling noise; rerun with a different seed and the AUC moves. K-fold cross-validation (five folds, each row held out exactly once) uses everything and gives you five estimates whose spread is itself informative. If your folds range from 0.72 to 0.86, no single number should be quoted without that range.

Two variants matter more than the choice of k:

• Grouped folds whenever rows share an entity: several loans per customer, several visits per patient. Same argument as the clustering lesson: without grouping, the model has seen the customer.
• Time-based folds for anything with a time dimension. Train on the past, test on the future, walking forward. A random split lets the model learn from March to predict February, which no deployed model will ever be able to do.

from sklearn.model_selection import GroupKFold, TimeSeriesSplit

Leakage, in the four forms you will actually meet

A column that could not exist yet. days_to_payment in a model predicting default. claim_closed_reason in a model predicting claim approval. The rule: for every feature, ask when its value becomes known, and delete anything known only after the target.

Preprocessing fitted outside the fold. Scalers, imputers, and especially target encoding fitted on the whole dataset leak the test rows’ information into training. The symptom is a cross-validation score better than any honest one, and the fix is to put every step inside a pipeline so it is refitted per fold:

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression())
cross_val_score(pipe, X, y, cv=GroupKFold(5), groups=g)   # fitted inside each fold

Duplicates across splits. Re-submitted forms, re-imported rows, the same customer under two IDs. Deduplicate on business keys before splitting.

An identifier that encodes the answer. Customer numbers issued in order encode signup date; branch codes encode region; a row index correlates with when the label was collected. Drop identifiers, or hash them and check the model does not improve when you do.

The tell for all four is the same: an implausible score. AUC 0.99 on churn, or 0.97 on fraud, is a bug until proven otherwise. Print feature importances the moment you see one: the leaking column is almost always at the top, and the whole investigation takes fifteen minutes.

Learning curves answer “should we collect more data”

Plot the score against training-set size, from 10% to 100%, with the same folds. Two readings:

• The curve is still climbing at 100%: more rows will help, and collecting them is the cheapest available improvement.
• The curve flattened at 40%: more rows will not help. You need better features, a better target definition, or to accept the ceiling.

This one plot has redirected more projects usefully than any model comparison, and it costs a loop over five sample sizes.

Baselines and the honest state of the field

Run DummyClassifier, then logistic regression, then a gradient-boosted tree: LightGBM or XGBoost, both free and both fine on a CPU. On tabular data, boosted trees remain very hard to beat, and the published comparisons with deep-learning approaches are still contested rather than settled; treat any claim in either direction as a claim about a particular set of datasets. What is not contested is that the boosted tree takes an hour, so it belongs in your comparison regardless.

The gap between the notebook and the service

The last failure is not statistical. Features computed in a notebook from a warehouse table are rarely computed identically by the serving code from live data: a different time zone, a different default for missing values, an aggregation window that is 30 days in one place and a calendar month in the other.

Check it directly. Take 100 rows, compute the features both ways, and compare column by column. Any mismatch is a silent accuracy loss that no offline evaluation will ever show you, and it is the most common reason a model that measured well arrives in production worse.