The whole loop, once, before we slow down

A map you can hold in your head

Everything in the next eight modules is one of nine steps. Seeing them together now means each later lesson lands somewhere rather than floating.

1. Frame. What decision, what row, what target, what moment. From the previous lessons.
2. Collect and split. Get the data, and put a portion away before you look at any of it.
3. Explore. Distributions, missingness, obvious errors, the target’s base rate.
4. Engineer features. The bulk of the work. Module 3.
5. Choose a model family and fit. Modules 2, 6, 8.
6. Tune against validation. Module 4.
7. Evaluate on test, once. Module 5.
8. Deploy. Module 9.
9. Monitor, and go back to step 1. Also module 9.

The arrows go both ways between 3 and 6 many times. They go one way into 7, and that is the discipline the whole field is built on.

The same loop in thirty lines

This runs on a free Colab or Kaggle notebook, on a laptop with 4 GB of RAM, and it contains a version of every step above.

import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.dummy import DummyClassifier
from sklearn.metrics import roc_auc_score

df = pd.read_csv("applications.csv")
y = df.pop("defaulted_90d")

# 2. split first, look second
X_tr, X_te, y_tr, y_te = train_test_split(
    df, y, test_size=0.2, stratify=y, random_state=0)

num = X_tr.select_dtypes("number").columns
cat = X_tr.select_dtypes("object").columns

prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), num),
    ("cat", OneHotEncoder(handle_unknown="ignore"), cat),
])

pipe = Pipeline([("prep", prep),
                 ("model", HistGradientBoostingClassifier(random_state=0))])

# 5-6. fit and tune using cross-validation on TRAIN only
print("cv auc", cross_val_score(pipe, X_tr, y_tr, cv=5, scoring="roc_auc").mean())

# baseline for comparison
base = DummyClassifier(strategy="prior").fit(X_tr, y_tr)
print("baseline auc", roc_auc_score(y_te, base.predict_proba(X_te)[:, 1]))

# 7. test set, once, at the end
pipe.fit(X_tr, y_tr)
print("test auc", roc_auc_score(y_te, pipe.predict_proba(X_te)[:, 1]))

Read it for the shape, not the details. Four things in there are load-bearing and worth naming now.

The split happens on line 12, before any exploration. Every transformation after it is fitted inside a Pipeline, which means the median used for imputation and the mean used for scaling are computed on training rows only. Do that by hand instead and you will one day compute the median on the whole dataset, leak test information into training, and get a score you cannot reproduce in production.

handle_unknown="ignore". A category appears in production that was not in training. Without this line, the pipeline raises and your service returns a 500. With it, that row gets all-zeros for the category block and a prediction still comes out. Neither is obviously right; the point is that you chose.

Cross-validation runs on training data only. The test set is not involved in any decision.

The test line appears once, at the end, after everything else is settled.

Where the time actually goes

If you have read that machine learning is 80% data cleaning, the number is folklore but the direction is right. A more useful breakdown, from projects that shipped:

• Deciding the question, the row and the target: more than people expect, and it is the highest-leverage time in the project.
• Getting the data and making it trustworthy: the largest single block, and the least visible.
• Feature engineering: the second largest, and where most accuracy is won on tabular problems.
• Model selection and tuning: usually far smaller than the excitement about it suggests. HistGradientBoostingClassifier with default settings is a strong model; tuning typically buys single-digit percentage improvements over a sensible default.
• Evaluation, deployment and monitoring: the part that decides whether the work survives a year.

The first three of those five are covered in modules 1, 3 and 5. That is deliberate.

What to do with this lesson

Run the code on any dataset you can find: the Titanic set, a CSV of your own expenses, anything with a column you could try to predict. It will fail in some interesting way. Getting it to run once, end to end, on your own machine or a free notebook, is worth more than the next three lessons read passively, and it gives every later lesson something concrete to attach to.