Fit on train, transform everywhere: the contract that keeps you honest

Two verbs, and the rule between them

Every preprocessing object in scikit-learn has two methods, and the whole discipline of this module comes down to when you call which.

• fit learns parameters from data. A mean. A median. A vocabulary. A set of PCA components.
• transform applies those learned parameters to data.

The rule: fit sees training data only. transform is applied to everything.

sc = StandardScaler()
X_train_s = sc.fit_transform(X_train)   # learns AND applies
X_test_s  = sc.transform(X_test)        # applies only

Calling fit_transform on the test set is the error. It computes a new mean from the test rows, so training and test are now on different scales, and the test rows have influenced their own transformation.

Why doing it by hand fails

It is not that manual preprocessing is impossible. It is that it fails under cross-validation, and cross-validation is where you will spend most of your time.

With 5-fold cross-validation, the training portion changes five times. The scaler’s mean must be recomputed five times, from five different subsets. Do the scaling once before the loop and every fold’s “held-out” portion contributed to the scaling, quietly, in all five folds, with no error and no warning.

Pipeline exists to make this structurally impossible:

from sklearn.pipeline import Pipeline
pipe = Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression())])
cross_val_score(pipe, X, y, cv=5)     # refits the scaler inside each fold

That is not a style preference. It changes the number you get.

ColumnTransformer, because real tables are mixed

Different columns need different treatment, and ColumnTransformer routes them:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer

prep = ColumnTransformer([
    ("num", Pipeline([
        ("imp", SimpleImputer(strategy="median", add_indicator=True)),
        ("sc", StandardScaler()),
    ]), numeric_cols),
    ("cat", Pipeline([
        ("imp", SimpleImputer(strategy="constant", fill_value="missing")),
        ("oh", OneHotEncoder(handle_unknown="ignore", min_frequency=20)),
    ]), categorical_cols),
], remainder="drop")

remainder="drop" is worth setting explicitly. The default drops unlisted columns, which is usually right, but writing it down means the next person can see that the omission was intentional rather than an oversight.

One caution: selecting columns by dtype (selector(dtype_include="number")) is convenient and fragile. A column that is numeric in training and arrives as a string in production (because one row had “N/A” in it) silently routes down the wrong branch. Explicit column lists are more typing and fewer 3am incidents.

Writing your own step

Anything you can express as a function of the columns can become a pipeline step:

from sklearn.preprocessing import FunctionTransformer
log_step = FunctionTransformer(np.log1p, feature_names_out="one-to-one")

FunctionTransformer is for stateless transformations: ones that learn nothing. If your step needs to remember something from training (a set of clip boundaries, a category list), write a small class with fit and transform. Inherit from BaseEstimator and TransformerMixin and it will work everywhere in scikit-learn, including inside a grid search.

The payoff at deployment

A fitted pipeline is one object containing every learned parameter: the medians, the category lists, the scaler’s means, the model’s coefficients.

import joblib
joblib.dump(pipe, "model.joblib")
# in the service
pipe = joblib.load("model.joblib")
pipe.predict_proba(new_rows_dataframe)

One artefact, one predict call, and the serving code cannot apply a different median from the training code because there is only one copy of it. This eliminates an entire category of production bug, training-serving skew, by construction rather than by discipline, which is the only way it stays eliminated after you leave the team.

Two cautions on saved model files. Version pinning matters: a model saved under scikit-learn 1.3 may fail to load under 1.6, so record the version alongside the file. And never load a model file from a source you do not trust: restoring one of these files can execute arbitrary code, so a model downloaded from the internet should be treated exactly like a downloaded executable.

Inspecting what came out

After one-hot encoding you have no idea which column is which. Ask:

pipe[:-1].get_feature_names_out()[:20]

This is how you connect a coefficient or an importance back to a human-readable name, and it is the difference between “feature 43 is important” and “applications from unverified accounts are important”.

The same call is your best defence against a silent column-order change. If an upstream table gains a column, a pipeline built from explicit lists will ignore it, while one built from positional indexing will shift every downstream feature by one and keep running. Print the first twenty names into your training log on every run; a diff between two runs then makes the change visible the day it happens rather than the week the metrics drift.