The feature computed two different ways

The bug that only exists in production

A model scores 0.91 on the held-out set. It ships. Live performance is closer to random, and nothing in the code is wrong in any way a test would find.

The usual cause is that the numbers going into the model at serving time are not the numbers it was trained on. The training pipeline computed days_since_signup from a batch table in pandas; the serving code computes it in the web application, in a different timezone, with a different definition of “today”. The model sees a feature it has never seen and returns confident nonsense.

This is training-serving skew, and it is the most common serious failure in deployed machine learning. It is an operations problem, not a modelling one, which is why it belongs in this course.

The four ways the two paths diverge

1. Two implementations. Training in SQL or pandas, serving in application code. Every difference in rounding, null handling, string normalisation and default value is a silent bug. This is the big one.

2. Different data. Training uses a warehouse table that has been cleaned, deduplicated and back-filled overnight. Serving uses the live row, which has nulls the warehouse fixed.

3. Time travel, in both directions. Training joins a value as it is today rather than as it was at the moment being predicted: the label leaks backwards, the offline score is beautiful, and no such information exists at serving time. Or the reverse: serving reads a value that has not been updated yet, because the job that computes it runs at 03:00.

4. Categories that did not exist. A new country code, a new plan name, a new device type. The encoder was fitted on last quarter’s values and maps the unknown one to whatever its default is: often index 0, which means “the most common category”, which is a lie.

The one habit that prevents most of it

Compute each feature in exactly one piece of code, and call that code from both paths.

# features.py — imported by the training job AND the serving app
def build(row, now):
    return {
        "days_since_signup": (now.date() - row.signup_at.date()).days,
        "posts_7d": row.posts_7d or 0,
        "country": normalise_country(row.country),
    }

This is unglamorous and it removes an entire class of failure. If the two paths cannot share code (different languages, different runtimes) then the tests below stop being optional.

The test that catches it in ten minutes

Take a hundred rows. Score them through the training pipeline and through the live serving path. Compare feature by feature, not just the final prediction.

# expect an empty diff; anything printed here is skew
diff <(jq -S . offline_features.json) <(jq -S . online_features.json)

Mismatches will be boring: a timezone, a null becoming zero in one path and the mean in the other, a string trimmed on one side. Boring and fatal. Run this as a scheduled job, not once: skew arrives when somebody edits one path.

Then log the served feature vector for a sample of live requests. Compare its distribution against the training distribution weekly; that comparison is also your drift monitor two lessons from now, so the work is shared.

Feature stores, and whether you need one

A feature store is a system that computes features once and serves them to both training and inference, with a record of what each value was at each point in time. Feast is the free, open-source one; every cloud has a managed version.

Be honest about the threshold. A feature store earns its complexity when several models share features, when features are computed by scheduled jobs rather than from the request, or when point-in-time correctness across many tables is hard to get right by hand. For one model with ten features derived from the current row, a shared Python module and the diff test above give you most of the benefit for none of the operational cost.

Save the fitted transforms with the model

The scaler, the encoder, the vocabulary and the imputation values are part of the model. Serialise them together (a scikit-learn Pipeline, an ONNX graph that includes preprocessing, or one directory containing all of them) and load them together.

The failure otherwise is subtle and common: the model file is deployed, a slightly older encoder is already on the box, the column order shifts by one, and every prediction is wrong in a way that no error message reports. A checksum of the preprocessing artifact, logged at start-up next to the model version, turns that into something you can see.