When training and serving quietly disagree

The bug the notebook cannot see

The model scored 0.89 offline and 0.71 in production. Nothing is broken, no errors are logged, and the difference is that the features computed at serving time are not the features the model was trained on.

This is training-serving skew. Google’s engineers report that it occurs frequently in production pipelines, most often because training and serving compute features in different code paths (Breck et al., 2019; Google’s Rules of Machine Learning, rules 29 and 37). It is almost never visible from inside the notebook.

Four ways it happens

1. Two implementations of the same feature. Training features are computed in Python by the data scientist. Serving features are computed in Java by the platform team, from a specification in a document. “Average order value over the last 30 days” turns out to mean calendar days in one and rolling 720 hours in the other; one excludes cancelled orders and the other does not.

The difference is small, systematic, and applies to every prediction.

2. Different time semantics. Training computes features from a table containing the whole history, so “transactions in the last 7 days” is computed with hindsight. Serving computes it from data that has arrived so far, which for a system with a two-hour ingestion delay is not the same window. This is the leakage from module 3 reappearing as a production discrepancy.

3. Different preprocessing. The scaler’s mean was fitted on training data and hardcoded into the training script. The serving code recomputes it from whatever batch it currently has. Or the category list differs, so one-hot columns are in a different order, which produces no error and completely scrambles the input.

4. Different data sources. Training joins a clean warehouse table; serving reads a live operational database where the same field has nulls, different casing, or a different unit.

The three defences

Ship one implementation. Serialise the whole fitted pipeline (every transformation plus the model, as module 3 described) and call it in both places. The scaler’s mean is a value stored in the artefact rather than a computation that happens twice. This eliminates cause 3 entirely and is the highest-value practice in this lesson.

Share the feature code. If features are computed before the pipeline, put that code in a library that both the training job and the serving path import. Not copied. Imported. A copy is a second implementation with a delayed divergence.

Log serving features and compare. For a sample of production requests, write the computed feature values to a table. Then run the training-time computation over the same entities and compare distributions, feature by feature.

That third one is what actually finds skew. Any feature whose serving distribution differs meaningfully from its training distribution is either drift or a bug, and both need investigating. Do it weekly, automatically, and the comparison becomes a report rather than an investigation.

Feature stores

The systematic answer to causes 1 and 2. A feature store computes each feature once and serves it to both training and inference, with two properties that matter:

• Point-in-time correctness. When building a training set, it returns each feature’s value as of the moment of the event, not the current value. This is the leakage guard from module 3 implemented as infrastructure.
• Consistency. The same computation feeds both paths, so they cannot diverge.

Feast is the free open-source option; the cloud vendors all sell one. They are genuine infrastructure with real operational cost, and they are worth it when you have many models sharing features and several teams. For one model and one team, shared feature code in a library plus a serialised pipeline gets you most of the benefit for a fraction of the work.

The test that catches it before launch

One procedure, and it takes an afternoon.

Take 1,000 rows from your test set. Score them through the training pipeline. Then send the same 1,000 rows through the serving path as if they were live requests. Compare the two sets of predictions.

They should be identical to floating-point precision. If they differ at all, find out why before you launch: a discrepancy in the fourth decimal place is usually a different library version, and a discrepancy in the first is a feature computed differently.

Make this an automated test that runs on every deploy. It is the production equivalent of the label-shuffle check from module 4: cheap, mechanical, and it catches a class of error that reasoning about the code does not.