{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"training-serving-skew","title":"When training and serving quietly disagree","url":"https://addaly.com/learn/machine-learning-foundations/training-serving-skew","position":85,"of":92,"minutes":9,"answer":"Training-serving skew is a systematic difference between how a feature is computed offline and online, and the reliable defences are shipping one serialised pipeline, importing shared feature code rather than copying it.","keyIdea":"Training-serving skew is a systematic difference between how a feature is computed offline and online, and the reliable defences are shipping one serialised pipeline, importing shared feature code rather than copying it, and an automated test asserting both paths produce identical predictions on the same rows."},"body":{"format":"markdown","text":"## The bug the notebook cannot see\n\nThe model scored 0.89 offline and 0.71 in production. Nothing is broken, no errors are logged, and the difference is that the features computed at serving time are not the features the model was trained on.\n\nThis is **training-serving skew**. Google's engineers report that it occurs frequently in production pipelines, most often because training and serving compute features in different code paths (Breck et al., 2019; Google's Rules of Machine Learning, rules 29 and 37). It is almost never visible from inside the notebook.\n\n## Four ways it happens\n\n**1. Two implementations of the same feature.** Training features are computed in Python by the data scientist. Serving features are computed in Java by the platform team, from a specification in a document. \"Average order value over the last 30 days\" turns out to mean calendar days in one and rolling 720 hours in the other; one excludes cancelled orders and the other does not.\n\nThe difference is small, systematic, and applies to every prediction.\n\n**2. Different time semantics.** Training computes features from a table containing the whole history, so \"transactions in the last 7 days\" is computed with hindsight. Serving computes it from data that has arrived so far, which for a system with a two-hour ingestion delay is not the same window. This is the leakage from module 3 reappearing as a production discrepancy.\n\n**3. Different preprocessing.** The scaler's mean was fitted on training data and hardcoded into the training script. The serving code recomputes it from whatever batch it currently has. Or the category list differs, so one-hot columns are in a different order, which produces no error and completely scrambles the input.\n\n**4. Different data sources.** Training joins a clean warehouse table; serving reads a live operational database where the same field has nulls, different casing, or a different unit.\n\n## The three defences\n\n**Ship one implementation.** Serialise the whole fitted pipeline (every transformation plus the model, as module 3 described) and call it in both places. The scaler's mean is a value stored in the artefact rather than a computation that happens twice. This eliminates cause 3 entirely and is the highest-value practice in this lesson.\n\n**Share the feature code.** If features are computed before the pipeline, put that code in a library that both the training job and the serving path import. Not copied. Imported. A copy is a second implementation with a delayed divergence.\n\n**Log serving features and compare.** For a sample of production requests, write the computed feature values to a table. Then run the training-time computation over the same entities and compare distributions, feature by feature.\n\nThat third one is what actually finds skew. Any feature whose serving distribution differs meaningfully from its training distribution is either drift or a bug, and both need investigating. Do it weekly, automatically, and the comparison becomes a report rather than an investigation.\n\n## Feature stores\n\nThe systematic answer to causes 1 and 2. A feature store computes each feature once and serves it to both training and inference, with two properties that matter:\n\n- **Point-in-time correctness.** When building a training set, it returns each feature's value *as of* the moment of the event, not the current value. This is the leakage guard from module 3 implemented as infrastructure.\n- **Consistency.** The same computation feeds both paths, so they cannot diverge.\n\nFeast is the free open-source option; the cloud vendors all sell one. They are genuine infrastructure with real operational cost, and they are worth it when you have many models sharing features and several teams. For one model and one team, shared feature code in a library plus a serialised pipeline gets you most of the benefit for a fraction of the work.\n\n## The test that catches it before launch\n\nOne procedure, and it takes an afternoon.\n\nTake 1,000 rows from your test set. Score them through the *training* pipeline. Then send the same 1,000 rows through the *serving* path as if they were live requests. Compare the two sets of predictions.\n\nThey should be identical to floating-point precision. If they differ at all, find out why before you launch: a discrepancy in the fourth decimal place is usually a different library version, and a discrepancy in the first is a feature computed differently.\n\nMake this an automated test that runs on every deploy. It is the production equivalent of the label-shuffle check from module 4: cheap, mechanical, and it catches a class of error that reasoning about the code does not."},"check":{"question":"A model performs well offline and poorly in production with no errors logged. Predictions from the training pipeline and the serving path on the same 1,000 rows differ in the second decimal place. What does that indicate?","options":[{"text":"Ordinary floating-point non-determinism between environments, which is expected at this magnitude.","correct":false,"tempting":"Two environments running the same model at the same precision differ only by rounding: around the seventh significant digit in 32-bit arithmetic, and far less in 64-bit. A gap in the second decimal place on identical rows is not rounding noise, unless the serving path deliberately runs at reduced precision (bfloat16, float16 or quantised integers)."},{"text":"Concept drift, since production data no longer resembles the training distribution.","correct":false,"tempting":"Drift would change performance without making the two paths disagree on identical input rows."},{"text":"At least one feature is computed differently in the two paths (a different window, unit, category ordering or preprocessing constant) which is a bug rather than a data problem.","correct":true},{"text":"The serving model is an older version, so the weights differ.","correct":false,"tempting":"A stale artefact would produce a difference, but it is ruled out cheaply by comparing version identifiers and would typically show a larger and more uniform gap."}]},"prev":{"slug":"serving-a-model","title":"Batch, online, and the latency budget","url":"https://addaly.com/learn/machine-learning-foundations/serving-a-model"},"next":{"slug":"monitoring-what-matters","title":"Monitoring: what to watch, and in what order","url":"https://addaly.com/learn/machine-learning-foundations/monitoring-what-matters"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When training and serving quietly disagree', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/training-serving-skew (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When training and serving quietly disagree","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"Training-serving skew is a systematic difference between how a feature is computed offline and online, and the reliable defences are shipping one serialised pipeline, importing shared feature code rather than copying it, and an automated test asserting both paths produce identical predictions on the same rows.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-bug-the-notebook-cannot-see","runs":[{"text":"The bug the notebook cannot see"}]},{"t":"p","runs":[{"text":"The model scored 0.89 offline and 0.71 in production. Nothing is broken, no errors are logged, and the difference is that the features computed at serving time are not the features the model was trained on."}]},{"t":"p","runs":[{"text":"This is "},{"text":"training-serving skew","b":true},{"text":". Google’s engineers report that it occurs frequently in production pipelines, most often because training and serving compute features in different code paths (Breck et al., 2019; Google’s Rules of Machine Learning, rules 29 and 37). It is almost never visible from inside the notebook."}]},{"t":"h2","id":"four-ways-it-happens","runs":[{"text":"Four ways it happens"}]},{"t":"p","runs":[{"text":"1. Two implementations of the same feature.","b":true},{"text":" Training features are computed in Python by the data scientist. Serving features are computed in Java by the platform team, from a specification in a document. “Average order value over the last 30 days” turns out to mean calendar days in one and rolling 720 hours in the other; one excludes cancelled orders and the other does not."}]},{"t":"p","runs":[{"text":"The difference is small, systematic, and applies to every prediction."}]},{"t":"p","runs":[{"text":"2. Different time semantics.","b":true},{"text":" Training computes features from a table containing the whole history, so “transactions in the last 7 days” is computed with hindsight. Serving computes it from data that has arrived so far, which for a system with a two-hour ingestion delay is not the same window. This is the leakage from module 3 reappearing as a production discrepancy."}]},{"t":"p","runs":[{"text":"3. Different preprocessing.","b":true},{"text":" The scaler’s mean was fitted on training data and hardcoded into the training script. The serving code recomputes it from whatever batch it currently has. Or the category list differs, so one-hot columns are in a different order, which produces no error and completely scrambles the input."}]},{"t":"p","runs":[{"text":"4. Different data sources.","b":true},{"text":" Training joins a clean warehouse table; serving reads a live operational database where the same field has nulls, different casing, or a different unit."}]},{"t":"h2","id":"the-three-defences","runs":[{"text":"The three defences"}]},{"t":"p","runs":[{"text":"Ship one implementation.","b":true},{"text":" Serialise the whole fitted pipeline (every transformation plus the model, as module 3 described) and call it in both places. The scaler’s mean is a value stored in the artefact rather than a computation that happens twice. This eliminates cause 3 entirely and is the highest-value practice in this lesson."}]},{"t":"p","runs":[{"text":"Share the feature code.","b":true},{"text":" If features are computed before the pipeline, put that code in a library that both the training job and the serving path import. Not copied. Imported. A copy is a second implementation with a delayed divergence."}]},{"t":"p","runs":[{"text":"Log serving features and compare.","b":true},{"text":" For a sample of production requests, write the computed feature values to a table. Then run the training-time computation over the same entities and compare distributions, feature by feature."}]},{"t":"p","runs":[{"text":"That third one is what actually finds skew. Any feature whose serving distribution differs meaningfully from its training distribution is either drift or a bug, and both need investigating. Do it weekly, automatically, and the comparison becomes a report rather than an investigation."}]},{"t":"h2","id":"feature-stores","runs":[{"text":"Feature stores"}]},{"t":"p","runs":[{"text":"The systematic answer to causes 1 and 2. A feature store computes each feature once and serves it to both training and inference, with two properties that matter:"}]},{"t":"ul","items":[[{"text":"Point-in-time correctness.","b":true},{"text":" When building a training set, it returns each feature’s value "},{"text":"as of","i":true},{"text":" the moment of the event, not the current value. This is the leakage guard from module 3 implemented as infrastructure."}],[{"text":"Consistency.","b":true},{"text":" The same computation feeds both paths, so they cannot diverge."}]]},{"t":"p","runs":[{"text":"Feast is the free open-source option; the cloud vendors all sell one. They are genuine infrastructure with real operational cost, and they are worth it when you have many models sharing features and several teams. For one model and one team, shared feature code in a library plus a serialised pipeline gets you most of the benefit for a fraction of the work."}]},{"t":"h2","id":"the-test-that-catches-it-before-launch","runs":[{"text":"The test that catches it before launch"}]},{"t":"p","runs":[{"text":"One procedure, and it takes an afternoon."}]},{"t":"p","runs":[{"text":"Take 1,000 rows from your test set. Score them through the "},{"text":"training","i":true},{"text":" pipeline. Then send the same 1,000 rows through the "},{"text":"serving","i":true},{"text":" path as if they were live requests. Compare the two sets of predictions."}]},{"t":"p","runs":[{"text":"They should be identical to floating-point precision. If they differ at all, find out why before you launch: a discrepancy in the fourth decimal place is usually a different library version, and a discrepancy in the first is a feature computed differently."}]},{"t":"p","runs":[{"text":"Make this an automated test that runs on every deploy. It is the production equivalent of the label-shuffle check from module 4: cheap, mechanical, and it catches a class of error that reasoning about the code does not."}]}],"check":{"question":[{"text":"A model performs well offline and poorly in production with no errors logged. Predictions from the training pipeline and the serving path on the same 1,000 rows differ in the second decimal place. What does that indicate?"}],"options":[{"text":[{"text":"Ordinary floating-point non-determinism between environments, which is expected at this magnitude."}],"correct":false,"tempting":[{"text":"Two environments running the same model at the same precision differ only by rounding: around the seventh significant digit in 32-bit arithmetic, and far less in 64-bit. A gap in the second decimal place on identical rows is not rounding noise, unless the serving path deliberately runs at reduced precision (bfloat16, float16 or quantised integers)."}]},{"text":[{"text":"Concept drift, since production data no longer resembles the training distribution."}],"correct":false,"tempting":[{"text":"Drift would change performance without making the two paths disagree on identical input rows."}]},{"text":[{"text":"At least one feature is computed differently in the two paths (a different window, unit, category ordering or preprocessing constant) which is a bug rather than a data problem."}],"correct":true,"tempting":null},{"text":[{"text":"The serving model is an older version, so the weights differ."}],"correct":false,"tempting":[{"text":"A stale artefact would produce a difference, but it is ruled out cheaply by comparing version identifiers and would typically show a larger and more uniform gap."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"serving-a-model","title":"Batch, online, and the latency budget","minutes":8},"next":{"slug":"monitoring-what-matters","title":"Monitoring: what to watch, and in what order","minutes":9}}}