{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"monitoring-what-matters","title":"Monitoring: what to watch, and in what order","url":"https://addaly.com/learn/machine-learning-foundations/monitoring-what-matters","position":86,"of":92,"minutes":9,"answer":"Monitor system health, input distributions, prediction distributions and finally actual performance: the first three need no labels and move earlier, and the alert must name which kind of shift it found because covariate.","keyIdea":"Monitor system health, input distributions, prediction distributions and finally actual performance: the first three need no labels and move earlier, and the alert must name which kind of shift it found because covariate, label and concept drift take different remedies."},"body":{"format":"markdown","text":"## Four layers, from cheapest to most valuable\n\n**1. System health.** Request rate, error rate, latency percentiles, memory. Standard software monitoring, and it catches the model container restarting every ten minutes. Necessary and not sufficient: a model returning confident nonsense is perfectly healthy by these measures.\n\n**2. Input distributions.** For each feature: mean, standard deviation, missing rate, and for categoricals, the set of levels and the rate of previously unseen ones. Compare against the training data.\n\nThis is where most problems appear first, because most problems are upstream data problems. A column that becomes 100% null overnight, a currency changing units, a category renamed: all visible here within hours.\n\n**3. Prediction distributions.** The mean predicted probability, the distribution's shape, the rate of predictions above your decision threshold. You do not need labels for any of this, and it moves before your accuracy metrics can even be computed.\n\nA flag rate that jumps from 2% to 9% overnight is an incident. It might be real fraud, or an upstream change, or a broken feature, and you want to know within the hour rather than at the next monthly review.\n\n**4. Actual performance.** Accuracy, AUC, calibration, per slice. The thing you truly care about, and the slowest to arrive, because labels lag: sometimes by 90 days for a default, sometimes forever for a loan you declined.\n\nBuild all four, and understand that the first three are early warnings for the fourth.\n\n> Figure: Four layers of monitoring, cheapest first. Layers, top to bottom: 1. System health (Request rate, errors, latency percentiles, memory. Catches a container restarting; misses a model returning confident nonsense.); 2. Input distributions (Means, missing rates, unseen categories, against the training data. Where most problems appear first, within hours.); 3. Prediction distributions (Mean score, shape, rate above threshold. A flag rate that goes from 2% to 9% overnight is an incident. Needs no labels.); 4. Actual performance (Accuracy, AUC, calibration, per slice. What you care about, and the slowest to arrive: 90 days for a default, never for a loan you declined.). The first three need no labels and move earlier; they are early warnings for the fourth, which is the only one that can see concept drift and the last to arrive, because labels lag by weeks or forever.\n\n## Detecting a change in a distribution\n\nThree standard measures, and the practical advice about each.\n\n**Population Stability Index.** The credit industry's standard. Bucket the feature into deciles based on training data, compare the proportion in each bucket now, and sum `(now - then) * ln(now / then)`. The conventional reading: under 0.1 is no meaningful shift, 0.1 to 0.25 is moderate, above 0.25 is significant. Those thresholds are convention rather than theory, and they are useful precisely because everyone uses the same ones.\n\n**Kolmogorov-Smirnov test** for continuous features, comparing two distributions. Its problem in production is sample size: with a million rows a week it declares tiny, meaningless differences significant. Look at the test statistic's magnitude and ignore the p-value.\n\n**Chi-squared** for categorical features, with the same caveat.\n\nMy practical recommendation: use PSI for its interpretable thresholds, and always plot the two distributions before acting. A number told you something changed; the plot tells you what.\n\n## The three shifts need different responses\n\nFrom module 3, and this is where the distinction pays off.\n\n**Covariate shift**: inputs moved, the relationship holds. Your users got younger. Detected by input monitoring. Often the model is still fine; check performance before retraining.\n\n**Label shift**: the base rate moved. Fraud went from 1% to 4%. Detected by prediction distribution and, later, by calibration. Recalibration usually fixes it and is much cheaper than retraining.\n\n**Concept drift**: the relationship changed. Fraudsters adapted. Detected only by actual performance, which is why layers 1 to 3 cannot substitute for layer 4. Requires new labelled data and retraining.\n\nAn alert saying \"drift detected\" without saying which of these it is sends the team to the wrong remedy. Name the type in the alert.\n\n## Getting labels\n\nThe hard part of layer 4, and the options are limited:\n\n- **Wait.** For a 30-day outcome, you know last month's performance today. Acceptable, and it means your monitoring is always a month behind.\n- **Sample and label by hand.** 200 predictions a week, reviewed. Expensive and it works, and it gives you a slice-level picture the automatic metrics cannot.\n- **Use a proxy.** Did the user click, did the flagged case get confirmed. Fast and biased, because you only observe outcomes for the cases your model acted on.\n- **Randomise a small slice.** The exploration policy from module 3. The only way to get unbiased labels in the region your model currently excludes, and it costs money.\n\n## Alert on the right things\n\nThe common failure is alerting on everything and being ignored within two weeks.\n\nAlert on: prediction rate moving beyond a stated band, a feature's missing rate jumping, unseen categories exceeding a threshold, and performance falling below the number you committed to.\n\nDo not alert on: every PSI above 0.1, every statistically significant difference, or every slice that looks poor this week. Those go on a weekly report that a human reads, not into a pager.\n\nAnd write down what you would do in response to each alert. An alert with no defined action is not monitoring; it is anxiety with a delivery mechanism."},"check":{"question":"A model's input distributions are unchanged, but predicted positive rate has climbed from 3% to 8% over two months and calibration has degraded while AUC is stable. What has most likely happened?","options":[{"text":"A data quality problem upstream, since a sudden rise in positive predictions usually means a corrupted feature.","correct":false,"tempting":"A corrupted feature would normally register in the input distribution monitoring, which the scenario states is unchanged."},{"text":"Concept drift, since the relationship between features and outcome has changed.","correct":false,"tempting":"Concept drift typically degrades ranking quality, and AUC has held steady, which points away from the relationship itself having changed."},{"text":"Training-serving skew, since the serving features must be computed differently from the training ones.","correct":false,"tempting":"Skew would show as a difference between the two paths on identical rows, and it would usually disturb the input distributions being monitored."},{"text":"Label shift: the base rate of the positive class has risen, which moves the predicted rate and breaks calibration while leaving the ranking, and therefore AUC, intact.","correct":true}]},"prev":{"slug":"training-serving-skew","title":"When training and serving quietly disagree","url":"https://addaly.com/learn/machine-learning-foundations/training-serving-skew"},"next":{"slug":"when-to-retrain","title":"When to retrain, and what breaks when you do","url":"https://addaly.com/learn/machine-learning-foundations/when-to-retrain"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Monitoring: what to watch, and in what order', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/monitoring-what-matters (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Monitoring: what to watch, and in what order","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"Monitor system health, input distributions, prediction distributions and finally actual performance: the first three need no labels and move earlier, and the alert must name which kind of shift it found because covariate, label and concept drift take different remedies.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Four layers of monitoring, cheapest first","layers":[{"label":"1. System health","detail":"Request rate, errors, latency percentiles, memory. Catches a container restarting; misses a model returning confident nonsense."},{"label":"2. Input distributions","detail":"Means, missing rates, unseen categories, against the training data. Where most problems appear first, within hours."},{"label":"3. Prediction distributions","detail":"Mean score, shape, rate above threshold. A flag rate that goes from 2% to 9% overnight is an incident. Needs no labels."},{"label":"4. Actual performance","detail":"Accuracy, AUC, calibration, per slice. What you care about, and the slowest to arrive: 90 days for a default, never for a loan you declined."}],"caption":"The first three need no labels and move earlier; they are early warnings for the fourth, which is the only one that can see concept drift and the last to arrive, because labels lag by weeks or forever."},"alt":"Figure: Four layers of monitoring, cheapest first. Layers, top to bottom: 1. System health (Request rate, errors, latency percentiles, memory. Catches a container restarting; misses a model returning confident nonsense.); 2. Input distributions (Means, missing rates, unseen categories, against the training data. Where most problems appear first, within hours.); 3. Prediction distributions (Mean score, shape, rate above threshold. A flag rate that goes from 2% to 9% overnight is an incident. Needs no labels.); 4. Actual performance (Accuracy, AUC, calibration, per slice. What you care about, and the slowest to arrive: 90 days for a default, never for a loan you declined.). The first three need no labels and move earlier; they are early warnings for the fourth, which is the only one that can see concept drift and the last to arrive, because labels lag by weeks or forever."}},"blocks":[{"t":"h2","id":"four-layers-from-cheapest-to-most-valuable","runs":[{"text":"Four layers, from cheapest to most valuable"}]},{"t":"p","runs":[{"text":"1. System health.","b":true},{"text":" Request rate, error rate, latency percentiles, memory. Standard software monitoring, and it catches the model container restarting every ten minutes. Necessary and not sufficient: a model returning confident nonsense is perfectly healthy by these measures."}]},{"t":"p","runs":[{"text":"2. Input distributions.","b":true},{"text":" For each feature: mean, standard deviation, missing rate, and for categoricals, the set of levels and the rate of previously unseen ones. Compare against the training data."}]},{"t":"p","runs":[{"text":"This is where most problems appear first, because most problems are upstream data problems. A column that becomes 100% null overnight, a currency changing units, a category renamed: all visible here within hours."}]},{"t":"p","runs":[{"text":"3. Prediction distributions.","b":true},{"text":" The mean predicted probability, the distribution’s shape, the rate of predictions above your decision threshold. You do not need labels for any of this, and it moves before your accuracy metrics can even be computed."}]},{"t":"p","runs":[{"text":"A flag rate that jumps from 2% to 9% overnight is an incident. It might be real fraud, or an upstream change, or a broken feature, and you want to know within the hour rather than at the next monthly review."}]},{"t":"p","runs":[{"text":"4. Actual performance.","b":true},{"text":" Accuracy, AUC, calibration, per slice. The thing you truly care about, and the slowest to arrive, because labels lag: sometimes by 90 days for a default, sometimes forever for a loan you declined."}]},{"t":"p","runs":[{"text":"Build all four, and understand that the first three are early warnings for the fourth."}]},{"t":"h2","id":"detecting-a-change-in-a-distribution","runs":[{"text":"Detecting a change in a distribution"}]},{"t":"p","runs":[{"text":"Three standard measures, and the practical advice about each."}]},{"t":"p","runs":[{"text":"Population Stability Index.","b":true},{"text":" The credit industry’s standard. Bucket the feature into deciles based on training data, compare the proportion in each bucket now, and sum "},{"text":"(now - then) * ln(now / then)","code":true},{"text":". The conventional reading: under 0.1 is no meaningful shift, 0.1 to 0.25 is moderate, above 0.25 is significant. Those thresholds are convention rather than theory, and they are useful precisely because everyone uses the same ones."}]},{"t":"p","runs":[{"text":"Kolmogorov-Smirnov test","b":true},{"text":" for continuous features, comparing two distributions. Its problem in production is sample size: with a million rows a week it declares tiny, meaningless differences significant. Look at the test statistic’s magnitude and ignore the p-value."}]},{"t":"p","runs":[{"text":"Chi-squared","b":true},{"text":" for categorical features, with the same caveat."}]},{"t":"p","runs":[{"text":"My practical recommendation: use PSI for its interpretable thresholds, and always plot the two distributions before acting. A number told you something changed; the plot tells you what."}]},{"t":"h2","id":"the-three-shifts-need-different-responses","runs":[{"text":"The three shifts need different responses"}]},{"t":"p","runs":[{"text":"From module 3, and this is where the distinction pays off."}]},{"t":"p","runs":[{"text":"Covariate shift","b":true},{"text":": inputs moved, the relationship holds. Your users got younger. Detected by input monitoring. Often the model is still fine; check performance before retraining."}]},{"t":"p","runs":[{"text":"Label shift","b":true},{"text":": the base rate moved. Fraud went from 1% to 4%. Detected by prediction distribution and, later, by calibration. Recalibration usually fixes it and is much cheaper than retraining."}]},{"t":"p","runs":[{"text":"Concept drift","b":true},{"text":": the relationship changed. Fraudsters adapted. Detected only by actual performance, which is why layers 1 to 3 cannot substitute for layer 4. Requires new labelled data and retraining."}]},{"t":"p","runs":[{"text":"An alert saying “drift detected” without saying which of these it is sends the team to the wrong remedy. Name the type in the alert."}]},{"t":"h2","id":"getting-labels","runs":[{"text":"Getting labels"}]},{"t":"p","runs":[{"text":"The hard part of layer 4, and the options are limited:"}]},{"t":"ul","items":[[{"text":"Wait.","b":true},{"text":" For a 30-day outcome, you know last month’s performance today. Acceptable, and it means your monitoring is always a month behind."}],[{"text":"Sample and label by hand.","b":true},{"text":" 200 predictions a week, reviewed. Expensive and it works, and it gives you a slice-level picture the automatic metrics cannot."}],[{"text":"Use a proxy.","b":true},{"text":" Did the user click, did the flagged case get confirmed. Fast and biased, because you only observe outcomes for the cases your model acted on."}],[{"text":"Randomise a small slice.","b":true},{"text":" The exploration policy from module 3. The only way to get unbiased labels in the region your model currently excludes, and it costs money."}]]},{"t":"h2","id":"alert-on-the-right-things","runs":[{"text":"Alert on the right things"}]},{"t":"p","runs":[{"text":"The common failure is alerting on everything and being ignored within two weeks."}]},{"t":"p","runs":[{"text":"Alert on: prediction rate moving beyond a stated band, a feature’s missing rate jumping, unseen categories exceeding a threshold, and performance falling below the number you committed to."}]},{"t":"p","runs":[{"text":"Do not alert on: every PSI above 0.1, every statistically significant difference, or every slice that looks poor this week. Those go on a weekly report that a human reads, not into a pager."}]},{"t":"p","runs":[{"text":"And write down what you would do in response to each alert. An alert with no defined action is not monitoring; it is anxiety with a delivery mechanism."}]}],"check":{"question":[{"text":"A model’s input distributions are unchanged, but predicted positive rate has climbed from 3% to 8% over two months and calibration has degraded while AUC is stable. What has most likely happened?"}],"options":[{"text":[{"text":"A data quality problem upstream, since a sudden rise in positive predictions usually means a corrupted feature."}],"correct":false,"tempting":[{"text":"A corrupted feature would normally register in the input distribution monitoring, which the scenario states is unchanged."}]},{"text":[{"text":"Concept drift, since the relationship between features and outcome has changed."}],"correct":false,"tempting":[{"text":"Concept drift typically degrades ranking quality, and AUC has held steady, which points away from the relationship itself having changed."}]},{"text":[{"text":"Training-serving skew, since the serving features must be computed differently from the training ones."}],"correct":false,"tempting":[{"text":"Skew would show as a difference between the two paths on identical rows, and it would usually disturb the input distributions being monitored."}]},{"text":[{"text":"Label shift: the base rate of the positive class has risen, which moves the predicted rate and breaks calibration while leaving the ranking, and therefore AUC, intact."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"training-serving-skew","title":"When training and serving quietly disagree","minutes":9},"next":{"slug":"when-to-retrain","title":"When to retrain, and what breaks when you do","minutes":8}}}