Monitoring: what to watch, and in what order

Four layers, from cheapest to most valuable

1. System health. Request rate, error rate, latency percentiles, memory. Standard software monitoring, and it catches the model container restarting every ten minutes. Necessary and not sufficient: a model returning confident nonsense is perfectly healthy by these measures.

2. Input distributions. For each feature: mean, standard deviation, missing rate, and for categoricals, the set of levels and the rate of previously unseen ones. Compare against the training data.

This is where most problems appear first, because most problems are upstream data problems. A column that becomes 100% null overnight, a currency changing units, a category renamed: all visible here within hours.

3. Prediction distributions. The mean predicted probability, the distribution’s shape, the rate of predictions above your decision threshold. You do not need labels for any of this, and it moves before your accuracy metrics can even be computed.

A flag rate that jumps from 2% to 9% overnight is an incident. It might be real fraud, or an upstream change, or a broken feature, and you want to know within the hour rather than at the next monthly review.

4. Actual performance. Accuracy, AUC, calibration, per slice. The thing you truly care about, and the slowest to arrive, because labels lag: sometimes by 90 days for a default, sometimes forever for a loan you declined.

Build all four, and understand that the first three are early warnings for the fourth.

Detecting a change in a distribution

Three standard measures, and the practical advice about each.

Population Stability Index. The credit industry’s standard. Bucket the feature into deciles based on training data, compare the proportion in each bucket now, and sum (now - then) * ln(now / then). The conventional reading: under 0.1 is no meaningful shift, 0.1 to 0.25 is moderate, above 0.25 is significant. Those thresholds are convention rather than theory, and they are useful precisely because everyone uses the same ones.

Kolmogorov-Smirnov test for continuous features, comparing two distributions. Its problem in production is sample size: with a million rows a week it declares tiny, meaningless differences significant. Look at the test statistic’s magnitude and ignore the p-value.

Chi-squared for categorical features, with the same caveat.

My practical recommendation: use PSI for its interpretable thresholds, and always plot the two distributions before acting. A number told you something changed; the plot tells you what.

The three shifts need different responses

From module 3, and this is where the distinction pays off.

Covariate shift: inputs moved, the relationship holds. Your users got younger. Detected by input monitoring. Often the model is still fine; check performance before retraining.

Label shift: the base rate moved. Fraud went from 1% to 4%. Detected by prediction distribution and, later, by calibration. Recalibration usually fixes it and is much cheaper than retraining.

Concept drift: the relationship changed. Fraudsters adapted. Detected only by actual performance, which is why layers 1 to 3 cannot substitute for layer 4. Requires new labelled data and retraining.

An alert saying “drift detected” without saying which of these it is sends the team to the wrong remedy. Name the type in the alert.

Getting labels

The hard part of layer 4, and the options are limited:

• Wait. For a 30-day outcome, you know last month’s performance today. Acceptable, and it means your monitoring is always a month behind.
• Sample and label by hand. 200 predictions a week, reviewed. Expensive and it works, and it gives you a slice-level picture the automatic metrics cannot.
• Use a proxy. Did the user click, did the flagged case get confirmed. Fast and biased, because you only observe outcomes for the cases your model acted on.
• Randomise a small slice. The exploration policy from module 3. The only way to get unbiased labels in the region your model currently excludes, and it costs money.

Alert on the right things

The common failure is alerting on everything and being ignored within two weeks.

Alert on: prediction rate moving beyond a stated band, a feature’s missing rate jumping, unseen categories exceeding a threshold, and performance falling below the number you committed to.

Do not alert on: every PSI above 0.1, every statistically significant difference, or every slice that looks poor this week. Those go on a weekly report that a human reads, not into a pager.

And write down what you would do in response to each alert. An alert with no defined action is not monitoring; it is anxiety with a delivery mechanism.