Monitoring a model that fails silently

The failure has no exception

A web service that breaks returns a 500. A fine-tuned model that has drifted returns a fluent, well-formatted, confident, wrong answer with a 200 status code. Every dashboard stays green.

So monitoring here is not about uptime. It is about building proxies for quality that fire without anybody labelling anything.

Input drift, which you can measure exactly

Your model was tuned on a distribution. Production moves: new products, a marketing campaign bringing a different audience, a seasonal pattern, a competitor’s outage, a new phrasing people picked up somewhere.

Embed a rolling sample of production inputs and compare against the training distribution:

centroid = train_embeddings.mean(axis=0)
today    = embed(sample_of_today_inputs)
print("mean cosine to training centroid:", cos(today, centroid).mean())
print("share beyond 95th pct of training distances:",
      (dist(today, centroid) > p95_train).mean())

That second number is the useful one. If 5% of training inputs were beyond that threshold by construction, and today 22% are, your traffic has moved somewhere your model was not trained for. This is cheap, requires no labels, and fires before quality complaints do.

The output-side proxies

All free, all computable on every request:

• Response length distribution. A shift is the earliest sign something changed.
• Refusal rate. Both directions matter: a rise means the model is declining things it should handle; a fall may mean the safety regression from the previous module has crept in through a model update.
• Format validity rate. If constrained decoding is on this is 100% and worth asserting; if not, it is a direct quality signal.
• Tool-call rate, where applicable. A sudden rise or fall points at something.
• Fallback rate, if you route on confidence. This one is doubly useful, because it also detects the calibration shift from the previous module.
• Empty or truncated outputs. Should be near zero. Alert on any rise.

The user-behaviour proxies, which are the best ones

What users do after an answer is the closest free thing to a quality label:

• Rephrase rate. The user immediately asks again in different words. This is the single strongest unlabelled signal of a bad answer that exists.
• Escalation rate. They ask for a human.
• Abandonment. They leave without completing the task.
• Explicit feedback, which is sparse and biased towards complaint, but free, and, as the preference lesson noted, it is also KTO training data.

Plot these weekly against your model versions. A step change that lines up with a deployment is your answer.

The shadow evaluation

Run your frozen held-out set against the production model on a schedule: weekly is usually right.

This catches what the proxies cannot: a configuration change, a silently updated dependency, a quantisation applied during a deployment, a model that was rolled back and forgotten. It takes minutes and it is the only check that directly measures quality rather than a proxy for it.

And label 50 real production items a week. It is an hour of somebody’s time and it gives you a running measurement on the current distribution, which no frozen set can provide, because your frozen set is, by construction, ageing.

From incident to permanent test

Every production failure becomes an evaluation item, that day, in the locked set.

This is the habit that compounds more than any other in this course. After a year, your evaluation set contains every way the system has ever failed, and no regression of a kind you have already seen can reach production again. It costs ten minutes per incident and it is the only mechanism that makes the evaluation set get better over time rather than staler.

Alert on the things that mean something

Do not alert on the model’s loss, which you are not computing in production. Alert on:

• Input drift beyond a threshold you chose from the training distribution.
• Rephrase or escalation rate rising more than a stated amount week on week.
• Shadow evaluation falling more than your noise floor.
• Format validity or empty-output rate moving at all.

Four alerts, each with a number behind it that somebody chose deliberately. A dashboard nobody has set thresholds on is a dashboard nobody reads.