Watching a system you can no longer test

Offline sets go stale; traffic does not

Your eval set was built from last quarter’s inputs. Production is receiving this morning’s. Between them sit a new customer segment, a marketing campaign that brought a different kind of question, a festival, a competitor’s outage, and a provider who updated a model without telling you.

Monitoring is the part of evaluation that runs continuously, and its job is narrower than people assume. It is not to measure quality: you cannot, without labels. It is to detect change fast enough to go and look.

Log this, per request

The minimum that makes everything else possible:

• request id, session id, turn index, timestamp
• model id including the exact version string, prompt version, retrieval config version
• input length, detected language, and the slice tags you defined
• output length, finish reason, refusal flag
• the result of every deterministic check
• latency (queue, first token, total), tokens in and out, cost
• downstream signal: regenerated, escalated, edited, abandoned

Store raw input and output text under a retention policy, redacted or hashed where the field is personal, with a stated window: 30 days is a common and defensible choice. Some of the most useful debugging is impossible without the text, and some of the worst incidents are caused by keeping it forever. Decide deliberately, write it down, and tell the people whose text it is.

Distribution shift, on both sides

You can watch the inputs and the outputs without any labels at all, and it is remarkable how much this catches.

Input distribution. Mean and p95 length. Language mix. Share per slice tag. Cluster share, if you embed a sample and track how the clusters divide. A sudden change means your users changed, and your eval set no longer represents them.

Output distribution. These are the leading indicators, and they move before anybody complains:

• refusal rate: the single most sensitive detector of a silent model change
• schema failure rate
• mean output length
• tool-call rate and step-limit-hit rate, for agents
• retrieval hit rate at your k
• p95 latency and cost per request

A refusal rate moving from 2% to 9% overnight, with no deployment on your side, is a provider-side change, and you will know within an hour instead of finding out from a customer in three weeks.

Alert on shape, not on a single reading

Rates are noisy at low volume. Three rules keep the alerts trustworthy:

• Compare against the same hour last week, not against the last hour. Traffic has a daily and weekly shape, and comparing to an hour ago generates an alert every Monday morning.
• Require a minimum denominator. A refusal rate computed on 12 requests is not a signal.
• Alert on a sustained change (say, three consecutive windows outside the band) rather than a single spike.

A simple band of mean plus or minus three standard deviations over the last four weeks of the same weekday-hour is usually enough. Fancier change-point detection is available and rarely necessary.

Sample the expensive checks

Deterministic checks are free; run them on 100% of traffic. Judge calls are not; run them on a sample (1 to 5% is typical) stratified so that rare slices are represented, and remember when reading the number that it is stratified.

A judge running on 2% of a million requests is 20,000 judged items a day, which is far more than any offline set, and it is enough to detect a genuine quality change within a day. That is a good use of a few dollars.

Pin the version, and watch for the swap

The recurring incident in this course is the model changing under you. Two defences, and you want both.

First, pin exact version strings everywhere: the system’s model, the judge’s model, the embedding model. Aliases like -latest are convenient and they are how a Tuesday becomes interesting.

Second, because pinning is not always possible, and because pinned models are deprecated on a schedule, keep a canary set: 20 items, run hourly, with outputs stored. When outputs that were stable for weeks all change on the same afternoon, something moved that nobody told you about. Twenty items an hour is about 14,400 calls a month: a few dollars on a small model with short prompts, tens of dollars on a mid-tier or frontier model. That is still the cheapest early warning available.

The two-line rule for dashboards

A quality dashboard nobody looks at is a cost. Keep the daily view to what a person can absorb in ten seconds: today’s numbers against last week’s, and a list of the alerts that fired. Everything else lives in a query you run when something looks wrong.

And give it an owner. A metric with no owner drifts out of date, then breaks, then is ignored.