{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"monitoring-in-production","title":"Watching a system you can no longer test","url":"https://addaly.com/learn/evaluating-ai/monitoring-in-production","position":70,"of":76,"minutes":9,"answer":"Without labels you can still watch the input and output distributions, and refusal rate is the fastest detector of a model that changed under you.","keyIdea":"Without labels you can still watch the input and output distributions, and refusal rate is the fastest detector of a model that changed under you."},"body":{"format":"markdown","text":"## Offline sets go stale; traffic does not\n\nYour eval set was built from last quarter's inputs. Production is receiving this morning's. Between them sit a new customer segment, a marketing campaign that brought a different kind of question, a festival, a competitor's outage, and a provider who updated a model without telling you.\n\nMonitoring is the part of evaluation that runs continuously, and its job is narrower than people assume. It is not to measure quality: you cannot, without labels. It is to **detect change fast enough to go and look**.\n\n## Log this, per request\n\nThe minimum that makes everything else possible:\n\n- request id, session id, turn index, timestamp\n- **model id including the exact version string**, prompt version, retrieval config version\n- input length, detected language, and the slice tags you defined\n- output length, finish reason, refusal flag\n- the result of every deterministic check\n- latency (queue, first token, total), tokens in and out, cost\n- downstream signal: regenerated, escalated, edited, abandoned\n\nStore raw input and output text under a retention policy, redacted or hashed where the field is personal, with a stated window: 30 days is a common and defensible choice. Some of the most useful debugging is impossible without the text, and some of the worst incidents are caused by keeping it forever. Decide deliberately, write it down, and tell the people whose text it is.\n\n## Distribution shift, on both sides\n\nYou can watch the inputs and the outputs without any labels at all, and it is remarkable how much this catches.\n\n**Input distribution.** Mean and p95 length. Language mix. Share per slice tag. Cluster share, if you embed a sample and track how the clusters divide. A sudden change means your users changed, and your eval set no longer represents them.\n\n**Output distribution.** These are the leading indicators, and they move before anybody complains:\n\n- **refusal rate**: the single most sensitive detector of a silent model change\n- **schema failure rate**\n- **mean output length**\n- **tool-call rate** and **step-limit-hit rate**, for agents\n- **retrieval hit rate** at your k\n- **p95 latency** and cost per request\n\nA refusal rate moving from 2% to 9% overnight, with no deployment on your side, is a provider-side change, and you will know within an hour instead of finding out from a customer in three weeks.\n\n## Alert on shape, not on a single reading\n\nRates are noisy at low volume. Three rules keep the alerts trustworthy:\n\n- Compare against the same hour last week, not against the last hour. Traffic has a daily and weekly shape, and comparing to an hour ago generates an alert every Monday morning.\n- Require a minimum denominator. A refusal rate computed on 12 requests is not a signal.\n- Alert on a sustained change (say, three consecutive windows outside the band) rather than a single spike.\n\nA simple band of mean plus or minus three standard deviations over the last four weeks of the same weekday-hour is usually enough. Fancier change-point detection is available and rarely necessary.\n\n## Sample the expensive checks\n\nDeterministic checks are free; run them on 100% of traffic. Judge calls are not; run them on a sample (1 to 5% is typical) stratified so that rare slices are represented, and remember when reading the number that it is stratified.\n\nA judge running on 2% of a million requests is 20,000 judged items a day, which is far more than any offline set, and it is enough to detect a genuine quality change within a day. That is a good use of a few dollars.\n\n## Pin the version, and watch for the swap\n\nThe recurring incident in this course is the model changing under you. Two defences, and you want both.\n\nFirst, pin exact version strings everywhere: the system's model, the judge's model, the embedding model. Aliases like `-latest` are convenient and they are how a Tuesday becomes interesting.\n\nSecond, because pinning is not always possible, and because pinned models are deprecated on a schedule, keep a **canary set**: 20 items, run hourly, with outputs stored. When outputs that were stable for weeks all change on the same afternoon, something moved that nobody told you about. Twenty items an hour is about 14,400 calls a month: a few dollars on a small model with short prompts, tens of dollars on a mid-tier or frontier model. That is still the cheapest early warning available.\n\n## The two-line rule for dashboards\n\nA quality dashboard nobody looks at is a cost. Keep the daily view to what a person can absorb in ten seconds: today's numbers against last week's, and a list of the alerts that fired. Everything else lives in a query you run when something looks wrong.\n\nAnd give it an owner. A metric with no owner drifts out of date, then breaks, then is ignored."},"check":{"question":"With no deployment on your side, refusal rate rises from 2% to 9% and mean output length falls 30% over one afternoon. Quality complaints have not yet appeared. What is the most likely cause?","options":[{"text":"A traffic shift: new users asking more out-of-scope questions, which would raise refusals legitimately.","correct":false,"tempting":"Is a real possibility worth ruling out, but it would not usually shorten every output at the same time on the same afternoon."},{"text":"Retrieval has degraded, so the system has less context and declines more often.","correct":false,"tempting":"Names a plausible internal cause that a retrieval hit-rate metric would show directly, and which a sudden step change fits poorly."},{"text":"The provider has changed the model behind an unpinned alias, and both metrics moved together because the underlying system changed.","correct":true},{"text":"The refusal detector itself is miscounting, since a 4.5x change in a day is implausible for real behaviour.","correct":false,"tempting":"Doubts the instrument, which is worth a minute's check, but dismisses a step change of exactly the size a model swap produces."}]},"prev":{"slug":"latency-is-a-distribution","title":"The tail is what your users feel","url":"https://addaly.com/learn/evaluating-ai/latency-is-a-distribution"},"next":{"slug":"feedback-loops-and-self-poisoning","title":"The system that trains on its own output","url":"https://addaly.com/learn/evaluating-ai/feedback-loops-and-self-poisoning"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Watching a system you can no longer test', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/monitoring-in-production (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Watching a system you can no longer test","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Without labels you can still watch the input and output distributions, and refusal rate is the fastest detector of a model that changed under you.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"offline-sets-go-stale-traffic-does-not","runs":[{"text":"Offline sets go stale; traffic does not"}]},{"t":"p","runs":[{"text":"Your eval set was built from last quarter’s inputs. Production is receiving this morning’s. Between them sit a new customer segment, a marketing campaign that brought a different kind of question, a festival, a competitor’s outage, and a provider who updated a model without telling you."}]},{"t":"p","runs":[{"text":"Monitoring is the part of evaluation that runs continuously, and its job is narrower than people assume. It is not to measure quality: you cannot, without labels. It is to "},{"text":"detect change fast enough to go and look","b":true},{"text":"."}]},{"t":"h2","id":"log-this-per-request","runs":[{"text":"Log this, per request"}]},{"t":"p","runs":[{"text":"The minimum that makes everything else possible:"}]},{"t":"ul","items":[[{"text":"request id, session id, turn index, timestamp"}],[{"text":"model id including the exact version string","b":true},{"text":", prompt version, retrieval config version"}],[{"text":"input length, detected language, and the slice tags you defined"}],[{"text":"output length, finish reason, refusal flag"}],[{"text":"the result of every deterministic check"}],[{"text":"latency (queue, first token, total), tokens in and out, cost"}],[{"text":"downstream signal: regenerated, escalated, edited, abandoned"}]]},{"t":"p","runs":[{"text":"Store raw input and output text under a retention policy, redacted or hashed where the field is personal, with a stated window: 30 days is a common and defensible choice. Some of the most useful debugging is impossible without the text, and some of the worst incidents are caused by keeping it forever. Decide deliberately, write it down, and tell the people whose text it is."}]},{"t":"h2","id":"distribution-shift-on-both-sides","runs":[{"text":"Distribution shift, on both sides"}]},{"t":"p","runs":[{"text":"You can watch the inputs and the outputs without any labels at all, and it is remarkable how much this catches."}]},{"t":"p","runs":[{"text":"Input distribution.","b":true},{"text":" Mean and p95 length. Language mix. Share per slice tag. Cluster share, if you embed a sample and track how the clusters divide. A sudden change means your users changed, and your eval set no longer represents them."}]},{"t":"p","runs":[{"text":"Output distribution.","b":true},{"text":" These are the leading indicators, and they move before anybody complains:"}]},{"t":"ul","items":[[{"text":"refusal rate","b":true},{"text":": the single most sensitive detector of a silent model change"}],[{"text":"schema failure rate","b":true}],[{"text":"mean output length","b":true}],[{"text":"tool-call rate","b":true},{"text":" and "},{"text":"step-limit-hit rate","b":true},{"text":", for agents"}],[{"text":"retrieval hit rate","b":true},{"text":" at your k"}],[{"text":"p95 latency","b":true},{"text":" and cost per request"}]]},{"t":"p","runs":[{"text":"A refusal rate moving from 2% to 9% overnight, with no deployment on your side, is a provider-side change, and you will know within an hour instead of finding out from a customer in three weeks."}]},{"t":"h2","id":"alert-on-shape-not-on-a-single-reading","runs":[{"text":"Alert on shape, not on a single reading"}]},{"t":"p","runs":[{"text":"Rates are noisy at low volume. Three rules keep the alerts trustworthy:"}]},{"t":"ul","items":[[{"text":"Compare against the same hour last week, not against the last hour. Traffic has a daily and weekly shape, and comparing to an hour ago generates an alert every Monday morning."}],[{"text":"Require a minimum denominator. A refusal rate computed on 12 requests is not a signal."}],[{"text":"Alert on a sustained change (say, three consecutive windows outside the band) rather than a single spike."}]]},{"t":"p","runs":[{"text":"A simple band of mean plus or minus three standard deviations over the last four weeks of the same weekday-hour is usually enough. Fancier change-point detection is available and rarely necessary."}]},{"t":"h2","id":"sample-the-expensive-checks","runs":[{"text":"Sample the expensive checks"}]},{"t":"p","runs":[{"text":"Deterministic checks are free; run them on 100% of traffic. Judge calls are not; run them on a sample (1 to 5% is typical) stratified so that rare slices are represented, and remember when reading the number that it is stratified."}]},{"t":"p","runs":[{"text":"A judge running on 2% of a million requests is 20,000 judged items a day, which is far more than any offline set, and it is enough to detect a genuine quality change within a day. That is a good use of a few dollars."}]},{"t":"h2","id":"pin-the-version-and-watch-for-the-swap","runs":[{"text":"Pin the version, and watch for the swap"}]},{"t":"p","runs":[{"text":"The recurring incident in this course is the model changing under you. Two defences, and you want both."}]},{"t":"p","runs":[{"text":"First, pin exact version strings everywhere: the system’s model, the judge’s model, the embedding model. Aliases like "},{"text":"-latest","code":true},{"text":" are convenient and they are how a Tuesday becomes interesting."}]},{"t":"p","runs":[{"text":"Second, because pinning is not always possible, and because pinned models are deprecated on a schedule, keep a "},{"text":"canary set","b":true},{"text":": 20 items, run hourly, with outputs stored. When outputs that were stable for weeks all change on the same afternoon, something moved that nobody told you about. Twenty items an hour is about 14,400 calls a month: a few dollars on a small model with short prompts, tens of dollars on a mid-tier or frontier model. That is still the cheapest early warning available."}]},{"t":"h2","id":"the-two-line-rule-for-dashboards","runs":[{"text":"The two-line rule for dashboards"}]},{"t":"p","runs":[{"text":"A quality dashboard nobody looks at is a cost. Keep the daily view to what a person can absorb in ten seconds: today’s numbers against last week’s, and a list of the alerts that fired. Everything else lives in a query you run when something looks wrong."}]},{"t":"p","runs":[{"text":"And give it an owner. A metric with no owner drifts out of date, then breaks, then is ignored."}]}],"check":{"question":[{"text":"With no deployment on your side, refusal rate rises from 2% to 9% and mean output length falls 30% over one afternoon. Quality complaints have not yet appeared. What is the most likely cause?"}],"options":[{"text":[{"text":"A traffic shift: new users asking more out-of-scope questions, which would raise refusals legitimately."}],"correct":false,"tempting":[{"text":"Is a real possibility worth ruling out, but it would not usually shorten every output at the same time on the same afternoon."}]},{"text":[{"text":"Retrieval has degraded, so the system has less context and declines more often."}],"correct":false,"tempting":[{"text":"Names a plausible internal cause that a retrieval hit-rate metric would show directly, and which a sudden step change fits poorly."}]},{"text":[{"text":"The provider has changed the model behind an unpinned alias, and both metrics moved together because the underlying system changed."}],"correct":true,"tempting":null},{"text":[{"text":"The refusal detector itself is miscounting, since a 4.5x change in a day is implausible for real behaviour."}],"correct":false,"tempting":[{"text":"Doubts the instrument, which is worth a minute’s check, but dismisses a step change of exactly the size a model swap produces."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"latency-is-a-distribution","title":"The tail is what your users feel","minutes":9},"next":{"slug":"feedback-loops-and-self-poisoning","title":"The system that trains on its own output","minutes":9}}}