{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"monitoring-a-tuned-model","title":"Monitoring a model that fails silently","url":"https://addaly.com/learn/fine-tuning/monitoring-a-tuned-model","position":80,"of":84,"minutes":9,"answer":"A drifted model returns a confident wrong answer with a 200 status, so monitoring means unlabelled proxies (embedding distance from the training distribution, response length, refusal and fallback rates.","keyIdea":"A drifted model returns a confident wrong answer with a 200 status, so monitoring means unlabelled proxies (embedding distance from the training distribution, response length, refusal and fallback rates, and above all the rate at which users rephrase) plus a weekly shadow run of the frozen set and every incident turned into a permanent evaluation item."},"body":{"format":"markdown","text":"## The failure has no exception\n\nA web service that breaks returns a 500. A fine-tuned model that has drifted returns a fluent, well-formatted, confident, wrong answer with a 200 status code. Every dashboard stays green.\n\nSo monitoring here is not about uptime. It is about building proxies for quality that fire without anybody labelling anything.\n\n## Input drift, which you can measure exactly\n\nYour model was tuned on a distribution. Production moves: new products, a marketing campaign bringing a different audience, a seasonal pattern, a competitor's outage, a new phrasing people picked up somewhere.\n\nEmbed a rolling sample of production inputs and compare against the training distribution:\n\n```python\ncentroid = train_embeddings.mean(axis=0)\ntoday    = embed(sample_of_today_inputs)\nprint(\"mean cosine to training centroid:\", cos(today, centroid).mean())\nprint(\"share beyond 95th pct of training distances:\",\n      (dist(today, centroid) > p95_train).mean())\n```\n\nThat second number is the useful one. If 5% of training inputs were beyond that threshold by construction, and today 22% are, your traffic has moved somewhere your model was not trained for. This is cheap, requires no labels, and fires before quality complaints do.\n\n## The output-side proxies\n\nAll free, all computable on every request:\n\n- **Response length distribution.** A shift is the earliest sign something changed.\n- **Refusal rate.** Both directions matter: a rise means the model is declining things it should handle; a fall may mean the safety regression from the previous module has crept in through a model update.\n- **Format validity rate.** If constrained decoding is on this is 100% and worth asserting; if not, it is a direct quality signal.\n- **Tool-call rate**, where applicable. A sudden rise or fall points at something.\n- **Fallback rate**, if you route on confidence. This one is doubly useful, because it also detects the calibration shift from the previous module.\n- **Empty or truncated outputs.** Should be near zero. Alert on any rise.\n\n## The user-behaviour proxies, which are the best ones\n\nWhat users do after an answer is the closest free thing to a quality label:\n\n- **Rephrase rate.** The user immediately asks again in different words. This is the single strongest unlabelled signal of a bad answer that exists.\n- **Escalation rate.** They ask for a human.\n- **Abandonment.** They leave without completing the task.\n- **Explicit feedback**, which is sparse and biased towards complaint, but free, and, as the preference lesson noted, it is also KTO training data.\n\nPlot these weekly against your model versions. A step change that lines up with a deployment is your answer.\n\n## The shadow evaluation\n\nRun your frozen held-out set against the production model on a schedule: weekly is usually right.\n\nThis catches what the proxies cannot: a configuration change, a silently updated dependency, a quantisation applied during a deployment, a model that was rolled back and forgotten. It takes minutes and it is the only check that directly measures quality rather than a proxy for it.\n\nAnd label 50 real production items a week. It is an hour of somebody's time and it gives you a running measurement on the current distribution, which no frozen set can provide, because your frozen set is, by construction, ageing.\n\n## From incident to permanent test\n\nEvery production failure becomes an evaluation item, that day, in the locked set.\n\nThis is the habit that compounds more than any other in this course. After a year, your evaluation set contains every way the system has ever failed, and no regression of a kind you have already seen can reach production again. It costs ten minutes per incident and it is the only mechanism that makes the evaluation set get *better* over time rather than staler.\n\n## Alert on the things that mean something\n\nDo not alert on the model's loss, which you are not computing in production. Alert on:\n\n- Input drift beyond a threshold you chose from the training distribution.\n- Rephrase or escalation rate rising more than a stated amount week on week.\n- Shadow evaluation falling more than your noise floor.\n- Format validity or empty-output rate moving at all.\n\nFour alerts, each with a number behind it that somebody chose deliberately. A dashboard nobody has set thresholds on is a dashboard nobody reads."},"check":{"question":"Which production signal is the strongest unlabelled indicator that a tuned model's answers have degraded?","options":[{"text":"Mean response latency, which rises when the model produces longer, less certain answers.","correct":false,"tempting":"Latency tracks length and load rather than correctness, and a confidently wrong short answer is fast."},{"text":"The share of outputs failing schema validation, which indicates the model is losing its trained format.","correct":false,"tempting":"Detects format breakage well but is pinned at 100% valid under constrained decoding, and says nothing about whether the content is right."},{"text":"The explicit thumbs-down rate, which directly captures user dissatisfaction.","correct":false,"tempting":"Is a real signal but an extremely sparse one that most dissatisfied users never produce, so it moves late and noisily."},{"text":"The rate at which users immediately rephrase and resubmit their question.","correct":true}]},"prev":{"slug":"cost-at-steady-state","title":"The cost at steady state","url":"https://addaly.com/learn/fine-tuning/cost-at-steady-state"},"next":{"slug":"the-base-model-moved","title":"The day the base model moves","url":"https://addaly.com/learn/fine-tuning/the-base-model-moved"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Monitoring a model that fails silently', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/monitoring-a-tuned-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Monitoring a model that fails silently","courseTitle":"Fine-Tuning, and When Not To","module":{"index":9,"slug":"shipping-and-living-with-it","title":"Shipping it, and living with it"},"answer":null,"keyIdea":"A drifted model returns a confident wrong answer with a 200 status, so monitoring means unlabelled proxies (embedding distance from the training distribution, response length, refusal and fallback rates, and above all the rate at which users rephrase) plus a weekly shadow run of the frozen set and every incident turned into a permanent evaluation item.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-failure-has-no-exception","runs":[{"text":"The failure has no exception"}]},{"t":"p","runs":[{"text":"A web service that breaks returns a 500. A fine-tuned model that has drifted returns a fluent, well-formatted, confident, wrong answer with a 200 status code. Every dashboard stays green."}]},{"t":"p","runs":[{"text":"So monitoring here is not about uptime. It is about building proxies for quality that fire without anybody labelling anything."}]},{"t":"h2","id":"input-drift-which-you-can-measure-exactly","runs":[{"text":"Input drift, which you can measure exactly"}]},{"t":"p","runs":[{"text":"Your model was tuned on a distribution. Production moves: new products, a marketing campaign bringing a different audience, a seasonal pattern, a competitor’s outage, a new phrasing people picked up somewhere."}]},{"t":"p","runs":[{"text":"Embed a rolling sample of production inputs and compare against the training distribution:"}]},{"t":"code","lang":"python","label":"Python","text":"centroid = train_embeddings.mean(axis=0)\ntoday    = embed(sample_of_today_inputs)\nprint(\"mean cosine to training centroid:\", cos(today, centroid).mean())\nprint(\"share beyond 95th pct of training distances:\",\n      (dist(today, centroid) > p95_train).mean())"},{"t":"p","runs":[{"text":"That second number is the useful one. If 5% of training inputs were beyond that threshold by construction, and today 22% are, your traffic has moved somewhere your model was not trained for. This is cheap, requires no labels, and fires before quality complaints do."}]},{"t":"h2","id":"the-output-side-proxies","runs":[{"text":"The output-side proxies"}]},{"t":"p","runs":[{"text":"All free, all computable on every request:"}]},{"t":"ul","items":[[{"text":"Response length distribution.","b":true},{"text":" A shift is the earliest sign something changed."}],[{"text":"Refusal rate.","b":true},{"text":" Both directions matter: a rise means the model is declining things it should handle; a fall may mean the safety regression from the previous module has crept in through a model update."}],[{"text":"Format validity rate.","b":true},{"text":" If constrained decoding is on this is 100% and worth asserting; if not, it is a direct quality signal."}],[{"text":"Tool-call rate","b":true},{"text":", where applicable. A sudden rise or fall points at something."}],[{"text":"Fallback rate","b":true},{"text":", if you route on confidence. This one is doubly useful, because it also detects the calibration shift from the previous module."}],[{"text":"Empty or truncated outputs.","b":true},{"text":" Should be near zero. Alert on any rise."}]]},{"t":"h2","id":"the-user-behaviour-proxies-which-are-the-best-ones","runs":[{"text":"The user-behaviour proxies, which are the best ones"}]},{"t":"p","runs":[{"text":"What users do after an answer is the closest free thing to a quality label:"}]},{"t":"ul","items":[[{"text":"Rephrase rate.","b":true},{"text":" The user immediately asks again in different words. This is the single strongest unlabelled signal of a bad answer that exists."}],[{"text":"Escalation rate.","b":true},{"text":" They ask for a human."}],[{"text":"Abandonment.","b":true},{"text":" They leave without completing the task."}],[{"text":"Explicit feedback","b":true},{"text":", which is sparse and biased towards complaint, but free, and, as the preference lesson noted, it is also KTO training data."}]]},{"t":"p","runs":[{"text":"Plot these weekly against your model versions. A step change that lines up with a deployment is your answer."}]},{"t":"h2","id":"the-shadow-evaluation","runs":[{"text":"The shadow evaluation"}]},{"t":"p","runs":[{"text":"Run your frozen held-out set against the production model on a schedule: weekly is usually right."}]},{"t":"p","runs":[{"text":"This catches what the proxies cannot: a configuration change, a silently updated dependency, a quantisation applied during a deployment, a model that was rolled back and forgotten. It takes minutes and it is the only check that directly measures quality rather than a proxy for it."}]},{"t":"p","runs":[{"text":"And label 50 real production items a week. It is an hour of somebody’s time and it gives you a running measurement on the current distribution, which no frozen set can provide, because your frozen set is, by construction, ageing."}]},{"t":"h2","id":"from-incident-to-permanent-test","runs":[{"text":"From incident to permanent test"}]},{"t":"p","runs":[{"text":"Every production failure becomes an evaluation item, that day, in the locked set."}]},{"t":"p","runs":[{"text":"This is the habit that compounds more than any other in this course. After a year, your evaluation set contains every way the system has ever failed, and no regression of a kind you have already seen can reach production again. It costs ten minutes per incident and it is the only mechanism that makes the evaluation set get "},{"text":"better","i":true},{"text":" over time rather than staler."}]},{"t":"h2","id":"alert-on-the-things-that-mean-something","runs":[{"text":"Alert on the things that mean something"}]},{"t":"p","runs":[{"text":"Do not alert on the model’s loss, which you are not computing in production. Alert on:"}]},{"t":"ul","items":[[{"text":"Input drift beyond a threshold you chose from the training distribution."}],[{"text":"Rephrase or escalation rate rising more than a stated amount week on week."}],[{"text":"Shadow evaluation falling more than your noise floor."}],[{"text":"Format validity or empty-output rate moving at all."}]]},{"t":"p","runs":[{"text":"Four alerts, each with a number behind it that somebody chose deliberately. A dashboard nobody has set thresholds on is a dashboard nobody reads."}]}],"check":{"question":[{"text":"Which production signal is the strongest unlabelled indicator that a tuned model’s answers have degraded?"}],"options":[{"text":[{"text":"Mean response latency, which rises when the model produces longer, less certain answers."}],"correct":false,"tempting":[{"text":"Latency tracks length and load rather than correctness, and a confidently wrong short answer is fast."}]},{"text":[{"text":"The share of outputs failing schema validation, which indicates the model is losing its trained format."}],"correct":false,"tempting":[{"text":"Detects format breakage well but is pinned at 100% valid under constrained decoding, and says nothing about whether the content is right."}]},{"text":[{"text":"The explicit thumbs-down rate, which directly captures user dissatisfaction."}],"correct":false,"tempting":[{"text":"Is a real signal but an extremely sparse one that most dissatisfied users never produce, so it moves late and noisily."}]},{"text":[{"text":"The rate at which users immediately rephrase and resubmit their question."}],"correct":true,"tempting":null}],"order":[1,2,3,0],"why":null},"prev":{"slug":"cost-at-steady-state","title":"The cost at steady state","minutes":9},"next":{"slug":"the-base-model-moved","title":"The day the base model moves","minutes":9}}}