{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"when-to-retrain","title":"When to retrain, and what breaks when you do","url":"https://addaly.com/learn/machine-learning-foundations/when-to-retrain","position":87,"of":92,"minutes":8,"answer":"Retraining is a deployment and needs the same gates as one, including re-choosing the decision threshold for the new score distribution, and it is the correct remedy only when the relationship has genuinely changed.","keyIdea":"Retraining is a deployment and needs the same gates as one, including re-choosing the decision threshold for the new score distribution, and it is the correct remedy only when the relationship has genuinely changed, not for upstream data faults or serving bugs."},"body":{"format":"markdown","text":"## Three policies\n\n**Scheduled.** Retrain every week, or month, or quarter, regardless. Predictable, easy to automate, and it does work nobody asked for and sometimes misses a fast change between runs.\n\n**Triggered.** Retrain when a monitored metric crosses a line: performance below a threshold, PSI above 0.25, a flag rate outside its band. Responsive, and it requires the monitoring to be trustworthy first.\n\n**Continuous.** Update the model incrementally as data arrives. Necessary for fast-moving problems such as ad ranking. It also makes debugging much harder, because \"which model made this prediction\" has a different answer every hour.\n\nMost teams should start scheduled (monthly is a reasonable default) and add triggers once the monitoring has earned trust. Continuous is a commitment, not an upgrade.\n\n## How much history to use\n\nA real decision with no universal answer.\n\n**All of it** if the relationship is stable. More data, better estimates.\n\n**A rolling window** (the last 12 months) if the world changes. Old data actively misleads when behaviour has moved on.\n\n**Weighted**: all of it, with recent rows weighted more. The middle path, and often the best: `sample_weight` decaying with age gives you both the volume and the recency.\n\nDecide it empirically. Train on the last 3, 6, 12 and 24 months, evaluate each on the most recent held-out period, and let the numbers choose. This experiment takes an hour and teams routinely guess instead.\n\n## Retraining is a deployment\n\nThe most common production mistake in this area is treating a retrained model as automatically safe because \"it is the same code with newer data\". It is a new model with different behaviour and it needs the same gates as any deploy.\n\n**The minimum checks before a retrained model goes live:**\n\n1. **Performance on a fixed reference set** (a held-out period that never changes) compared against the current production model. If the new one is worse, stop.\n2. **Prediction distribution comparison.** Score the last week of production traffic with both models. A large shift in the distribution needs an explanation before it needs a deployment.\n3. **Slice checks.** From module 5. A model that improves overall while collapsing on one group must not ship silently.\n4. **The fixed test sets** you built from past failure modes.\n5. **Shadow mode** for a period, if the stakes justify it.\n\nAutomate all five. A retraining pipeline that trains and deploys without gates will eventually deploy a model trained on a corrupted extract, and it will do so at 4am on a Sunday.\n\n## The threshold moves too\n\nEasy to forget: your decision threshold was chosen for the old model's score distribution. A retrained model's scores are distributed differently, so the same threshold produces a different flag rate.\n\nRe-choose the threshold on validation data as part of every retraining, using the same cost or capacity criterion from module 5. Otherwise the model improves and the system behaves worse, which is a confusing failure to debug and a common one.\n\n## Keep the old model\n\nRollback should take minutes, which means the previous model artefact stays available and the serving layer can point at either. This is the same discipline as any deploy and it is the single thing most likely to save you.\n\nAlso keep a record: which model version made which prediction. When someone asks in March why a decision was made in January, you need to know which model was live and what it was trained on. In regulated settings this is a requirement rather than a convenience.\n\n## The honest counterweight\n\nRetraining is not always the answer, and it is reached for reflexively.\n\nIf performance dropped because an upstream table changed, retraining teaches the model the broken data. If it dropped because the population shifted into a region you have no examples of, retraining on data that lacks that region changes nothing. If it dropped because of a serving bug, retraining hides the symptom while the bug remains.\n\n**Diagnose before retraining.** Look at the errors, check the inputs, compare the two serving paths. Retraining is a remedy for one specific cause (the relationship in the data has changed and you now have labels reflecting that) and applying it to any other cause wastes a week and leaves the fault in place."},"check":{"question":"After an automated monthly retrain, a fraud model's offline AUC improves slightly but the number of cases sent for review triples overnight. What is the most likely cause?","options":[{"text":"The new model is overfitting, so it flags many more cases while scoring well offline.","correct":false,"tempting":"Overfitting would usually show as a worse offline score on held-out data, and the offline metric improved."},{"text":"The retraining window included a period of elevated fraud, so the model learned a higher base rate.","correct":false,"tempting":"A higher learned base rate would shift scores upward, but the review volume is set by the threshold applied to those scores rather than by the model alone."},{"text":"The threshold was carried over from the previous model, but the new model's scores are distributed differently, so the same cut point now selects a much larger fraction of traffic.","correct":true},{"text":"Training-serving skew was introduced by the retrain, so serving features no longer match training features.","correct":false,"tempting":"Skew is a real hazard but would normally degrade the offline-to-production relationship rather than appear as a clean tripling of volume with an improved offline score."}]},"prev":{"slug":"monitoring-what-matters","title":"Monitoring: what to watch, and in what order","url":"https://addaly.com/learn/machine-learning-foundations/monitoring-what-matters"},"next":{"slug":"reproducibility","title":"Being able to build the same model twice","url":"https://addaly.com/learn/machine-learning-foundations/reproducibility"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When to retrain, and what breaks when you do', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/when-to-retrain (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When to retrain, and what breaks when you do","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"Retraining is a deployment and needs the same gates as one, including re-choosing the decision threshold for the new score distribution, and it is the correct remedy only when the relationship has genuinely changed, not for upstream data faults or serving bugs.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-policies","runs":[{"text":"Three policies"}]},{"t":"p","runs":[{"text":"Scheduled.","b":true},{"text":" Retrain every week, or month, or quarter, regardless. Predictable, easy to automate, and it does work nobody asked for and sometimes misses a fast change between runs."}]},{"t":"p","runs":[{"text":"Triggered.","b":true},{"text":" Retrain when a monitored metric crosses a line: performance below a threshold, PSI above 0.25, a flag rate outside its band. Responsive, and it requires the monitoring to be trustworthy first."}]},{"t":"p","runs":[{"text":"Continuous.","b":true},{"text":" Update the model incrementally as data arrives. Necessary for fast-moving problems such as ad ranking. It also makes debugging much harder, because “which model made this prediction” has a different answer every hour."}]},{"t":"p","runs":[{"text":"Most teams should start scheduled (monthly is a reasonable default) and add triggers once the monitoring has earned trust. Continuous is a commitment, not an upgrade."}]},{"t":"h2","id":"how-much-history-to-use","runs":[{"text":"How much history to use"}]},{"t":"p","runs":[{"text":"A real decision with no universal answer."}]},{"t":"p","runs":[{"text":"All of it","b":true},{"text":" if the relationship is stable. More data, better estimates."}]},{"t":"p","runs":[{"text":"A rolling window","b":true},{"text":" (the last 12 months) if the world changes. Old data actively misleads when behaviour has moved on."}]},{"t":"p","runs":[{"text":"Weighted","b":true},{"text":": all of it, with recent rows weighted more. The middle path, and often the best: "},{"text":"sample_weight","code":true},{"text":" decaying with age gives you both the volume and the recency."}]},{"t":"p","runs":[{"text":"Decide it empirically. Train on the last 3, 6, 12 and 24 months, evaluate each on the most recent held-out period, and let the numbers choose. This experiment takes an hour and teams routinely guess instead."}]},{"t":"h2","id":"retraining-is-a-deployment","runs":[{"text":"Retraining is a deployment"}]},{"t":"p","runs":[{"text":"The most common production mistake in this area is treating a retrained model as automatically safe because “it is the same code with newer data”. It is a new model with different behaviour and it needs the same gates as any deploy."}]},{"t":"p","runs":[{"text":"The minimum checks before a retrained model goes live:","b":true}]},{"t":"ol","start":1,"items":[[{"text":"Performance on a fixed reference set","b":true},{"text":" (a held-out period that never changes) compared against the current production model. If the new one is worse, stop."}],[{"text":"Prediction distribution comparison.","b":true},{"text":" Score the last week of production traffic with both models. A large shift in the distribution needs an explanation before it needs a deployment."}],[{"text":"Slice checks.","b":true},{"text":" From module 5. A model that improves overall while collapsing on one group must not ship silently."}],[{"text":"The fixed test sets","b":true},{"text":" you built from past failure modes."}],[{"text":"Shadow mode","b":true},{"text":" for a period, if the stakes justify it."}]]},{"t":"p","runs":[{"text":"Automate all five. A retraining pipeline that trains and deploys without gates will eventually deploy a model trained on a corrupted extract, and it will do so at 4am on a Sunday."}]},{"t":"h2","id":"the-threshold-moves-too","runs":[{"text":"The threshold moves too"}]},{"t":"p","runs":[{"text":"Easy to forget: your decision threshold was chosen for the old model’s score distribution. A retrained model’s scores are distributed differently, so the same threshold produces a different flag rate."}]},{"t":"p","runs":[{"text":"Re-choose the threshold on validation data as part of every retraining, using the same cost or capacity criterion from module 5. Otherwise the model improves and the system behaves worse, which is a confusing failure to debug and a common one."}]},{"t":"h2","id":"keep-the-old-model","runs":[{"text":"Keep the old model"}]},{"t":"p","runs":[{"text":"Rollback should take minutes, which means the previous model artefact stays available and the serving layer can point at either. This is the same discipline as any deploy and it is the single thing most likely to save you."}]},{"t":"p","runs":[{"text":"Also keep a record: which model version made which prediction. When someone asks in March why a decision was made in January, you need to know which model was live and what it was trained on. In regulated settings this is a requirement rather than a convenience."}]},{"t":"h2","id":"the-honest-counterweight","runs":[{"text":"The honest counterweight"}]},{"t":"p","runs":[{"text":"Retraining is not always the answer, and it is reached for reflexively."}]},{"t":"p","runs":[{"text":"If performance dropped because an upstream table changed, retraining teaches the model the broken data. If it dropped because the population shifted into a region you have no examples of, retraining on data that lacks that region changes nothing. If it dropped because of a serving bug, retraining hides the symptom while the bug remains."}]},{"t":"p","runs":[{"text":"Diagnose before retraining.","b":true},{"text":" Look at the errors, check the inputs, compare the two serving paths. Retraining is a remedy for one specific cause (the relationship in the data has changed and you now have labels reflecting that) and applying it to any other cause wastes a week and leaves the fault in place."}]}],"check":{"question":[{"text":"After an automated monthly retrain, a fraud model’s offline AUC improves slightly but the number of cases sent for review triples overnight. What is the most likely cause?"}],"options":[{"text":[{"text":"The new model is overfitting, so it flags many more cases while scoring well offline."}],"correct":false,"tempting":[{"text":"Overfitting would usually show as a worse offline score on held-out data, and the offline metric improved."}]},{"text":[{"text":"The retraining window included a period of elevated fraud, so the model learned a higher base rate."}],"correct":false,"tempting":[{"text":"A higher learned base rate would shift scores upward, but the review volume is set by the threshold applied to those scores rather than by the model alone."}]},{"text":[{"text":"The threshold was carried over from the previous model, but the new model’s scores are distributed differently, so the same cut point now selects a much larger fraction of traffic."}],"correct":true,"tempting":null},{"text":[{"text":"Training-serving skew was introduced by the retrain, so serving features no longer match training features."}],"correct":false,"tempting":[{"text":"Skew is a real hazard but would normally degrade the offline-to-production relationship rather than appear as a clean tripling of volume with an improved offline score."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"monitoring-what-matters","title":"Monitoring: what to watch, and in what order","minutes":9},"next":{"slug":"reproducibility","title":"Being able to build the same model twice","minutes":8}}}