When to retrain, and what breaks when you do

Three policies

Scheduled. Retrain every week, or month, or quarter, regardless. Predictable, easy to automate, and it does work nobody asked for and sometimes misses a fast change between runs.

Triggered. Retrain when a monitored metric crosses a line: performance below a threshold, PSI above 0.25, a flag rate outside its band. Responsive, and it requires the monitoring to be trustworthy first.

Continuous. Update the model incrementally as data arrives. Necessary for fast-moving problems such as ad ranking. It also makes debugging much harder, because “which model made this prediction” has a different answer every hour.

Most teams should start scheduled (monthly is a reasonable default) and add triggers once the monitoring has earned trust. Continuous is a commitment, not an upgrade.

How much history to use

A real decision with no universal answer.

All of it if the relationship is stable. More data, better estimates.

A rolling window (the last 12 months) if the world changes. Old data actively misleads when behaviour has moved on.

Weighted: all of it, with recent rows weighted more. The middle path, and often the best: sample_weight decaying with age gives you both the volume and the recency.

Decide it empirically. Train on the last 3, 6, 12 and 24 months, evaluate each on the most recent held-out period, and let the numbers choose. This experiment takes an hour and teams routinely guess instead.

Retraining is a deployment

The most common production mistake in this area is treating a retrained model as automatically safe because “it is the same code with newer data”. It is a new model with different behaviour and it needs the same gates as any deploy.

The minimum checks before a retrained model goes live:

1. Performance on a fixed reference set (a held-out period that never changes) compared against the current production model. If the new one is worse, stop.
2. Prediction distribution comparison. Score the last week of production traffic with both models. A large shift in the distribution needs an explanation before it needs a deployment.
3. Slice checks. From module 5. A model that improves overall while collapsing on one group must not ship silently.
4. The fixed test sets you built from past failure modes.
5. Shadow mode for a period, if the stakes justify it.

Automate all five. A retraining pipeline that trains and deploys without gates will eventually deploy a model trained on a corrupted extract, and it will do so at 4am on a Sunday.

The threshold moves too

Easy to forget: your decision threshold was chosen for the old model’s score distribution. A retrained model’s scores are distributed differently, so the same threshold produces a different flag rate.

Re-choose the threshold on validation data as part of every retraining, using the same cost or capacity criterion from module 5. Otherwise the model improves and the system behaves worse, which is a confusing failure to debug and a common one.

Keep the old model

Rollback should take minutes, which means the previous model artefact stays available and the serving layer can point at either. This is the same discipline as any deploy and it is the single thing most likely to save you.

Also keep a record: which model version made which prediction. When someone asks in March why a decision was made in January, you need to know which model was live and what it was trained on. In regulated settings this is a requirement rather than a convenience.

The honest counterweight

Retraining is not always the answer, and it is reached for reflexively.

If performance dropped because an upstream table changed, retraining teaches the model the broken data. If it dropped because the population shifted into a region you have no examples of, retraining on data that lacks that region changes nothing. If it dropped because of a serving bug, retraining hides the symptom while the bug remains.

Diagnose before retraining. Look at the errors, check the inputs, compare the two serving paths. Retraining is a remedy for one specific cause (the relationship in the data has changed and you now have labels reflecting that) and applying it to any other cause wastes a week and leaves the fault in place.