Shadow, then canary, then everybody

An offline improvement is a hypothesis

The new model beats the old one on the held-out set. That is evidence about a fixed dataset, collected in the past, scored by a metric you chose. It is not evidence about live traffic, which contains inputs your set does not, arrives at a different rate, and is followed by user behaviour your metric never measured.

The gap between offline and online results is one of the most consistently reported findings in applied machine learning: a substantial share of changes that improve an offline metric fail to improve the online one, and some make it worse. Nothing about that is a criticism of offline evaluation. It is a reason to ship models the way you ship code: gradually, with a way back.

Stage one: shadow

Send real traffic to the new model, discard its answers, and record them beside the old model’s.

The user sees only the current model, so the risk is close to zero. What you learn is not quality but the things that break first: latency under real input sizes, memory under real concurrency, unexpected inputs, error rates, and how often the two models actually disagree. Disagreement rate is the most useful number here: if the new model differs on 0.4% of requests, the online effect will be small whatever the offline table said; if it differs on 30%, be careful.

Shadowing costs double inference for its duration, which is a real bill, so run it for hours or a day rather than a fortnight, and shadow a sample of traffic rather than all of it.

One rule: shadow requests must not have side effects. No emails, no writes, no charges. Route them through a flag that disables everything downstream, and test that the flag works before you turn shadowing on.

Stage two: canary

Give the new model a small share of real traffic (1%, then 5%, then 25%) and compare the same metrics for both groups.

Watch three families of number: technical (latency, errors, cost per request), model (score distribution, confidence, refusal or fallback rate), and product (the thing you actually care about: clicks, completions, reports, purchases). Automate the rollback: if error rate or latency exceeds a threshold for five minutes, revert without waiting for a human.

Split by a stable hash of the user ID, not per request. A user who gets the new model on one page and the old one on the next has an incoherent experience and pollutes your comparison.

Stage three: the honest experiment

If the decision matters, run it as a proper A/B test with the metric and duration fixed in advance. Two warnings specific to models:

• Peeking. Checking the result daily and stopping when it looks good manufactures significance out of noise. Fix the sample size before you start, or use a method designed for continuous monitoring.
• Novelty and learning effects. A changed recommendation feed produces a short-term bump because it is different, then settles. A week of data on a two-week effect is a wrong answer delivered confidently.

And for many small products the sober truth is that traffic is too low to detect the difference you are arguing about. When that is the case, say so, choose on other grounds (cost, latency, simplicity) and stop pretending the experiment settled it.

Keep a holdback

When the new model wins and goes to everybody, keep a small slice (1 to 5%) on the old one for a while. It costs almost nothing and it is the only way to answer the question that always arrives in month three: is this system actually better than what we had, or has everything just drifted?

Holdbacks are also how you notice slow degradation that affects both groups equally, because the comparison stays live.

Rollback is different for models

Code rolls back cleanly. A model has three complications:

• The old artifact must still exist, with its preprocessing, at a version you can deploy. Keep the last two or three, and their evaluation numbers.
• Data it produced does not roll back. Predictions written to rows, embeddings stored in an index, moderation decisions applied to content: those persist. If the model wrote to a shared index, reverting the code does not revert the index.
• The feedback it created stays in your training data. Which is the subject of the feedback-loop lesson later in this module.

So the deployment note for a model is the same as for a schema change: separate the reversible part from the irreversible one, and know which is which before you press the button.