{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"shipping-a-model-safely","title":"Shadow, then canary, then everybody","url":"https://addaly.com/learn/cloud-and-shipping/shipping-a-model-safely","position":54,"of":74,"minutes":9,"answer":"Ship a model in three stages (shadow to find what breaks, canary on a user-hashed slice with automatic rollback, then a properly sized experiment) and keep a small holdback afterwards.","keyIdea":"Ship a model in three stages (shadow to find what breaks, canary on a user-hashed slice with automatic rollback, then a properly sized experiment) and keep a small holdback afterwards, because the artifact rolls back but the data it wrote does not."},"body":{"format":"markdown","text":"## An offline improvement is a hypothesis\n\nThe new model beats the old one on the held-out set. That is evidence about a fixed dataset, collected in the past, scored by a metric you chose. It is not evidence about live traffic, which contains inputs your set does not, arrives at a different rate, and is followed by user behaviour your metric never measured.\n\nThe gap between offline and online results is one of the most consistently reported findings in applied machine learning: a substantial share of changes that improve an offline metric fail to improve the online one, and some make it worse. Nothing about that is a criticism of offline evaluation. It is a reason to ship models the way you ship code: gradually, with a way back.\n\n> Figure: Shadow, then canary, then everybody. Steps: 1. Shadow (real traffic, answers discarded and logged; no side effects; watch the disagreement rate); 2. Canary at 1%, 5%, 25% (split by a stable hash of the user; automatic rollback on error or latency); 3. Experiment (metric and sample size fixed in advance, no peeking); 4. Everybody, with a 1 to 5% holdback (the only way to answer, in month three, whether it is still better). Each stage answers a different question. Shadow finds what breaks, canary finds what the users do, the experiment settles whether it is better, and the holdback keeps that comparison alive after the launch.\n\n## Stage one: shadow\n\nSend real traffic to the new model, discard its answers, and record them beside the old model's.\n\nThe user sees only the current model, so the risk is close to zero. What you learn is not quality but the things that break first: latency under real input sizes, memory under real concurrency, unexpected inputs, error rates, and how often the two models actually disagree. Disagreement rate is the most useful number here: if the new model differs on 0.4% of requests, the online effect will be small whatever the offline table said; if it differs on 30%, be careful.\n\nShadowing costs double inference for its duration, which is a real bill, so run it for hours or a day rather than a fortnight, and shadow a sample of traffic rather than all of it.\n\nOne rule: shadow requests must not have side effects. No emails, no writes, no charges. Route them through a flag that disables everything downstream, and test that the flag works before you turn shadowing on.\n\n## Stage two: canary\n\nGive the new model a small share of real traffic (1%, then 5%, then 25%) and compare the same metrics for both groups.\n\nWatch three families of number: technical (latency, errors, cost per request), model (score distribution, confidence, refusal or fallback rate), and product (the thing you actually care about: clicks, completions, reports, purchases). Automate the rollback: if error rate or latency exceeds a threshold for five minutes, revert without waiting for a human.\n\nSplit by a stable hash of the user ID, not per request. A user who gets the new model on one page and the old one on the next has an incoherent experience and pollutes your comparison.\n\n## Stage three: the honest experiment\n\nIf the decision matters, run it as a proper A/B test with the metric and duration fixed in advance. Two warnings specific to models:\n\n- **Peeking.** Checking the result daily and stopping when it looks good manufactures significance out of noise. Fix the sample size before you start, or use a method designed for continuous monitoring.\n- **Novelty and learning effects.** A changed recommendation feed produces a short-term bump because it is different, then settles. A week of data on a two-week effect is a wrong answer delivered confidently.\n\nAnd for many small products the sober truth is that traffic is too low to detect the difference you are arguing about. When that is the case, say so, choose on other grounds (cost, latency, simplicity) and stop pretending the experiment settled it.\n\n## Keep a holdback\n\nWhen the new model wins and goes to everybody, keep a small slice (1 to 5%) on the old one for a while. It costs almost nothing and it is the only way to answer the question that always arrives in month three: is this system actually better than what we had, or has everything just drifted?\n\nHoldbacks are also how you notice slow degradation that affects both groups equally, because the comparison stays live.\n\n## Rollback is different for models\n\nCode rolls back cleanly. A model has three complications:\n\n- **The old artifact must still exist**, with its preprocessing, at a version you can deploy. Keep the last two or three, and their evaluation numbers.\n- **Data it produced does not roll back.** Predictions written to rows, embeddings stored in an index, moderation decisions applied to content: those persist. If the model wrote to a shared index, reverting the code does not revert the index.\n- **The feedback it created stays in your training data.** Which is the subject of the feedback-loop lesson later in this module.\n\nSo the deployment note for a model is the same as for a schema change: separate the reversible part from the irreversible one, and know which is which before you press the button."},"check":{"question":"A team shadows a new ranking model for a day. Latency, errors and cost all look fine, and it disagrees with the current model on 0.3% of requests. What is the reasonable conclusion?","options":[{"text":"It is operationally safe, but a 0.3% disagreement rate means any online effect will be small and may be undetectable at this traffic","correct":true},{"text":"The new model is safe to deploy to everybody, since shadowing showed no problems","correct":false,"tempting":"Shadowing tests operational behaviour, not quality; no user outcome was observed, so nothing has been learnt about whether it is better."},{"text":"The shadow test is invalid because the new model's answers were discarded","correct":false,"tempting":"Discarding the answers is the design, not a flaw; it is what makes shadowing risk-free."},{"text":"A low disagreement rate proves the offline improvement was measurement error","correct":false,"tempting":"Small disagreement is consistent with a real but narrow improvement; it bounds the effect size rather than disproving it."}]},"prev":{"slug":"training-serving-skew","title":"The feature computed two different ways","url":"https://addaly.com/learn/cloud-and-shipping/training-serving-skew"},"next":{"slug":"when-the-inputs-change","title":"The model that quietly stopped fitting","url":"https://addaly.com/learn/cloud-and-shipping/when-the-inputs-change"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Shadow, then canary, then everybody', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/shipping-a-model-safely (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Shadow, then canary, then everybody","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"Ship a model in three stages (shadow to find what breaks, canary on a user-hashed slice with automatic rollback, then a properly sized experiment) and keep a small holdback afterwards, because the artifact rolls back but the data it wrote does not.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Shadow","detail":"real traffic, answers discarded and logged; no side effects; watch the disagreement rate"},{"label":"Canary at 1%, 5%, 25%","detail":"split by a stable hash of the user; automatic rollback on error or latency"},{"label":"Experiment","detail":"metric and sample size fixed in advance, no peeking"},{"label":"Everybody, with a 1 to 5% holdback","detail":"the only way to answer, in month three, whether it is still better"}],"title":"Shadow, then canary, then everybody","caption":"Each stage answers a different question. Shadow finds what breaks, canary finds what the users do, the experiment settles whether it is better, and the holdback keeps that comparison alive after the launch."},"alt":"Figure: Shadow, then canary, then everybody. Steps: 1. Shadow (real traffic, answers discarded and logged; no side effects; watch the disagreement rate); 2. Canary at 1%, 5%, 25% (split by a stable hash of the user; automatic rollback on error or latency); 3. Experiment (metric and sample size fixed in advance, no peeking); 4. Everybody, with a 1 to 5% holdback (the only way to answer, in month three, whether it is still better). Each stage answers a different question. Shadow finds what breaks, canary finds what the users do, the experiment settles whether it is better, and the holdback keeps that comparison alive after the launch."}},"blocks":[{"t":"h2","id":"an-offline-improvement-is-a-hypothesis","runs":[{"text":"An offline improvement is a hypothesis"}]},{"t":"p","runs":[{"text":"The new model beats the old one on the held-out set. That is evidence about a fixed dataset, collected in the past, scored by a metric you chose. It is not evidence about live traffic, which contains inputs your set does not, arrives at a different rate, and is followed by user behaviour your metric never measured."}]},{"t":"p","runs":[{"text":"The gap between offline and online results is one of the most consistently reported findings in applied machine learning: a substantial share of changes that improve an offline metric fail to improve the online one, and some make it worse. Nothing about that is a criticism of offline evaluation. It is a reason to ship models the way you ship code: gradually, with a way back."}]},{"t":"h2","id":"stage-one-shadow","runs":[{"text":"Stage one: shadow"}]},{"t":"p","runs":[{"text":"Send real traffic to the new model, discard its answers, and record them beside the old model’s."}]},{"t":"p","runs":[{"text":"The user sees only the current model, so the risk is close to zero. What you learn is not quality but the things that break first: latency under real input sizes, memory under real concurrency, unexpected inputs, error rates, and how often the two models actually disagree. Disagreement rate is the most useful number here: if the new model differs on 0.4% of requests, the online effect will be small whatever the offline table said; if it differs on 30%, be careful."}]},{"t":"p","runs":[{"text":"Shadowing costs double inference for its duration, which is a real bill, so run it for hours or a day rather than a fortnight, and shadow a sample of traffic rather than all of it."}]},{"t":"p","runs":[{"text":"One rule: shadow requests must not have side effects. No emails, no writes, no charges. Route them through a flag that disables everything downstream, and test that the flag works before you turn shadowing on."}]},{"t":"h2","id":"stage-two-canary","runs":[{"text":"Stage two: canary"}]},{"t":"p","runs":[{"text":"Give the new model a small share of real traffic (1%, then 5%, then 25%) and compare the same metrics for both groups."}]},{"t":"p","runs":[{"text":"Watch three families of number: technical (latency, errors, cost per request), model (score distribution, confidence, refusal or fallback rate), and product (the thing you actually care about: clicks, completions, reports, purchases). Automate the rollback: if error rate or latency exceeds a threshold for five minutes, revert without waiting for a human."}]},{"t":"p","runs":[{"text":"Split by a stable hash of the user ID, not per request. A user who gets the new model on one page and the old one on the next has an incoherent experience and pollutes your comparison."}]},{"t":"h2","id":"stage-three-the-honest-experiment","runs":[{"text":"Stage three: the honest experiment"}]},{"t":"p","runs":[{"text":"If the decision matters, run it as a proper A/B test with the metric and duration fixed in advance. Two warnings specific to models:"}]},{"t":"ul","items":[[{"text":"Peeking.","b":true},{"text":" Checking the result daily and stopping when it looks good manufactures significance out of noise. Fix the sample size before you start, or use a method designed for continuous monitoring."}],[{"text":"Novelty and learning effects.","b":true},{"text":" A changed recommendation feed produces a short-term bump because it is different, then settles. A week of data on a two-week effect is a wrong answer delivered confidently."}]]},{"t":"p","runs":[{"text":"And for many small products the sober truth is that traffic is too low to detect the difference you are arguing about. When that is the case, say so, choose on other grounds (cost, latency, simplicity) and stop pretending the experiment settled it."}]},{"t":"h2","id":"keep-a-holdback","runs":[{"text":"Keep a holdback"}]},{"t":"p","runs":[{"text":"When the new model wins and goes to everybody, keep a small slice (1 to 5%) on the old one for a while. It costs almost nothing and it is the only way to answer the question that always arrives in month three: is this system actually better than what we had, or has everything just drifted?"}]},{"t":"p","runs":[{"text":"Holdbacks are also how you notice slow degradation that affects both groups equally, because the comparison stays live."}]},{"t":"h2","id":"rollback-is-different-for-models","runs":[{"text":"Rollback is different for models"}]},{"t":"p","runs":[{"text":"Code rolls back cleanly. A model has three complications:"}]},{"t":"ul","items":[[{"text":"The old artifact must still exist","b":true},{"text":", with its preprocessing, at a version you can deploy. Keep the last two or three, and their evaluation numbers."}],[{"text":"Data it produced does not roll back.","b":true},{"text":" Predictions written to rows, embeddings stored in an index, moderation decisions applied to content: those persist. If the model wrote to a shared index, reverting the code does not revert the index."}],[{"text":"The feedback it created stays in your training data.","b":true},{"text":" Which is the subject of the feedback-loop lesson later in this module."}]]},{"t":"p","runs":[{"text":"So the deployment note for a model is the same as for a schema change: separate the reversible part from the irreversible one, and know which is which before you press the button."}]}],"check":{"question":[{"text":"A team shadows a new ranking model for a day. Latency, errors and cost all look fine, and it disagrees with the current model on 0.3% of requests. What is the reasonable conclusion?"}],"options":[{"text":[{"text":"It is operationally safe, but a 0.3% disagreement rate means any online effect will be small and may be undetectable at this traffic"}],"correct":true,"tempting":null},{"text":[{"text":"The new model is safe to deploy to everybody, since shadowing showed no problems"}],"correct":false,"tempting":[{"text":"Shadowing tests operational behaviour, not quality; no user outcome was observed, so nothing has been learnt about whether it is better."}]},{"text":[{"text":"The shadow test is invalid because the new model’s answers were discarded"}],"correct":false,"tempting":[{"text":"Discarding the answers is the design, not a flaw; it is what makes shadowing risk-free."}]},{"text":[{"text":"A low disagreement rate proves the offline improvement was measurement error"}],"correct":false,"tempting":[{"text":"Small disagreement is consistent with a real but narrow improvement; it bounds the effect size rather than disproving it."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"training-serving-skew","title":"The feature computed two different ways","minutes":10},"next":{"slug":"when-the-inputs-change","title":"The model that quietly stopped fitting","minutes":10}}}