{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"shadow-and-canary-releases","title":"Getting evidence from live traffic without exposing users","url":"https://addaly.com/learn/evaluating-ai/shadow-and-canary-releases","position":56,"of":76,"minutes":8,"answer":"Run the new system on real inputs with its side effects disabled to compare distributions before anyone sees it, then ramp real traffic in stages with pre-agreed guardrails and a rollback you have rehearsed.","keyIdea":"Run the new system on real inputs with its side effects disabled to compare distributions before anyone sees it, then ramp real traffic in stages with pre-agreed guardrails and a rollback you have rehearsed."},"body":{"format":"markdown","text":"## Three stages between the eval set and everyone\n\nAn offline set says the change is probably good. Live traffic says whether it is. The gap between them is crossed in three steps, each answering a different question.\n\n**Shadow**: the new system processes real inputs and its outputs are discarded. Answers: does it behave sanely on the real distribution.\n\n**Canary**: a small share of real users get the new system. Answers: does anything break at volume, and do the guardrails hold.\n\n**Staged ramp**: the share grows on a schedule with soak time. Answers: does it survive a full day, a Monday, a month-end.\n\n## Shadow mode, and the incident everybody has once\n\nShadow runs are cheap evidence and they have one sharp edge: **side effects must be disabled**. The new agent must not send the email, write the record, call the payment API or post the message. Every team that skips this learns it in the same way, and the second copy of a customer email is the polite version of the lesson.\n\nEnforce it structurally, not by convention: a shadow request carries a flag that the tool layer checks, and effectful tools return a simulated result. Then verify it, by running the shadow path against a staging tool endpoint that records calls and asserts none arrived.\n\nWhat shadow gives you, with no labels at all:\n\n- **Distribution comparisons on real inputs**: output length, refusal rate, groundedness rate, schema validity, tool-call frequency, cost, latency percentiles. A shift in any of these on real traffic is a real finding.\n- **A paired comparison on thousands of items**, since old and new ran on identical inputs. Every label-free check from earlier in the course applies directly.\n- **Error discovery**: the inputs that crash the new path, which your set never contained.\n\nIt cannot tell you whether users prefer the result, and it doubles inference cost while it runs. Run it for a day or two on a sample, not indefinitely on everything.\n\n## Canary, sized by the guardrail\n\nThe share is chosen by arithmetic, not by feel: how many events do you need before the guardrail metric could plausibly trigger. If your guardrail is a safety-violation rate that occurs in one request per thousand, a 1% canary on 50,000 daily requests gives you 500 requests a day and roughly one expected event: far too slow. Either raise the share or pick a guardrail with a higher base rate.\n\nTwo practical rules. **Keep assignment sticky** per user, or somebody gets two different behaviours in one session and reports a bug that is your experiment. And **check whether the canary population is representative**: a canary routed to one region, one platform or one data centre is a biased sample, and the mix-shift arithmetic from the statistics block applies exactly.\n\n## Guardrails, thresholds and a rehearsed rollback\n\nWrite these before launch, with numbers:\n\n- The metrics that can stop the rollout: error rate, safety violations, refusal rate, p95 latency, cost per request, a business guardrail such as escalation rate.\n- The threshold and the window for each, sized so that normal fluctuation does not trigger them daily.\n- Who is paged, and what they are authorised to do without a meeting.\n\nThen rehearse the rollback. A rollback that has never been executed is a hypothesis, and the moment you discover the config takes ten minutes to propagate should not be during an incident. Roll back once on purpose, in the middle of a quiet afternoon, and time it.\n\n## The ramp, and why soak time matters\n\n1%, 5%, 25%, 100%, with a minimum soak at each step: usually a full business day, and at least one weekly cycle before the final step. Some failures only appear with time or volume: rate limits at peak, cache eviction under load, a nightly batch that now takes longer than its window, a weekly report built on a field the new version stopped writing.\n\n> Figure: From the eval set to everyone. Timeline: Shadow, 1 to 2 days, The new system reads real inputs; outputs are discarded and side effects are disabled in the tool layer. Compare distributions: refusal rate, length, schema validity, latency, cost.; Canary, 1%, Real users. The share is sized so the guardrail metric can plausibly trigger. Sticky assignment per user; check the canary population is representative.; 5%, one business day, Soak time. Rate limits at peak, cache eviction under load.; 25%, one weekly cycle, The nightly batch, the Monday, the weekly report built on a field the new version stopped writing.; 100%, Rollback already rehearsed once, on a quiet afternoon, and timed.. This machinery detects fast, visible harms. Slow ones, such as a corpus filling with the system's own output or users quietly giving up, need the monitoring in the final module.\n\nThe limit worth stating: this machinery detects fast, visible harms. It does not detect slow ones: users gradually trusting an answer that is subtly wrong, a corpus filling with the system's own output, a group of users quietly giving up. Those are the subject of the final block, and no ramp schedule substitutes for them."},"check":{"question":"What is the specific risk that makes shadow mode dangerous if implemented carelessly?","options":[{"text":"It doubles inference cost, which can exhaust a budget before the comparison is complete.","correct":false,"tempting":"A real operational cost that argues for sampling and time-boxing, not a safety risk to users."},{"text":"Shadow traffic can trigger rate limits that degrade the live path serving real users.","correct":false,"tempting":"A genuine capacity concern worth planning for, and it is a performance issue rather than the irreversible one."},{"text":"The shadow system executes real side effects (sending messages, writing records, calling payment APIs) because only its returned output was discarded.","correct":true},{"text":"Shadow outputs are never seen by users, so quality problems can persist unnoticed for weeks.","correct":false,"tempting":"Describes a limitation of what shadow mode measures; it is precisely why the canary stage follows, and it is not a danger created by shadow running."}]},"prev":{"slug":"ab-testing-honestly","title":"A/B testing a feature honestly","url":"https://addaly.com/learn/evaluating-ai/ab-testing-honestly"},"next":{"slug":"online-metrics-and-proxies","title":"The signals users give you without being asked","url":"https://addaly.com/learn/evaluating-ai/online-metrics-and-proxies"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Getting evidence from live traffic without exposing users', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/shadow-and-canary-releases (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Getting evidence from live traffic without exposing users","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Run the new system on real inputs with its side effects disabled to compare distributions before anyone sees it, then ramp real traffic in stages with pre-agreed guardrails and a rollback you have rehearsed.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"From the eval set to everyone","events":[{"when":"Shadow, 1 to 2 days","what":"The new system reads real inputs; outputs are discarded and side effects are disabled in the tool layer. Compare distributions: refusal rate, length, schema validity, latency, cost."},{"when":"Canary, 1%","what":"Real users. The share is sized so the guardrail metric can plausibly trigger. Sticky assignment per user; check the canary population is representative."},{"when":"5%, one business day","what":"Soak time. Rate limits at peak, cache eviction under load."},{"when":"25%, one weekly cycle","what":"The nightly batch, the Monday, the weekly report built on a field the new version stopped writing."},{"when":"100%","what":"Rollback already rehearsed once, on a quiet afternoon, and timed."}],"caption":"This machinery detects fast, visible harms. Slow ones, such as a corpus filling with the system’s own output or users quietly giving up, need the monitoring in the final module."},"alt":"Figure: From the eval set to everyone. Timeline: Shadow, 1 to 2 days, The new system reads real inputs; outputs are discarded and side effects are disabled in the tool layer. Compare distributions: refusal rate, length, schema validity, latency, cost.; Canary, 1%, Real users. The share is sized so the guardrail metric can plausibly trigger. Sticky assignment per user; check the canary population is representative.; 5%, one business day, Soak time. Rate limits at peak, cache eviction under load.; 25%, one weekly cycle, The nightly batch, the Monday, the weekly report built on a field the new version stopped writing.; 100%, Rollback already rehearsed once, on a quiet afternoon, and timed.. This machinery detects fast, visible harms. Slow ones, such as a corpus filling with the system’s own output or users quietly giving up, need the monitoring in the final module."}},"blocks":[{"t":"h2","id":"three-stages-between-the-eval-set-and-everyone","runs":[{"text":"Three stages between the eval set and everyone"}]},{"t":"p","runs":[{"text":"An offline set says the change is probably good. Live traffic says whether it is. The gap between them is crossed in three steps, each answering a different question."}]},{"t":"p","runs":[{"text":"Shadow","b":true},{"text":": the new system processes real inputs and its outputs are discarded. Answers: does it behave sanely on the real distribution."}]},{"t":"p","runs":[{"text":"Canary","b":true},{"text":": a small share of real users get the new system. Answers: does anything break at volume, and do the guardrails hold."}]},{"t":"p","runs":[{"text":"Staged ramp","b":true},{"text":": the share grows on a schedule with soak time. Answers: does it survive a full day, a Monday, a month-end."}]},{"t":"h2","id":"shadow-mode-and-the-incident-everybody-has-once","runs":[{"text":"Shadow mode, and the incident everybody has once"}]},{"t":"p","runs":[{"text":"Shadow runs are cheap evidence and they have one sharp edge: "},{"text":"side effects must be disabled","b":true},{"text":". The new agent must not send the email, write the record, call the payment API or post the message. Every team that skips this learns it in the same way, and the second copy of a customer email is the polite version of the lesson."}]},{"t":"p","runs":[{"text":"Enforce it structurally, not by convention: a shadow request carries a flag that the tool layer checks, and effectful tools return a simulated result. Then verify it, by running the shadow path against a staging tool endpoint that records calls and asserts none arrived."}]},{"t":"p","runs":[{"text":"What shadow gives you, with no labels at all:"}]},{"t":"ul","items":[[{"text":"Distribution comparisons on real inputs","b":true},{"text":": output length, refusal rate, groundedness rate, schema validity, tool-call frequency, cost, latency percentiles. A shift in any of these on real traffic is a real finding."}],[{"text":"A paired comparison on thousands of items","b":true},{"text":", since old and new ran on identical inputs. Every label-free check from earlier in the course applies directly."}],[{"text":"Error discovery","b":true},{"text":": the inputs that crash the new path, which your set never contained."}]]},{"t":"p","runs":[{"text":"It cannot tell you whether users prefer the result, and it doubles inference cost while it runs. Run it for a day or two on a sample, not indefinitely on everything."}]},{"t":"h2","id":"canary-sized-by-the-guardrail","runs":[{"text":"Canary, sized by the guardrail"}]},{"t":"p","runs":[{"text":"The share is chosen by arithmetic, not by feel: how many events do you need before the guardrail metric could plausibly trigger. If your guardrail is a safety-violation rate that occurs in one request per thousand, a 1% canary on 50,000 daily requests gives you 500 requests a day and roughly one expected event: far too slow. Either raise the share or pick a guardrail with a higher base rate."}]},{"t":"p","runs":[{"text":"Two practical rules. "},{"text":"Keep assignment sticky","b":true},{"text":" per user, or somebody gets two different behaviours in one session and reports a bug that is your experiment. And "},{"text":"check whether the canary population is representative","b":true},{"text":": a canary routed to one region, one platform or one data centre is a biased sample, and the mix-shift arithmetic from the statistics block applies exactly."}]},{"t":"h2","id":"guardrails-thresholds-and-a-rehearsed-rollback","runs":[{"text":"Guardrails, thresholds and a rehearsed rollback"}]},{"t":"p","runs":[{"text":"Write these before launch, with numbers:"}]},{"t":"ul","items":[[{"text":"The metrics that can stop the rollout: error rate, safety violations, refusal rate, p95 latency, cost per request, a business guardrail such as escalation rate."}],[{"text":"The threshold and the window for each, sized so that normal fluctuation does not trigger them daily."}],[{"text":"Who is paged, and what they are authorised to do without a meeting."}]]},{"t":"p","runs":[{"text":"Then rehearse the rollback. A rollback that has never been executed is a hypothesis, and the moment you discover the config takes ten minutes to propagate should not be during an incident. Roll back once on purpose, in the middle of a quiet afternoon, and time it."}]},{"t":"h2","id":"the-ramp-and-why-soak-time-matters","runs":[{"text":"The ramp, and why soak time matters"}]},{"t":"p","runs":[{"text":"1%, 5%, 25%, 100%, with a minimum soak at each step: usually a full business day, and at least one weekly cycle before the final step. Some failures only appear with time or volume: rate limits at peak, cache eviction under load, a nightly batch that now takes longer than its window, a weekly report built on a field the new version stopped writing."}]},{"t":"p","runs":[{"text":"The limit worth stating: this machinery detects fast, visible harms. It does not detect slow ones: users gradually trusting an answer that is subtly wrong, a corpus filling with the system’s own output, a group of users quietly giving up. Those are the subject of the final block, and no ramp schedule substitutes for them."}]}],"check":{"question":[{"text":"What is the specific risk that makes shadow mode dangerous if implemented carelessly?"}],"options":[{"text":[{"text":"It doubles inference cost, which can exhaust a budget before the comparison is complete."}],"correct":false,"tempting":[{"text":"A real operational cost that argues for sampling and time-boxing, not a safety risk to users."}]},{"text":[{"text":"Shadow traffic can trigger rate limits that degrade the live path serving real users."}],"correct":false,"tempting":[{"text":"A genuine capacity concern worth planning for, and it is a performance issue rather than the irreversible one."}]},{"text":[{"text":"The shadow system executes real side effects (sending messages, writing records, calling payment APIs) because only its returned output was discarded."}],"correct":true,"tempting":null},{"text":[{"text":"Shadow outputs are never seen by users, so quality problems can persist unnoticed for weeks."}],"correct":false,"tempting":[{"text":"Describes a limitation of what shadow mode measures; it is precisely why the canary stage follows, and it is not a danger created by shadow running."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"ab-testing-honestly","title":"A/B testing a feature honestly","minutes":8},"next":{"slug":"online-metrics-and-proxies","title":"The signals users give you without being asked","minutes":9}}}