{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"rollout-and-rollback","title":"Shadow, canary, and the off switch","url":"https://addaly.com/learn/agents-and-automation/rollout-and-rollback","position":61,"of":72,"minutes":9,"answer":"Assign a canary by hashing a stable identifier rather than rolling a die per request, or the same item will be handled by both versions and the comparison becomes meaningless.","keyIdea":"Assign a canary by hashing a stable identifier rather than rolling a die per request, or the same item will be handled by both versions and the comparison becomes meaningless."},"body":{"format":"markdown","text":"## Shadow mode first\n\nBefore an automation touches anything, run it alongside the people doing the work. It receives the same inputs, produces its outputs into a table, and changes nothing. Then compare: where did it agree, where did it differ, and on the differences, who was right.\n\nA week of this is the most informative week the project will have. You get a real accuracy number on real traffic, you find the input categories nobody mentioned, and you get it without a single consequence. It also converts the conversation with the team whose work this touches from a debate about whether it will work into a table of what it did.\n\nIt costs real money (you pay for every run and get no output) and it is worth it. Budget for a shadow week explicitly rather than discovering there is no line for it.\n\n> Figure: Shadow, then canary, with the stopping rule written first. Timeline: Week 1, Shadow. Same inputs, outputs into a table, nothing changed. Pure cost, and the most informative week of the project.; Week 2, Read the disagreements with the team who do the work. Two input categories nobody had mentioned turn up.; Before the canary, Write the stopping rule down: stop if escalation on canary tickets exceeds control by three points over two hundred tickets.; Week 3, Five per cent, assigned by hashing the ticket id, so an item keeps one version for its whole life.; Week 5, Twenty-five per cent. The off switch is tested from a phone, by somebody who did not build it: nineteen seconds.; Week 7, Everything, with the previous config version still deployed so a rollback is changing a value.. Assigning the canary by a die roll per request lets one ticket meet both versions, which makes every per-item metric uninterpretable. Hash the entity the outcome belongs to.\n\n## Then a canary, assigned by hash\n\nSend a small slice of live traffic to the new version. Five per cent is a reasonable start.\n\nAssign the slice by hashing a stable identifier, not by rolling a die per request:\n\n```python\nuse_new = (int(hashlib.sha256(ticket_id.encode()).hexdigest()[:8], 16) % 100) < 5\n```\n\nThe reason matters. With a per-request random choice, the same ticket can be handled by the old version on its first touch and the new one on its second. Users see inconsistent behaviour on one item, your comparison mixes both versions within a single case, and any per-item metric (resolution rate, edits needed, escalation) becomes uninterpretable. Hashing a stable id gives each item one version for its whole life, and the two groups stay clean.\n\nHash the entity the outcome belongs to. For a support ticket that is the ticket, or the customer if the customer's experience across tickets is what you are measuring.\n\n## Agree the stopping rule before you start\n\nWrite down, before the canary begins: the metric, the value that means stop, and how long you will watch. \"We stop if escalation rate on canary tickets exceeds the control by more than three points over two hundred tickets.\"\n\nWritten afterwards, every number becomes an argument. Written before, the decision takes ten seconds and nobody has to be talked out of their own project.\n\n## The off switch\n\nOne flag, readable at run time, that stops the automation.\n\nPut it in a database row or a configuration service, not in an environment variable, because an environment variable requires a deployment and a deployment takes longer than you have. Check it in two places: at the start of every run, and immediately before every irreversible action, so a run already in flight cannot complete a payment after somebody has pulled the lever.\n\nThree requirements that are easy to skip and decide whether it works:\n\n- **Under thirty seconds** from decision to stopped.\n- **Operable by somebody who is not the author**: the person on shift at 2am, with the procedure written down.\n- **Tested monthly.** An untested switch is a claim, and this is the claim you will make to a regulator, a customer or your own board.\n\n## Rollback is a value, not a rebuild\n\nBecause the model id, prompt and tool schema are a single config version, rollback is setting that value back. Keep the previous version deployed and reachable rather than deleting it, and make sure the rollback path is exercised: a rollback that requires a build, a review and a deploy is not a rollback, it is a fix.\n\n## Prompt changes are deployments\n\nThe most common way an automation regresses in production is a prompt edit that nobody treated as a release, because it is only text and it took two minutes.\n\nGive prompt changes the same path as code: version control, review, the golden task set, canary, and a note in the deployment log. The two-minute edit that goes straight to everyone is how a team ends up unable to explain a change in behaviour a week later, and it is the single most common cause of \"it used to work\".\n\n## Tell the people whose work changes\n\nBefore the canary, not after. A support team that notices its queue behaving differently and was not told has had an incident, whatever your metrics say, and the trust you lose there is slower to rebuild than any technical fault."},"check":{"question":"A team canaries a new version on 5 per cent of support tickets by generating a random number on each request. Their per-ticket comparison comes out uninterpretable. What went wrong?","options":[{"text":"Five per cent is too small a slice to produce a usable signal in the time available.","correct":false,"tempting":"Sample size affects the width of the error bars; it does not make a comparison structurally uninterpretable."},{"text":"Random assignment introduces selection bias, so the groups are not comparable.","correct":false,"tempting":"Inverts the statistics: random assignment is what removes selection bias: the flaw here is assigning repeatedly rather than once."},{"text":"A ticket touched several times gets different versions on different touches, so both versions act on the same case and no per-item outcome belongs to either.","correct":true},{"text":"Random numbers generated per process are correlated across workers, so the split is not really 5 per cent.","correct":false,"tempting":"Seeding faults can skew a split, but the reported problem is interpretability of per-ticket outcomes, not the size of the groups."}]},"prev":{"slug":"pinning-and-migrating-models","title":"The model you tested is not the model you deployed","url":"https://addaly.com/learn/agents-and-automation/pinning-and-migrating-models"},"next":{"slug":"alerts-and-the-runbook","title":"The 3am version","url":"https://addaly.com/learn/agents-and-automation/alerts-and-the-runbook"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Shadow, canary, and the off switch', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/rollout-and-rollback (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Shadow, canary, and the off switch","courseTitle":"Agents and Automation","module":{"index":7,"slug":"shipping-and-operating","title":"Shipping it, and keeping it alive"},"answer":null,"keyIdea":"Assign a canary by hashing a stable identifier rather than rolling a die per request, or the same item will be handled by both versions and the comparison becomes meaningless.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"Shadow, then canary, with the stopping rule written first","events":[{"when":"Week 1","what":"Shadow. Same inputs, outputs into a table, nothing changed. Pure cost, and the most informative week of the project."},{"when":"Week 2","what":"Read the disagreements with the team who do the work. Two input categories nobody had mentioned turn up."},{"when":"Before the canary","what":"Write the stopping rule down: stop if escalation on canary tickets exceeds control by three points over two hundred tickets."},{"when":"Week 3","what":"Five per cent, assigned by hashing the ticket id, so an item keeps one version for its whole life."},{"when":"Week 5","what":"Twenty-five per cent. The off switch is tested from a phone, by somebody who did not build it: nineteen seconds."},{"when":"Week 7","what":"Everything, with the previous config version still deployed so a rollback is changing a value."}],"caption":"Assigning the canary by a die roll per request lets one ticket meet both versions, which makes every per-item metric uninterpretable. Hash the entity the outcome belongs to."},"alt":"Figure: Shadow, then canary, with the stopping rule written first. Timeline: Week 1, Shadow. Same inputs, outputs into a table, nothing changed. Pure cost, and the most informative week of the project.; Week 2, Read the disagreements with the team who do the work. Two input categories nobody had mentioned turn up.; Before the canary, Write the stopping rule down: stop if escalation on canary tickets exceeds control by three points over two hundred tickets.; Week 3, Five per cent, assigned by hashing the ticket id, so an item keeps one version for its whole life.; Week 5, Twenty-five per cent. The off switch is tested from a phone, by somebody who did not build it: nineteen seconds.; Week 7, Everything, with the previous config version still deployed so a rollback is changing a value.. Assigning the canary by a die roll per request lets one ticket meet both versions, which makes every per-item metric uninterpretable. Hash the entity the outcome belongs to."}},"blocks":[{"t":"h2","id":"shadow-mode-first","runs":[{"text":"Shadow mode first"}]},{"t":"p","runs":[{"text":"Before an automation touches anything, run it alongside the people doing the work. It receives the same inputs, produces its outputs into a table, and changes nothing. Then compare: where did it agree, where did it differ, and on the differences, who was right."}]},{"t":"p","runs":[{"text":"A week of this is the most informative week the project will have. You get a real accuracy number on real traffic, you find the input categories nobody mentioned, and you get it without a single consequence. It also converts the conversation with the team whose work this touches from a debate about whether it will work into a table of what it did."}]},{"t":"p","runs":[{"text":"It costs real money (you pay for every run and get no output) and it is worth it. Budget for a shadow week explicitly rather than discovering there is no line for it."}]},{"t":"h2","id":"then-a-canary-assigned-by-hash","runs":[{"text":"Then a canary, assigned by hash"}]},{"t":"p","runs":[{"text":"Send a small slice of live traffic to the new version. Five per cent is a reasonable start."}]},{"t":"p","runs":[{"text":"Assign the slice by hashing a stable identifier, not by rolling a die per request:"}]},{"t":"code","lang":"python","label":"Python","text":"use_new = (int(hashlib.sha256(ticket_id.encode()).hexdigest()[:8], 16) % 100) < 5"},{"t":"p","runs":[{"text":"The reason matters. With a per-request random choice, the same ticket can be handled by the old version on its first touch and the new one on its second. Users see inconsistent behaviour on one item, your comparison mixes both versions within a single case, and any per-item metric (resolution rate, edits needed, escalation) becomes uninterpretable. Hashing a stable id gives each item one version for its whole life, and the two groups stay clean."}]},{"t":"p","runs":[{"text":"Hash the entity the outcome belongs to. For a support ticket that is the ticket, or the customer if the customer’s experience across tickets is what you are measuring."}]},{"t":"h2","id":"agree-the-stopping-rule-before-you-start","runs":[{"text":"Agree the stopping rule before you start"}]},{"t":"p","runs":[{"text":"Write down, before the canary begins: the metric, the value that means stop, and how long you will watch. “We stop if escalation rate on canary tickets exceeds the control by more than three points over two hundred tickets.”"}]},{"t":"p","runs":[{"text":"Written afterwards, every number becomes an argument. Written before, the decision takes ten seconds and nobody has to be talked out of their own project."}]},{"t":"h2","id":"the-off-switch","runs":[{"text":"The off switch"}]},{"t":"p","runs":[{"text":"One flag, readable at run time, that stops the automation."}]},{"t":"p","runs":[{"text":"Put it in a database row or a configuration service, not in an environment variable, because an environment variable requires a deployment and a deployment takes longer than you have. Check it in two places: at the start of every run, and immediately before every irreversible action, so a run already in flight cannot complete a payment after somebody has pulled the lever."}]},{"t":"p","runs":[{"text":"Three requirements that are easy to skip and decide whether it works:"}]},{"t":"ul","items":[[{"text":"Under thirty seconds","b":true},{"text":" from decision to stopped."}],[{"text":"Operable by somebody who is not the author","b":true},{"text":": the person on shift at 2am, with the procedure written down."}],[{"text":"Tested monthly.","b":true},{"text":" An untested switch is a claim, and this is the claim you will make to a regulator, a customer or your own board."}]]},{"t":"h2","id":"rollback-is-a-value-not-a-rebuild","runs":[{"text":"Rollback is a value, not a rebuild"}]},{"t":"p","runs":[{"text":"Because the model id, prompt and tool schema are a single config version, rollback is setting that value back. Keep the previous version deployed and reachable rather than deleting it, and make sure the rollback path is exercised: a rollback that requires a build, a review and a deploy is not a rollback, it is a fix."}]},{"t":"h2","id":"prompt-changes-are-deployments","runs":[{"text":"Prompt changes are deployments"}]},{"t":"p","runs":[{"text":"The most common way an automation regresses in production is a prompt edit that nobody treated as a release, because it is only text and it took two minutes."}]},{"t":"p","runs":[{"text":"Give prompt changes the same path as code: version control, review, the golden task set, canary, and a note in the deployment log. The two-minute edit that goes straight to everyone is how a team ends up unable to explain a change in behaviour a week later, and it is the single most common cause of “it used to work”."}]},{"t":"h2","id":"tell-the-people-whose-work-changes","runs":[{"text":"Tell the people whose work changes"}]},{"t":"p","runs":[{"text":"Before the canary, not after. A support team that notices its queue behaving differently and was not told has had an incident, whatever your metrics say, and the trust you lose there is slower to rebuild than any technical fault."}]}],"check":{"question":[{"text":"A team canaries a new version on 5 per cent of support tickets by generating a random number on each request. Their per-ticket comparison comes out uninterpretable. What went wrong?"}],"options":[{"text":[{"text":"Five per cent is too small a slice to produce a usable signal in the time available."}],"correct":false,"tempting":[{"text":"Sample size affects the width of the error bars; it does not make a comparison structurally uninterpretable."}]},{"text":[{"text":"Random assignment introduces selection bias, so the groups are not comparable."}],"correct":false,"tempting":[{"text":"Inverts the statistics: random assignment is what removes selection bias: the flaw here is assigning repeatedly rather than once."}]},{"text":[{"text":"A ticket touched several times gets different versions on different touches, so both versions act on the same case and no per-item outcome belongs to either."}],"correct":true,"tempting":null},{"text":[{"text":"Random numbers generated per process are correlated across workers, so the split is not really 5 per cent."}],"correct":false,"tempting":[{"text":"Seeding faults can skew a split, but the reported problem is interpretability of per-ticket outcomes, not the size of the groups."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"pinning-and-migrating-models","title":"The model you tested is not the model you deployed","minutes":9},"next":{"slug":"alerts-and-the-runbook","title":"The 3am version","minutes":8}}}