{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"shipping-a-change-that-can-be-stopped","title":"Shipping a change that can be stopped","url":"https://addaly.com/learn/building-with-ai/shipping-a-change-that-can-be-stopped","position":60,"of":80,"minutes":8,"answer":"A prompt or model change goes out behind a flag, is run in shadow against real traffic first, then reaches a sticky canary slice with a stopping metric decided in advance.","keyIdea":"A prompt or model change goes out behind a flag, is run in shadow against real traffic first, then reaches a sticky canary slice with a stopping metric decided in advance, because the evaluation set catches small regressions and the canary catches disasters, and neither does the other's job."},"body":{"format":"markdown","text":"## Prompts are configuration\n\nThe prompts-in-version-control lesson made prompts code. This lesson makes them configuration at run time as well: a prompt version selected per request by a flag, so that switching versions is a flag flip that takes seconds and no deployment, and rolling back is the same flip in reverse. Every trace records which version served the request, so a complaint can be tied to a version and a rollback can be verified by looking at traces, not by hoping.\n\nThe same holds for the model identifier, the retrieval configuration and the tool definitions. Anything whose change alters the model's behaviour is a version behind a flag.\n\n## Shadow mode\n\nBefore anyone sees the new version, run it beside the old one. For a slice of real requests, call both; show the user the old response; log the new one. Then compare the pairs offline, with the judge from earlier in this block, and read the disagreements.\n\nShadow mode costs the model calls for the shadowed slice, doubled. It buys the thing the evaluation set cannot: the distribution of real traffic today, including the inputs nobody wrote a case for. A change that passes fifty cases and behaves differently on a fifth of live requests is found here and nowhere else. Run it for a few days or a few thousand requests, whichever comes second.\n\n## The canary\n\nThen a small slice of real users (five per cent) gets the new version for real. Assign by user, not by request, and make the assignment sticky: a conversational product whose assistant changes voice between one message and the next is experienced as broken, and per-request assignment also contaminates the comparison, since a conversation's later turns depend on its earlier ones.\n\nWrite the stopping rule before the canary starts, with numbers: the thumbs-down rate, the regenerate rate, the refusal rate, error rate, cost per request, latency at the 95th percentile. Each with a threshold, each with the comparison arm it is measured against, and a named person who can stop it. Automate the stop where you can; a rule that requires someone to be watching a dashboard on a Saturday is not a rule.\n\n## What a canary can and cannot see\n\nThe arithmetic from the evaluation course applies with force here. A change in thumbs-down rate from two per cent to three per cent needs thousands of responses in each arm before it is distinguishable from noise. At five per cent of a thousand requests a day, that is weeks. The canary is therefore not the instrument for small regressions; the evaluation set is. The canary catches the disaster: the version that errors on a tenth of requests, that refuses everything in Hindi, that triples cost. Its thresholds should be set for disasters (a doubling, not a tenth) and its job is to stop those in hours.\n\n## When you want to prove an improvement\n\nA change that is meant to move a metric, rather than merely not break things, needs an experiment: two arms, a metric chosen in advance, a sample size computed in advance, sticky assignment, and a run to the planned size without peeking and stopping early when the number looks good. The evaluation course's lesson on whether a difference is real is the arithmetic. Most prompt changes do not need this; they need the harness and a canary. Know which kind of change you are making before you decide how to ship it.\n\n## Model upgrades are changes\n\nA new model version behind the same alias is a larger behaviour change than most prompt edits, and it takes the same path: the full suite against the pinned new identifier, a shadow run, a canary. Teams that upgrade the model by editing one string and deploying discover, days later, that refusals rose or JSON stopped validating on a case the old model handled. The later module on deprecations covers the calendar; this lesson's point is that the path is the same.\n\n## The checklist for one change\n\n- Harness green on the new version, report attached.\n- Shadow run complete, disagreements read, count recorded.\n- Canary stopping rule written, with thresholds and an owner.\n- Rollback tested: flip the flag back once, confirm traces show the old version.\n- Version visible in every trace.\n\nFive lines. A change that has all five can be shipped on a Friday afternoon, which is the standard a release process should meet.\n\n## Say what changed\n\nKeep a changelog of behaviour, internal to the team, one line per version: what changed, why, the harness result, the canary outcome. When a user-visible behaviour changes (the assistant now declines a kind of request it used to answer) tell users, in the product, in plain words. A change explained is a change; a change discovered is a fault.\n\n> Figure: Two weeks to ship one prompt version. Timeline: Monday, Harness green against the pinned model snapshot: 94 per cent overall, golden cases at 100, cost and latency in the report attached to the pull request; Tuesday to Friday, Shadow mode on five per cent of live traffic. Both versions called, the old one shown, 2,100 pairs compared by the calibrated judge and the disagreements read by a person; Monday, week two, Canary to five per cent of users, assigned by user and sticky, with the stopping thresholds and a named owner written down before it started; Wednesday, week two, Rollback tested on purpose: the flag flipped back once, and traces confirmed the old version served the next requests; Friday, week two, Full rollout, and one line in the behaviour changelog saying what changed and why. The evaluation set catches small regressions and the canary catches disasters, and neither does the other's job. Every step is a flag flip rather than a deployment, which is what makes the rollback on Wednesday a ten-second operation instead of a release."},"check":{"question":"A team runs a canary by assigning each request, rather than each user, to the old or new prompt version with 5 per cent probability. Users of the conversational assistant complain that it keeps changing its mind mid-conversation. What went wrong?","options":[{"text":"The canary percentage was too high; at 1 per cent the changes would be rare enough not to notice.","correct":false,"tempting":"Lowering the share reduces how often it happens, not whether it happens; the flaw is the unit of assignment."},{"text":"The two prompt versions are too different and should have been made more similar before the canary.","correct":false,"tempting":"Versions are supposed to differ; the canary exists to compare them, and similarity is not the fix."},{"text":"Assignment must be per user and sticky: later turns depend on earlier ones, so switching versions mid-conversation breaks the experience and the comparison.","correct":true},{"text":"The shadow-mode phase was skipped, which would have revealed the inconsistency before launch by comparing both versions' outputs on the same live requests over several days.","correct":false,"tempting":"Shadow mode compares outputs offline and cannot reveal a problem that arises only from live per-request assignment."}]},"prev":{"slug":"error-analysis-as-a-weekly-habit","title":"Error analysis as a weekly habit","url":"https://addaly.com/learn/building-with-ai/error-analysis-as-a-weekly-habit"},"next":{"slug":"designing-for-failure","title":"The failure modes to design for","url":"https://addaly.com/learn/building-with-ai/designing-for-failure"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Shipping a change that can be stopped', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/shipping-a-change-that-can-be-stopped (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Shipping a change that can be stopped","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"A prompt or model change goes out behind a flag, is run in shadow against real traffic first, then reaches a sticky canary slice with a stopping metric decided in advance, because the evaluation set catches small regressions and the canary catches disasters, and neither does the other’s job.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"Two weeks to ship one prompt version","events":[{"when":"Monday","what":"Harness green against the pinned model snapshot: 94 per cent overall, golden cases at 100, cost and latency in the report attached to the pull request"},{"when":"Tuesday to Friday","what":"Shadow mode on five per cent of live traffic. Both versions called, the old one shown, 2,100 pairs compared by the calibrated judge and the disagreements read by a person"},{"when":"Monday, week two","what":"Canary to five per cent of users, assigned by user and sticky, with the stopping thresholds and a named owner written down before it started"},{"when":"Wednesday, week two","what":"Rollback tested on purpose: the flag flipped back once, and traces confirmed the old version served the next requests"},{"when":"Friday, week two","what":"Full rollout, and one line in the behaviour changelog saying what changed and why"}],"caption":"The evaluation set catches small regressions and the canary catches disasters, and neither does the other’s job. Every step is a flag flip rather than a deployment, which is what makes the rollback on Wednesday a ten-second operation instead of a release."},"alt":"Figure: Two weeks to ship one prompt version. Timeline: Monday, Harness green against the pinned model snapshot: 94 per cent overall, golden cases at 100, cost and latency in the report attached to the pull request; Tuesday to Friday, Shadow mode on five per cent of live traffic. Both versions called, the old one shown, 2,100 pairs compared by the calibrated judge and the disagreements read by a person; Monday, week two, Canary to five per cent of users, assigned by user and sticky, with the stopping thresholds and a named owner written down before it started; Wednesday, week two, Rollback tested on purpose: the flag flipped back once, and traces confirmed the old version served the next requests; Friday, week two, Full rollout, and one line in the behaviour changelog saying what changed and why. The evaluation set catches small regressions and the canary catches disasters, and neither does the other’s job. Every step is a flag flip rather than a deployment, which is what makes the rollback on Wednesday a ten-second operation instead of a release."}},"blocks":[{"t":"h2","id":"prompts-are-configuration","runs":[{"text":"Prompts are configuration"}]},{"t":"p","runs":[{"text":"The prompts-in-version-control lesson made prompts code. This lesson makes them configuration at run time as well: a prompt version selected per request by a flag, so that switching versions is a flag flip that takes seconds and no deployment, and rolling back is the same flip in reverse. Every trace records which version served the request, so a complaint can be tied to a version and a rollback can be verified by looking at traces, not by hoping."}]},{"t":"p","runs":[{"text":"The same holds for the model identifier, the retrieval configuration and the tool definitions. Anything whose change alters the model’s behaviour is a version behind a flag."}]},{"t":"h2","id":"shadow-mode","runs":[{"text":"Shadow mode"}]},{"t":"p","runs":[{"text":"Before anyone sees the new version, run it beside the old one. For a slice of real requests, call both; show the user the old response; log the new one. Then compare the pairs offline, with the judge from earlier in this block, and read the disagreements."}]},{"t":"p","runs":[{"text":"Shadow mode costs the model calls for the shadowed slice, doubled. It buys the thing the evaluation set cannot: the distribution of real traffic today, including the inputs nobody wrote a case for. A change that passes fifty cases and behaves differently on a fifth of live requests is found here and nowhere else. Run it for a few days or a few thousand requests, whichever comes second."}]},{"t":"h2","id":"the-canary","runs":[{"text":"The canary"}]},{"t":"p","runs":[{"text":"Then a small slice of real users (five per cent) gets the new version for real. Assign by user, not by request, and make the assignment sticky: a conversational product whose assistant changes voice between one message and the next is experienced as broken, and per-request assignment also contaminates the comparison, since a conversation’s later turns depend on its earlier ones."}]},{"t":"p","runs":[{"text":"Write the stopping rule before the canary starts, with numbers: the thumbs-down rate, the regenerate rate, the refusal rate, error rate, cost per request, latency at the 95th percentile. Each with a threshold, each with the comparison arm it is measured against, and a named person who can stop it. Automate the stop where you can; a rule that requires someone to be watching a dashboard on a Saturday is not a rule."}]},{"t":"h2","id":"what-a-canary-can-and-cannot-see","runs":[{"text":"What a canary can and cannot see"}]},{"t":"p","runs":[{"text":"The arithmetic from the evaluation course applies with force here. A change in thumbs-down rate from two per cent to three per cent needs thousands of responses in each arm before it is distinguishable from noise. At five per cent of a thousand requests a day, that is weeks. The canary is therefore not the instrument for small regressions; the evaluation set is. The canary catches the disaster: the version that errors on a tenth of requests, that refuses everything in Hindi, that triples cost. Its thresholds should be set for disasters (a doubling, not a tenth) and its job is to stop those in hours."}]},{"t":"h2","id":"when-you-want-to-prove-an-improvement","runs":[{"text":"When you want to prove an improvement"}]},{"t":"p","runs":[{"text":"A change that is meant to move a metric, rather than merely not break things, needs an experiment: two arms, a metric chosen in advance, a sample size computed in advance, sticky assignment, and a run to the planned size without peeking and stopping early when the number looks good. The evaluation course’s lesson on whether a difference is real is the arithmetic. Most prompt changes do not need this; they need the harness and a canary. Know which kind of change you are making before you decide how to ship it."}]},{"t":"h2","id":"model-upgrades-are-changes","runs":[{"text":"Model upgrades are changes"}]},{"t":"p","runs":[{"text":"A new model version behind the same alias is a larger behaviour change than most prompt edits, and it takes the same path: the full suite against the pinned new identifier, a shadow run, a canary. Teams that upgrade the model by editing one string and deploying discover, days later, that refusals rose or JSON stopped validating on a case the old model handled. The later module on deprecations covers the calendar; this lesson’s point is that the path is the same."}]},{"t":"h2","id":"the-checklist-for-one-change","runs":[{"text":"The checklist for one change"}]},{"t":"ul","items":[[{"text":"Harness green on the new version, report attached."}],[{"text":"Shadow run complete, disagreements read, count recorded."}],[{"text":"Canary stopping rule written, with thresholds and an owner."}],[{"text":"Rollback tested: flip the flag back once, confirm traces show the old version."}],[{"text":"Version visible in every trace."}]]},{"t":"p","runs":[{"text":"Five lines. A change that has all five can be shipped on a Friday afternoon, which is the standard a release process should meet."}]},{"t":"h2","id":"say-what-changed","runs":[{"text":"Say what changed"}]},{"t":"p","runs":[{"text":"Keep a changelog of behaviour, internal to the team, one line per version: what changed, why, the harness result, the canary outcome. When a user-visible behaviour changes (the assistant now declines a kind of request it used to answer) tell users, in the product, in plain words. A change explained is a change; a change discovered is a fault."}]}],"check":{"question":[{"text":"A team runs a canary by assigning each request, rather than each user, to the old or new prompt version with 5 per cent probability. Users of the conversational assistant complain that it keeps changing its mind mid-conversation. What went wrong?"}],"options":[{"text":[{"text":"The canary percentage was too high; at 1 per cent the changes would be rare enough not to notice."}],"correct":false,"tempting":[{"text":"Lowering the share reduces how often it happens, not whether it happens; the flaw is the unit of assignment."}]},{"text":[{"text":"The two prompt versions are too different and should have been made more similar before the canary."}],"correct":false,"tempting":[{"text":"Versions are supposed to differ; the canary exists to compare them, and similarity is not the fix."}]},{"text":[{"text":"Assignment must be per user and sticky: later turns depend on earlier ones, so switching versions mid-conversation breaks the experience and the comparison."}],"correct":true,"tempting":null},{"text":[{"text":"The shadow-mode phase was skipped, which would have revealed the inconsistency before launch by comparing both versions’ outputs on the same live requests over several days."}],"correct":false,"tempting":[{"text":"Shadow mode compares outputs offline and cannot reveal a problem that arises only from live per-request assignment."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"error-analysis-as-a-weekly-habit","title":"Error analysis as a weekly habit","minutes":8},"next":{"slug":"designing-for-failure","title":"The failure modes to design for","minutes":9}}}