{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"release-as-verification","title":"Verification that continues after merge","url":"https://addaly.com/learn/how-software-changed/release-as-verification","position":29,"of":87,"minutes":9,"answer":"Production is the last and most informative oracle, so design releases to be reversible and pick the halt metric and canary duration from your actual traffic rate rather than by feel.","keyIdea":"Production is the last and most informative oracle, so design releases to be reversible and pick the halt metric and canary duration from your actual traffic rate rather than by feel."},"body":{"format":"markdown","text":"## Everything before deployment is a model of production\n\nYour tests run against a fixture. Your staging environment has a thousand rows and production has forty million. Your integration tests mock the payment provider, which in real life returns a 502 for ninety seconds every few weeks.\n\nThat gap is permanent. No amount of pre-merge verification closes it, which means the last and most informative oracle is the release itself, provided the release is designed so that being wrong is survivable.\n\nThis is the layer that most teams skip and it is the one that gets cheaper every year.\n\n> Figure: Four release mechanisms, weakest first. Layers, top to bottom: Feature flags (Deployed and inactive, so deployment and release become separate events. Every flag needs an owner and a removal date, or the configuration space stops being reasonable about.); Canary (A small share of traffic, for long enough to see the thing you are watching for. At two requests a second, one per cent for five minutes has told you nothing at all.); Shadow mode (The new path runs on real traffic, its output is discarded and compared. All the information and none of the risk, provided the shadow path writes nothing.); Progressive rollout with an automatic stop (One, five, twenty-five, fifty, a hundred per cent, with the halt condition wired up in advance: error rate, p99 latency, and one business metric that is specific to the change.). Production is the last and most informative oracle, and the only one that has forty million rows in it. It is worth reaching for only if being wrong there is survivable.\n\n## The four mechanisms, weakest to strongest\n\n**Feature flags.** The code is deployed and inactive. Deployment and release become separate events, which is the whole point: you can deploy at 11am on a Tuesday with the flag off, and turn it on when the right people are watching.\n\n```python\nif flags.enabled(\"new_pricing\", user=user):\n    return new_pricing(order)\nreturn old_pricing(order)\n```\n\nTwo disciplines make this work. Every flag has an owner and a removal date, because a codebase with 200 stale flags has 2^200 configurations and no one can reason about any of them. And the flag check goes in one place, not scattered through the call stack.\n\n**Canary.** Route a small share of traffic to the new version and watch. The useful question is how small and how long, and there is arithmetic for it: to observe an event that happens once in a thousand requests, you need on the order of a few thousand requests through the canary before its absence means anything. At 1% of a service doing 100 requests per second, that is a few minutes. At 1% of a service doing two requests per second, it is most of a day, and a five-minute canary there has told you nothing at all.\n\nWork out that number for your traffic before you choose a canary duration. Most canary windows are chosen by feel and are far too short to detect what they are nominally watching for.\n\n**Shadow mode.** Run the new path on real traffic, discard its output, compare it with the old path's. All the information, none of the risk. This is the same technique as differential testing, applied at full production scale, and it is the strongest verification available for a rewrite. The same rule applies: no side effects on the shadow path.\n\n**Progressive rollout with an automatic stop.** 1%, 5%, 25%, 50%, 100%, with a check between each step that can halt or reverse it.\n\n## The metric that stops the rollout\n\nA rollout that only a human can stop is stopped late, because the human is asleep, in a meeting, or looking at a different dashboard. Pick the halt condition in advance and wire it up.\n\nWhat works: error rate on the affected endpoints, p99 latency, and one business metric that is specific to the change: completed checkouts, successful logins, messages delivered. That last one catches the failures that do not raise exceptions, which are the expensive ones. A pricing bug that charges everyone zero produces a perfect error rate and a perfect latency graph.\n\nWhat does not work: CPU, memory, general dashboards nobody has calibrated, and any alert with a history of firing spuriously.\n\n## Reversibility is the property that makes all of this work\n\nBefore any release, ask the question plainly: **how do we undo this, and how long does it take?**\n\nCode is easy: redeploy the previous image, seconds to minutes. The things that are not reversible are what should shape the plan:\n\n- **A migration that drops a column.** Do it in two releases: stop writing the column, deploy, wait, then drop it in a later change once you are sure nothing needs it. The gap should be days, not minutes.\n- **A message sent to a user.** Cannot be recalled. Rate-limit new notification paths on their first day and have somebody watch the first hundred.\n- **A charge to a card.** Refundable, expensively and visibly.\n- **A cache format change** that the old version cannot read, which makes rollback impossible until the cache expires. Version your serialised formats.\n- **Anything written to another company's system.**\n\nA change made mostly of reversible parts can be released quickly and verified in production. A change containing one irreversible part needs the full weight of pre-merge verification on that part specifically, and the rest can go fast.\n\nThat is the practical shape of the whole module: know which of your oracles applies where, spend the expensive human attention on the part that cannot be undone, and let the cheap automatic ones cover everything else."},"check":{"question":"A service handling two requests per second canaries a change at 1% of traffic for five minutes, sees no errors, and rolls out fully. What is wrong with this as verification?","options":[{"text":"Five minutes is too short for latency effects to appear, since caches and connection pools need time to warm.","correct":false,"tempting":"Warm-up is a real confounder in canaries, but it would produce misleading signal rather than the near-total absence of signal here."},{"text":"The canary received roughly six requests, so seeing no errors carries almost no information.","correct":true},{"text":"One percent is too small a share to expose the change to representative traffic, and 5% would have been a sound choice.","correct":false,"tempting":"Adjusts the percentage while leaving the underlying problem (absolute request count) unaddressed at this traffic rate."},{"text":"Canarying is unsuitable for low-traffic services, so the team should have used a feature flag with an internal-users-only rule instead.","correct":false,"tempting":"Names a genuinely better technique here, but frames canarying as inapplicable rather than as needing a duration derived from traffic."}]},"prev":{"slug":"ci-as-the-gate","title":"Designing a pipeline that is actually a gate","url":"https://addaly.com/learn/how-software-changed/ci-as-the-gate"},"next":{"slug":"specification-got-more-valuable","title":"Why writing it down matters more now","url":"https://addaly.com/learn/how-software-changed/specification-got-more-valuable"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Verification that continues after merge', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/release-as-verification (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Verification that continues after merge","courseTitle":"How Software Development Changed","module":{"index":3,"slug":"oracles-and-verification","title":"Building oracles that stay ahead"},"answer":null,"keyIdea":"Production is the last and most informative oracle, so design releases to be reversible and pick the halt metric and canary duration from your actual traffic rate rather than by feel.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Four release mechanisms, weakest first","layers":[{"label":"Feature flags","detail":"Deployed and inactive, so deployment and release become separate events. Every flag needs an owner and a removal date, or the configuration space stops being reasonable about."},{"label":"Canary","detail":"A small share of traffic, for long enough to see the thing you are watching for. At two requests a second, one per cent for five minutes has told you nothing at all."},{"label":"Shadow mode","detail":"The new path runs on real traffic, its output is discarded and compared. All the information and none of the risk, provided the shadow path writes nothing."},{"label":"Progressive rollout with an automatic stop","detail":"One, five, twenty-five, fifty, a hundred per cent, with the halt condition wired up in advance: error rate, p99 latency, and one business metric that is specific to the change."}],"caption":"Production is the last and most informative oracle, and the only one that has forty million rows in it. It is worth reaching for only if being wrong there is survivable."},"alt":"Figure: Four release mechanisms, weakest first. Layers, top to bottom: Feature flags (Deployed and inactive, so deployment and release become separate events. Every flag needs an owner and a removal date, or the configuration space stops being reasonable about.); Canary (A small share of traffic, for long enough to see the thing you are watching for. At two requests a second, one per cent for five minutes has told you nothing at all.); Shadow mode (The new path runs on real traffic, its output is discarded and compared. All the information and none of the risk, provided the shadow path writes nothing.); Progressive rollout with an automatic stop (One, five, twenty-five, fifty, a hundred per cent, with the halt condition wired up in advance: error rate, p99 latency, and one business metric that is specific to the change.). Production is the last and most informative oracle, and the only one that has forty million rows in it. It is worth reaching for only if being wrong there is survivable."}},"blocks":[{"t":"h2","id":"everything-before-deployment-is-a-model-of-production","runs":[{"text":"Everything before deployment is a model of production"}]},{"t":"p","runs":[{"text":"Your tests run against a fixture. Your staging environment has a thousand rows and production has forty million. Your integration tests mock the payment provider, which in real life returns a 502 for ninety seconds every few weeks."}]},{"t":"p","runs":[{"text":"That gap is permanent. No amount of pre-merge verification closes it, which means the last and most informative oracle is the release itself, provided the release is designed so that being wrong is survivable."}]},{"t":"p","runs":[{"text":"This is the layer that most teams skip and it is the one that gets cheaper every year."}]},{"t":"h2","id":"the-four-mechanisms-weakest-to-strongest","runs":[{"text":"The four mechanisms, weakest to strongest"}]},{"t":"p","runs":[{"text":"Feature flags.","b":true},{"text":" The code is deployed and inactive. Deployment and release become separate events, which is the whole point: you can deploy at 11am on a Tuesday with the flag off, and turn it on when the right people are watching."}]},{"t":"code","lang":"python","label":"Python","text":"if flags.enabled(\"new_pricing\", user=user):\n    return new_pricing(order)\nreturn old_pricing(order)"},{"t":"p","runs":[{"text":"Two disciplines make this work. Every flag has an owner and a removal date, because a codebase with 200 stale flags has 2^200 configurations and no one can reason about any of them. And the flag check goes in one place, not scattered through the call stack."}]},{"t":"p","runs":[{"text":"Canary.","b":true},{"text":" Route a small share of traffic to the new version and watch. The useful question is how small and how long, and there is arithmetic for it: to observe an event that happens once in a thousand requests, you need on the order of a few thousand requests through the canary before its absence means anything. At 1% of a service doing 100 requests per second, that is a few minutes. At 1% of a service doing two requests per second, it is most of a day, and a five-minute canary there has told you nothing at all."}]},{"t":"p","runs":[{"text":"Work out that number for your traffic before you choose a canary duration. Most canary windows are chosen by feel and are far too short to detect what they are nominally watching for."}]},{"t":"p","runs":[{"text":"Shadow mode.","b":true},{"text":" Run the new path on real traffic, discard its output, compare it with the old path’s. All the information, none of the risk. This is the same technique as differential testing, applied at full production scale, and it is the strongest verification available for a rewrite. The same rule applies: no side effects on the shadow path."}]},{"t":"p","runs":[{"text":"Progressive rollout with an automatic stop.","b":true},{"text":" 1%, 5%, 25%, 50%, 100%, with a check between each step that can halt or reverse it."}]},{"t":"h2","id":"the-metric-that-stops-the-rollout","runs":[{"text":"The metric that stops the rollout"}]},{"t":"p","runs":[{"text":"A rollout that only a human can stop is stopped late, because the human is asleep, in a meeting, or looking at a different dashboard. Pick the halt condition in advance and wire it up."}]},{"t":"p","runs":[{"text":"What works: error rate on the affected endpoints, p99 latency, and one business metric that is specific to the change: completed checkouts, successful logins, messages delivered. That last one catches the failures that do not raise exceptions, which are the expensive ones. A pricing bug that charges everyone zero produces a perfect error rate and a perfect latency graph."}]},{"t":"p","runs":[{"text":"What does not work: CPU, memory, general dashboards nobody has calibrated, and any alert with a history of firing spuriously."}]},{"t":"h2","id":"reversibility-is-the-property-that-makes-all-of-this-work","runs":[{"text":"Reversibility is the property that makes all of this work"}]},{"t":"p","runs":[{"text":"Before any release, ask the question plainly: "},{"text":"how do we undo this, and how long does it take?","b":true}]},{"t":"p","runs":[{"text":"Code is easy: redeploy the previous image, seconds to minutes. The things that are not reversible are what should shape the plan:"}]},{"t":"ul","items":[[{"text":"A migration that drops a column.","b":true},{"text":" Do it in two releases: stop writing the column, deploy, wait, then drop it in a later change once you are sure nothing needs it. The gap should be days, not minutes."}],[{"text":"A message sent to a user.","b":true},{"text":" Cannot be recalled. Rate-limit new notification paths on their first day and have somebody watch the first hundred."}],[{"text":"A charge to a card.","b":true},{"text":" Refundable, expensively and visibly."}],[{"text":"A cache format change","b":true},{"text":" that the old version cannot read, which makes rollback impossible until the cache expires. Version your serialised formats."}],[{"text":"Anything written to another company’s system.","b":true}]]},{"t":"p","runs":[{"text":"A change made mostly of reversible parts can be released quickly and verified in production. A change containing one irreversible part needs the full weight of pre-merge verification on that part specifically, and the rest can go fast."}]},{"t":"p","runs":[{"text":"That is the practical shape of the whole module: know which of your oracles applies where, spend the expensive human attention on the part that cannot be undone, and let the cheap automatic ones cover everything else."}]}],"check":{"question":[{"text":"A service handling two requests per second canaries a change at 1% of traffic for five minutes, sees no errors, and rolls out fully. What is wrong with this as verification?"}],"options":[{"text":[{"text":"Five minutes is too short for latency effects to appear, since caches and connection pools need time to warm."}],"correct":false,"tempting":[{"text":"Warm-up is a real confounder in canaries, but it would produce misleading signal rather than the near-total absence of signal here."}]},{"text":[{"text":"The canary received roughly six requests, so seeing no errors carries almost no information."}],"correct":true,"tempting":null},{"text":[{"text":"One percent is too small a share to expose the change to representative traffic, and 5% would have been a sound choice."}],"correct":false,"tempting":[{"text":"Adjusts the percentage while leaving the underlying problem (absolute request count) unaddressed at this traffic rate."}]},{"text":[{"text":"Canarying is unsuitable for low-traffic services, so the team should have used a feature flag with an internal-users-only rule instead."}],"correct":false,"tempting":[{"text":"Names a genuinely better technique here, but frames canarying as inapplicable rather than as needing a duration derived from traffic."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"ci-as-the-gate","title":"Designing a pipeline that is actually a gate","minutes":9},"next":{"slug":"specification-got-more-valuable","title":"Why writing it down matters more now","minutes":8}}}