Verification that continues after merge

Everything before deployment is a model of production

Your tests run against a fixture. Your staging environment has a thousand rows and production has forty million. Your integration tests mock the payment provider, which in real life returns a 502 for ninety seconds every few weeks.

That gap is permanent. No amount of pre-merge verification closes it, which means the last and most informative oracle is the release itself, provided the release is designed so that being wrong is survivable.

This is the layer that most teams skip and it is the one that gets cheaper every year.

The four mechanisms, weakest to strongest

Feature flags. The code is deployed and inactive. Deployment and release become separate events, which is the whole point: you can deploy at 11am on a Tuesday with the flag off, and turn it on when the right people are watching.

if flags.enabled("new_pricing", user=user):
    return new_pricing(order)
return old_pricing(order)

Two disciplines make this work. Every flag has an owner and a removal date, because a codebase with 200 stale flags has 2^200 configurations and no one can reason about any of them. And the flag check goes in one place, not scattered through the call stack.

Canary. Route a small share of traffic to the new version and watch. The useful question is how small and how long, and there is arithmetic for it: to observe an event that happens once in a thousand requests, you need on the order of a few thousand requests through the canary before its absence means anything. At 1% of a service doing 100 requests per second, that is a few minutes. At 1% of a service doing two requests per second, it is most of a day, and a five-minute canary there has told you nothing at all.

Work out that number for your traffic before you choose a canary duration. Most canary windows are chosen by feel and are far too short to detect what they are nominally watching for.

Shadow mode. Run the new path on real traffic, discard its output, compare it with the old path’s. All the information, none of the risk. This is the same technique as differential testing, applied at full production scale, and it is the strongest verification available for a rewrite. The same rule applies: no side effects on the shadow path.

Progressive rollout with an automatic stop. 1%, 5%, 25%, 50%, 100%, with a check between each step that can halt or reverse it.

The metric that stops the rollout

A rollout that only a human can stop is stopped late, because the human is asleep, in a meeting, or looking at a different dashboard. Pick the halt condition in advance and wire it up.

What works: error rate on the affected endpoints, p99 latency, and one business metric that is specific to the change: completed checkouts, successful logins, messages delivered. That last one catches the failures that do not raise exceptions, which are the expensive ones. A pricing bug that charges everyone zero produces a perfect error rate and a perfect latency graph.

What does not work: CPU, memory, general dashboards nobody has calibrated, and any alert with a history of firing spuriously.

Reversibility is the property that makes all of this work

Before any release, ask the question plainly: how do we undo this, and how long does it take?

Code is easy: redeploy the previous image, seconds to minutes. The things that are not reversible are what should shape the plan:

• A migration that drops a column. Do it in two releases: stop writing the column, deploy, wait, then drop it in a later change once you are sure nothing needs it. The gap should be days, not minutes.
• A message sent to a user. Cannot be recalled. Rate-limit new notification paths on their first day and have somebody watch the first hundred.
• A charge to a card. Refundable, expensively and visibly.
• A cache format change that the old version cannot read, which makes rollback impossible until the cache expires. Version your serialised formats.
• Anything written to another company’s system.

A change made mostly of reversible parts can be released quickly and verified in production. A change containing one irreversible part needs the full weight of pre-merge verification on that part specifically, and the rest can go fast.

That is the practical shape of the whole module: know which of your oracles applies where, spend the expensive human attention on the part that cannot be undone, and let the cheap automatic ones cover everything else.