Shadow, canary, and the off switch

Shadow mode first

Before an automation touches anything, run it alongside the people doing the work. It receives the same inputs, produces its outputs into a table, and changes nothing. Then compare: where did it agree, where did it differ, and on the differences, who was right.

A week of this is the most informative week the project will have. You get a real accuracy number on real traffic, you find the input categories nobody mentioned, and you get it without a single consequence. It also converts the conversation with the team whose work this touches from a debate about whether it will work into a table of what it did.

It costs real money (you pay for every run and get no output) and it is worth it. Budget for a shadow week explicitly rather than discovering there is no line for it.

Then a canary, assigned by hash

Send a small slice of live traffic to the new version. Five per cent is a reasonable start.

Assign the slice by hashing a stable identifier, not by rolling a die per request:

use_new = (int(hashlib.sha256(ticket_id.encode()).hexdigest()[:8], 16) % 100) < 5

The reason matters. With a per-request random choice, the same ticket can be handled by the old version on its first touch and the new one on its second. Users see inconsistent behaviour on one item, your comparison mixes both versions within a single case, and any per-item metric (resolution rate, edits needed, escalation) becomes uninterpretable. Hashing a stable id gives each item one version for its whole life, and the two groups stay clean.

Hash the entity the outcome belongs to. For a support ticket that is the ticket, or the customer if the customer’s experience across tickets is what you are measuring.

Agree the stopping rule before you start

Write down, before the canary begins: the metric, the value that means stop, and how long you will watch. “We stop if escalation rate on canary tickets exceeds the control by more than three points over two hundred tickets.”

Written afterwards, every number becomes an argument. Written before, the decision takes ten seconds and nobody has to be talked out of their own project.

The off switch

One flag, readable at run time, that stops the automation.

Put it in a database row or a configuration service, not in an environment variable, because an environment variable requires a deployment and a deployment takes longer than you have. Check it in two places: at the start of every run, and immediately before every irreversible action, so a run already in flight cannot complete a payment after somebody has pulled the lever.

Three requirements that are easy to skip and decide whether it works:

• Under thirty seconds from decision to stopped.
• Operable by somebody who is not the author: the person on shift at 2am, with the procedure written down.
• Tested monthly. An untested switch is a claim, and this is the claim you will make to a regulator, a customer or your own board.

Rollback is a value, not a rebuild

Because the model id, prompt and tool schema are a single config version, rollback is setting that value back. Keep the previous version deployed and reachable rather than deleting it, and make sure the rollback path is exercised: a rollback that requires a build, a review and a deploy is not a rollback, it is a fix.

Prompt changes are deployments

The most common way an automation regresses in production is a prompt edit that nobody treated as a release, because it is only text and it took two minutes.

Give prompt changes the same path as code: version control, review, the golden task set, canary, and a note in the deployment log. The two-minute edit that goes straight to everyone is how a team ends up unable to explain a change in behaviour a week later, and it is the single most common cause of “it used to work”.

Tell the people whose work changes

Before the canary, not after. A support team that notices its queue behaving differently and was not told has had an incident, whatever your metrics say, and the trust you lose there is slower to rebuild than any technical fault.