Shipping a change that can be stopped

Prompts are configuration

The prompts-in-version-control lesson made prompts code. This lesson makes them configuration at run time as well: a prompt version selected per request by a flag, so that switching versions is a flag flip that takes seconds and no deployment, and rolling back is the same flip in reverse. Every trace records which version served the request, so a complaint can be tied to a version and a rollback can be verified by looking at traces, not by hoping.

The same holds for the model identifier, the retrieval configuration and the tool definitions. Anything whose change alters the model’s behaviour is a version behind a flag.

Shadow mode

Before anyone sees the new version, run it beside the old one. For a slice of real requests, call both; show the user the old response; log the new one. Then compare the pairs offline, with the judge from earlier in this block, and read the disagreements.

Shadow mode costs the model calls for the shadowed slice, doubled. It buys the thing the evaluation set cannot: the distribution of real traffic today, including the inputs nobody wrote a case for. A change that passes fifty cases and behaves differently on a fifth of live requests is found here and nowhere else. Run it for a few days or a few thousand requests, whichever comes second.

The canary

Then a small slice of real users (five per cent) gets the new version for real. Assign by user, not by request, and make the assignment sticky: a conversational product whose assistant changes voice between one message and the next is experienced as broken, and per-request assignment also contaminates the comparison, since a conversation’s later turns depend on its earlier ones.

Write the stopping rule before the canary starts, with numbers: the thumbs-down rate, the regenerate rate, the refusal rate, error rate, cost per request, latency at the 95th percentile. Each with a threshold, each with the comparison arm it is measured against, and a named person who can stop it. Automate the stop where you can; a rule that requires someone to be watching a dashboard on a Saturday is not a rule.

What a canary can and cannot see

The arithmetic from the evaluation course applies with force here. A change in thumbs-down rate from two per cent to three per cent needs thousands of responses in each arm before it is distinguishable from noise. At five per cent of a thousand requests a day, that is weeks. The canary is therefore not the instrument for small regressions; the evaluation set is. The canary catches the disaster: the version that errors on a tenth of requests, that refuses everything in Hindi, that triples cost. Its thresholds should be set for disasters (a doubling, not a tenth) and its job is to stop those in hours.

When you want to prove an improvement

A change that is meant to move a metric, rather than merely not break things, needs an experiment: two arms, a metric chosen in advance, a sample size computed in advance, sticky assignment, and a run to the planned size without peeking and stopping early when the number looks good. The evaluation course’s lesson on whether a difference is real is the arithmetic. Most prompt changes do not need this; they need the harness and a canary. Know which kind of change you are making before you decide how to ship it.

Model upgrades are changes

A new model version behind the same alias is a larger behaviour change than most prompt edits, and it takes the same path: the full suite against the pinned new identifier, a shadow run, a canary. Teams that upgrade the model by editing one string and deploying discover, days later, that refusals rose or JSON stopped validating on a case the old model handled. The later module on deprecations covers the calendar; this lesson’s point is that the path is the same.

The checklist for one change

• Harness green on the new version, report attached.
• Shadow run complete, disagreements read, count recorded.
• Canary stopping rule written, with thresholds and an owner.
• Rollback tested: flip the flag back once, confirm traces show the old version.
• Version visible in every trace.

Five lines. A change that has all five can be shipped on a Friday afternoon, which is the standard a release process should meet.

Say what changed

Keep a changelog of behaviour, internal to the team, one line per version: what changed, why, the harness result, the canary outcome. When a user-visible behaviour changes (the assistant now declines a kind of request it used to answer) tell users, in the product, in plain words. A change explained is a change; a change discovered is a fault.