Rolling out a prompt like a deployment

The asymmetry nobody plans for

A bad code deploy usually announces itself. Exceptions, failing health checks, a graph falling over. Somebody notices within minutes.

A bad prompt deploy produces a system that runs perfectly, returns two hundreds, and answers slightly worse. No exception. No alert. The graph of request volume is identical. You find out from a support ticket eleven days later, and by then eleven days of changes sit between you and the cause.

So the controls have to be structural, because the feedback will not arrive in time to be useful.

One object, one version

Everything that shapes the window belongs in a single versioned configuration: the prompt text, k, the chunk header fields, the reranker setting, the model id, the temperature, the truncation limits, the cache breakpoint position.

CTX_V38 = ContextConfig(
    version="ctx-v38",
    system=SYSTEM_TEXT_V12,
    k=6, rerank=True, header_fields=["path", "handle"],
    model="provider/model-2026-08-14",
    max_tool_tokens=3000, output_reserve=1200,
)

One object means one thing to log, one thing to revert, one thing to compare. The failure mode it prevents is the common one: the prompt lives in a template file, k lives in an environment variable, the model id lives in a different service’s config, and when quality drops nobody can state what the system was doing last Tuesday, because the answer is spread across three repositories and a deploy history.

Shadow before you serve

For a fraction of live traffic (five percent is plenty) assemble and call both configurations. Serve the old answer. Log both.

This is the most informative test available, because it runs on real traffic rather than on your golden set, which by construction contains only the cases you thought of. It costs double tokens on the shadowed fraction and nothing else, and it carries no user risk at all.

What to compare: token counts and latency directly, cache hit rate directly, and answer differences by sampling. Most requests will produce near-identical answers. The ones that diverge are your review queue, and fifty of them read carefully tells you more than any aggregate score.

Canary, then widen

Serve the new configuration to a small percentage. One percent, then five, then twenty-five, then all, with time between steps.

Watch the outcome metrics, not the evaluation metrics. Evaluation told you it should be better; the canary tells you what actually happened to the things you care about. Escalation rate, thumbs-down rate, abandonment, repeat-question rate, cost per resolved conversation. These are noisy and slow, which is why you spend a day at each step rather than an hour.

Hold the assignment stable per user. A user flipping between two prompts mid-conversation gets an incoherent experience and pollutes your comparison.

The kill switch is the important part

If you take one thing from this lesson: you must be able to revert a context change without a deploy.

The configuration id lives somewhere you can change in seconds: a feature flag, a config service, a database row read per request with a short cache. Reverting is setting it back.

The reason this matters more here than for code is the detection lag. A prompt regression is found late, often out of hours, often by somebody who is not the person who shipped it. If reverting requires a build, a review and a deploy pipeline, the outage is extended by an hour of process at the worst possible time. If it requires changing a value, it is over in a minute and you diagnose in the morning.

Test the revert. Not once, on a schedule, the way you would test a backup restore: a kill switch nobody has pulled since it was built is an untested path in the one situation where you need it to work first time.

Change one thing per version

The rule from the significance lesson applies with more force here, because production is noisier than an evaluation.

A version that changes the prompt, raises k and switches model has three possible causes for any regression and no way to separate them. Untangling it under pressure means reverting everything and re-shipping one at a time, which is exactly the work you were trying to avoid by bundling.

Small versions ship more often, revert cleanly, and produce a history someone can read. ctx-v38: k 8→6 is a changelog entry that will still make sense in a year. ctx-v38: prompt improvements is not.

Stamp it on the response

Every response carries its configuration version back to the client, and the client shows it somewhere unobtrusive: a tooltip, a debug panel, the support form.

This single habit turns “the assistant has been worse lately” into “three complaints, all on ctx-v38, none on v37”. It costs one field, and it converts the vaguest class of bug report into the most actionable one.