{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"shipping-a-context-change","title":"Rolling out a prompt like a deployment","url":"https://addaly.com/learn/context-engineering/shipping-a-context-change","position":85,"of":87,"minutes":9,"answer":"A prompt change is a production change with no type system and no compiler, so it needs the same machinery any risky deployment gets: one versioned configuration object, a shadow run on real traffic.","keyIdea":"A prompt change is a production change with no type system and no compiler, so it needs the same machinery any risky deployment gets: one versioned configuration object, a shadow run on real traffic, a percentage rollout, and a revert that does not require a deploy."},"body":{"format":"markdown","text":"## The asymmetry nobody plans for\n\nA bad code deploy usually announces itself. Exceptions, failing health checks, a graph falling over. Somebody notices within minutes.\n\nA bad prompt deploy produces a system that runs perfectly, returns two hundreds, and answers slightly worse. No exception. No alert. The graph of request volume is identical. You find out from a support ticket eleven days later, and by then eleven days of changes sit between you and the cause.\n\nSo the controls have to be structural, because the feedback will not arrive in time to be useful.\n\n## One object, one version\n\nEverything that shapes the window belongs in a single versioned configuration: the prompt text, k, the chunk header fields, the reranker setting, the model id, the temperature, the truncation limits, the cache breakpoint position.\n\n```python\nCTX_V38 = ContextConfig(\n    version=\"ctx-v38\",\n    system=SYSTEM_TEXT_V12,\n    k=6, rerank=True, header_fields=[\"path\", \"handle\"],\n    model=\"provider/model-2026-08-14\",\n    max_tool_tokens=3000, output_reserve=1200,\n)\n```\n\nOne object means one thing to log, one thing to revert, one thing to compare. The failure mode it prevents is the common one: the prompt lives in a template file, k lives in an environment variable, the model id lives in a different service's config, and when quality drops nobody can state what the system was doing last Tuesday, because the answer is spread across three repositories and a deploy history.\n\n## Shadow before you serve\n\nFor a fraction of live traffic (five percent is plenty) assemble and call both configurations. Serve the old answer. Log both.\n\nThis is the most informative test available, because it runs on real traffic rather than on your golden set, which by construction contains only the cases you thought of. It costs double tokens on the shadowed fraction and nothing else, and it carries no user risk at all.\n\nWhat to compare: token counts and latency directly, cache hit rate directly, and answer differences by sampling. Most requests will produce near-identical answers. The ones that diverge are your review queue, and fifty of them read carefully tells you more than any aggregate score.\n\n## Canary, then widen\n\nServe the new configuration to a small percentage. One percent, then five, then twenty-five, then all, with time between steps.\n\nWatch the outcome metrics, not the evaluation metrics. Evaluation told you it should be better; the canary tells you what actually happened to the things you care about. Escalation rate, thumbs-down rate, abandonment, repeat-question rate, cost per resolved conversation. These are noisy and slow, which is why you spend a day at each step rather than an hour.\n\nHold the assignment stable per user. A user flipping between two prompts mid-conversation gets an incoherent experience and pollutes your comparison.\n\n> Figure: Shipping one context version. Timeline: Day 0, One configuration object: prompt text, k, header fields, reranker, model id, truncation limits. One change from the version before it; Day 0, The held-out golden set, and the offline assertions on the assembled request, which need no model call; Days 1 to 3, Shadow run on five per cent of live traffic. Both configurations assembled and called, the old answer served, both logged; Day 4, One per cent of users, with the assignment held stable per user so nobody flips mid-conversation; Days 5 to 8, Five, then twenty-five, then all, watching escalation rate and thumbs-down rather than the evaluation score; Any hour, Revert by setting the configuration id back, with no deploy. Test that path on a schedule, as you would a backup restore. A bad prompt deploy returns two hundreds and answers slightly worse, so the feedback will not arrive in time to be useful and the controls have to be structural. Stamp the version on every response and the vaguest class of bug report becomes the most actionable: three complaints, all on v38, none on v37.\n\n## The kill switch is the important part\n\nIf you take one thing from this lesson: **you must be able to revert a context change without a deploy.**\n\nThe configuration id lives somewhere you can change in seconds: a feature flag, a config service, a database row read per request with a short cache. Reverting is setting it back.\n\nThe reason this matters more here than for code is the detection lag. A prompt regression is found late, often out of hours, often by somebody who is not the person who shipped it. If reverting requires a build, a review and a deploy pipeline, the outage is extended by an hour of process at the worst possible time. If it requires changing a value, it is over in a minute and you diagnose in the morning.\n\nTest the revert. Not once, on a schedule, the way you would test a backup restore: a kill switch nobody has pulled since it was built is an untested path in the one situation where you need it to work first time.\n\n## Change one thing per version\n\nThe rule from the significance lesson applies with more force here, because production is noisier than an evaluation.\n\nA version that changes the prompt, raises k and switches model has three possible causes for any regression and no way to separate them. Untangling it under pressure means reverting everything and re-shipping one at a time, which is exactly the work you were trying to avoid by bundling.\n\nSmall versions ship more often, revert cleanly, and produce a history someone can read. `ctx-v38: k 8→6` is a changelog entry that will still make sense in a year. `ctx-v38: prompt improvements` is not.\n\n## Stamp it on the response\n\nEvery response carries its configuration version back to the client, and the client shows it somewhere unobtrusive: a tooltip, a debug panel, the support form.\n\nThis single habit turns \"the assistant has been worse lately\" into \"three complaints, all on ctx-v38, none on v37\". It costs one field, and it converts the vaguest class of bug report into the most actionable one."},"check":{"question":"Why does a prompt regression need a revert path that avoids the deploy pipeline more urgently than a code regression does?","options":[{"text":"Prompt changes cannot be rolled back by redeploying, because the previous text is not kept in version control.","correct":false,"tempting":"It should be in version control, and if it is not that is the bug to fix; the pipeline's speed is the real issue here."},{"text":"Prompt changes affect all users at once, whereas code deploys roll out gradually by instance.","correct":false,"tempting":"Both can be staged or not; this is a property of your rollout mechanism rather than of what changed."},{"text":"It is found late and often out of hours, so a deploy adds an hour to an already extended problem.","correct":true},{"text":"Providers cache prompt prefixes, so an old prompt keeps being served until the cache expires.","correct":false,"tempting":"Caching returns computed state for a matching prefix; sending different text simply misses the cache rather than serving the old version."}]},"prev":{"slug":"logging-a-request-you-can-replay","title":"Logging enough to rebuild the request","url":"https://addaly.com/learn/context-engineering/logging-a-request-you-can-replay"},"next":{"slug":"the-model-changed-under-you","title":"The dependency you do not control","url":"https://addaly.com/learn/context-engineering/the-model-changed-under-you"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rolling out a prompt like a deployment', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/shipping-a-context-change (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rolling out a prompt like a deployment","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"A prompt change is a production change with no type system and no compiler, so it needs the same machinery any risky deployment gets: one versioned configuration object, a shadow run on real traffic, a percentage rollout, and a revert that does not require a deploy.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"Shipping one context version","events":[{"when":"Day 0","what":"One configuration object: prompt text, k, header fields, reranker, model id, truncation limits. One change from the version before it"},{"when":"Day 0","what":"The held-out golden set, and the offline assertions on the assembled request, which need no model call"},{"when":"Days 1 to 3","what":"Shadow run on five per cent of live traffic. Both configurations assembled and called, the old answer served, both logged"},{"when":"Day 4","what":"One per cent of users, with the assignment held stable per user so nobody flips mid-conversation"},{"when":"Days 5 to 8","what":"Five, then twenty-five, then all, watching escalation rate and thumbs-down rather than the evaluation score"},{"when":"Any hour","what":"Revert by setting the configuration id back, with no deploy. Test that path on a schedule, as you would a backup restore"}],"caption":"A bad prompt deploy returns two hundreds and answers slightly worse, so the feedback will not arrive in time to be useful and the controls have to be structural. Stamp the version on every response and the vaguest class of bug report becomes the most actionable: three complaints, all on v38, none on v37."},"alt":"Figure: Shipping one context version. Timeline: Day 0, One configuration object: prompt text, k, header fields, reranker, model id, truncation limits. One change from the version before it; Day 0, The held-out golden set, and the offline assertions on the assembled request, which need no model call; Days 1 to 3, Shadow run on five per cent of live traffic. Both configurations assembled and called, the old answer served, both logged; Day 4, One per cent of users, with the assignment held stable per user so nobody flips mid-conversation; Days 5 to 8, Five, then twenty-five, then all, watching escalation rate and thumbs-down rather than the evaluation score; Any hour, Revert by setting the configuration id back, with no deploy. Test that path on a schedule, as you would a backup restore. A bad prompt deploy returns two hundreds and answers slightly worse, so the feedback will not arrive in time to be useful and the controls have to be structural. Stamp the version on every response and the vaguest class of bug report becomes the most actionable: three complaints, all on v38, none on v37."}},"blocks":[{"t":"h2","id":"the-asymmetry-nobody-plans-for","runs":[{"text":"The asymmetry nobody plans for"}]},{"t":"p","runs":[{"text":"A bad code deploy usually announces itself. Exceptions, failing health checks, a graph falling over. Somebody notices within minutes."}]},{"t":"p","runs":[{"text":"A bad prompt deploy produces a system that runs perfectly, returns two hundreds, and answers slightly worse. No exception. No alert. The graph of request volume is identical. You find out from a support ticket eleven days later, and by then eleven days of changes sit between you and the cause."}]},{"t":"p","runs":[{"text":"So the controls have to be structural, because the feedback will not arrive in time to be useful."}]},{"t":"h2","id":"one-object-one-version","runs":[{"text":"One object, one version"}]},{"t":"p","runs":[{"text":"Everything that shapes the window belongs in a single versioned configuration: the prompt text, k, the chunk header fields, the reranker setting, the model id, the temperature, the truncation limits, the cache breakpoint position."}]},{"t":"code","lang":"python","label":"Python","text":"CTX_V38 = ContextConfig(\n    version=\"ctx-v38\",\n    system=SYSTEM_TEXT_V12,\n    k=6, rerank=True, header_fields=[\"path\", \"handle\"],\n    model=\"provider/model-2026-08-14\",\n    max_tool_tokens=3000, output_reserve=1200,\n)"},{"t":"p","runs":[{"text":"One object means one thing to log, one thing to revert, one thing to compare. The failure mode it prevents is the common one: the prompt lives in a template file, k lives in an environment variable, the model id lives in a different service’s config, and when quality drops nobody can state what the system was doing last Tuesday, because the answer is spread across three repositories and a deploy history."}]},{"t":"h2","id":"shadow-before-you-serve","runs":[{"text":"Shadow before you serve"}]},{"t":"p","runs":[{"text":"For a fraction of live traffic (five percent is plenty) assemble and call both configurations. Serve the old answer. Log both."}]},{"t":"p","runs":[{"text":"This is the most informative test available, because it runs on real traffic rather than on your golden set, which by construction contains only the cases you thought of. It costs double tokens on the shadowed fraction and nothing else, and it carries no user risk at all."}]},{"t":"p","runs":[{"text":"What to compare: token counts and latency directly, cache hit rate directly, and answer differences by sampling. Most requests will produce near-identical answers. The ones that diverge are your review queue, and fifty of them read carefully tells you more than any aggregate score."}]},{"t":"h2","id":"canary-then-widen","runs":[{"text":"Canary, then widen"}]},{"t":"p","runs":[{"text":"Serve the new configuration to a small percentage. One percent, then five, then twenty-five, then all, with time between steps."}]},{"t":"p","runs":[{"text":"Watch the outcome metrics, not the evaluation metrics. Evaluation told you it should be better; the canary tells you what actually happened to the things you care about. Escalation rate, thumbs-down rate, abandonment, repeat-question rate, cost per resolved conversation. These are noisy and slow, which is why you spend a day at each step rather than an hour."}]},{"t":"p","runs":[{"text":"Hold the assignment stable per user. A user flipping between two prompts mid-conversation gets an incoherent experience and pollutes your comparison."}]},{"t":"h2","id":"the-kill-switch-is-the-important-part","runs":[{"text":"The kill switch is the important part"}]},{"t":"p","runs":[{"text":"If you take one thing from this lesson: "},{"text":"you must be able to revert a context change without a deploy.","b":true}]},{"t":"p","runs":[{"text":"The configuration id lives somewhere you can change in seconds: a feature flag, a config service, a database row read per request with a short cache. Reverting is setting it back."}]},{"t":"p","runs":[{"text":"The reason this matters more here than for code is the detection lag. A prompt regression is found late, often out of hours, often by somebody who is not the person who shipped it. If reverting requires a build, a review and a deploy pipeline, the outage is extended by an hour of process at the worst possible time. If it requires changing a value, it is over in a minute and you diagnose in the morning."}]},{"t":"p","runs":[{"text":"Test the revert. Not once, on a schedule, the way you would test a backup restore: a kill switch nobody has pulled since it was built is an untested path in the one situation where you need it to work first time."}]},{"t":"h2","id":"change-one-thing-per-version","runs":[{"text":"Change one thing per version"}]},{"t":"p","runs":[{"text":"The rule from the significance lesson applies with more force here, because production is noisier than an evaluation."}]},{"t":"p","runs":[{"text":"A version that changes the prompt, raises k and switches model has three possible causes for any regression and no way to separate them. Untangling it under pressure means reverting everything and re-shipping one at a time, which is exactly the work you were trying to avoid by bundling."}]},{"t":"p","runs":[{"text":"Small versions ship more often, revert cleanly, and produce a history someone can read. "},{"text":"ctx-v38: k 8→6","code":true},{"text":" is a changelog entry that will still make sense in a year. "},{"text":"ctx-v38: prompt improvements","code":true},{"text":" is not."}]},{"t":"h2","id":"stamp-it-on-the-response","runs":[{"text":"Stamp it on the response"}]},{"t":"p","runs":[{"text":"Every response carries its configuration version back to the client, and the client shows it somewhere unobtrusive: a tooltip, a debug panel, the support form."}]},{"t":"p","runs":[{"text":"This single habit turns “the assistant has been worse lately” into “three complaints, all on ctx-v38, none on v37”. It costs one field, and it converts the vaguest class of bug report into the most actionable one."}]}],"check":{"question":[{"text":"Why does a prompt regression need a revert path that avoids the deploy pipeline more urgently than a code regression does?"}],"options":[{"text":[{"text":"Prompt changes cannot be rolled back by redeploying, because the previous text is not kept in version control."}],"correct":false,"tempting":[{"text":"It should be in version control, and if it is not that is the bug to fix; the pipeline’s speed is the real issue here."}]},{"text":[{"text":"Prompt changes affect all users at once, whereas code deploys roll out gradually by instance."}],"correct":false,"tempting":[{"text":"Both can be staged or not; this is a property of your rollout mechanism rather than of what changed."}]},{"text":[{"text":"It is found late and often out of hours, so a deploy adds an hour to an already extended problem."}],"correct":true,"tempting":null},{"text":[{"text":"Providers cache prompt prefixes, so an old prompt keeps being served until the cache expires."}],"correct":false,"tempting":[{"text":"Caching returns computed state for a matching prefix; sending different text simply misses the cache rather than serving the old version."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"logging-a-request-you-can-replay","title":"Logging enough to rebuild the request","minutes":9},"next":{"slug":"the-model-changed-under-you","title":"The dependency you do not control","minutes":8}}}