{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"prompts-you-can-roll-back","title":"A prompt you can version, test and roll back","url":"https://addaly.com/learn/context-engineering/prompts-you-can-roll-back","position":78,"of":87,"minutes":10,"answer":"Version the whole call and log its hash, or you will not be able to explain your own regressions.","keyIdea":"Version the whole call and log its hash, or you will not be able to explain your own regressions."},"body":{"format":"markdown","text":"## A prompt is a deploy\n\nA prompt change alters production behaviour for every user immediately. That makes it a deploy. It should have a diff, a review, a test run, a version and a rollback path, and in most organisations it has none of those because it lives in a database field that a product manager edits at four in the afternoon.\n\nThe fix is not ceremony. It is treating the prompt as an artefact with the same lifecycle as the code that parses its output, because those two things must change together and any system that lets them drift apart will break.\n\n## Version the whole call, not the string\n\nThe text is one input among several. All of these change behaviour:\n\n```python\n@dataclass(frozen=True)\nclass PromptConfig:\n    template: str\n    model: str                 # pinned snapshot, not an alias\n    temperature: float\n    schema: dict               # output contract\n    tools: list                # definitions, order-stable\n    retrieval: RetrievalCfg    # k, reranker, hybrid weights, filters\n    truncation: TruncationCfg  # what gets dropped, in what order\n\n    def hash(self) -> str:\n        return sha256(json.dumps(asdict(self), sort_keys=True).encode()).hexdigest()[:12]\n```\n\nLog `config.hash()` on every request alongside the trace id. Six weeks later, when accuracy dropped on a Tuesday, you can join complaints to configs. Without it you are reading commit history and guessing.\n\n**Pin the model.** Aliases like `gpt-4o` or `claude-sonnet-4` move underneath you. Pin the dated snapshot. Upgrading the model is a change that gets evaluated, not a thing that happens to you.\n\n## Build the eval set from your failures\n\nThirty to fifty real cases beat five hundred synthetic ones, because synthetic cases encode what you already imagined going wrong.\n\nEvery production bug becomes a case. Someone reports the assistant confused two products with similar names: that conversation, trimmed, with the expected answer, goes into the set. This is the discipline. The set grows from reality, and a prompt fix that does not add a case is not finished.\n\nKeep a held-out slice you do not look at. If you iterate against the whole set for a month, you have fitted the prompt to it, and the score no longer predicts anything.\n\n## Grade cheaply where you can\n\nIn order of preference:\n\n1. **Deterministic assertions.** Schema validates. Extracted quote appears verbatim in the source. Cited chunk ids are real. Currency unchanged. No refusal. These are free, fast and never drift.\n2. **Exact or fuzzy match** against a known answer, where the task has one.\n3. **A model judge**, only for what the first two cannot reach: tone, completeness, whether an explanation is actually responsive.\n\nIf you use a judge, calibrate it. Label 50 cases by hand, run the judge, report agreement. A judge you have not measured is a random number generator with good manners. Known biases: judges prefer longer answers, prefer the response shown first, and prefer output from their own model family. Randomise order, and strip length cues where the task allows.\n\n## Take small differences seriously as noise\n\nPrompt B scores 87% against prompt A's 82% on your 40 cases. That is two cases. Run it again tomorrow and it may reverse.\n\nDo the paired version instead: same cases, both prompts, several runs each, count wins, losses and ties per case. Report an interval. A bootstrap over 40 items gives roughly a ±10 point band; that band is the honest answer, and it usually means you need more cases before you can call anything an improvement.\n\nMore repeats of the same 40 cases reduce sampling noise from decoding. They do nothing about the possibility that your 40 cases are unrepresentative. Those are different uncertainties and only one of them is fixed by running it again.\n\n## Ship like code\n\n- Prompts in files in the repo, next to the parser and the tests.\n- CI runs the eval set on every prompt diff and fails on a regression beyond the interval.\n- Roll out behind a flag at 5%, then 50%, comparing live signals: refusal rate, retry rate, schema failure rate, escalation to a human, latency, cost per request.\n- Rollback is `git revert` plus a deploy, and it should take the same three minutes a code rollback takes. A bad prompt is a production incident. If reverting requires someone to find the right row in an admin panel, you do not have rollback.\n\n## Log what the model saw\n\nNot the template. The assembled prompt, byte for byte, with the config hash, sampled at whatever rate your privacy rules and storage allow.\n\nAt three in the morning, the question is never \"what does the template say\". It is \"what actually went in\". Templates are rendered by code with defaults and truncation and retrieval and a date, and the gap between what you think you sent and what you sent is where the bug lives."},"check":{"question":"A team compares two prompts on their 40-case evaluation set, one run each. Prompt B scores 87%, prompt A scores 82%. What is the sound conclusion?","options":[{"text":"B is better. Five points on real production cases is a clear signal, and waiting for more data delays a working improvement.","correct":false,"tempting":"Percentages read as precise regardless of how few items produced them, and five points sounds decisive until you convert it back into two cases."},{"text":"Set temperature to 0 so the comparison becomes deterministic, which makes the five-point gap a real measurement.","correct":false,"tempting":"Greedy decoding does remove sampling variation, so it feels like it removes uncertainty, but it cannot tell you whether these particular 40 cases represent your traffic."},{"text":"Almost nothing follows. Five points on 40 cases is two cases, inside the run-to-run band; you need a paired comparison with repeats and more cases before calling it an improvement.","correct":true},{"text":"Rerun each prompt twenty times and average the scores; the averaged numbers settle which prompt is better.","correct":false,"tempting":"Averaging repeats genuinely shrinks one source of noise, which makes it feel rigorous, but it leaves the uncertainty from having only 40 items completely untouched."}]},"prev":{"slug":"testing-the-boundary","title":"Making attack success a number you track","url":"https://addaly.com/learn/context-engineering/testing-the-boundary"},"next":{"slug":"the-golden-set","title":"The set of examples you are allowed to trust","url":"https://addaly.com/learn/context-engineering/the-golden-set"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A prompt you can version, test and roll back', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/prompts-you-can-roll-back (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A prompt you can version, test and roll back","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"Version the whole call and log its hash, or you will not be able to explain your own regressions.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-prompt-is-a-deploy","runs":[{"text":"A prompt is a deploy"}]},{"t":"p","runs":[{"text":"A prompt change alters production behaviour for every user immediately. That makes it a deploy. It should have a diff, a review, a test run, a version and a rollback path, and in most organisations it has none of those because it lives in a database field that a product manager edits at four in the afternoon."}]},{"t":"p","runs":[{"text":"The fix is not ceremony. It is treating the prompt as an artefact with the same lifecycle as the code that parses its output, because those two things must change together and any system that lets them drift apart will break."}]},{"t":"h2","id":"version-the-whole-call-not-the-string","runs":[{"text":"Version the whole call, not the string"}]},{"t":"p","runs":[{"text":"The text is one input among several. All of these change behaviour:"}]},{"t":"code","lang":"python","label":"Python","text":"@dataclass(frozen=True)\nclass PromptConfig:\n    template: str\n    model: str                 # pinned snapshot, not an alias\n    temperature: float\n    schema: dict               # output contract\n    tools: list                # definitions, order-stable\n    retrieval: RetrievalCfg    # k, reranker, hybrid weights, filters\n    truncation: TruncationCfg  # what gets dropped, in what order\n\n    def hash(self) -> str:\n        return sha256(json.dumps(asdict(self), sort_keys=True).encode()).hexdigest()[:12]"},{"t":"p","runs":[{"text":"Log "},{"text":"config.hash()","code":true},{"text":" on every request alongside the trace id. Six weeks later, when accuracy dropped on a Tuesday, you can join complaints to configs. Without it you are reading commit history and guessing."}]},{"t":"p","runs":[{"text":"Pin the model.","b":true},{"text":" Aliases like "},{"text":"gpt-4o","code":true},{"text":" or "},{"text":"claude-sonnet-4","code":true},{"text":" move underneath you. Pin the dated snapshot. Upgrading the model is a change that gets evaluated, not a thing that happens to you."}]},{"t":"h2","id":"build-the-eval-set-from-your-failures","runs":[{"text":"Build the eval set from your failures"}]},{"t":"p","runs":[{"text":"Thirty to fifty real cases beat five hundred synthetic ones, because synthetic cases encode what you already imagined going wrong."}]},{"t":"p","runs":[{"text":"Every production bug becomes a case. Someone reports the assistant confused two products with similar names: that conversation, trimmed, with the expected answer, goes into the set. This is the discipline. The set grows from reality, and a prompt fix that does not add a case is not finished."}]},{"t":"p","runs":[{"text":"Keep a held-out slice you do not look at. If you iterate against the whole set for a month, you have fitted the prompt to it, and the score no longer predicts anything."}]},{"t":"h2","id":"grade-cheaply-where-you-can","runs":[{"text":"Grade cheaply where you can"}]},{"t":"p","runs":[{"text":"In order of preference:"}]},{"t":"ol","start":1,"items":[[{"text":"Deterministic assertions.","b":true},{"text":" Schema validates. Extracted quote appears verbatim in the source. Cited chunk ids are real. Currency unchanged. No refusal. These are free, fast and never drift."}],[{"text":"Exact or fuzzy match","b":true},{"text":" against a known answer, where the task has one."}],[{"text":"A model judge","b":true},{"text":", only for what the first two cannot reach: tone, completeness, whether an explanation is actually responsive."}]]},{"t":"p","runs":[{"text":"If you use a judge, calibrate it. Label 50 cases by hand, run the judge, report agreement. A judge you have not measured is a random number generator with good manners. Known biases: judges prefer longer answers, prefer the response shown first, and prefer output from their own model family. Randomise order, and strip length cues where the task allows."}]},{"t":"h2","id":"take-small-differences-seriously-as-noise","runs":[{"text":"Take small differences seriously as noise"}]},{"t":"p","runs":[{"text":"Prompt B scores 87% against prompt A’s 82% on your 40 cases. That is two cases. Run it again tomorrow and it may reverse."}]},{"t":"p","runs":[{"text":"Do the paired version instead: same cases, both prompts, several runs each, count wins, losses and ties per case. Report an interval. A bootstrap over 40 items gives roughly a ±10 point band; that band is the honest answer, and it usually means you need more cases before you can call anything an improvement."}]},{"t":"p","runs":[{"text":"More repeats of the same 40 cases reduce sampling noise from decoding. They do nothing about the possibility that your 40 cases are unrepresentative. Those are different uncertainties and only one of them is fixed by running it again."}]},{"t":"h2","id":"ship-like-code","runs":[{"text":"Ship like code"}]},{"t":"ul","items":[[{"text":"Prompts in files in the repo, next to the parser and the tests."}],[{"text":"CI runs the eval set on every prompt diff and fails on a regression beyond the interval."}],[{"text":"Roll out behind a flag at 5%, then 50%, comparing live signals: refusal rate, retry rate, schema failure rate, escalation to a human, latency, cost per request."}],[{"text":"Rollback is "},{"text":"git revert","code":true},{"text":" plus a deploy, and it should take the same three minutes a code rollback takes. A bad prompt is a production incident. If reverting requires someone to find the right row in an admin panel, you do not have rollback."}]]},{"t":"h2","id":"log-what-the-model-saw","runs":[{"text":"Log what the model saw"}]},{"t":"p","runs":[{"text":"Not the template. The assembled prompt, byte for byte, with the config hash, sampled at whatever rate your privacy rules and storage allow."}]},{"t":"p","runs":[{"text":"At three in the morning, the question is never “what does the template say”. It is “what actually went in”. Templates are rendered by code with defaults and truncation and retrieval and a date, and the gap between what you think you sent and what you sent is where the bug lives."}]}],"check":{"question":[{"text":"A team compares two prompts on their 40-case evaluation set, one run each. Prompt B scores 87%, prompt A scores 82%. What is the sound conclusion?"}],"options":[{"text":[{"text":"B is better. Five points on real production cases is a clear signal, and waiting for more data delays a working improvement."}],"correct":false,"tempting":[{"text":"Percentages read as precise regardless of how few items produced them, and five points sounds decisive until you convert it back into two cases."}]},{"text":[{"text":"Set temperature to 0 so the comparison becomes deterministic, which makes the five-point gap a real measurement."}],"correct":false,"tempting":[{"text":"Greedy decoding does remove sampling variation, so it feels like it removes uncertainty, but it cannot tell you whether these particular 40 cases represent your traffic."}]},{"text":[{"text":"Almost nothing follows. Five points on 40 cases is two cases, inside the run-to-run band; you need a paired comparison with repeats and more cases before calling it an improvement."}],"correct":true,"tempting":null},{"text":[{"text":"Rerun each prompt twenty times and average the scores; the averaged numbers settle which prompt is better."}],"correct":false,"tempting":[{"text":"Averaging repeats genuinely shrinks one source of noise, which makes it feel rigorous, but it leaves the uncertainty from having only 40 items completely untouched."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"testing-the-boundary","title":"Making attack success a number you track","minutes":9},"next":{"slug":"the-golden-set","title":"The set of examples you are allowed to trust","minutes":9}}}