{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"regression-testing-a-prompt","title":"Regression testing a prompt","url":"https://addaly.com/learn/evaluating-ai/regression-testing-a-prompt","position":53,"of":76,"minutes":8,"answer":"Track which items flipped, not just the total; an unchanged score can hide a rewritten system.","keyIdea":"Track which items flipped, not just the total; an unchanged score can hide a rewritten system."},"body":{"format":"markdown","text":"## A prompt is code with no type system\n\nEvery edit is a refactor with unknown blast radius. You add one line to fix how it formats Indian dates and it starts writing Mexican peso amounts with a full stop where the comma should be. Nothing errors. Nothing warns you. The change reaches production and looks fine for three weeks.\n\nSo treat prompt edits the way you treat edits to a function with a thousand callers: run the tests.\n\n## The setup, concretely\n\nPut the eval set in the repository. Add a command (`npm run eval`, `pytest tests/eval`) that runs the frozen set against the current prompt and writes results per item. Wire it into CI on any pull request that touches a prompt file, a rubric, or a model id.\n\nKeep a fast lane and a slow lane. Ten items that run in twenty seconds for local iteration, the full hundred-plus in CI. Cache results keyed on a hash of `(prompt, input, model, temperature)` so re-running after an unrelated change costs nothing.\n\nPin everything. An explicit dated model id, never an alias like `-latest`. Temperature 0 for evals, while being honest that temperature 0 is not actually deterministic: for items that flip between runs, run them three times and record the flakiness rather than pretending it away.\n\n## The report that matters is not the score\n\nThis is the part teams get wrong. Your run prints 84%, the previous run printed 84%, and everyone moves on. But an aggregate can hold perfectly still while a third of the individual verdicts change places.\n\nSo print the flips:\n\n```\neval: address-eval-100    84% -> 84%   (no change)\n\n  newly passing (7):  addr-003 addr-018 addr-044 addr-051\n                      addr-067 addr-072 addr-090\n  newly failing (7):  addr-009 addr-021 addr-033 addr-058\n                      addr-061 addr-079 addr-095\n\n> Figure: 84% to 84%, and fourteen verdicts moved. A two by two grid, rows Passed before and Failed before, columns Passes now and Fails now: Passed before and Passes now: 77; Passed before and Fails now: 7; Failed before and Passes now: 7; Failed before and Fails now: 9. Seven items in, seven out, and the total holds perfectly still. By tag, landmark-only fell from 61% to 44% while street-address rose from 92% to 99%. That is a trade, and the aggregate hid it.\n\n  by tag:\n    landmark-only   61% -> 44%   (-17)\n    street-address  92% -> 99%   (+7)\n```\n\nNow you can see it. You did not make a neutral change; you traded away the landmark cases to gain the easy ones. That trade might be correct, but it should be a decision, not an accident, and the aggregate would have hidden it completely.\n\n## Golden outputs in the diff\n\nThe highest-value habit in this whole lesson: commit the actual generated outputs for a small set of items, so a prompt change shows up as a readable text diff in the pull request.\n\n```diff\n- Your refund will be processed within 5-7 business days.\n+ ## Refund status\n+ Your refund will be processed within 5-7 business days.\n+ Let me know if there's anything else I can help with.\n```\n\nNo metric caught that. A reviewer catches it in one second, and it is the kind of change (headings appearing, sign-offs appearing, personality drifting) that annoys users long before it moves a score.\n\n## Thresholds, used lightly\n\nFail the build on a drop of more than a few points, sure. But a hard gate on a hundred-item set will produce false alarms from normal variance, and a team that gets three false alarms stops reading the output entirely. The gate is a prompt to look, not a verdict. The flip list and the golden diff are what you actually read.\n\n## What this buys you\n\nSpeed, mostly. With this in place you can make a risky prompt change on a Tuesday afternoon and know within four minutes whether it broke the Portuguese cases. Without it, every change is a small act of faith, and after enough of them nobody on the team is willing to touch the prompt at all."},"check":{"question":"After a prompt edit, the eval run reports 84% before and 84% after on the same frozen set. What should the team conclude?","options":[{"text":"Nothing yet: the same total is produced both by \"nothing changed\" and by a batch of items swapping places, and only the per-item diff distinguishes them.","correct":true},{"text":"The edit is behaviour-neutral on their test set, so it is safe to ship.","correct":false,"tempting":"Reads an unchanged aggregate as unchanged behaviour, which is exactly the case where aggregates are blind."},{"text":"The edit had no effect and should be reverted to keep the prompt short.","correct":false,"tempting":"Concludes the change is a no-op, when it may have traded one group of failures for a different group."},{"text":"The set is too small to detect the change and should be expanded to 300 items before deciding.","correct":false,"tempting":"Reaches for more data when the information needed is already sitting in the run they just did."}]},"prev":{"slug":"tracing-and-logging","title":"Recording enough to diagnose a failure you did not see happen","url":"https://addaly.com/learn/evaluating-ai/tracing-and-logging"},"next":{"slug":"error-analysis","title":"Reading fifty failures","url":"https://addaly.com/learn/evaluating-ai/error-analysis"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Regression testing a prompt', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/regression-testing-a-prompt (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Regression testing a prompt","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Track which items flipped, not just the total; an unchanged score can hide a rewritten system.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Passes now","Fails now"],"rows":["Passed before","Failed before"],"type":"matrix","cells":[["77","7"],["7","9"]],"title":"84% to 84%, and fourteen verdicts moved","caption":"Seven items in, seven out, and the total holds perfectly still. By tag, landmark-only fell from 61% to 44% while street-address rose from 92% to 99%. That is a trade, and the aggregate hid it.","cellLabels":[["unchanged","newly failing"],["newly passing","unchanged"]]},"alt":"Figure: 84% to 84%, and fourteen verdicts moved. A two by two grid, rows Passed before and Failed before, columns Passes now and Fails now: Passed before and Passes now: 77; Passed before and Fails now: 7; Failed before and Passes now: 7; Failed before and Fails now: 9. Seven items in, seven out, and the total holds perfectly still. By tag, landmark-only fell from 61% to 44% while street-address rose from 92% to 99%. That is a trade, and the aggregate hid it."}},"blocks":[{"t":"h2","id":"a-prompt-is-code-with-no-type-system","runs":[{"text":"A prompt is code with no type system"}]},{"t":"p","runs":[{"text":"Every edit is a refactor with unknown blast radius. You add one line to fix how it formats Indian dates and it starts writing Mexican peso amounts with a full stop where the comma should be. Nothing errors. Nothing warns you. The change reaches production and looks fine for three weeks."}]},{"t":"p","runs":[{"text":"So treat prompt edits the way you treat edits to a function with a thousand callers: run the tests."}]},{"t":"h2","id":"the-setup-concretely","runs":[{"text":"The setup, concretely"}]},{"t":"p","runs":[{"text":"Put the eval set in the repository. Add a command ("},{"text":"npm run eval","code":true},{"text":", "},{"text":"pytest tests/eval","code":true},{"text":") that runs the frozen set against the current prompt and writes results per item. Wire it into CI on any pull request that touches a prompt file, a rubric, or a model id."}]},{"t":"p","runs":[{"text":"Keep a fast lane and a slow lane. Ten items that run in twenty seconds for local iteration, the full hundred-plus in CI. Cache results keyed on a hash of "},{"text":"(prompt, input, model, temperature)","code":true},{"text":" so re-running after an unrelated change costs nothing."}]},{"t":"p","runs":[{"text":"Pin everything. An explicit dated model id, never an alias like "},{"text":"-latest","code":true},{"text":". Temperature 0 for evals, while being honest that temperature 0 is not actually deterministic: for items that flip between runs, run them three times and record the flakiness rather than pretending it away."}]},{"t":"h2","id":"the-report-that-matters-is-not-the-score","runs":[{"text":"The report that matters is not the score"}]},{"t":"p","runs":[{"text":"This is the part teams get wrong. Your run prints 84%, the previous run printed 84%, and everyone moves on. But an aggregate can hold perfectly still while a third of the individual verdicts change places."}]},{"t":"p","runs":[{"text":"So print the flips:"}]},{"t":"code","lang":"","label":"Text","text":"eval: address-eval-100    84% -> 84%   (no change)\n\n  newly passing (7):  addr-003 addr-018 addr-044 addr-051\n                      addr-067 addr-072 addr-090\n  newly failing (7):  addr-009 addr-021 addr-033 addr-058\n                      addr-061 addr-079 addr-095\n\n  by tag:\n    landmark-only   61% -> 44%   (-17)\n    street-address  92% -> 99%   (+7)"},{"t":"p","runs":[{"text":"Now you can see it. You did not make a neutral change; you traded away the landmark cases to gain the easy ones. That trade might be correct, but it should be a decision, not an accident, and the aggregate would have hidden it completely."}]},{"t":"h2","id":"golden-outputs-in-the-diff","runs":[{"text":"Golden outputs in the diff"}]},{"t":"p","runs":[{"text":"The highest-value habit in this whole lesson: commit the actual generated outputs for a small set of items, so a prompt change shows up as a readable text diff in the pull request."}]},{"t":"code","lang":"diff","label":"Diff","text":"- Your refund will be processed within 5-7 business days.\n+ ## Refund status\n+ Your refund will be processed within 5-7 business days.\n+ Let me know if there's anything else I can help with."},{"t":"p","runs":[{"text":"No metric caught that. A reviewer catches it in one second, and it is the kind of change (headings appearing, sign-offs appearing, personality drifting) that annoys users long before it moves a score."}]},{"t":"h2","id":"thresholds-used-lightly","runs":[{"text":"Thresholds, used lightly"}]},{"t":"p","runs":[{"text":"Fail the build on a drop of more than a few points, sure. But a hard gate on a hundred-item set will produce false alarms from normal variance, and a team that gets three false alarms stops reading the output entirely. The gate is a prompt to look, not a verdict. The flip list and the golden diff are what you actually read."}]},{"t":"h2","id":"what-this-buys-you","runs":[{"text":"What this buys you"}]},{"t":"p","runs":[{"text":"Speed, mostly. With this in place you can make a risky prompt change on a Tuesday afternoon and know within four minutes whether it broke the Portuguese cases. Without it, every change is a small act of faith, and after enough of them nobody on the team is willing to touch the prompt at all."}]}],"check":{"question":[{"text":"After a prompt edit, the eval run reports 84% before and 84% after on the same frozen set. What should the team conclude?"}],"options":[{"text":[{"text":"Nothing yet: the same total is produced both by “nothing changed” and by a batch of items swapping places, and only the per-item diff distinguishes them."}],"correct":true,"tempting":null},{"text":[{"text":"The edit is behaviour-neutral on their test set, so it is safe to ship."}],"correct":false,"tempting":[{"text":"Reads an unchanged aggregate as unchanged behaviour, which is exactly the case where aggregates are blind."}]},{"text":[{"text":"The edit had no effect and should be reverted to keep the prompt short."}],"correct":false,"tempting":[{"text":"Concludes the change is a no-op, when it may have traded one group of failures for a different group."}]},{"text":[{"text":"The set is too small to detect the change and should be expanded to 300 items before deciding."}],"correct":false,"tempting":[{"text":"Reaches for more data when the information needed is already sitting in the run they just did."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"tracing-and-logging","title":"Recording enough to diagnose a failure you did not see happen","minutes":9},"next":{"slug":"error-analysis","title":"Reading fifty failures","minutes":9}}}