Regression testing a prompt

A prompt is code with no type system

Every edit is a refactor with unknown blast radius. You add one line to fix how it formats Indian dates and it starts writing Mexican peso amounts with a full stop where the comma should be. Nothing errors. Nothing warns you. The change reaches production and looks fine for three weeks.

So treat prompt edits the way you treat edits to a function with a thousand callers: run the tests.

The setup, concretely

Put the eval set in the repository. Add a command — npm run eval, pytest tests/eval — that runs the frozen set against the current prompt and writes results per item. Wire it into CI on any pull request that touches a prompt file, a rubric, or a model id.

Keep a fast lane and a slow lane. Ten items that run in twenty seconds for local iteration, the full hundred-plus in CI. Cache results keyed on a hash of (prompt, input, model, temperature) so re-running after an unrelated change costs nothing.

Pin everything. An explicit dated model id, never an alias like -latest. Temperature 0 for evals, while being honest that temperature 0 is not actually deterministic — for items that flip between runs, run them three times and record the flakiness rather than pretending it away.

The report that matters is not the score

This is the part teams get wrong. Your run prints 84%, the previous run printed 84%, and everyone moves on. But an aggregate can hold perfectly still while a third of the individual verdicts change places.

So print the flips:

eval: address-eval-100    84% -> 84%   (no change)

  newly passing (7):  addr-003 addr-018 addr-044 addr-051
                      addr-067 addr-072 addr-090
  newly failing (7):  addr-009 addr-021 addr-033 addr-058
                      addr-061 addr-079 addr-095

  by tag:
    landmark-only   61% -> 44%   (-17)
    street-address  92% -> 99%   (+7)

Now you can see it. You did not make a neutral change; you traded away the landmark cases to gain the easy ones. That trade might be correct — but it should be a decision, not an accident, and the aggregate would have hidden it completely.

Golden outputs in the diff

The highest-value habit in this whole lesson: commit the actual generated outputs for a small set of items, so a prompt change shows up as a readable text diff in the pull request.

- Your refund will be processed within 5-7 business days.
+ ## Refund status
+ Your refund will be processed within 5-7 business days.
+ Let me know if there's anything else I can help with.

No metric caught that. A reviewer catches it in one second, and it is the kind of change — headings appearing, sign-offs appearing, personality drifting — that annoys users long before it moves a score.

Thresholds, used lightly

Fail the build on a drop of more than a few points, sure. But a hard gate on a hundred-item set will produce false alarms from normal variance, and a team that gets three false alarms stops reading the output entirely. The gate is a prompt to look, not a verdict. The flip list and the golden diff are what you actually read.

What this buys you

Speed, mostly. With this in place you can make a risky prompt change on a Tuesday afternoon and know within four minutes whether it broke the Portuguese cases. Without it, every change is a small act of faith, and after enough of them nobody on the team is willing to touch the prompt at all.