A prompt you can version, test and roll back

A prompt is a deploy

A prompt change alters production behaviour for every user immediately. That makes it a deploy. It should have a diff, a review, a test run, a version and a rollback path, and in most organisations it has none of those because it lives in a database field that a product manager edits at four in the afternoon.

The fix is not ceremony. It is treating the prompt as an artefact with the same lifecycle as the code that parses its output — because those two things must change together and any system that lets them drift apart will break.

Version the whole call, not the string

The text is one input among several. All of these change behaviour:

@dataclass(frozen=True)
class PromptConfig:
    template: str
    model: str                 # pinned snapshot, not an alias
    temperature: float
    schema: dict               # output contract
    tools: list                # definitions, order-stable
    retrieval: RetrievalCfg    # k, reranker, hybrid weights, filters
    truncation: TruncationCfg  # what gets dropped, in what order

    def hash(self) -> str:
        return sha256(json.dumps(asdict(self), sort_keys=True).encode()).hexdigest()[:12]

Log config.hash() on every request alongside the trace id. Six weeks later, when accuracy dropped on a Tuesday, you can join complaints to configs. Without it you are reading commit history and guessing.

Pin the model. Aliases like gpt-4o or claude-sonnet-4 move underneath you. Pin the dated snapshot. Upgrading the model is a change that gets evaluated, not a thing that happens to you.

Build the eval set from your failures

Thirty to fifty real cases beat five hundred synthetic ones, because synthetic cases encode what you already imagined going wrong.

Every production bug becomes a case. Someone reports the assistant confused two products with similar names: that conversation, trimmed, with the expected answer, goes into the set. This is the discipline. The set grows from reality, and a prompt fix that does not add a case is not finished.

Keep a held-out slice you do not look at. If you iterate against the whole set for a month, you have fitted the prompt to it, and the score no longer predicts anything.

Grade cheaply where you can

In order of preference:

1. 1Deterministic assertions. Schema validates. Extracted quote appears verbatim in the source. Cited chunk ids are real. Currency unchanged. No refusal. These are free, fast and never drift.
2. 2Exact or fuzzy match against a known answer, where the task has one.
3. 3A model judge, only for what the first two cannot reach — tone, completeness, whether an explanation is actually responsive.

If you use a judge, calibrate it. Label 50 cases by hand, run the judge, report agreement. A judge you have not measured is a random number generator with good manners. Known biases: judges prefer longer answers, prefer the response shown first, and prefer output from their own model family. Randomise order, and strip length cues where the task allows.

Take small differences seriously as noise

Prompt B scores 87% against prompt A's 82% on your 40 cases. That is two cases. Run it again tomorrow and it may reverse.

Do the paired version instead: same cases, both prompts, several runs each, count wins, losses and ties per case. Report an interval. A bootstrap over 40 items gives roughly a ±10 point band; that band is the honest answer, and it usually means you need more cases before you can call anything an improvement.

More repeats of the same 40 cases reduce sampling noise from decoding. They do nothing about the possibility that your 40 cases are unrepresentative. Those are different uncertainties and only one of them is fixed by running it again.

Ship like code

• Prompts in files in the repo, next to the parser and the tests.
• CI runs the eval set on every prompt diff and fails on a regression beyond the interval.
• Roll out behind a flag at 5%, then 50%, comparing live signals — refusal rate, retry rate, schema failure rate, escalation to a human, latency, cost per request.
• Rollback is git revert plus a deploy, and it should take the same three minutes a code rollback takes. A bad prompt is a production incident. If reverting requires someone to find the right row in an admin panel, you do not have rollback.

Log what the model saw

Not the template. The assembled prompt, byte for byte, with the config hash, sampled at whatever rate your privacy rules and storage allow.

At three in the morning, the question is never "what does the template say". It is "what actually went in". Templates are rendered by code with defaults and truncation and retrieval and a date, and the gap between what you think you sent and what you sent is where the bug lives.