The first fifty cases, and where they come from

Fifty, not five hundred

The earlier lesson in this course made the claim: if you cannot re-run fifty saved cases, you are guessing. This lesson is about assembling those fifty. Not five hundred; that comes later, from production, one failure at a time. Fifty is what one person builds in a day, and it is enough to catch the regressions that matter, because most prompt changes that break something break it visibly on a handful of cases.

The evaluation course in this catalogue covers sampling, labelling and statistics in depth. Here the goal is narrower: a set you can build this week for a feature you own.

Three sources, roughly equal

From the logs. If the feature has shipped, pull real inputs. Not the most recent hundred (the evaluation course explains why that is the worst sample you own) but a spread: different intents, different lengths, different languages if you serve several, a few from each week. Redact identifiers before anything lands in a file. Real inputs are the only source that contains the things you did not think of.

From failures. Every complaint, every thumbs-down with a category, every bug report that included the input. These are the most valuable cases because they are known to have failed once. A set built only from happy paths measures whether the feature still works on the things it already handled.

From the specification. Write cases for what the feature is supposed to do, including the edges: an empty input, an input in the wrong language, a question the corpus cannot answer, a request that should be refused, an input containing an instruction to the model. These are cases the logs will not contain until the day they matter.

Fifteen or twenty from each source is the target. If the feature has not shipped, the first source is replaced by inputs written by two people separately, compared afterwards: the overlap is the obvious, the difference is the interesting.

The shape of a case

One line of JSON per case, in a file beside the prompt it tests:

{"id": "refund-window-annual", "source": "logs",
 "tags": ["refunds", "annual-plan"],
 "input": {"message": "how long do I have to cancel an annual plan?",
           "history": []},
 "expect": {"must_contain": ["14 days"],
            "must_not_contain": ["30 days"],
            "cites": ["policy-refunds-v3"],
            "rubric": "States the 14-day window and that it applies from purchase, not renewal."}}

The source field is the provenance the evaluation course insists on; six months from now nobody remembers where a case came from. Tags let you report by slice, which the next lessons rely on. The expect block is the part people get wrong.

Writing an expectation you can check

For a closed answer (a classification, a number, an extracted field) the expectation is the value, and a check is equality. For open text the expectation cannot be the exact wording, because the model will phrase it differently every run. Three tools cover almost everything:

• Must-contain and must-not-contain: strings or patterns that any correct answer includes, and strings whose presence proves the answer wrong. “14 days” must appear; “30 days” must not.
• Structural checks: valid JSON, a citation that resolves, a length under a limit, the right language. The next lesson is a catalogue of these.
• A rubric: one or two sentences saying what a correct answer does, for a judge (human or model) to apply. Write it as a statement to be marked true or false, not as a score out of ten.

An expectation that is only a rubric is expensive to check. An expectation that is only a string match misses wrong answers that happen to contain the right words. Use both where you can.

Golden and regression

Split the set in two by purpose. Golden cases define correct behaviour and rarely change. Regression cases each encode one specific past failure, and their number only grows. When a prompt change makes a golden case fail, the change is probably wrong. When it makes a regression case fail, a bug you fixed has come back, and the case’s id tells you which.

The case whose expectation is wrong

Some fraction of your fifty will turn out to have a bad expectation: the “correct” answer you wrote was itself mistaken, or the policy changed. This is normal, and it is a reason to keep cases in version control with the prompt: a change to an expectation is a reviewed change with a reason, not a silent edit in a spreadsheet. When a case fails, the first question is always whether the case or the system is wrong, and the evaluation course’s error-analysis habit, applied here in a later lesson, makes “expectation wrong” an explicit category rather than an embarrassment.

Keep it beside the prompt

prompts/triage/v7.md and prompts/triage/cases.jsonl in the same directory, in the same commit. The earlier lesson on prompts in version control gave the reason; this lesson gives the file. A change to either without the other should look odd in review.