Turning a complaint into a permanent test

Every incident is a gap in the set

Something went wrong in production. A customer was told the wrong refund window; the assistant answered in the wrong language; a document nobody should see was retrieved and quoted.

The fix is the obvious part. The valuable part is the loop that makes sure that specific failure never returns unnoticed, and it is a five-step habit that most teams do not have.

The five steps

1. Capture the exact input. Not a description: the input, verbatim, with the retrieval context, the model version, the prompt version, and the timestamp. This is the reason the logging in the previous lesson exists. “A customer asked something about refunds” cannot be reproduced.

2. Reproduce it. Run the same input through the current system. Three outcomes, all informative:

• It reproduces. Good: you have a test.
• It does not, because the system is non-deterministic. Run it twenty times and record the rate. A failure that occurs one time in five is still a failure, and a one-shot test would have called it fixed.
• It does not, because something has already changed. Find out what, or your fix addresses a system that no longer exists.

3. Minimise. Strip the input to the smallest thing that still fails. A 40-turn conversation usually reduces to two turns. A minimal case is faster to run, easier to reason about, and much more likely to generalise to the family of inputs that share the cause.

4. Add it to the set, with provenance.

{"id": "inc-2026-08-14-refund-window",
 "input": "...",
 "expected": "states the 14-day window from the policy, or asks",
 "assert": [{"type": "not_contains", "value": "30 days"}],
 "origin": "incident", "incident": "INC-2291",
 "added": "2026-08-14", "severity": "high"}

The origin field is what makes this manageable later. It lets you count how much of your set is history and how much is traffic.

5. Ask what would have caught it a week earlier. This is the question that produces the durable improvement, and the answer is always a mechanism, never a person. Sometimes it is a deterministic check. Sometimes it is a new rubric rule. Sometimes it is a monitor on an output metric nobody was watching. Write the answer down and build it, or the next incident of the same kind will also be found by a customer.

Keep the set from becoming a museum

Here is the failure mode of doing this well. After eighteen months, 60% of your set is incident items (rare, hard, historical cases) and the set no longer looks anything like your traffic. Every number it produces is pessimistic, its slices are meaningless, and the team stops believing it.

Two controls:

• Cap the proportion. Keep incident-derived items under about a third of the main set. Move the overflow to a separate regression suite that runs in CI as a pass/fail gate and does not contribute to the quality percentage. Two files, two purposes: one estimates quality on real traffic, one prevents known bugs returning.
• Refresh the base. Every quarter, replace a slice of the ordinary items with a fresh sample of recent traffic. Keep the old ones in the frozen set for historical comparison, but let the live set track what people are actually asking.

The distinction is worth being firm about. A regression suite answers “has anything we already fixed come back?” and every item in it must pass. An eval set answers “how good is this on the work we actually do?” and a percentage is the right output. Merging them gives you a number that is neither.

The review, and its tone

Hold a short review for anything customer-visible. Twenty minutes, four questions:

1. What happened, in one paragraph, with the timeline.
2. Why did the eval set not contain this case?
3. What check, monitor or rule would have caught it earlier?
4. What is now in the set, and who added it?

Question two is the one that improves the evaluation rather than the system, and it is usually the most interesting. The answer is often that the case existed in traffic and was never sampled, which points straight at your sampling, not at the model.

Keep it blameless, and not as a slogan. The moment an incident review becomes a search for who wrote the prompt, people stop reporting the small failures, and the small failures are your entire early-warning system.