{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"incident-to-eval-item","title":"Turning a complaint into a permanent test","url":"https://addaly.com/learn/evaluating-ai/incident-to-eval-item","position":74,"of":76,"minutes":8,"answer":"Every incident becomes a minimised, provenanced test item, but keep those in a regression suite so the eval set still resembles real traffic rather than a museum of old bugs.","keyIdea":"Every incident becomes a minimised, provenanced test item, but keep those in a regression suite so the eval set still resembles real traffic rather than a museum of old bugs."},"body":{"format":"markdown","text":"## Every incident is a gap in the set\n\nSomething went wrong in production. A customer was told the wrong refund window; the assistant answered in the wrong language; a document nobody should see was retrieved and quoted.\n\nThe fix is the obvious part. The valuable part is the loop that makes sure that specific failure never returns unnoticed, and it is a five-step habit that most teams do not have.\n\n## The five steps\n\n**1. Capture the exact input.** Not a description: the input, verbatim, with the retrieval context, the model version, the prompt version, and the timestamp. This is the reason the logging in the previous lesson exists. \"A customer asked something about refunds\" cannot be reproduced.\n\n**2. Reproduce it.** Run the same input through the current system. Three outcomes, all informative:\n\n- It reproduces. Good: you have a test.\n- It does not, because the system is non-deterministic. Run it twenty times and record the rate. A failure that occurs one time in five is still a failure, and a one-shot test would have called it fixed.\n- It does not, because something has already changed. Find out what, or your fix addresses a system that no longer exists.\n\n**3. Minimise.** Strip the input to the smallest thing that still fails. A 40-turn conversation usually reduces to two turns. A minimal case is faster to run, easier to reason about, and much more likely to generalise to the family of inputs that share the cause.\n\n**4. Add it to the set, with provenance.**\n\n```jsonl\n{\"id\": \"inc-2026-08-14-refund-window\",\n \"input\": \"...\",\n \"expected\": \"states the 14-day window from the policy, or asks\",\n \"assert\": [{\"type\": \"not_contains\", \"value\": \"30 days\"}],\n \"origin\": \"incident\", \"incident\": \"INC-2291\",\n \"added\": \"2026-08-14\", \"severity\": \"high\"}\n```\n\nThe `origin` field is what makes this manageable later. It lets you count how much of your set is history and how much is traffic.\n\n**5. Ask what would have caught it a week earlier.** This is the question that produces the durable improvement, and the answer is always a mechanism, never a person. Sometimes it is a deterministic check. Sometimes it is a new rubric rule. Sometimes it is a monitor on an output metric nobody was watching. Write the answer down and build it, or the next incident of the same kind will also be found by a customer.\n\n> Figure: From a complaint to a permanent test. Steps: 1. Capture (The exact input, the retrieval context, model and prompt versions, the timestamp.); 2. Reproduce (Run it now. If it flips, run it twenty times and record the rate.); 3. Minimise (Strip to the smallest input that still fails: forty turns usually become two.); 4. Add with provenance (origin: incident, the incident id, severity, the date added.); 5. Ask what would have caught it (A week earlier: a check, a rule, a monitor. Always a mechanism, never a person.). The origin field is what keeps the set from becoming a museum: it lets you count how much of it is history and how much is traffic, and hold incident items under a third.\n\n## Keep the set from becoming a museum\n\nHere is the failure mode of doing this well. After eighteen months, 60% of your set is incident items (rare, hard, historical cases) and the set no longer looks anything like your traffic. Every number it produces is pessimistic, its slices are meaningless, and the team stops believing it.\n\nTwo controls:\n\n- **Cap the proportion.** Keep incident-derived items under about a third of the main set. Move the overflow to a separate regression suite that runs in CI as a pass/fail gate and does not contribute to the quality percentage. Two files, two purposes: one estimates quality on real traffic, one prevents known bugs returning.\n- **Refresh the base.** Every quarter, replace a slice of the ordinary items with a fresh sample of recent traffic. Keep the old ones in the frozen set for historical comparison, but let the live set track what people are actually asking.\n\nThe distinction is worth being firm about. A regression suite answers \"has anything we already fixed come back?\" and every item in it must pass. An eval set answers \"how good is this on the work we actually do?\" and a percentage is the right output. Merging them gives you a number that is neither.\n\n## The review, and its tone\n\nHold a short review for anything customer-visible. Twenty minutes, four questions:\n\n1. What happened, in one paragraph, with the timeline.\n2. Why did the eval set not contain this case?\n3. What check, monitor or rule would have caught it earlier?\n4. What is now in the set, and who added it?\n\nQuestion two is the one that improves the evaluation rather than the system, and it is usually the most interesting. The answer is often that the case existed in traffic and was never sampled, which points straight at your sampling, not at the model.\n\nKeep it blameless, and not as a slogan. The moment an incident review becomes a search for who wrote the prompt, people stop reporting the small failures, and the small failures are your entire early-warning system."},"check":{"question":"After eighteen months of diligently adding every incident to the eval set, a team finds their score has drifted down and no longer matches user satisfaction. What is the most likely explanation?","options":[{"text":"The judge has drifted over eighteen months and is now scoring more harshly than it did at the start.","correct":false,"tempting":"Names a real hazard that regular calibration exists to catch, but it would not by itself decouple the score from user satisfaction in this direction."},{"text":"The set is now dominated by rare historical failures, so it measures a distribution of hard cases rather than the traffic users actually send.","correct":true},{"text":"The system has genuinely got worse, and user satisfaction is a lagging indicator that has not caught up.","correct":false,"tempting":"Trusts the offline number over the user signal without first asking whether the offline set still represents the same population."},{"text":"Incident items were added without expected outputs, so many of them are being scored as failures incorrectly.","correct":false,"tempting":"Points at a labelling defect that would show up as a fixed set of always-failing items rather than a gradual drift."}]},"prev":{"slug":"fairness-you-can-measure","title":"Fairness as arithmetic, and the choice you cannot avoid","url":"https://addaly.com/learn/evaluating-ai/fairness-you-can-measure"},"next":{"slug":"reporting-a-number-honestly","title":"Writing the result down","url":"https://addaly.com/learn/evaluating-ai/reporting-a-number-honestly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Turning a complaint into a permanent test', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/incident-to-eval-item (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Turning a complaint into a permanent test","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Every incident becomes a minimised, provenanced test item, but keep those in a regression suite so the eval set still resembles real traffic rather than a museum of old bugs.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Capture","detail":"The exact input, the retrieval context, model and prompt versions, the timestamp."},{"label":"Reproduce","detail":"Run it now. If it flips, run it twenty times and record the rate."},{"label":"Minimise","detail":"Strip to the smallest input that still fails: forty turns usually become two."},{"label":"Add with provenance","detail":"origin: incident, the incident id, severity, the date added."},{"label":"Ask what would have caught it","detail":"A week earlier: a check, a rule, a monitor. Always a mechanism, never a person."}],"title":"From a complaint to a permanent test","caption":"The origin field is what keeps the set from becoming a museum: it lets you count how much of it is history and how much is traffic, and hold incident items under a third."},"alt":"Figure: From a complaint to a permanent test. Steps: 1. Capture (The exact input, the retrieval context, model and prompt versions, the timestamp.); 2. Reproduce (Run it now. If it flips, run it twenty times and record the rate.); 3. Minimise (Strip to the smallest input that still fails: forty turns usually become two.); 4. Add with provenance (origin: incident, the incident id, severity, the date added.); 5. Ask what would have caught it (A week earlier: a check, a rule, a monitor. Always a mechanism, never a person.). The origin field is what keeps the set from becoming a museum: it lets you count how much of it is history and how much is traffic, and hold incident items under a third."}},"blocks":[{"t":"h2","id":"every-incident-is-a-gap-in-the-set","runs":[{"text":"Every incident is a gap in the set"}]},{"t":"p","runs":[{"text":"Something went wrong in production. A customer was told the wrong refund window; the assistant answered in the wrong language; a document nobody should see was retrieved and quoted."}]},{"t":"p","runs":[{"text":"The fix is the obvious part. The valuable part is the loop that makes sure that specific failure never returns unnoticed, and it is a five-step habit that most teams do not have."}]},{"t":"h2","id":"the-five-steps","runs":[{"text":"The five steps"}]},{"t":"p","runs":[{"text":"1. Capture the exact input.","b":true},{"text":" Not a description: the input, verbatim, with the retrieval context, the model version, the prompt version, and the timestamp. This is the reason the logging in the previous lesson exists. “A customer asked something about refunds” cannot be reproduced."}]},{"t":"p","runs":[{"text":"2. Reproduce it.","b":true},{"text":" Run the same input through the current system. Three outcomes, all informative:"}]},{"t":"ul","items":[[{"text":"It reproduces. Good: you have a test."}],[{"text":"It does not, because the system is non-deterministic. Run it twenty times and record the rate. A failure that occurs one time in five is still a failure, and a one-shot test would have called it fixed."}],[{"text":"It does not, because something has already changed. Find out what, or your fix addresses a system that no longer exists."}]]},{"t":"p","runs":[{"text":"3. Minimise.","b":true},{"text":" Strip the input to the smallest thing that still fails. A 40-turn conversation usually reduces to two turns. A minimal case is faster to run, easier to reason about, and much more likely to generalise to the family of inputs that share the cause."}]},{"t":"p","runs":[{"text":"4. Add it to the set, with provenance.","b":true}]},{"t":"code","lang":"jsonl","label":"JSON Lines","text":"{\"id\": \"inc-2026-08-14-refund-window\",\n \"input\": \"...\",\n \"expected\": \"states the 14-day window from the policy, or asks\",\n \"assert\": [{\"type\": \"not_contains\", \"value\": \"30 days\"}],\n \"origin\": \"incident\", \"incident\": \"INC-2291\",\n \"added\": \"2026-08-14\", \"severity\": \"high\"}"},{"t":"p","runs":[{"text":"The "},{"text":"origin","code":true},{"text":" field is what makes this manageable later. It lets you count how much of your set is history and how much is traffic."}]},{"t":"p","runs":[{"text":"5. Ask what would have caught it a week earlier.","b":true},{"text":" This is the question that produces the durable improvement, and the answer is always a mechanism, never a person. Sometimes it is a deterministic check. Sometimes it is a new rubric rule. Sometimes it is a monitor on an output metric nobody was watching. Write the answer down and build it, or the next incident of the same kind will also be found by a customer."}]},{"t":"h2","id":"keep-the-set-from-becoming-a-museum","runs":[{"text":"Keep the set from becoming a museum"}]},{"t":"p","runs":[{"text":"Here is the failure mode of doing this well. After eighteen months, 60% of your set is incident items (rare, hard, historical cases) and the set no longer looks anything like your traffic. Every number it produces is pessimistic, its slices are meaningless, and the team stops believing it."}]},{"t":"p","runs":[{"text":"Two controls:"}]},{"t":"ul","items":[[{"text":"Cap the proportion.","b":true},{"text":" Keep incident-derived items under about a third of the main set. Move the overflow to a separate regression suite that runs in CI as a pass/fail gate and does not contribute to the quality percentage. Two files, two purposes: one estimates quality on real traffic, one prevents known bugs returning."}],[{"text":"Refresh the base.","b":true},{"text":" Every quarter, replace a slice of the ordinary items with a fresh sample of recent traffic. Keep the old ones in the frozen set for historical comparison, but let the live set track what people are actually asking."}]]},{"t":"p","runs":[{"text":"The distinction is worth being firm about. A regression suite answers “has anything we already fixed come back?” and every item in it must pass. An eval set answers “how good is this on the work we actually do?” and a percentage is the right output. Merging them gives you a number that is neither."}]},{"t":"h2","id":"the-review-and-its-tone","runs":[{"text":"The review, and its tone"}]},{"t":"p","runs":[{"text":"Hold a short review for anything customer-visible. Twenty minutes, four questions:"}]},{"t":"ol","start":1,"items":[[{"text":"What happened, in one paragraph, with the timeline."}],[{"text":"Why did the eval set not contain this case?"}],[{"text":"What check, monitor or rule would have caught it earlier?"}],[{"text":"What is now in the set, and who added it?"}]]},{"t":"p","runs":[{"text":"Question two is the one that improves the evaluation rather than the system, and it is usually the most interesting. The answer is often that the case existed in traffic and was never sampled, which points straight at your sampling, not at the model."}]},{"t":"p","runs":[{"text":"Keep it blameless, and not as a slogan. The moment an incident review becomes a search for who wrote the prompt, people stop reporting the small failures, and the small failures are your entire early-warning system."}]}],"check":{"question":[{"text":"After eighteen months of diligently adding every incident to the eval set, a team finds their score has drifted down and no longer matches user satisfaction. What is the most likely explanation?"}],"options":[{"text":[{"text":"The judge has drifted over eighteen months and is now scoring more harshly than it did at the start."}],"correct":false,"tempting":[{"text":"Names a real hazard that regular calibration exists to catch, but it would not by itself decouple the score from user satisfaction in this direction."}]},{"text":[{"text":"The set is now dominated by rare historical failures, so it measures a distribution of hard cases rather than the traffic users actually send."}],"correct":true,"tempting":null},{"text":[{"text":"The system has genuinely got worse, and user satisfaction is a lagging indicator that has not caught up."}],"correct":false,"tempting":[{"text":"Trusts the offline number over the user signal without first asking whether the offline set still represents the same population."}]},{"text":[{"text":"Incident items were added without expected outputs, so many of them are being scored as failures incorrectly."}],"correct":false,"tempting":[{"text":"Points at a labelling defect that would show up as a fixed set of always-failing items rather than a gradual drift."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"fairness-you-can-measure","title":"Fairness as arithmetic, and the choice you cannot avoid","minutes":10},"next":{"slug":"reporting-a-number-honestly","title":"Writing the result down","minutes":8}}}