{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-first-fifty-cases","title":"The first fifty cases, and where they come from","url":"https://addaly.com/learn/building-with-ai/the-first-fifty-cases","position":53,"of":80,"minutes":8,"answer":"An evaluation set for a feature is fifty cases with provenance (drawn from logs, from failures.","keyIdea":"An evaluation set for a feature is fifty cases with provenance (drawn from logs, from failures, and from the specification in roughly equal parts) each with an expectation precise enough to check by code or rubric, kept in version control beside the prompt it tests."},"body":{"format":"markdown","text":"## Fifty, not five hundred\n\nThe earlier lesson in this course made the claim: if you cannot re-run fifty saved cases, you are guessing. This lesson is about assembling those fifty. Not five hundred; that comes later, from production, one failure at a time. Fifty is what one person builds in a day, and it is enough to catch the regressions that matter, because most prompt changes that break something break it visibly on a handful of cases.\n\nThe evaluation course in this catalogue covers sampling, labelling and statistics in depth. Here the goal is narrower: a set you can build this week for a feature you own.\n\n## Three sources, roughly equal\n\n**From the logs.** If the feature has shipped, pull real inputs. Not the most recent hundred (the evaluation course explains why that is the worst sample you own) but a spread: different intents, different lengths, different languages if you serve several, a few from each week. Redact identifiers before anything lands in a file. Real inputs are the only source that contains the things you did not think of.\n\n**From failures.** Every complaint, every thumbs-down with a category, every bug report that included the input. These are the most valuable cases because they are known to have failed once. A set built only from happy paths measures whether the feature still works on the things it already handled.\n\n**From the specification.** Write cases for what the feature is supposed to do, including the edges: an empty input, an input in the wrong language, a question the corpus cannot answer, a request that should be refused, an input containing an instruction to the model. These are cases the logs will not contain until the day they matter.\n\nFifteen or twenty from each source is the target. If the feature has not shipped, the first source is replaced by inputs written by two people separately, compared afterwards: the overlap is the obvious, the difference is the interesting.\n\n## The shape of a case\n\nOne line of JSON per case, in a file beside the prompt it tests:\n\n```json\n{\"id\": \"refund-window-annual\", \"source\": \"logs\",\n \"tags\": [\"refunds\", \"annual-plan\"],\n \"input\": {\"message\": \"how long do I have to cancel an annual plan?\",\n           \"history\": []},\n \"expect\": {\"must_contain\": [\"14 days\"],\n            \"must_not_contain\": [\"30 days\"],\n            \"cites\": [\"policy-refunds-v3\"],\n            \"rubric\": \"States the 14-day window and that it applies from purchase, not renewal.\"}}\n```\n\nThe `source` field is the provenance the evaluation course insists on; six months from now nobody remembers where a case came from. Tags let you report by slice, which the next lessons rely on. The `expect` block is the part people get wrong.\n\n## Writing an expectation you can check\n\nFor a closed answer (a classification, a number, an extracted field) the expectation is the value, and a check is equality. For open text the expectation cannot be the exact wording, because the model will phrase it differently every run. Three tools cover almost everything:\n\n- **Must-contain and must-not-contain**: strings or patterns that any correct answer includes, and strings whose presence proves the answer wrong. \"14 days\" must appear; \"30 days\" must not.\n- **Structural checks**: valid JSON, a citation that resolves, a length under a limit, the right language. The next lesson is a catalogue of these.\n- **A rubric**: one or two sentences saying what a correct answer does, for a judge (human or model) to apply. Write it as a statement to be marked true or false, not as a score out of ten.\n\nAn expectation that is only a rubric is expensive to check. An expectation that is only a string match misses wrong answers that happen to contain the right words. Use both where you can.\n\n## Golden and regression\n\nSplit the set in two by purpose. Golden cases define correct behaviour and rarely change. Regression cases each encode one specific past failure, and their number only grows. When a prompt change makes a golden case fail, the change is probably wrong. When it makes a regression case fail, a bug you fixed has come back, and the case's id tells you which.\n\n## The case whose expectation is wrong\n\nSome fraction of your fifty will turn out to have a bad expectation: the \"correct\" answer you wrote was itself mistaken, or the policy changed. This is normal, and it is a reason to keep cases in version control with the prompt: a change to an expectation is a reviewed change with a reason, not a silent edit in a spreadsheet. When a case fails, the first question is always whether the case or the system is wrong, and the evaluation course's error-analysis habit, applied here in a later lesson, makes \"expectation wrong\" an explicit category rather than an embarrassment.\n\n## Keep it beside the prompt\n\n`prompts/triage/v7.md` and `prompts/triage/cases.jsonl` in the same directory, in the same commit. The earlier lesson on prompts in version control gave the reason; this lesson gives the file. A change to either without the other should look odd in review."},"check":{"question":"A team builds its evaluation set entirely from conversations that users rated with a thumbs-up. Prompt changes pass the set and still cause complaints in production. What is wrong with the set?","options":[{"text":"Fifty cases is too few; the set needs several hundred before it can predict production behaviour reliably across the range of real inputs.","correct":false,"tempting":"Size is not the issue; a larger set drawn from the same source would have the same blind spot."},{"text":"It holds only inputs the feature already handled well, so it cannot catch regressions on the failures that generate complaints.","correct":true},{"text":"Thumbs-up conversations are longer than average, so the set over-represents multi-turn cases.","correct":false,"tempting":"A plausible skew, but it does not explain why known failure types are missed entirely."},{"text":"The expectations were written by the same people who wrote the prompt, so they are biased towards its behaviour.","correct":false,"tempting":"A real risk in rubric writing, but the deeper problem is the source of the inputs, not the authorship of the expectations."}]},"prev":{"slug":"evaluating-your-feature","title":"Evaluating your own AI feature","url":"https://addaly.com/learn/building-with-ai/evaluating-your-feature"},"next":{"slug":"assertions-before-judges","title":"Assertions before judges: the checks that cost nothing","url":"https://addaly.com/learn/building-with-ai/assertions-before-judges"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The first fifty cases, and where they come from', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/the-first-fifty-cases (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The first fifty cases, and where they come from","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"An evaluation set for a feature is fifty cases with provenance (drawn from logs, from failures, and from the specification in roughly equal parts) each with an expectation precise enough to check by code or rubric, kept in version control beside the prompt it tests.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"fifty-not-five-hundred","runs":[{"text":"Fifty, not five hundred"}]},{"t":"p","runs":[{"text":"The earlier lesson in this course made the claim: if you cannot re-run fifty saved cases, you are guessing. This lesson is about assembling those fifty. Not five hundred; that comes later, from production, one failure at a time. Fifty is what one person builds in a day, and it is enough to catch the regressions that matter, because most prompt changes that break something break it visibly on a handful of cases."}]},{"t":"p","runs":[{"text":"The evaluation course in this catalogue covers sampling, labelling and statistics in depth. Here the goal is narrower: a set you can build this week for a feature you own."}]},{"t":"h2","id":"three-sources-roughly-equal","runs":[{"text":"Three sources, roughly equal"}]},{"t":"p","runs":[{"text":"From the logs.","b":true},{"text":" If the feature has shipped, pull real inputs. Not the most recent hundred (the evaluation course explains why that is the worst sample you own) but a spread: different intents, different lengths, different languages if you serve several, a few from each week. Redact identifiers before anything lands in a file. Real inputs are the only source that contains the things you did not think of."}]},{"t":"p","runs":[{"text":"From failures.","b":true},{"text":" Every complaint, every thumbs-down with a category, every bug report that included the input. These are the most valuable cases because they are known to have failed once. A set built only from happy paths measures whether the feature still works on the things it already handled."}]},{"t":"p","runs":[{"text":"From the specification.","b":true},{"text":" Write cases for what the feature is supposed to do, including the edges: an empty input, an input in the wrong language, a question the corpus cannot answer, a request that should be refused, an input containing an instruction to the model. These are cases the logs will not contain until the day they matter."}]},{"t":"p","runs":[{"text":"Fifteen or twenty from each source is the target. If the feature has not shipped, the first source is replaced by inputs written by two people separately, compared afterwards: the overlap is the obvious, the difference is the interesting."}]},{"t":"h2","id":"the-shape-of-a-case","runs":[{"text":"The shape of a case"}]},{"t":"p","runs":[{"text":"One line of JSON per case, in a file beside the prompt it tests:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"id\": \"refund-window-annual\", \"source\": \"logs\",\n \"tags\": [\"refunds\", \"annual-plan\"],\n \"input\": {\"message\": \"how long do I have to cancel an annual plan?\",\n           \"history\": []},\n \"expect\": {\"must_contain\": [\"14 days\"],\n            \"must_not_contain\": [\"30 days\"],\n            \"cites\": [\"policy-refunds-v3\"],\n            \"rubric\": \"States the 14-day window and that it applies from purchase, not renewal.\"}}"},{"t":"p","runs":[{"text":"The "},{"text":"source","code":true},{"text":" field is the provenance the evaluation course insists on; six months from now nobody remembers where a case came from. Tags let you report by slice, which the next lessons rely on. The "},{"text":"expect","code":true},{"text":" block is the part people get wrong."}]},{"t":"h2","id":"writing-an-expectation-you-can-check","runs":[{"text":"Writing an expectation you can check"}]},{"t":"p","runs":[{"text":"For a closed answer (a classification, a number, an extracted field) the expectation is the value, and a check is equality. For open text the expectation cannot be the exact wording, because the model will phrase it differently every run. Three tools cover almost everything:"}]},{"t":"ul","items":[[{"text":"Must-contain and must-not-contain","b":true},{"text":": strings or patterns that any correct answer includes, and strings whose presence proves the answer wrong. “14 days” must appear; “30 days” must not."}],[{"text":"Structural checks","b":true},{"text":": valid JSON, a citation that resolves, a length under a limit, the right language. The next lesson is a catalogue of these."}],[{"text":"A rubric","b":true},{"text":": one or two sentences saying what a correct answer does, for a judge (human or model) to apply. Write it as a statement to be marked true or false, not as a score out of ten."}]]},{"t":"p","runs":[{"text":"An expectation that is only a rubric is expensive to check. An expectation that is only a string match misses wrong answers that happen to contain the right words. Use both where you can."}]},{"t":"h2","id":"golden-and-regression","runs":[{"text":"Golden and regression"}]},{"t":"p","runs":[{"text":"Split the set in two by purpose. Golden cases define correct behaviour and rarely change. Regression cases each encode one specific past failure, and their number only grows. When a prompt change makes a golden case fail, the change is probably wrong. When it makes a regression case fail, a bug you fixed has come back, and the case’s id tells you which."}]},{"t":"h2","id":"the-case-whose-expectation-is-wrong","runs":[{"text":"The case whose expectation is wrong"}]},{"t":"p","runs":[{"text":"Some fraction of your fifty will turn out to have a bad expectation: the “correct” answer you wrote was itself mistaken, or the policy changed. This is normal, and it is a reason to keep cases in version control with the prompt: a change to an expectation is a reviewed change with a reason, not a silent edit in a spreadsheet. When a case fails, the first question is always whether the case or the system is wrong, and the evaluation course’s error-analysis habit, applied here in a later lesson, makes “expectation wrong” an explicit category rather than an embarrassment."}]},{"t":"h2","id":"keep-it-beside-the-prompt","runs":[{"text":"Keep it beside the prompt"}]},{"t":"p","runs":[{"text":"prompts/triage/v7.md","code":true},{"text":" and "},{"text":"prompts/triage/cases.jsonl","code":true},{"text":" in the same directory, in the same commit. The earlier lesson on prompts in version control gave the reason; this lesson gives the file. A change to either without the other should look odd in review."}]}],"check":{"question":[{"text":"A team builds its evaluation set entirely from conversations that users rated with a thumbs-up. Prompt changes pass the set and still cause complaints in production. What is wrong with the set?"}],"options":[{"text":[{"text":"Fifty cases is too few; the set needs several hundred before it can predict production behaviour reliably across the range of real inputs."}],"correct":false,"tempting":[{"text":"Size is not the issue; a larger set drawn from the same source would have the same blind spot."}]},{"text":[{"text":"It holds only inputs the feature already handled well, so it cannot catch regressions on the failures that generate complaints."}],"correct":true,"tempting":null},{"text":[{"text":"Thumbs-up conversations are longer than average, so the set over-represents multi-turn cases."}],"correct":false,"tempting":[{"text":"A plausible skew, but it does not explain why known failure types are missed entirely."}]},{"text":[{"text":"The expectations were written by the same people who wrote the prompt, so they are biased towards its behaviour."}],"correct":false,"tempting":[{"text":"A real risk in rubric writing, but the deeper problem is the source of the inputs, not the authorship of the expectations."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"evaluating-your-feature","title":"Evaluating your own AI feature","minutes":8},"next":{"slug":"assertions-before-judges","title":"Assertions before judges: the checks that cost nothing","minutes":8}}}