{"course":{"slug":"prompting","title":"Getting Real Work out of a Model","url":"https://addaly.com/learn/prompting","level":"beginner","track":"start"},"lesson":{"slug":"the-regression-check","title":"The five items you keep forever","url":"https://addaly.com/learn/prompting/the-regression-check","position":49,"of":72,"minutes":8,"answer":"Keep five to ten fixed items with expected answers, including one that should produce a refusal, and run them after any prompt change and after any model update: decay is invisible without them.","keyIdea":"Keep five to ten fixed items with expected answers, including one that should produce a refusal, and run them after any prompt change and after any model update: decay is invisible without them."},"body":{"format":"markdown","text":"## The problem this solves\n\nYou improve a prompt to handle a tricky case. Two weeks later it handles that case and has stopped handling something it used to get right.\n\nNobody notices, because nobody re-checks the old cases. This is the single most common way a working prompt decays, and the fix is small: a fixed set of items you run every time anything changes.\n\n## Building it\n\nFive to ten items. Each item is an input plus the output you would accept.\n\n- Two ordinary cases, so you notice if the everyday path breaks.\n- Three or four hard cases, from real failures you have already fixed.\n- One case that should produce a refusal or \"not stated\": the ability to correctly not answer is a behaviour that regresses like any other, and nobody ever tests it.\n\nStore it as a plain text file next to the prompt. No tooling required.\n\n```\n# ITEM 3 (was failing 2026-08-14: read the order date as the invoice date)\nINPUT:  [the invoice text]\nEXPECT: invoice_date = 2026-07-30, order_date = 2026-07-12,\n        total = 45200.00, currency = KES\n```\n\nThe comment matters. Six months on, \"why is this item here?\" is a real question, and the answer stops somebody deleting the test that protects the fix.\n\n## When to run it\n\n**After any change to the prompt.** That is the previous lesson.\n\n**When the model version changes.** Providers update models, sometimes without a version bump you would notice. Behaviour shifts. A prompt tuned around an old model's habit can lose several points overnight, and the failure appears as \"it's been a bit worse lately\", which is not a bug report anybody can act on.\n\n**When something upstream changes.** A new invoice template, a new product line, a supplier who formats dates differently. Add the new case as item eleven while you are there.\n\n**Monthly, if the prompt runs unattended.** Ten minutes.\n\n> Figure: Ten items, run monthly, on an unattended prompt. Line chart of Items correct, out of ten against Month: Invoice extraction prompt (1, 9) (2, 9) (3, 9) (4, 6) (5, 9) (6, 9). Month four: the provider updated the model, the refusal item stopped firing, and a missing field began receiving a plausible value. Without the fixed items this would have read as 'a bit worse lately', which is not a report anybody can act on.\n\n## The refusal item is the one people miss\n\nIf your prompt has an escape hatch (write \"not stated\", say \"cannot answer\", ask a clarifying question) you have a behaviour that can silently disappear.\n\nIt disappears in a specific way: you add examples to improve extraction, all of them successful extractions, and the model learns that extraction always succeeds. Now the missing field gets a plausible value instead of `NOT_STATED`, and the failure is invisible precisely because it produces a full-looking result.\n\nOne test item with a deliberately incomplete document catches it.\n\n## Scoring, kept crude\n\nRight or wrong per item, counted out of ten, three runs. Resist elaborating this. A percentage to one decimal place from ten items is false precision, and the point is to notice a drop of two or three items, which crude counting shows perfectly well.\n\nWhere the output is prose, use a three-line checklist per item and mark each line. Same crudeness, same purpose.\n\n## What to do when it drops\n\nDo not immediately rewrite. Ask the diagnostic question first: **did the prompt change, or did something else?**\n\nIf nothing in the prompt changed, look at the model version, the product settings, the memory, the input format, and whether the source data changed shape. Rewriting a prompt to compensate for an upstream change you have not identified produces a prompt that is wrong twice.\n\n## The honest limit\n\nTen items cannot detect a small change. They will catch a fall from 9 to 5 and will not reliably distinguish 9 from 8. That is fine, because the failures that matter in everyday work are large: a behaviour disappeared, a format broke, an escape hatch stopped firing.\n\nIf you need to detect a small change, you need many more items and a way to score them automatically: a real evaluation, which is a subject of its own. For everything most people do, ten items and three runs is the whole of what is needed, and it is ten items and three runs more than almost anyone has.\n\n## Where the items come from\n\nYou do not sit down and invent them. Every one of them is a failure you already had.\n\nSomething goes wrong. You fix it. Before moving on, spend sixty seconds saving the input that broke it and the answer you wanted. That is one item, and by the time you have five you have a check that covers the specific ways your work actually goes wrong, which is a far better test than any set you could have designed in advance, because you would have designed for the failures you could imagine.\n\nThe habit is the same one from the examples module, with a different purpose: keep the corrections. Nearly everything durable in this course comes from that single habit applied to different files."},"check":{"question":"A prompt that reliably wrote NOT_STATED for missing invoice fields starts filling them with plausible values, after the team added six successful extraction examples. Which test item would have caught this?","options":[{"text":"An item whose document is deliberately missing a field, with NOT_STATED as the expected answer.","correct":true},{"text":"An item with an unusual invoice layout, testing robustness to format changes.","correct":false,"tempting":"Tests a different failure; a well-formed unusual layout does not exercise the missing-field path."},{"text":"An item with a very long invoice, testing whether all fields are found.","correct":false,"tempting":"Length tests coverage, not the behaviour when a field genuinely is not there."},{"text":"Three runs of the same standard invoice, checking for variation between runs.","correct":false,"tempting":"Measures stability on a case that was never broken."}]},"prev":{"slug":"change-one-thing","title":"Change one thing at a time","url":"https://addaly.com/learn/prompting/change-one-thing"},"next":{"slug":"your-own-prompt-file","title":"The file that makes this compound","url":"https://addaly.com/learn/prompting/your-own-prompt-file"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The five items you keep forever', in Getting Real Work out of a Model. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/prompting/the-regression-check (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The five items you keep forever","courseTitle":"Getting Real Work out of a Model","module":{"index":6,"slug":"from-lucky-to-reliable","title":"From lucky to reliable"},"answer":null,"keyIdea":"Keep five to ten fixed items with expected answers, including one that should produce a refusal, and run them after any prompt change and after any model update: decay is invisible without them.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Ten items, run monthly, on an unattended prompt","series":[{"label":"Invoice extraction prompt","points":[[1,9],[2,9],[3,9],[4,6],[5,9],[6,9]]}],"xLabel":"Month","yLabel":"Items correct, out of ten","caption":"Month four: the provider updated the model, the refusal item stopped firing, and a missing field began receiving a plausible value. Without the fixed items this would have read as ‘a bit worse lately’, which is not a report anybody can act on."},"alt":"Figure: Ten items, run monthly, on an unattended prompt. Line chart of Items correct, out of ten against Month: Invoice extraction prompt (1, 9) (2, 9) (3, 9) (4, 6) (5, 9) (6, 9). Month four: the provider updated the model, the refusal item stopped firing, and a missing field began receiving a plausible value. Without the fixed items this would have read as ‘a bit worse lately’, which is not a report anybody can act on."}},"blocks":[{"t":"h2","id":"the-problem-this-solves","runs":[{"text":"The problem this solves"}]},{"t":"p","runs":[{"text":"You improve a prompt to handle a tricky case. Two weeks later it handles that case and has stopped handling something it used to get right."}]},{"t":"p","runs":[{"text":"Nobody notices, because nobody re-checks the old cases. This is the single most common way a working prompt decays, and the fix is small: a fixed set of items you run every time anything changes."}]},{"t":"h2","id":"building-it","runs":[{"text":"Building it"}]},{"t":"p","runs":[{"text":"Five to ten items. Each item is an input plus the output you would accept."}]},{"t":"ul","items":[[{"text":"Two ordinary cases, so you notice if the everyday path breaks."}],[{"text":"Three or four hard cases, from real failures you have already fixed."}],[{"text":"One case that should produce a refusal or “not stated”: the ability to correctly not answer is a behaviour that regresses like any other, and nobody ever tests it."}]]},{"t":"p","runs":[{"text":"Store it as a plain text file next to the prompt. No tooling required."}]},{"t":"code","lang":"","label":"Text","text":"# ITEM 3 (was failing 2026-08-14: read the order date as the invoice date)\nINPUT:  [the invoice text]\nEXPECT: invoice_date = 2026-07-30, order_date = 2026-07-12,\n        total = 45200.00, currency = KES"},{"t":"p","runs":[{"text":"The comment matters. Six months on, “why is this item here?” is a real question, and the answer stops somebody deleting the test that protects the fix."}]},{"t":"h2","id":"when-to-run-it","runs":[{"text":"When to run it"}]},{"t":"p","runs":[{"text":"After any change to the prompt.","b":true},{"text":" That is the previous lesson."}]},{"t":"p","runs":[{"text":"When the model version changes.","b":true},{"text":" Providers update models, sometimes without a version bump you would notice. Behaviour shifts. A prompt tuned around an old model’s habit can lose several points overnight, and the failure appears as “it’s been a bit worse lately”, which is not a bug report anybody can act on."}]},{"t":"p","runs":[{"text":"When something upstream changes.","b":true},{"text":" A new invoice template, a new product line, a supplier who formats dates differently. Add the new case as item eleven while you are there."}]},{"t":"p","runs":[{"text":"Monthly, if the prompt runs unattended.","b":true},{"text":" Ten minutes."}]},{"t":"h2","id":"the-refusal-item-is-the-one-people-miss","runs":[{"text":"The refusal item is the one people miss"}]},{"t":"p","runs":[{"text":"If your prompt has an escape hatch (write “not stated”, say “cannot answer”, ask a clarifying question) you have a behaviour that can silently disappear."}]},{"t":"p","runs":[{"text":"It disappears in a specific way: you add examples to improve extraction, all of them successful extractions, and the model learns that extraction always succeeds. Now the missing field gets a plausible value instead of "},{"text":"NOT_STATED","code":true},{"text":", and the failure is invisible precisely because it produces a full-looking result."}]},{"t":"p","runs":[{"text":"One test item with a deliberately incomplete document catches it."}]},{"t":"h2","id":"scoring-kept-crude","runs":[{"text":"Scoring, kept crude"}]},{"t":"p","runs":[{"text":"Right or wrong per item, counted out of ten, three runs. Resist elaborating this. A percentage to one decimal place from ten items is false precision, and the point is to notice a drop of two or three items, which crude counting shows perfectly well."}]},{"t":"p","runs":[{"text":"Where the output is prose, use a three-line checklist per item and mark each line. Same crudeness, same purpose."}]},{"t":"h2","id":"what-to-do-when-it-drops","runs":[{"text":"What to do when it drops"}]},{"t":"p","runs":[{"text":"Do not immediately rewrite. Ask the diagnostic question first: "},{"text":"did the prompt change, or did something else?","b":true}]},{"t":"p","runs":[{"text":"If nothing in the prompt changed, look at the model version, the product settings, the memory, the input format, and whether the source data changed shape. Rewriting a prompt to compensate for an upstream change you have not identified produces a prompt that is wrong twice."}]},{"t":"h2","id":"the-honest-limit","runs":[{"text":"The honest limit"}],"role":"limit"},{"t":"p","runs":[{"text":"Ten items cannot detect a small change. They will catch a fall from 9 to 5 and will not reliably distinguish 9 from 8. That is fine, because the failures that matter in everyday work are large: a behaviour disappeared, a format broke, an escape hatch stopped firing."}]},{"t":"p","runs":[{"text":"If you need to detect a small change, you need many more items and a way to score them automatically: a real evaluation, which is a subject of its own. For everything most people do, ten items and three runs is the whole of what is needed, and it is ten items and three runs more than almost anyone has."}]},{"t":"h2","id":"where-the-items-come-from","runs":[{"text":"Where the items come from"}]},{"t":"p","runs":[{"text":"You do not sit down and invent them. Every one of them is a failure you already had."}]},{"t":"p","runs":[{"text":"Something goes wrong. You fix it. Before moving on, spend sixty seconds saving the input that broke it and the answer you wanted. That is one item, and by the time you have five you have a check that covers the specific ways your work actually goes wrong, which is a far better test than any set you could have designed in advance, because you would have designed for the failures you could imagine."}]},{"t":"p","runs":[{"text":"The habit is the same one from the examples module, with a different purpose: keep the corrections. Nearly everything durable in this course comes from that single habit applied to different files."}]}],"check":{"question":[{"text":"A prompt that reliably wrote NOT_STATED for missing invoice fields starts filling them with plausible values, after the team added six successful extraction examples. Which test item would have caught this?"}],"options":[{"text":[{"text":"An item whose document is deliberately missing a field, with NOT_STATED as the expected answer."}],"correct":true,"tempting":null},{"text":[{"text":"An item with an unusual invoice layout, testing robustness to format changes."}],"correct":false,"tempting":[{"text":"Tests a different failure; a well-formed unusual layout does not exercise the missing-field path."}]},{"text":[{"text":"An item with a very long invoice, testing whether all fields are found."}],"correct":false,"tempting":[{"text":"Length tests coverage, not the behaviour when a field genuinely is not there."}]},{"text":[{"text":"Three runs of the same standard invoice, checking for variation between runs."}],"correct":false,"tempting":[{"text":"Measures stability on a case that was never broken."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"change-one-thing","title":"Change one thing at a time","minutes":8},"next":{"slug":"your-own-prompt-file","title":"The file that makes this compound","minutes":8}}}