The five items you keep forever

The problem this solves

You improve a prompt to handle a tricky case. Two weeks later it handles that case and has stopped handling something it used to get right.

Nobody notices, because nobody re-checks the old cases. This is the single most common way a working prompt decays, and the fix is small: a fixed set of items you run every time anything changes.

Building it

Five to ten items. Each item is an input plus the output you would accept.

• Two ordinary cases, so you notice if the everyday path breaks.
• Three or four hard cases, from real failures you have already fixed.
• One case that should produce a refusal or “not stated”: the ability to correctly not answer is a behaviour that regresses like any other, and nobody ever tests it.

Store it as a plain text file next to the prompt. No tooling required.

# ITEM 3 (was failing 2026-08-14: read the order date as the invoice date)
INPUT:  [the invoice text]
EXPECT: invoice_date = 2026-07-30, order_date = 2026-07-12,
        total = 45200.00, currency = KES

The comment matters. Six months on, “why is this item here?” is a real question, and the answer stops somebody deleting the test that protects the fix.

When to run it

After any change to the prompt. That is the previous lesson.

When the model version changes. Providers update models, sometimes without a version bump you would notice. Behaviour shifts. A prompt tuned around an old model’s habit can lose several points overnight, and the failure appears as “it’s been a bit worse lately”, which is not a bug report anybody can act on.

When something upstream changes. A new invoice template, a new product line, a supplier who formats dates differently. Add the new case as item eleven while you are there.

Monthly, if the prompt runs unattended. Ten minutes.

The refusal item is the one people miss

If your prompt has an escape hatch (write “not stated”, say “cannot answer”, ask a clarifying question) you have a behaviour that can silently disappear.

It disappears in a specific way: you add examples to improve extraction, all of them successful extractions, and the model learns that extraction always succeeds. Now the missing field gets a plausible value instead of NOT_STATED, and the failure is invisible precisely because it produces a full-looking result.

One test item with a deliberately incomplete document catches it.

Scoring, kept crude

Right or wrong per item, counted out of ten, three runs. Resist elaborating this. A percentage to one decimal place from ten items is false precision, and the point is to notice a drop of two or three items, which crude counting shows perfectly well.

Where the output is prose, use a three-line checklist per item and mark each line. Same crudeness, same purpose.

What to do when it drops

Do not immediately rewrite. Ask the diagnostic question first: did the prompt change, or did something else?

If nothing in the prompt changed, look at the model version, the product settings, the memory, the input format, and whether the source data changed shape. Rewriting a prompt to compensate for an upstream change you have not identified produces a prompt that is wrong twice.

The honest limit

Ten items cannot detect a small change. They will catch a fall from 9 to 5 and will not reliably distinguish 9 from 8. That is fine, because the failures that matter in everyday work are large: a behaviour disappeared, a format broke, an escape hatch stopped firing.

If you need to detect a small change, you need many more items and a way to score them automatically: a real evaluation, which is a subject of its own. For everything most people do, ten items and three runs is the whole of what is needed, and it is ten items and three runs more than almost anyone has.

Where the items come from

You do not sit down and invent them. Every one of them is a failure you already had.

Something goes wrong. You fix it. Before moving on, spend sixty seconds saving the input that broke it and the answer you wanted. That is one item, and by the time you have five you have a check that covers the specific ways your work actually goes wrong, which is a far better test than any set you could have designed in advance, because you would have designed for the failures you could imagine.

The habit is the same one from the examples module, with a different purpose: keep the corrections. Nearly everything durable in this course comes from that single habit applied to different files.