Ten cases before you trust it

One good answer is not evidence

You wrote a prompt, tried it on a message, and it produced something excellent. That is encouraging and it is not a reason to start using it on customers.

The model samples. Run the same prompt on the same input three times and you get three answers of varying quality. A single result tells you the prompt can work, not that it does.

So before anything goes into daily use, run it against ten cases. Twenty minutes, once.

Choosing the ten

Do not invent them. Take real items from your own records (the last thirty enquiries, or a spread across the last three months) and pick to a shape:

• Four ordinary ones. The bread and butter. If it fails here, nothing else matters.
• Two where information is missing. The customer did not say which size, or asked about an area not on your sheet. You are testing whether it flags the gap or invents a figure. This is the most important pair.
• Two awkward ones. Angry, rambling, three questions at once, or in a mix of two languages.
• One outside your range. Something you do not sell. The right answer is a clean no, not a helpful invention.
• One with a trap. A message containing something like “my friend said you’d do it for half price” or an instruction aimed at the assistant. You are checking that it does not simply agree.

Keep these ten in a file. They are now your test set, and you will use them again every time something changes.

Judging by behaviour, not wording

Do not compare against a model answer you wrote. The output will differ every time and that is not a failure.

Score each case pass or fail against behaviour:

1. Every fact traces to your facts sheet.
2. Nothing was promised that you did not authorise.
3. Gaps were flagged rather than filled.
4. Tone is one you would send.
5. Format matches what you asked for.

A case passes only if all five hold. This is strict on purpose: a reply with perfect tone and an invented delivery date is a failure, and scoring it as half a pass is how businesses talk themselves into shipping something unsafe.

Reading the score

Ten out of ten: use it, and keep spot-checking weekly.

Eight or nine: look at what failed. If the two failures are both the missing-information cases, your grounding instruction is too weak; strengthen it and re-run. If they are scattered, the prompt is probably fine for drafting with a person reading, and not fine for anything automatic.

Six or seven: the prompt needs work, and it is usually the facts rather than the wording. Fix and re-run the whole ten, not just the failures. Fixing one case often breaks another, and testing only the failures hides that completely.

Below six: this may not be a task for a model. Go back to the process map.

Re-running the ten

The test set is worth much more the second time you use it. Re-run when:

• Your vendor changes or updates the model, which they do without asking you.
• You edit the prompt or the facts sheet.
• Something odd happens in real use, in which case that case becomes the eleventh item permanently.

That last habit is the one that compounds. Every real failure you meet gets added to the file, so the same failure can never surprise you twice. After a year you have thirty cases drawn entirely from things that actually went wrong in your business, which is a better test set than any vendor could have given you.

The proportion

Twenty minutes to build, ten to re-run. For a prompt you will use several hundred times a year, that is a trivial cost and it is the only thing standing between “it seemed fine when I tried it” and knowing.

Somebody else should mark it

The person who wrote the prompt is the worst judge of its output. They know what it was meant to do, so they read the intention rather than the text, and they forgive near-misses because they can see how the near-miss happened.

Hand the ten outputs to somebody who does not know which prompt produced them and does know the business: a partner, a colleague, the person who normally answers the messages. Ask one question: would you send this to a customer, yes or no?

The scores usually come back lower, and the disagreements are the interesting part. Where the author says pass and the marker says fail, the reason is almost always an assumption the author was supplying from their own head rather than from the facts sheet.

Keep the file with the prompt

The ten cases live next to the prompt in the library, with the date of the last run and the score. A prompt without a test set is an opinion. A prompt with one is something you can hand to somebody else, re-check after a model update, and defend to a client who asks how you know your replies are accurate.