Proving the examples were worth it

Nobody checks, and the examples cost real money

You added five examples. The output feels better. Did it get better?

You cannot tell by reading one output, for two reasons you already know. Sampling means two runs of the same prompt differ, so a good result may be a good draw. And you wrote the examples, so you are inclined to see their influence.

The test is not hard. It takes twenty minutes once, and it is the difference between prompting as craft and prompting as superstition.

The smallest honest test

1. Build ten items with known answers. Real ones, from the archive lesson. For classification, ten messages with the label you would give. For extraction, ten documents with the three fields written out. For writing, ten briefs plus the output you would accept; this last one is harder to score, and we come to it below.

Hold these out. They must not be your examples. An item that appears in the prompt is not a test.

2. Run version A (no examples) over all ten. Run version B (with examples) over the same ten.

3. Do it three times each. Sampling, again. Thirty runs sounds like a lot; it is ten minutes of pasting, or five lines of script if you are using an API.

4. Count. How many of ten correct, in each of the three runs, for each version.

Reading the result honestly

You will get something like: A scored 6, 7, 6. B scored 8, 8, 9.

That is a real difference. Not because of a statistical test (with ten items you have no business running one) but because the ranges do not overlap and the gap is large relative to the wobble.

Now the more common result: A scored 7, 6, 8. B scored 7, 8, 7.

That is nothing. The ranges overlap completely. Your examples cost you four hundred tokens per call and bought you noise. Take them out, or find better ones, probably from the four items that failed in both versions.

Ten items cannot detect a small improvement. If A gets 7 and B gets 8, one item moved, and one item is well inside the range of chance. Believing a one-item difference is the most common error people make with small test sets, and knowing that you cannot see it is the useful skill. If the difference matters enough to argue about, you need more items, not more confidence.

Scoring when the output is prose

Harder, and still doable. Two workable approaches:

A checklist per item. Write the three or four things a good output must contain, for each of your ten briefs. Score out of four. This is crude and it is consistent, which is the property that matters when comparing two versions.

Blind pairwise comparison. Put A and B outputs side by side without labels, shuffle which side each is on, and pick the better one. Do all ten. If you cannot tell which is which more than half the time, there is no difference worth paying for. Hiding the labels from yourself is the entire point; unblinded, you will pick the version you built.

What this habit is for

Not rigour for its own sake. It settles arguments, including the ones with yourself at eleven at night, and it stops a prompt accumulating decoration that nobody has ever tested.

Almost every long prompt you will inherit from a colleague contains three or four things that were added once, felt right, and have never been checked. Some of them are helping. Some are costing space and narrowing outputs. Twenty minutes tells you which.

There is a second reason to do it, less obvious and probably more valuable: the four items that fail in both versions are the most informative thing you own. They are not an examples problem, so no amount of example-tuning will move them. They are telling you that the task, the material or the model is the constraint, and that is where your next hour should go.

And once you have the ten items, you have them forever. Every future change to that prompt (a new model, a new constraint, a rewrite) gets checked against the same ten in ten minutes. That is the whole idea, and it is coming back in a later module as the thing that makes a prompt maintainable rather than lucky.