{"course":{"slug":"prompting","title":"Getting Real Work out of a Model","url":"https://addaly.com/learn/prompting","level":"beginner","track":"start"},"lesson":{"slug":"testing-whether-examples-helped","title":"Proving the examples were worth it","url":"https://addaly.com/learn/prompting/testing-whether-examples-helped","position":27,"of":72,"minutes":8,"answer":"Compare a prompt with and without its examples on ten held-out items, three runs each, and treat a difference smaller than the run-to-run wobble as no difference at all.","keyIdea":"Compare a prompt with and without its examples on ten held-out items, three runs each, and treat a difference smaller than the run-to-run wobble as no difference at all."},"body":{"format":"markdown","text":"## Nobody checks, and the examples cost real money\n\nYou added five examples. The output feels better. Did it get better?\n\nYou cannot tell by reading one output, for two reasons you already know. Sampling means two runs of the same prompt differ, so a good result may be a good draw. And you wrote the examples, so you are inclined to see their influence.\n\nThe test is not hard. It takes twenty minutes once, and it is the difference between prompting as craft and prompting as superstition.\n\n## The smallest honest test\n\n**1. Build ten items with known answers.** Real ones, from the archive lesson. For classification, ten messages with the label you would give. For extraction, ten documents with the three fields written out. For writing, ten briefs plus the output you would accept; this last one is harder to score, and we come to it below.\n\nHold these out. They must not be your examples. An item that appears in the prompt is not a test.\n\n**2. Run version A** (no examples) over all ten. **Run version B** (with examples) over the same ten.\n\n**3. Do it three times each.** Sampling, again. Thirty runs sounds like a lot; it is ten minutes of pasting, or five lines of script if you are using an API.\n\n**4. Count.** How many of ten correct, in each of the three runs, for each version.\n\n## Reading the result honestly\n\nYou will get something like: A scored 6, 7, 6. B scored 8, 8, 9.\n\nThat is a real difference. Not because of a statistical test (with ten items you have no business running one) but because the ranges do not overlap and the gap is large relative to the wobble.\n\nNow the more common result: A scored 7, 6, 8. B scored 7, 8, 7.\n\nThat is nothing. The ranges overlap completely. Your examples cost you four hundred tokens per call and bought you noise. Take them out, or find better ones, probably from the four items that failed in both versions.\n\n> Figure: Two prompt versions over the same ten items, three runs each. Line chart of Items correct, out of ten against Run: A: no examples (1, 6) (2, 7) (3, 6); B: with examples (1, 8) (2, 8) (3, 9). The ranges do not overlap and the gap is large relative to the wobble, so the examples earned their space. Had B scored 7, 8, 7 against A's 7, 6, 8, the lines would have crossed and the examples would have bought nothing but noise.\n\n**Ten items cannot detect a small improvement.** If A gets 7 and B gets 8, one item moved, and one item is well inside the range of chance. Believing a one-item difference is the most common error people make with small test sets, and knowing that you cannot see it is the useful skill. If the difference matters enough to argue about, you need more items, not more confidence.\n\n## Scoring when the output is prose\n\nHarder, and still doable. Two workable approaches:\n\n**A checklist per item.** Write the three or four things a good output must contain, for each of your ten briefs. Score out of four. This is crude and it is consistent, which is the property that matters when comparing two versions.\n\n**Blind pairwise comparison.** Put A and B outputs side by side without labels, shuffle which side each is on, and pick the better one. Do all ten. If you cannot tell which is which more than half the time, there is no difference worth paying for. Hiding the labels from yourself is the entire point; unblinded, you will pick the version you built.\n\n## What this habit is for\n\nNot rigour for its own sake. It settles arguments, including the ones with yourself at eleven at night, and it stops a prompt accumulating decoration that nobody has ever tested.\n\nAlmost every long prompt you will inherit from a colleague contains three or four things that were added once, felt right, and have never been checked. Some of them are helping. Some are costing space and narrowing outputs. Twenty minutes tells you which.\n\nThere is a second reason to do it, less obvious and probably more valuable: the four items that fail in both versions are the most informative thing you own. They are not an examples problem, so no amount of example-tuning will move them. They are telling you that the task, the material or the model is the constraint, and that is where your next hour should go.\n\nAnd once you have the ten items, you have them forever. Every future change to that prompt (a new model, a new constraint, a rewrite) gets checked against the same ten in ten minutes. That is the whole idea, and it is coming back in a later module as the thing that makes a prompt maintainable rather than lucky."},"check":{"question":"A prompt with examples scores 8 out of 10 on a test set; without them it scores 7. Both were run once. What is the right conclusion?","options":[{"text":"The examples help by about 10%, which justifies keeping them.","correct":false,"tempting":"Converts a one-item difference in a single run into a percentage, which implies a precision the test cannot support."},{"text":"The examples do not help, since the difference is only one item.","correct":false,"tempting":"Equally unjustified in the other direction; one run cannot establish absence any more than presence."},{"text":"The test set is too small to be worth using at all.","correct":false,"tempting":"Ten items are useful for catching large differences and for regression testing; the problem here is the single run, not the size."},{"text":"Nothing yet: one run of each cannot separate a real difference from ordinary sampling variation, so run both several times first.","correct":true}]},"prev":{"slug":"examples-from-your-own-archive","title":"Your best examples already exist","url":"https://addaly.com/learn/prompting/examples-from-your-own-archive"},"next":{"slug":"ask-for-a-usable-shape","title":"Ask for the shape you can actually use","url":"https://addaly.com/learn/prompting/ask-for-a-usable-shape"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Proving the examples were worth it', in Getting Real Work out of a Model. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/prompting/testing-whether-examples-helped (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Proving the examples were worth it","courseTitle":"Getting Real Work out of a Model","module":{"index":3,"slug":"teaching-by-example","title":"Teaching by example"},"answer":null,"keyIdea":"Compare a prompt with and without its examples on ten held-out items, three runs each, and treat a difference smaller than the run-to-run wobble as no difference at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Two prompt versions over the same ten items, three runs each","series":[{"label":"A: no examples","points":[[1,6],[2,7],[3,6]]},{"label":"B: with examples","points":[[1,8],[2,8],[3,9]]}],"xLabel":"Run","yLabel":"Items correct, out of ten","caption":"The ranges do not overlap and the gap is large relative to the wobble, so the examples earned their space. Had B scored 7, 8, 7 against A’s 7, 6, 8, the lines would have crossed and the examples would have bought nothing but noise."},"alt":"Figure: Two prompt versions over the same ten items, three runs each. Line chart of Items correct, out of ten against Run: A: no examples (1, 6) (2, 7) (3, 6); B: with examples (1, 8) (2, 8) (3, 9). The ranges do not overlap and the gap is large relative to the wobble, so the examples earned their space. Had B scored 7, 8, 7 against A’s 7, 6, 8, the lines would have crossed and the examples would have bought nothing but noise."}},"blocks":[{"t":"h2","id":"nobody-checks-and-the-examples-cost-real-money","runs":[{"text":"Nobody checks, and the examples cost real money"}]},{"t":"p","runs":[{"text":"You added five examples. The output feels better. Did it get better?"}]},{"t":"p","runs":[{"text":"You cannot tell by reading one output, for two reasons you already know. Sampling means two runs of the same prompt differ, so a good result may be a good draw. And you wrote the examples, so you are inclined to see their influence."}]},{"t":"p","runs":[{"text":"The test is not hard. It takes twenty minutes once, and it is the difference between prompting as craft and prompting as superstition."}]},{"t":"h2","id":"the-smallest-honest-test","runs":[{"text":"The smallest honest test"}]},{"t":"p","runs":[{"text":"1. Build ten items with known answers.","b":true},{"text":" Real ones, from the archive lesson. For classification, ten messages with the label you would give. For extraction, ten documents with the three fields written out. For writing, ten briefs plus the output you would accept; this last one is harder to score, and we come to it below."}]},{"t":"p","runs":[{"text":"Hold these out. They must not be your examples. An item that appears in the prompt is not a test."}]},{"t":"p","runs":[{"text":"2. Run version A","b":true},{"text":" (no examples) over all ten. "},{"text":"Run version B","b":true},{"text":" (with examples) over the same ten."}]},{"t":"p","runs":[{"text":"3. Do it three times each.","b":true},{"text":" Sampling, again. Thirty runs sounds like a lot; it is ten minutes of pasting, or five lines of script if you are using an API."}]},{"t":"p","runs":[{"text":"4. Count.","b":true},{"text":" How many of ten correct, in each of the three runs, for each version."}]},{"t":"h2","id":"reading-the-result-honestly","runs":[{"text":"Reading the result honestly"}]},{"t":"p","runs":[{"text":"You will get something like: A scored 6, 7, 6. B scored 8, 8, 9."}]},{"t":"p","runs":[{"text":"That is a real difference. Not because of a statistical test (with ten items you have no business running one) but because the ranges do not overlap and the gap is large relative to the wobble."}]},{"t":"p","runs":[{"text":"Now the more common result: A scored 7, 6, 8. B scored 7, 8, 7."}]},{"t":"p","runs":[{"text":"That is nothing. The ranges overlap completely. Your examples cost you four hundred tokens per call and bought you noise. Take them out, or find better ones, probably from the four items that failed in both versions."}]},{"t":"p","runs":[{"text":"Ten items cannot detect a small improvement.","b":true},{"text":" If A gets 7 and B gets 8, one item moved, and one item is well inside the range of chance. Believing a one-item difference is the most common error people make with small test sets, and knowing that you cannot see it is the useful skill. If the difference matters enough to argue about, you need more items, not more confidence."}]},{"t":"h2","id":"scoring-when-the-output-is-prose","runs":[{"text":"Scoring when the output is prose"}]},{"t":"p","runs":[{"text":"Harder, and still doable. Two workable approaches:"}]},{"t":"p","runs":[{"text":"A checklist per item.","b":true},{"text":" Write the three or four things a good output must contain, for each of your ten briefs. Score out of four. This is crude and it is consistent, which is the property that matters when comparing two versions."}]},{"t":"p","runs":[{"text":"Blind pairwise comparison.","b":true},{"text":" Put A and B outputs side by side without labels, shuffle which side each is on, and pick the better one. Do all ten. If you cannot tell which is which more than half the time, there is no difference worth paying for. Hiding the labels from yourself is the entire point; unblinded, you will pick the version you built."}]},{"t":"h2","id":"what-this-habit-is-for","runs":[{"text":"What this habit is for"}]},{"t":"p","runs":[{"text":"Not rigour for its own sake. It settles arguments, including the ones with yourself at eleven at night, and it stops a prompt accumulating decoration that nobody has ever tested."}]},{"t":"p","runs":[{"text":"Almost every long prompt you will inherit from a colleague contains three or four things that were added once, felt right, and have never been checked. Some of them are helping. Some are costing space and narrowing outputs. Twenty minutes tells you which."}]},{"t":"p","runs":[{"text":"There is a second reason to do it, less obvious and probably more valuable: the four items that fail in both versions are the most informative thing you own. They are not an examples problem, so no amount of example-tuning will move them. They are telling you that the task, the material or the model is the constraint, and that is where your next hour should go."}]},{"t":"p","runs":[{"text":"And once you have the ten items, you have them forever. Every future change to that prompt (a new model, a new constraint, a rewrite) gets checked against the same ten in ten minutes. That is the whole idea, and it is coming back in a later module as the thing that makes a prompt maintainable rather than lucky."}]}],"check":{"question":[{"text":"A prompt with examples scores 8 out of 10 on a test set; without them it scores 7. Both were run once. What is the right conclusion?"}],"options":[{"text":[{"text":"The examples help by about 10%, which justifies keeping them."}],"correct":false,"tempting":[{"text":"Converts a one-item difference in a single run into a percentage, which implies a precision the test cannot support."}]},{"text":[{"text":"The examples do not help, since the difference is only one item."}],"correct":false,"tempting":[{"text":"Equally unjustified in the other direction; one run cannot establish absence any more than presence."}]},{"text":[{"text":"The test set is too small to be worth using at all."}],"correct":false,"tempting":[{"text":"Ten items are useful for catching large differences and for regression testing; the problem here is the single run, not the size."}]},{"text":[{"text":"Nothing yet: one run of each cannot separate a real difference from ordinary sampling variation, so run both several times first."}],"correct":true,"tempting":null}],"order":[2,3,1,0],"why":null},"prev":{"slug":"examples-from-your-own-archive","title":"Your best examples already exist","minutes":8},"next":{"slug":"ask-for-a-usable-shape","title":"Ask for the shape you can actually use","minutes":7}}}