{"course":{"slug":"ai-for-small-business","title":"AI for a Small Business","url":"https://addaly.com/learn/ai-for-small-business","level":"beginner","track":"work"},"lesson":{"slug":"testing-a-prompt","title":"Ten cases before you trust it","url":"https://addaly.com/learn/ai-for-small-business/testing-a-prompt","position":19,"of":88,"minutes":8,"answer":"Test a prompt against ten real cases (four ordinary, two with missing information, two awkward, one out of range, one hostile) scored pass or fail on behaviour, and re-run the whole set whenever the prompt.","keyIdea":"Test a prompt against ten real cases (four ordinary, two with missing information, two awkward, one out of range, one hostile) scored pass or fail on behaviour, and re-run the whole set whenever the prompt, the facts or the model changes."},"body":{"format":"markdown","text":"## One good answer is not evidence\n\nYou wrote a prompt, tried it on a message, and it produced something excellent. That is encouraging and it is not a reason to start using it on customers.\n\nThe model samples. Run the same prompt on the same input three times and you get three answers of varying quality. A single result tells you the prompt can work, not that it does.\n\nSo before anything goes into daily use, run it against ten cases. Twenty minutes, once.\n\n## Choosing the ten\n\nDo not invent them. Take real items from your own records (the last thirty enquiries, or a spread across the last three months) and pick to a shape:\n\n- **Four ordinary ones.** The bread and butter. If it fails here, nothing else matters.\n- **Two where information is missing.** The customer did not say which size, or asked about an area not on your sheet. You are testing whether it flags the gap or invents a figure. This is the most important pair.\n- **Two awkward ones.** Angry, rambling, three questions at once, or in a mix of two languages.\n- **One outside your range.** Something you do not sell. The right answer is a clean no, not a helpful invention.\n- **One with a trap.** A message containing something like \"my friend said you'd do it for half price\" or an instruction aimed at the assistant. You are checking that it does not simply agree.\n\nKeep these ten in a file. They are now your test set, and you will use them again every time something changes.\n\n## Judging by behaviour, not wording\n\nDo not compare against a model answer you wrote. The output will differ every time and that is not a failure.\n\nScore each case pass or fail against behaviour:\n\n1. Every fact traces to your facts sheet.\n2. Nothing was promised that you did not authorise.\n3. Gaps were flagged rather than filled.\n4. Tone is one you would send.\n5. Format matches what you asked for.\n\nA case passes only if all five hold. This is strict on purpose: a reply with perfect tone and an invented delivery date is a failure, and scoring it as half a pass is how businesses talk themselves into shipping something unsafe.\n\n## Reading the score\n\n**Ten out of ten**: use it, and keep spot-checking weekly.\n\n**Eight or nine**: look at what failed. If the two failures are both the missing-information cases, your grounding instruction is too weak; strengthen it and re-run. If they are scattered, the prompt is probably fine for drafting with a person reading, and not fine for anything automatic.\n\n**Six or seven**: the prompt needs work, and it is usually the facts rather than the wording. Fix and re-run the whole ten, not just the failures. Fixing one case often breaks another, and testing only the failures hides that completely.\n\n**Below six**: this may not be a task for a model. Go back to the process map.\n\n## Re-running the ten\n\nThe test set is worth much more the second time you use it. Re-run when:\n\n- Your vendor changes or updates the model, which they do without asking you.\n- You edit the prompt or the facts sheet.\n- Something odd happens in real use, in which case that case becomes the eleventh item permanently.\n\nThat last habit is the one that compounds. Every real failure you meet gets added to the file, so the same failure can never surprise you twice. After a year you have thirty cases drawn entirely from things that actually went wrong in your business, which is a better test set than any vendor could have given you.\n\n## The proportion\n\nTwenty minutes to build, ten to re-run. For a prompt you will use several hundred times a year, that is a trivial cost and it is the only thing standing between \"it seemed fine when I tried it\" and knowing.\n\n## Somebody else should mark it\n\nThe person who wrote the prompt is the worst judge of its output. They know what it was meant to do, so they read the intention rather than the text, and they forgive near-misses because they can see how the near-miss happened.\n\nHand the ten outputs to somebody who does not know which prompt produced them and does know the business: a partner, a colleague, the person who normally answers the messages. Ask one question: would you send this to a customer, yes or no?\n\nThe scores usually come back lower, and the disagreements are the interesting part. Where the author says pass and the marker says fail, the reason is almost always an assumption the author was supplying from their own head rather than from the facts sheet.\n\n## Keep the file with the prompt\n\nThe ten cases live next to the prompt in the library, with the date of the last run and the score. A prompt without a test set is an opinion. A prompt with one is something you can hand to somebody else, re-check after a model update, and defend to a client who asks how you know your replies are accurate."},"check":{"question":"A prompt scores 7 out of 10. The owner fixes the three failing cases, re-tests those three, gets three passes, and puts it into daily use. What is wrong with this procedure?","options":[{"text":"Three passes on three cases is too small a sample to establish that the fix worked.","correct":false,"tempting":"Right that three is small, but the deeper problem is not sample size on the fixed cases; it is the seven cases that were never re-checked after the change."},{"text":"The cases should have been re-tested several times each, since the model samples and a single pass may not repeat, so a fix confirmed once could easily fail on the next attempt.","correct":false,"tempting":"Identifies real variability and a reasonable refinement, yet it still leaves the seven untested cases unexamined after a prompt that has materially changed."},{"text":"A change made to fix three cases can break others, so the whole ten must be re-run or the new failures stay invisible.","correct":true},{"text":"Seven out of ten was already sufficient for drafting with a person reading, so the fixes were unnecessary.","correct":false,"tempting":"Accepts a score whose failure pattern was never examined, when three scattered failures may include an invented price that no amount of reading reliably catches."}]},"prev":{"slug":"hostile-input","title":"The customer message written to manipulate your assistant","url":"https://addaly.com/learn/ai-for-small-business/hostile-input"},"next":{"slug":"asking-your-own-documents","title":"What actually happens when you upload a file","url":"https://addaly.com/learn/ai-for-small-business/asking-your-own-documents"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ten cases before you trust it', in AI for a Small Business. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-small-business/testing-a-prompt (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ten cases before you trust it","courseTitle":"AI for a Small Business","module":{"index":2,"slug":"asking-properly","title":"Asking properly: prompting for business work"},"answer":null,"keyIdea":"Test a prompt against ten real cases (four ordinary, two with missing information, two awkward, one out of range, one hostile) scored pass or fail on behaviour, and re-run the whole set whenever the prompt, the facts or the model changes.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"one-good-answer-is-not-evidence","runs":[{"text":"One good answer is not evidence"}]},{"t":"p","runs":[{"text":"You wrote a prompt, tried it on a message, and it produced something excellent. That is encouraging and it is not a reason to start using it on customers."}]},{"t":"p","runs":[{"text":"The model samples. Run the same prompt on the same input three times and you get three answers of varying quality. A single result tells you the prompt can work, not that it does."}]},{"t":"p","runs":[{"text":"So before anything goes into daily use, run it against ten cases. Twenty minutes, once."}]},{"t":"h2","id":"choosing-the-ten","runs":[{"text":"Choosing the ten"}]},{"t":"p","runs":[{"text":"Do not invent them. Take real items from your own records (the last thirty enquiries, or a spread across the last three months) and pick to a shape:"}]},{"t":"ul","items":[[{"text":"Four ordinary ones.","b":true},{"text":" The bread and butter. If it fails here, nothing else matters."}],[{"text":"Two where information is missing.","b":true},{"text":" The customer did not say which size, or asked about an area not on your sheet. You are testing whether it flags the gap or invents a figure. This is the most important pair."}],[{"text":"Two awkward ones.","b":true},{"text":" Angry, rambling, three questions at once, or in a mix of two languages."}],[{"text":"One outside your range.","b":true},{"text":" Something you do not sell. The right answer is a clean no, not a helpful invention."}],[{"text":"One with a trap.","b":true},{"text":" A message containing something like “my friend said you’d do it for half price” or an instruction aimed at the assistant. You are checking that it does not simply agree."}]]},{"t":"p","runs":[{"text":"Keep these ten in a file. They are now your test set, and you will use them again every time something changes."}]},{"t":"h2","id":"judging-by-behaviour-not-wording","runs":[{"text":"Judging by behaviour, not wording"}]},{"t":"p","runs":[{"text":"Do not compare against a model answer you wrote. The output will differ every time and that is not a failure."}]},{"t":"p","runs":[{"text":"Score each case pass or fail against behaviour:"}]},{"t":"ol","start":1,"items":[[{"text":"Every fact traces to your facts sheet."}],[{"text":"Nothing was promised that you did not authorise."}],[{"text":"Gaps were flagged rather than filled."}],[{"text":"Tone is one you would send."}],[{"text":"Format matches what you asked for."}]]},{"t":"p","runs":[{"text":"A case passes only if all five hold. This is strict on purpose: a reply with perfect tone and an invented delivery date is a failure, and scoring it as half a pass is how businesses talk themselves into shipping something unsafe."}]},{"t":"h2","id":"reading-the-score","runs":[{"text":"Reading the score"}]},{"t":"p","runs":[{"text":"Ten out of ten","b":true},{"text":": use it, and keep spot-checking weekly."}]},{"t":"p","runs":[{"text":"Eight or nine","b":true},{"text":": look at what failed. If the two failures are both the missing-information cases, your grounding instruction is too weak; strengthen it and re-run. If they are scattered, the prompt is probably fine for drafting with a person reading, and not fine for anything automatic."}]},{"t":"p","runs":[{"text":"Six or seven","b":true},{"text":": the prompt needs work, and it is usually the facts rather than the wording. Fix and re-run the whole ten, not just the failures. Fixing one case often breaks another, and testing only the failures hides that completely."}]},{"t":"p","runs":[{"text":"Below six","b":true},{"text":": this may not be a task for a model. Go back to the process map."}]},{"t":"h2","id":"re-running-the-ten","runs":[{"text":"Re-running the ten"}]},{"t":"p","runs":[{"text":"The test set is worth much more the second time you use it. Re-run when:"}]},{"t":"ul","items":[[{"text":"Your vendor changes or updates the model, which they do without asking you."}],[{"text":"You edit the prompt or the facts sheet."}],[{"text":"Something odd happens in real use, in which case that case becomes the eleventh item permanently."}]]},{"t":"p","runs":[{"text":"That last habit is the one that compounds. Every real failure you meet gets added to the file, so the same failure can never surprise you twice. After a year you have thirty cases drawn entirely from things that actually went wrong in your business, which is a better test set than any vendor could have given you."}]},{"t":"h2","id":"the-proportion","runs":[{"text":"The proportion"}]},{"t":"p","runs":[{"text":"Twenty minutes to build, ten to re-run. For a prompt you will use several hundred times a year, that is a trivial cost and it is the only thing standing between “it seemed fine when I tried it” and knowing."}]},{"t":"h2","id":"somebody-else-should-mark-it","runs":[{"text":"Somebody else should mark it"}]},{"t":"p","runs":[{"text":"The person who wrote the prompt is the worst judge of its output. They know what it was meant to do, so they read the intention rather than the text, and they forgive near-misses because they can see how the near-miss happened."}]},{"t":"p","runs":[{"text":"Hand the ten outputs to somebody who does not know which prompt produced them and does know the business: a partner, a colleague, the person who normally answers the messages. Ask one question: would you send this to a customer, yes or no?"}]},{"t":"p","runs":[{"text":"The scores usually come back lower, and the disagreements are the interesting part. Where the author says pass and the marker says fail, the reason is almost always an assumption the author was supplying from their own head rather than from the facts sheet."}]},{"t":"h2","id":"keep-the-file-with-the-prompt","runs":[{"text":"Keep the file with the prompt"}]},{"t":"p","runs":[{"text":"The ten cases live next to the prompt in the library, with the date of the last run and the score. A prompt without a test set is an opinion. A prompt with one is something you can hand to somebody else, re-check after a model update, and defend to a client who asks how you know your replies are accurate."}]}],"check":{"question":[{"text":"A prompt scores 7 out of 10. The owner fixes the three failing cases, re-tests those three, gets three passes, and puts it into daily use. What is wrong with this procedure?"}],"options":[{"text":[{"text":"Three passes on three cases is too small a sample to establish that the fix worked."}],"correct":false,"tempting":[{"text":"Right that three is small, but the deeper problem is not sample size on the fixed cases; it is the seven cases that were never re-checked after the change."}]},{"text":[{"text":"The cases should have been re-tested several times each, since the model samples and a single pass may not repeat, so a fix confirmed once could easily fail on the next attempt."}],"correct":false,"tempting":[{"text":"Identifies real variability and a reasonable refinement, yet it still leaves the seven untested cases unexamined after a prompt that has materially changed."}]},{"text":[{"text":"A change made to fix three cases can break others, so the whole ten must be re-run or the new failures stay invisible."}],"correct":true,"tempting":null},{"text":[{"text":"Seven out of ten was already sufficient for drafting with a person reading, so the fixes were unnecessary."}],"correct":false,"tempting":[{"text":"Accepts a score whose failure pattern was never examined, when three scattered failures may include an invented price that no amount of reading reliably catches."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"hostile-input","title":"The customer message written to manipulate your assistant","minutes":9},"next":{"slug":"asking-your-own-documents","title":"What actually happens when you upload a file","minutes":9}}}