{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"synthetic-test-cases","title":"Test cases you generate, and what they cannot tell you","url":"https://addaly.com/learn/evaluating-ai/synthetic-test-cases","position":7,"of":76,"minutes":8,"answer":"Generated items measure whether a system can handle a shape of input; they cannot tell you how often that shape occurs, so they belong in a capability suite and never in the headline number.","keyIdea":"Generated items measure whether a system can handle a shape of input; they cannot tell you how often that shape occurs, so they belong in a capability suite and never in the headline number."},"body":{"format":"markdown","text":"## Why anybody reaches for this\n\nThere are three honest reasons to generate test items rather than collect them: the feature has not shipped and there are no logs; the case you care about is real but rare, so a natural sample contains two of them; or privacy rules forbid copying user text into an evaluation file at all.\n\nThere is also a dishonest reason, which is that generating 500 items takes ten minutes and labelling 100 takes two days. That one produces sets that look impressive and measure nothing.\n\n## Two kinds of generation, and one is much safer\n\n**Templates with slots** are deterministic. You write the shapes and enumerate the values.\n\n```python\nTEMPLATES = [\"cancel my {product} subscription before {date}\",\n             \"{product} ka subscription band karna hai {date} se pehle\"]\nPRODUCTS  = [\"pro\", \"family\", \"annual plan\"]\nDATES     = [\"31 March\", \"31/03\", \"next Tuesday\", \"aaj raat\"]\n\ncases = [t.format(product=p, date=d)\n         for t in TEMPLATES for p in PRODUCTS for d in DATES]\n```\n\nYou know precisely what varies, so a failure localises: it fails on `31/03` and not on `31 March`, which is a date-parsing bug you can hand to somebody. Templates are boring and they are the most useful synthetic data most teams ever make. Free tools are enough: Python's `itertools` and the `Faker` package generate names, addresses and phone numbers in dozens of locales.\n\n**Model-generated variation** gives you fluency and surprise: \"rewrite this request forty ways, as an angry customer, a terse one, one typing on a phone with autocorrect\". It reads like real traffic. It is also the risky kind, for a specific mechanical reason.\n\n## The blind-spot mechanism\n\nA language model generates text from its own high-probability region. Ask a model to invent hard cases for a system built on that same model family, and it produces the difficulties it can already represent, which are disproportionately the ones it handles. The failures you most need are the inputs the model would not think to write, because not thinking to write them and not handling them come from the same gap in the model.\n\nYou do not have to take this on faith. Measure it. Run the system over 100 real items and 100 generated ones, both labelled the same way.\n\n- If the generated set's pass rate is within a few points of the real one, it is a usable stand-in for that slice.\n- If it is fifteen points higher (the common result) your synthetic set is easier than reality and cannot carry a headline number.\n\nRecord that gap in the report. It is one number and it converts an argument into a fact.\n\n## What generated items are good for\n\n- **Structural coverage.** Empty input, 20,000 characters, emoji-only, mixed scripts, right-to-left text, code-switching between Hindi and English mid-sentence, a CSV pasted into a chat box. Real traffic contains all of these and your sample of 300 contains none of them.\n- **Controlled variation.** The same request with the amount changed from 100 to 100,000 to -100, to see where the behaviour breaks.\n- **Adversarial seeds.** A starting list of injection and jailbreak attempts, which you then extend by hand. The safety lesson later in the course treats this properly.\n- **Privacy-safe stand-ins.** Same shape, invented content, so the file can live somewhere with fewer restrictions.\n\n## What they cannot do\n\nSynthetic items answer \"can it\", never \"how often\". Frequency is a property of your users, and no generator has access to your users. This one distinction settles most disputes about whether a synthetic set is acceptable:\n\n- \"Can the system handle Devanagari input?\": a generated set answers this.\n- \"What fraction of our failures come from Devanagari input?\": only sampled traffic answers this.\n\nSo keep two files. A **capability suite** of generated and hand-written items, which you run as a pass or fail gate and which is allowed to be unrepresentative on purpose. A **traffic set** of sampled real items, which produces the number you quote to anybody outside the team. Tag every item with its `source` field, and make the harness refuse to average across sources.\n\n## The failure to watch for\n\nGenerated sets rot in a particular way: somebody adds items whenever a model does something amusing, the file grows to 4,000 items, nobody has read most of them, and a third of them are near-duplicates of each other. Then a change that fixes one template appears to fix eighty items and the score jumps six points for no reason.\n\nDeduplicate on normalised text, cap the number of variants per template, and review the file the way you would review code. A test file nobody reads is not a test."},"check":{"question":"A team generates 400 test questions with a model, measures 91% correct, and reports it as the system's accuracy. Real traffic later shows 74%. What is the mechanism behind the gap?","options":[{"text":"The generated set was too small, and 4,000 generated items would have converged on the true figure.","correct":false,"tempting":"Treats a distribution mismatch as sampling error; more items from the same generator move the number nowhere, because the generator's distribution is what differs from traffic."},{"text":"Generated questions come from the model's own high-probability region, so they over-represent the phrasings the system handles and carry no information about how often hard inputs occur.","correct":true},{"text":"The generated questions were probably mislabelled, since nobody checked the expected answers by hand.","correct":false,"tempting":"Possible in practice but a different failure; even with perfect labels the generated set would still misstate the traffic mix, which is what produced this 17-point gap."},{"text":"Real traffic contains adversarial users, and the generated set contained none, which accounts for the whole difference.","correct":false,"tempting":"Names one plausible slice as the whole cause; the dominant effect is the easier distribution overall, and abuse is usually a small fraction of live volume."}]},"prev":{"slug":"agreement-arithmetic","title":"How much your labellers actually agree","url":"https://addaly.com/learn/evaluating-ai/agreement-arithmetic"},"next":{"slug":"splits-and-leakage","title":"Two sets, and the ways they get contaminated","url":"https://addaly.com/learn/evaluating-ai/splits-and-leakage"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Test cases you generate, and what they cannot tell you', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/synthetic-test-cases (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Test cases you generate, and what they cannot tell you","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"Generated items measure whether a system can handle a shape of input; they cannot tell you how often that shape occurs, so they belong in a capability suite and never in the headline number.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-anybody-reaches-for-this","runs":[{"text":"Why anybody reaches for this"}]},{"t":"p","runs":[{"text":"There are three honest reasons to generate test items rather than collect them: the feature has not shipped and there are no logs; the case you care about is real but rare, so a natural sample contains two of them; or privacy rules forbid copying user text into an evaluation file at all."}]},{"t":"p","runs":[{"text":"There is also a dishonest reason, which is that generating 500 items takes ten minutes and labelling 100 takes two days. That one produces sets that look impressive and measure nothing."}]},{"t":"h2","id":"two-kinds-of-generation-and-one-is-much-safer","runs":[{"text":"Two kinds of generation, and one is much safer"}]},{"t":"p","runs":[{"text":"Templates with slots","b":true},{"text":" are deterministic. You write the shapes and enumerate the values."}]},{"t":"code","lang":"python","label":"Python","text":"TEMPLATES = [\"cancel my {product} subscription before {date}\",\n             \"{product} ka subscription band karna hai {date} se pehle\"]\nPRODUCTS  = [\"pro\", \"family\", \"annual plan\"]\nDATES     = [\"31 March\", \"31/03\", \"next Tuesday\", \"aaj raat\"]\n\ncases = [t.format(product=p, date=d)\n         for t in TEMPLATES for p in PRODUCTS for d in DATES]"},{"t":"p","runs":[{"text":"You know precisely what varies, so a failure localises: it fails on "},{"text":"31/03","code":true},{"text":" and not on "},{"text":"31 March","code":true},{"text":", which is a date-parsing bug you can hand to somebody. Templates are boring and they are the most useful synthetic data most teams ever make. Free tools are enough: Python’s "},{"text":"itertools","code":true},{"text":" and the "},{"text":"Faker","code":true},{"text":" package generate names, addresses and phone numbers in dozens of locales."}]},{"t":"p","runs":[{"text":"Model-generated variation","b":true},{"text":" gives you fluency and surprise: “rewrite this request forty ways, as an angry customer, a terse one, one typing on a phone with autocorrect”. It reads like real traffic. It is also the risky kind, for a specific mechanical reason."}]},{"t":"h2","id":"the-blind-spot-mechanism","runs":[{"text":"The blind-spot mechanism"}]},{"t":"p","runs":[{"text":"A language model generates text from its own high-probability region. Ask a model to invent hard cases for a system built on that same model family, and it produces the difficulties it can already represent, which are disproportionately the ones it handles. The failures you most need are the inputs the model would not think to write, because not thinking to write them and not handling them come from the same gap in the model."}]},{"t":"p","runs":[{"text":"You do not have to take this on faith. Measure it. Run the system over 100 real items and 100 generated ones, both labelled the same way."}]},{"t":"ul","items":[[{"text":"If the generated set’s pass rate is within a few points of the real one, it is a usable stand-in for that slice."}],[{"text":"If it is fifteen points higher (the common result) your synthetic set is easier than reality and cannot carry a headline number."}]]},{"t":"p","runs":[{"text":"Record that gap in the report. It is one number and it converts an argument into a fact."}]},{"t":"h2","id":"what-generated-items-are-good-for","runs":[{"text":"What generated items are good for"}]},{"t":"ul","items":[[{"text":"Structural coverage.","b":true},{"text":" Empty input, 20,000 characters, emoji-only, mixed scripts, right-to-left text, code-switching between Hindi and English mid-sentence, a CSV pasted into a chat box. Real traffic contains all of these and your sample of 300 contains none of them."}],[{"text":"Controlled variation.","b":true},{"text":" The same request with the amount changed from 100 to 100,000 to -100, to see where the behaviour breaks."}],[{"text":"Adversarial seeds.","b":true},{"text":" A starting list of injection and jailbreak attempts, which you then extend by hand. The safety lesson later in the course treats this properly."}],[{"text":"Privacy-safe stand-ins.","b":true},{"text":" Same shape, invented content, so the file can live somewhere with fewer restrictions."}]]},{"t":"h2","id":"what-they-cannot-do","runs":[{"text":"What they cannot do"}]},{"t":"p","runs":[{"text":"Synthetic items answer “can it”, never “how often”. Frequency is a property of your users, and no generator has access to your users. This one distinction settles most disputes about whether a synthetic set is acceptable:"}]},{"t":"ul","items":[[{"text":"“Can the system handle Devanagari input?”: a generated set answers this."}],[{"text":"“What fraction of our failures come from Devanagari input?”: only sampled traffic answers this."}]]},{"t":"p","runs":[{"text":"So keep two files. A "},{"text":"capability suite","b":true},{"text":" of generated and hand-written items, which you run as a pass or fail gate and which is allowed to be unrepresentative on purpose. A "},{"text":"traffic set","b":true},{"text":" of sampled real items, which produces the number you quote to anybody outside the team. Tag every item with its "},{"text":"source","code":true},{"text":" field, and make the harness refuse to average across sources."}]},{"t":"h2","id":"the-failure-to-watch-for","runs":[{"text":"The failure to watch for"}]},{"t":"p","runs":[{"text":"Generated sets rot in a particular way: somebody adds items whenever a model does something amusing, the file grows to 4,000 items, nobody has read most of them, and a third of them are near-duplicates of each other. Then a change that fixes one template appears to fix eighty items and the score jumps six points for no reason."}]},{"t":"p","runs":[{"text":"Deduplicate on normalised text, cap the number of variants per template, and review the file the way you would review code. A test file nobody reads is not a test."}]}],"check":{"question":[{"text":"A team generates 400 test questions with a model, measures 91% correct, and reports it as the system’s accuracy. Real traffic later shows 74%. What is the mechanism behind the gap?"}],"options":[{"text":[{"text":"The generated set was too small, and 4,000 generated items would have converged on the true figure."}],"correct":false,"tempting":[{"text":"Treats a distribution mismatch as sampling error; more items from the same generator move the number nowhere, because the generator’s distribution is what differs from traffic."}]},{"text":[{"text":"Generated questions come from the model’s own high-probability region, so they over-represent the phrasings the system handles and carry no information about how often hard inputs occur."}],"correct":true,"tempting":null},{"text":[{"text":"The generated questions were probably mislabelled, since nobody checked the expected answers by hand."}],"correct":false,"tempting":[{"text":"Possible in practice but a different failure; even with perfect labels the generated set would still misstate the traffic mix, which is what produced this 17-point gap."}]},{"text":[{"text":"Real traffic contains adversarial users, and the generated set contained none, which accounts for the whole difference."}],"correct":false,"tempting":[{"text":"Names one plausible slice as the whole cause; the dominant effect is the easier distribution overall, and abuse is usually a small fraction of live volume."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"agreement-arithmetic","title":"How much your labellers actually agree","minutes":9},"next":{"slug":"splits-and-leakage","title":"Two sets, and the ways they get contaminated","minutes":9}}}