Test cases you generate, and what they cannot tell you

Why anybody reaches for this

There are three honest reasons to generate test items rather than collect them: the feature has not shipped and there are no logs; the case you care about is real but rare, so a natural sample contains two of them; or privacy rules forbid copying user text into an evaluation file at all.

There is also a dishonest reason, which is that generating 500 items takes ten minutes and labelling 100 takes two days. That one produces sets that look impressive and measure nothing.

Two kinds of generation, and one is much safer

Templates with slots are deterministic. You write the shapes and enumerate the values.

TEMPLATES = ["cancel my {product} subscription before {date}",
             "{product} ka subscription band karna hai {date} se pehle"]
PRODUCTS  = ["pro", "family", "annual plan"]
DATES     = ["31 March", "31/03", "next Tuesday", "aaj raat"]

cases = [t.format(product=p, date=d)
         for t in TEMPLATES for p in PRODUCTS for d in DATES]

You know precisely what varies, so a failure localises: it fails on 31/03 and not on 31 March, which is a date-parsing bug you can hand to somebody. Templates are boring and they are the most useful synthetic data most teams ever make. Free tools are enough: Python’s itertools and the Faker package generate names, addresses and phone numbers in dozens of locales.

Model-generated variation gives you fluency and surprise: “rewrite this request forty ways, as an angry customer, a terse one, one typing on a phone with autocorrect”. It reads like real traffic. It is also the risky kind, for a specific mechanical reason.

The blind-spot mechanism

A language model generates text from its own high-probability region. Ask a model to invent hard cases for a system built on that same model family, and it produces the difficulties it can already represent, which are disproportionately the ones it handles. The failures you most need are the inputs the model would not think to write, because not thinking to write them and not handling them come from the same gap in the model.

You do not have to take this on faith. Measure it. Run the system over 100 real items and 100 generated ones, both labelled the same way.

• If the generated set’s pass rate is within a few points of the real one, it is a usable stand-in for that slice.
• If it is fifteen points higher (the common result) your synthetic set is easier than reality and cannot carry a headline number.

Record that gap in the report. It is one number and it converts an argument into a fact.

What generated items are good for

• Structural coverage. Empty input, 20,000 characters, emoji-only, mixed scripts, right-to-left text, code-switching between Hindi and English mid-sentence, a CSV pasted into a chat box. Real traffic contains all of these and your sample of 300 contains none of them.
• Controlled variation. The same request with the amount changed from 100 to 100,000 to -100, to see where the behaviour breaks.
• Adversarial seeds. A starting list of injection and jailbreak attempts, which you then extend by hand. The safety lesson later in the course treats this properly.
• Privacy-safe stand-ins. Same shape, invented content, so the file can live somewhere with fewer restrictions.

What they cannot do

Synthetic items answer “can it”, never “how often”. Frequency is a property of your users, and no generator has access to your users. This one distinction settles most disputes about whether a synthetic set is acceptable:

• “Can the system handle Devanagari input?”: a generated set answers this.
• “What fraction of our failures come from Devanagari input?”: only sampled traffic answers this.

So keep two files. A capability suite of generated and hand-written items, which you run as a pass or fail gate and which is allowed to be unrepresentative on purpose. A traffic set of sampled real items, which produces the number you quote to anybody outside the team. Tag every item with its source field, and make the harness refuse to average across sources.

The failure to watch for

Generated sets rot in a particular way: somebody adds items whenever a model does something amusing, the file grows to 4,000 items, nobody has read most of them, and a third of them are near-duplicates of each other. Then a change that fixes one template appears to fix eighty items and the score jumps six points for no reason.

Deduplicate on normalised text, cap the number of variants per template, and review the file the way you would review code. A test file nobody reads is not a test.