Generating a dataset from a teacher

Start from real inputs

The most common mistake in synthetic data generation is synthesising the inputs as well as the outputs. Ask a model for “200 customer support questions” and you get 200 well-formed, grammatical, politely-phrased questions that resemble nothing your users type. Train on those and you have a model tuned for a population that does not exist.

Inputs from traffic, outputs from the teacher. That is the shape. If you have no traffic, use adjacent real text (forum posts, search queries, tickets) before you resort to invented inputs. If you must invent, invent from a taxonomy you derived from real examples, and include the ugly ones: the two-word question, the paragraph with no question mark, the one in the other language.

Generating at volume, cheaply

For 50,000 outputs, per-request API calls are slow and often expensive. Two better routes:

Batch endpoints. Most hosted providers offer an asynchronous batch mode at roughly half price with a turnaround measured in hours. If your generation is not interactive, and it is not, this halves the bill for the cost of waiting.

A local teacher. Rent one large card for a day, serve an open-weight 70B model with vLLM, and generate offline:

from vllm import LLM, SamplingParams
llm = LLM(model="Qwen/Qwen2.5-72B-Instruct", tensor_parallel_size=2)
outs = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=512, n=4))

vLLM’s continuous batching gives throughput that makes this cheap: tens of thousands of completions per GPU-hour for short outputs. And a permissively licensed teacher removes the question about whether you may use its outputs at all.

Temperature, and the diversity question

Generate at temperature 0 and every similar input gets a near-identical answer; your dataset has less variety than its row count suggests, and the student learns a narrow band of phrasing.

Generate at temperature 1.0 and you get variety including the teacher’s bad days.

The usual compromise is 0.7 with n=4 samples per input, then a filter to pick among them. Which brings us to the part that matters.

Filtering is where the quality comes from

Ranked by strength:

1. A verifier. Where correctness is checkable by a program (code that must run, JSON that must validate against a schema, arithmetic you can recompute, SQL you can execute against a test database) keep only outputs that pass. This is the strongest filter available and it costs nothing but CPU. Where you can construct one, do.

keep = [o for o in outputs if schema_validates(o) and no_placeholder(o)]

2. Self-consistency. Sample four answers. If three agree on the substantive claim, keep that one; if all four disagree, drop the item. This works well for anything with a determinate answer and works not at all for open-ended writing.

3. A scoring model. Have a model rate its own or another model’s outputs and keep the top-scoring one. Effective on writing tasks. Be aware of self-preference: a model asked to rank candidates tends to prefer its own, which matters if you are mixing teachers.

4. A human on a sample. Not a filter for 50,000 items, but read 100 of the kept ones. You are looking for a systematic teacher habit that survived the filter: a recurring preamble, a stock hedge, a wrong assumption the teacher makes consistently. You will find at least one in ten minutes, and finding it is worth more than the other three filters.

Strip the teacher’s tics before training

Hosted models have characteristic openings and closings. If you train on unstripped outputs, your model inherits them, and they are exactly the phrases that make a product feel like a thin wrapper.

PREFIXES = ["Certainly! ", "Of course! ", "Sure, ", "I'd be happy to help"]
SUFFIXES = ["Let me know if you need anything else",
            "Is there anything else I can help with"]

Strip these deterministically before writing the training file, then check a sample by hand for the ones your list missed.

Two things to measure afterwards

Diversity. Count distinct 4-grams across the outputs and divide by total 4-grams. If that ratio is low, your dataset is more repetitive than its size suggests and the student will be narrower than you expect.

Coverage. Cluster the inputs you generated from (embeddings and k-means is enough), and check that the clusters roughly match the shape of your real traffic. A dataset that covers 80% of your traffic’s volume but only 40% of its distinct intents will produce a model that is excellent on the head and useless on everything else. That failure arrives in production, where it looks like a quality problem and is actually a sampling one.