{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"distillation-datasets-in-practice","title":"Generating a dataset from a teacher","url":"https://addaly.com/learn/fine-tuning/distillation-datasets-in-practice","position":33,"of":84,"minutes":10,"answer":"Draw the inputs from real traffic and only the outputs from the teacher, sample several completions and filter them with a verifier wherever correctness is checkable.","keyIdea":"Draw the inputs from real traffic and only the outputs from the teacher, sample several completions and filter them with a verifier wherever correctness is checkable, then strip the teacher's stock phrases and check that your generated set covers your traffic's distinct intents rather than just its volume."},"body":{"format":"markdown","text":"## Start from real inputs\n\nThe most common mistake in synthetic data generation is synthesising the inputs as well as the outputs. Ask a model for \"200 customer support questions\" and you get 200 well-formed, grammatical, politely-phrased questions that resemble nothing your users type. Train on those and you have a model tuned for a population that does not exist.\n\n**Inputs from traffic, outputs from the teacher.** That is the shape. If you have no traffic, use adjacent real text (forum posts, search queries, tickets) before you resort to invented inputs. If you must invent, invent from a taxonomy you derived from real examples, and include the ugly ones: the two-word question, the paragraph with no question mark, the one in the other language.\n\n## Generating at volume, cheaply\n\nFor 50,000 outputs, per-request API calls are slow and often expensive. Two better routes:\n\n**Batch endpoints.** Most hosted providers offer an asynchronous batch mode at roughly half price with a turnaround measured in hours. If your generation is not interactive, and it is not, this halves the bill for the cost of waiting.\n\n**A local teacher.** Rent one large card for a day, serve an open-weight 70B model with vLLM, and generate offline:\n\n```python\nfrom vllm import LLM, SamplingParams\nllm = LLM(model=\"Qwen/Qwen2.5-72B-Instruct\", tensor_parallel_size=2)\nouts = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=512, n=4))\n```\n\nvLLM's continuous batching gives throughput that makes this cheap: tens of thousands of completions per GPU-hour for short outputs. And a permissively licensed teacher removes the question about whether you may use its outputs at all.\n\n## Temperature, and the diversity question\n\nGenerate at temperature 0 and every similar input gets a near-identical answer; your dataset has less variety than its row count suggests, and the student learns a narrow band of phrasing.\n\nGenerate at temperature 1.0 and you get variety including the teacher's bad days.\n\nThe usual compromise is 0.7 with `n=4` samples per input, then a filter to pick among them. Which brings us to the part that matters.\n\n## Filtering is where the quality comes from\n\nRanked by strength:\n\n**1. A verifier.** Where correctness is checkable by a program (code that must run, JSON that must validate against a schema, arithmetic you can recompute, SQL you can execute against a test database) keep only outputs that pass. This is the strongest filter available and it costs nothing but CPU. Where you can construct one, do.\n\n```python\nkeep = [o for o in outputs if schema_validates(o) and no_placeholder(o)]\n```\n\n**2. Self-consistency.** Sample four answers. If three agree on the substantive claim, keep that one; if all four disagree, drop the item. This works well for anything with a determinate answer and works not at all for open-ended writing.\n\n**3. A scoring model.** Have a model rate its own or another model's outputs and keep the top-scoring one. Effective on writing tasks. Be aware of self-preference: a model asked to rank candidates tends to prefer its own, which matters if you are mixing teachers.\n\n**4. A human on a sample.** Not a filter for 50,000 items, but read 100 of the *kept* ones. You are looking for a systematic teacher habit that survived the filter: a recurring preamble, a stock hedge, a wrong assumption the teacher makes consistently. You will find at least one in ten minutes, and finding it is worth more than the other three filters.\n\n## Strip the teacher's tics before training\n\nHosted models have characteristic openings and closings. If you train on unstripped outputs, your model inherits them, and they are exactly the phrases that make a product feel like a thin wrapper.\n\n```python\nPREFIXES = [\"Certainly! \", \"Of course! \", \"Sure, \", \"I'd be happy to help\"]\nSUFFIXES = [\"Let me know if you need anything else\",\n            \"Is there anything else I can help with\"]\n```\n\nStrip these deterministically before writing the training file, then check a sample by hand for the ones your list missed.\n\n## Two things to measure afterwards\n\n**Diversity.** Count distinct 4-grams across the outputs and divide by total 4-grams. If that ratio is low, your dataset is more repetitive than its size suggests and the student will be narrower than you expect.\n\n**Coverage.** Cluster the *inputs* you generated from (embeddings and k-means is enough), and check that the clusters roughly match the shape of your real traffic. A dataset that covers 80% of your traffic's volume but only 40% of its distinct intents will produce a model that is excellent on the head and useless on everything else. That failure arrives in production, where it looks like a quality problem and is actually a sampling one."},"check":{"question":"A team generates 50,000 training examples by asking a teacher model to invent both the customer questions and the answers. The student performs well in evaluation on the same synthetic distribution and poorly in production. What is the mechanism?","options":[{"text":"The synthetic inputs are well-formed in a way real user text is not, so production is a different distribution.","correct":true},{"text":"Synthetic data always contains subtle factual errors that compound during training, degrading the student below the teacher.","correct":false,"tempting":"Blames output quality when the described failure is specifically distributional, and would persist even with a perfectly accurate teacher."},{"text":"The student memorised 50,000 examples, and memorisation does not transfer to new inputs.","correct":false,"tempting":"Invokes memorisation, which would show as poor performance on held-out synthetic items too, not just on production traffic."},{"text":"Teacher-generated answers are longer than human ones, and production users abandon long answers before reading them.","correct":false,"tempting":"Shifts to a user-behaviour explanation that would not show up as the model performing poorly on the task."}]},"prev":{"slug":"writing-and-rewriting-outputs","title":"Writing the answers, and keeping them consistent","url":"https://addaly.com/learn/fine-tuning/writing-and-rewriting-outputs"},"next":{"slug":"data-mixtures-and-replay","title":"Mixtures and replay","url":"https://addaly.com/learn/fine-tuning/data-mixtures-and-replay"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Generating a dataset from a teacher', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/distillation-datasets-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Generating a dataset from a teacher","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Draw the inputs from real traffic and only the outputs from the teacher, sample several completions and filter them with a verifier wherever correctness is checkable, then strip the teacher’s stock phrases and check that your generated set covers your traffic’s distinct intents rather than just its volume.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"start-from-real-inputs","runs":[{"text":"Start from real inputs"}]},{"t":"p","runs":[{"text":"The most common mistake in synthetic data generation is synthesising the inputs as well as the outputs. Ask a model for “200 customer support questions” and you get 200 well-formed, grammatical, politely-phrased questions that resemble nothing your users type. Train on those and you have a model tuned for a population that does not exist."}]},{"t":"p","runs":[{"text":"Inputs from traffic, outputs from the teacher.","b":true},{"text":" That is the shape. If you have no traffic, use adjacent real text (forum posts, search queries, tickets) before you resort to invented inputs. If you must invent, invent from a taxonomy you derived from real examples, and include the ugly ones: the two-word question, the paragraph with no question mark, the one in the other language."}]},{"t":"h2","id":"generating-at-volume-cheaply","runs":[{"text":"Generating at volume, cheaply"}]},{"t":"p","runs":[{"text":"For 50,000 outputs, per-request API calls are slow and often expensive. Two better routes:"}]},{"t":"p","runs":[{"text":"Batch endpoints.","b":true},{"text":" Most hosted providers offer an asynchronous batch mode at roughly half price with a turnaround measured in hours. If your generation is not interactive, and it is not, this halves the bill for the cost of waiting."}]},{"t":"p","runs":[{"text":"A local teacher.","b":true},{"text":" Rent one large card for a day, serve an open-weight 70B model with vLLM, and generate offline:"}]},{"t":"code","lang":"python","label":"Python","text":"from vllm import LLM, SamplingParams\nllm = LLM(model=\"Qwen/Qwen2.5-72B-Instruct\", tensor_parallel_size=2)\nouts = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=512, n=4))"},{"t":"p","runs":[{"text":"vLLM’s continuous batching gives throughput that makes this cheap: tens of thousands of completions per GPU-hour for short outputs. And a permissively licensed teacher removes the question about whether you may use its outputs at all."}]},{"t":"h2","id":"temperature-and-the-diversity-question","runs":[{"text":"Temperature, and the diversity question"}]},{"t":"p","runs":[{"text":"Generate at temperature 0 and every similar input gets a near-identical answer; your dataset has less variety than its row count suggests, and the student learns a narrow band of phrasing."}]},{"t":"p","runs":[{"text":"Generate at temperature 1.0 and you get variety including the teacher’s bad days."}]},{"t":"p","runs":[{"text":"The usual compromise is 0.7 with "},{"text":"n=4","code":true},{"text":" samples per input, then a filter to pick among them. Which brings us to the part that matters."}]},{"t":"h2","id":"filtering-is-where-the-quality-comes-from","runs":[{"text":"Filtering is where the quality comes from"}]},{"t":"p","runs":[{"text":"Ranked by strength:"}]},{"t":"p","runs":[{"text":"1. A verifier.","b":true},{"text":" Where correctness is checkable by a program (code that must run, JSON that must validate against a schema, arithmetic you can recompute, SQL you can execute against a test database) keep only outputs that pass. This is the strongest filter available and it costs nothing but CPU. Where you can construct one, do."}]},{"t":"code","lang":"python","label":"Python","text":"keep = [o for o in outputs if schema_validates(o) and no_placeholder(o)]"},{"t":"p","runs":[{"text":"2. Self-consistency.","b":true},{"text":" Sample four answers. If three agree on the substantive claim, keep that one; if all four disagree, drop the item. This works well for anything with a determinate answer and works not at all for open-ended writing."}]},{"t":"p","runs":[{"text":"3. A scoring model.","b":true},{"text":" Have a model rate its own or another model’s outputs and keep the top-scoring one. Effective on writing tasks. Be aware of self-preference: a model asked to rank candidates tends to prefer its own, which matters if you are mixing teachers."}]},{"t":"p","runs":[{"text":"4. A human on a sample.","b":true},{"text":" Not a filter for 50,000 items, but read 100 of the "},{"text":"kept","i":true},{"text":" ones. You are looking for a systematic teacher habit that survived the filter: a recurring preamble, a stock hedge, a wrong assumption the teacher makes consistently. You will find at least one in ten minutes, and finding it is worth more than the other three filters."}]},{"t":"h2","id":"strip-the-teacher-s-tics-before-training","runs":[{"text":"Strip the teacher’s tics before training"}]},{"t":"p","runs":[{"text":"Hosted models have characteristic openings and closings. If you train on unstripped outputs, your model inherits them, and they are exactly the phrases that make a product feel like a thin wrapper."}]},{"t":"code","lang":"python","label":"Python","text":"PREFIXES = [\"Certainly! \", \"Of course! \", \"Sure, \", \"I'd be happy to help\"]\nSUFFIXES = [\"Let me know if you need anything else\",\n            \"Is there anything else I can help with\"]"},{"t":"p","runs":[{"text":"Strip these deterministically before writing the training file, then check a sample by hand for the ones your list missed."}]},{"t":"h2","id":"two-things-to-measure-afterwards","runs":[{"text":"Two things to measure afterwards"}]},{"t":"p","runs":[{"text":"Diversity.","b":true},{"text":" Count distinct 4-grams across the outputs and divide by total 4-grams. If that ratio is low, your dataset is more repetitive than its size suggests and the student will be narrower than you expect."}]},{"t":"p","runs":[{"text":"Coverage.","b":true},{"text":" Cluster the "},{"text":"inputs","i":true},{"text":" you generated from (embeddings and k-means is enough), and check that the clusters roughly match the shape of your real traffic. A dataset that covers 80% of your traffic’s volume but only 40% of its distinct intents will produce a model that is excellent on the head and useless on everything else. That failure arrives in production, where it looks like a quality problem and is actually a sampling one."}]}],"check":{"question":[{"text":"A team generates 50,000 training examples by asking a teacher model to invent both the customer questions and the answers. The student performs well in evaluation on the same synthetic distribution and poorly in production. What is the mechanism?"}],"options":[{"text":[{"text":"The synthetic inputs are well-formed in a way real user text is not, so production is a different distribution."}],"correct":true,"tempting":null},{"text":[{"text":"Synthetic data always contains subtle factual errors that compound during training, degrading the student below the teacher."}],"correct":false,"tempting":[{"text":"Blames output quality when the described failure is specifically distributional, and would persist even with a perfectly accurate teacher."}]},{"text":[{"text":"The student memorised 50,000 examples, and memorisation does not transfer to new inputs."}],"correct":false,"tempting":[{"text":"Invokes memorisation, which would show as poor performance on held-out synthetic items too, not just on production traffic."}]},{"text":[{"text":"Teacher-generated answers are longer than human ones, and production users abandon long answers before reading them."}],"correct":false,"tempting":[{"text":"Shifts to a user-behaviour explanation that would not show up as the model performing poorly on the task."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"writing-and-rewriting-outputs","title":"Writing the answers, and keeping them consistent","minutes":9},"next":{"slug":"data-mixtures-and-replay","title":"Mixtures and replay","minutes":9}}}