{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"distilling-a-bigger-model","title":"Distilling a larger model into a smaller one","url":"https://addaly.com/learn/open-models/distilling-a-bigger-model","position":57,"of":71,"minutes":9,"answer":"Response distillation is supervised fine-tuning on a teacher's outputs, so the student inherits the teacher's errors and its style, cannot exceed it.","keyIdea":"Response distillation is supervised fine-tuning on a teacher's outputs, so the student inherits the teacher's errors and its style, cannot exceed it, and must be measured against ground truth rather than against agreement with the teacher."},"body":{"format":"markdown","text":"## The idea, and the two versions of it\n\nYou have a large model that does the task well and is too expensive to serve. Distillation trains a small model to imitate it.\n\n**Response distillation** is what almost everybody does. Run a set of prompts through the teacher, keep the outputs, fine-tune the student on those pairs. It is ordinary supervised fine-tuning where the labels came from a model instead of a person.\n\n**Logit distillation** is stronger and rarer. Instead of the teacher's chosen token, you train against the teacher's full probability distribution at each step, minimising the divergence between student and teacher. The student learns not just what the teacher said but how uncertain it was, which carries much more information per example. It requires the same tokenizer for both models and access to the teacher's logits, which rules out any closed API and most convenient setups.\n\nFor most people, response distillation with a good filter on the outputs is the practical choice.\n\n## The legal part, which is not optional\n\n**Closed commercial APIs.** Terms of service typically prohibit using outputs to train competing models. A great many popular community fine-tunes did exactly this, which is one reason their provenance sections are empty. Read the terms of the specific service; do not assume, and do not assume that because others did it, it is fine.\n\n**Open-weights teachers.** Generally permitted, subject to the base licence. Apache 2.0 and MIT teachers impose nothing. Llama's terms permit training on outputs while requiring the resulting model's name to begin with Llama, plus the attribution obligations. Some licences restrict it outright. This is a five-minute check with a real answer, unlike most of the copyright material in this course.\n\n**Your own outputs.** If the teacher is a model you are entitled to use commercially, distilling it is a straightforwardly clean route, and it is why open-weight teachers are worth the small quality gap.\n\n## A recipe that works\n\n1. **Collect 5,000 to 50,000 real prompts.** From logs, ideally. Coverage matters more than count.\n2. **Generate with the teacher at low temperature**, with the prompt you would actually ship.\n3. **Filter.** This step is what separates a good distillation from a mediocre one. Drop outputs that fail schema validation, that are empty or truncated, that are refusals when a refusal was wrong, or that a cheap check finds inconsistent with the input. Expect to discard 10 to 30%.\n4. **Have a person read 100 of them.** You are about to train on this; if 15% are wrong, your student learns to be wrong 15% of the time with total confidence.\n5. **Fine-tune the student** with the previous lessons' settings.\n6. **Evaluate against ground truth, not against the teacher.** Agreement with the teacher measures imitation. Only your labelled held-out set measures whether the student is any good.\n\n## The ceiling, and the inheritance\n\nThe student will not exceed the teacher on the distilled task, and normally lands somewhere below it. What it gains is cost: a 3B student serving at a fraction of a 70B's price, running where the teacher cannot.\n\nIt also inherits the teacher's **errors** and its **style**, and the second one is worth watching. Distilled models pick up the teacher's structure (the headings, the hedging, the closing summary) which makes them read as more capable than they are. Somebody evaluating by skimming five outputs will overrate a distilled model, which is exactly the failure the earlier vetting lesson described from the other side.\n\n## Reasoning distillation, which is a special case\n\nTraining a small model on a reasoning model's chains of thought produces a student that reasons in the same style. The well-known distills of DeepSeek-R1 into Qwen and Llama bases are the public examples, and they are better at multi-step problems than the bases they came from.\n\nTwo honest caveats. The gain is largest on problems resembling the distilled traces, mostly mathematics and code. And the student produces long chains, so it inherits the teacher's token cost per answer even though it is a smaller model, which removes part of the reason you wanted a smaller model."},"check":{"question":"A distilled 3B student agrees with its 70B teacher on 94% of held-out prompts. What does that number establish?","options":[{"text":"The student is 94% as capable as the teacher, so the remaining gap is small.","correct":false,"tempting":"Agreement is not a capability ratio, and disagreements concentrate on the hard cases where capability differences show."},{"text":"It measures imitation only; if the teacher was wrong on 12% of these prompts, the student is confidently wrong on most of those too, and only a labelled set reveals it.","correct":true},{"text":"94% agreement indicates the distillation dataset was too small, since a larger one would push agreement higher.","correct":false,"tempting":"Higher agreement would mean closer imitation, which is not the same as better task performance."},{"text":"The 6% disagreement represents the student's improvements over the teacher.","correct":false,"tempting":"A student trained to imitate has no mechanism for exceeding its teacher on the distilled behaviour."}]},"prev":{"slug":"preference-tuning","title":"Teaching it which answer is better","url":"https://addaly.com/learn/open-models/preference-tuning"},"next":{"slug":"catastrophic-forgetting","title":"Measuring what you broke","url":"https://addaly.com/learn/open-models/catastrophic-forgetting"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Distilling a larger model into a smaller one', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/distilling-a-bigger-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Distilling a larger model into a smaller one","courseTitle":"The Open Model Ecosystem","module":{"index":6,"slug":"adapting-a-model","title":"Adapting a model to your work"},"answer":null,"keyIdea":"Response distillation is supervised fine-tuning on a teacher’s outputs, so the student inherits the teacher’s errors and its style, cannot exceed it, and must be measured against ground truth rather than against agreement with the teacher.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-idea-and-the-two-versions-of-it","runs":[{"text":"The idea, and the two versions of it"}]},{"t":"p","runs":[{"text":"You have a large model that does the task well and is too expensive to serve. Distillation trains a small model to imitate it."}]},{"t":"p","runs":[{"text":"Response distillation","b":true},{"text":" is what almost everybody does. Run a set of prompts through the teacher, keep the outputs, fine-tune the student on those pairs. It is ordinary supervised fine-tuning where the labels came from a model instead of a person."}]},{"t":"p","runs":[{"text":"Logit distillation","b":true},{"text":" is stronger and rarer. Instead of the teacher’s chosen token, you train against the teacher’s full probability distribution at each step, minimising the divergence between student and teacher. The student learns not just what the teacher said but how uncertain it was, which carries much more information per example. It requires the same tokenizer for both models and access to the teacher’s logits, which rules out any closed API and most convenient setups."}]},{"t":"p","runs":[{"text":"For most people, response distillation with a good filter on the outputs is the practical choice."}]},{"t":"h2","id":"the-legal-part-which-is-not-optional","runs":[{"text":"The legal part, which is not optional"}]},{"t":"p","runs":[{"text":"Closed commercial APIs.","b":true},{"text":" Terms of service typically prohibit using outputs to train competing models. A great many popular community fine-tunes did exactly this, which is one reason their provenance sections are empty. Read the terms of the specific service; do not assume, and do not assume that because others did it, it is fine."}]},{"t":"p","runs":[{"text":"Open-weights teachers.","b":true},{"text":" Generally permitted, subject to the base licence. Apache 2.0 and MIT teachers impose nothing. Llama’s terms permit training on outputs while requiring the resulting model’s name to begin with Llama, plus the attribution obligations. Some licences restrict it outright. This is a five-minute check with a real answer, unlike most of the copyright material in this course."}]},{"t":"p","runs":[{"text":"Your own outputs.","b":true},{"text":" If the teacher is a model you are entitled to use commercially, distilling it is a straightforwardly clean route, and it is why open-weight teachers are worth the small quality gap."}]},{"t":"h2","id":"a-recipe-that-works","runs":[{"text":"A recipe that works"}]},{"t":"ol","start":1,"items":[[{"text":"Collect 5,000 to 50,000 real prompts.","b":true},{"text":" From logs, ideally. Coverage matters more than count."}],[{"text":"Generate with the teacher at low temperature","b":true},{"text":", with the prompt you would actually ship."}],[{"text":"Filter.","b":true},{"text":" This step is what separates a good distillation from a mediocre one. Drop outputs that fail schema validation, that are empty or truncated, that are refusals when a refusal was wrong, or that a cheap check finds inconsistent with the input. Expect to discard 10 to 30%."}],[{"text":"Have a person read 100 of them.","b":true},{"text":" You are about to train on this; if 15% are wrong, your student learns to be wrong 15% of the time with total confidence."}],[{"text":"Fine-tune the student","b":true},{"text":" with the previous lessons’ settings."}],[{"text":"Evaluate against ground truth, not against the teacher.","b":true},{"text":" Agreement with the teacher measures imitation. Only your labelled held-out set measures whether the student is any good."}]]},{"t":"h2","id":"the-ceiling-and-the-inheritance","runs":[{"text":"The ceiling, and the inheritance"}]},{"t":"p","runs":[{"text":"The student will not exceed the teacher on the distilled task, and normally lands somewhere below it. What it gains is cost: a 3B student serving at a fraction of a 70B’s price, running where the teacher cannot."}]},{"t":"p","runs":[{"text":"It also inherits the teacher’s "},{"text":"errors","b":true},{"text":" and its "},{"text":"style","b":true},{"text":", and the second one is worth watching. Distilled models pick up the teacher’s structure (the headings, the hedging, the closing summary) which makes them read as more capable than they are. Somebody evaluating by skimming five outputs will overrate a distilled model, which is exactly the failure the earlier vetting lesson described from the other side."}]},{"t":"h2","id":"reasoning-distillation-which-is-a-special-case","runs":[{"text":"Reasoning distillation, which is a special case"}]},{"t":"p","runs":[{"text":"Training a small model on a reasoning model’s chains of thought produces a student that reasons in the same style. The well-known distills of DeepSeek-R1 into Qwen and Llama bases are the public examples, and they are better at multi-step problems than the bases they came from."}]},{"t":"p","runs":[{"text":"Two honest caveats. The gain is largest on problems resembling the distilled traces, mostly mathematics and code. And the student produces long chains, so it inherits the teacher’s token cost per answer even though it is a smaller model, which removes part of the reason you wanted a smaller model."}]}],"check":{"question":[{"text":"A distilled 3B student agrees with its 70B teacher on 94% of held-out prompts. What does that number establish?"}],"options":[{"text":[{"text":"The student is 94% as capable as the teacher, so the remaining gap is small."}],"correct":false,"tempting":[{"text":"Agreement is not a capability ratio, and disagreements concentrate on the hard cases where capability differences show."}]},{"text":[{"text":"It measures imitation only; if the teacher was wrong on 12% of these prompts, the student is confidently wrong on most of those too, and only a labelled set reveals it."}],"correct":true,"tempting":null},{"text":[{"text":"94% agreement indicates the distillation dataset was too small, since a larger one would push agreement higher."}],"correct":false,"tempting":[{"text":"Higher agreement would mean closer imitation, which is not the same as better task performance."}]},{"text":[{"text":"The 6% disagreement represents the student’s improvements over the teacher."}],"correct":false,"tempting":[{"text":"A student trained to imitate has no mechanism for exceeding its teacher on the distilled behaviour."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"preference-tuning","title":"Teaching it which answer is better","minutes":9},"next":{"slug":"catastrophic-forgetting","title":"Measuring what you broke","minutes":8}}}