{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"distillation-as-the-real-case","title":"Distillation, the case that usually survives","url":"https://addaly.com/learn/fine-tuning/distillation-as-the-real-case","position":6,"of":84,"minutes":10,"answer":"Distillation is the fine-tuning case with no cheaper rival, but the student is capped by the teacher on the traffic you sampled, so the filter between them (verifiers, rejection sampling.","keyIdea":"Distillation is the fine-tuning case with no cheaper rival, but the student is capped by the teacher on the traffic you sampled, so the filter between them (verifiers, rejection sampling, a hundred pairs read by a person) is where the quality actually comes from."},"body":{"format":"markdown","text":"## The one argument that keeps working\n\nEvery other reason to fine-tune has a cheaper rival. Format has constrained decoding, knowledge has retrieval, style has few-shot examples. Distillation has none: you want the behaviour of an expensive model at the price of a cheap one, and there is no prompt that makes a 3B model cost what a 3B model costs while answering like a 200B one, other than training it.\n\nThe shape is simple. Run the large model over inputs drawn from your real traffic. Keep the good outputs. Train a small model on those pairs. Serve the small model.\n\n## The arithmetic that makes it worth it\n\nSuppose you handle two million requests a month, each about 1,200 input tokens and 300 output tokens.\n\n- **Hosted frontier model**, at roughly $3 per million input and $15 per million output tokens: about $7,200 input plus $9,000 output, so on the order of $16,000 a month.\n- **A 7B model you serve yourself**, on a single rented L4 or A10G at roughly $0.60 to $1.00 an hour: about $430 to $720 a month, plus the engineering to keep it up. Batched, one such card handles that volume with room to spare for 300-token replies.\n\nThe gap is twenty to thirty times, and it is recurring. Against it you have a one-off cost: generating the training data (say 50,000 teacher outputs, which at the same prices is around $300), the training run itself (single-digit dollars for a LoRA), and the evaluation, which is the part that actually costs you: days of someone's attention.\n\nThat is the honest shape of the deal: a four-figure one-off and a headcount cost, against a five-figure monthly saving. It is a good deal at two million requests and a terrible one at twenty thousand.\n\n## Where the ceiling is\n\nA distilled student is bounded by its teacher on the distribution you sampled. Three consequences follow, and each has bitten real teams:\n\n1. **The teacher's mistakes become the student's convictions.** If the teacher is wrong on 6% of your inputs and you do not filter, the student learns those errors as correct answers, and it produces them with less hedging than the teacher did, because it never learned the teacher's uncertainty: only its words.\n2. **Off-distribution, the student collapses faster.** The teacher degrades gracefully on inputs unlike anything it saw; the student, having been trained narrowly, often produces confident nonsense. Sample your training inputs from real traffic, including the weird tail, not from a synthetic list of nice questions.\n3. **The student does not inherit reasoning it cannot represent.** Distilling a chain-of-thought model into a 0.5B model gives you a model that writes something that looks like reasoning and gets the arithmetic wrong. There is a floor of capability below which imitation is mimicry.\n\n## Filtering is the whole craft\n\nThe difference between a distillation that works and one that does not is almost entirely the filter between teacher and student.\n\n- **Verifier filtering.** Where correctness is checkable (code that must compile, JSON that must validate, arithmetic you can recompute, a SQL query you can run) keep only outputs that pass. This is the strongest filter in the field and it costs nothing.\n- **Rejection sampling.** Generate four teacher outputs per input, score them, keep the best one. Roughly quadruples the generation bill and reliably improves the student.\n- **Human spot-check.** Read 100 of the kept pairs yourself. You will find a systematic teacher habit you did not want (a preamble, a hedge, a recurring wrong assumption) and you will find it in ten minutes.\n\n## The part people skip\n\nCheck the licence. Some model providers' terms restrict using their outputs to train other models, and some open-weight licences pass obligations down to anything trained on the model's outputs. The terms differ by provider and have changed more than once; several have relaxed, some have tightened. Read the current terms for the specific model you are calling, and if the answer is commercially important, put it in front of a lawyer rather than a forum post. This is also an area where the underlying law (what rights, if any, attach to model outputs at all) is unsettled and differs between jurisdictions. Nobody can currently tell you the answer with confidence, and anybody who does is telling you their opinion.\n\n## The free path\n\nNone of this requires a big budget. You can distil from an open-weight teacher you run yourself (a 70B model rented by the hour for a day generates a large dataset) and the licence question becomes much simpler, because permissive open-weight licences such as Apache-2.0 impose no downstream restriction on outputs at all. The student trains on a free Colab or Kaggle T4 if it is small enough, and `llama.cpp` serves it on a laptop CPU afterwards."},"check":{"question":"A team distils a frontier model into a 7B student using 50,000 teacher outputs with no filtering. The teacher is correct on 94% of their traffic. The student scores 91% but its wrong answers sound far more certain than the teacher's did. What explains the increased confidence?","options":[{"text":"Fifty thousand examples is too many, and the student memorised the teacher's phrasing rather than generalising from it.","correct":false,"tempting":"Blames dataset size for an effect caused by what the data contains; the same confidence appears with 5,000 unfiltered examples."},{"text":"Smaller models are inherently more confident, because fewer parameters produce sharper probability distributions.","correct":false,"tempting":"States a property of model size that does not hold; sharpness comes from the training objective and data, not from parameter count."},{"text":"The 3-point accuracy drop caused the confidence rise, since accuracy and calibration always move together.","correct":false,"tempting":"Assumes calibration tracks accuracy; a model can lose accuracy and gain or lose confidence independently, which is why they are measured separately."},{"text":"It learned the teacher's wording, but the teacher's internal uncertainty was never in the training signal.","correct":true}]},"prev":{"slug":"classifiers-and-encoders","title":"When the answer is one of six labels","url":"https://addaly.com/learn/fine-tuning/classifiers-and-encoders"},"next":{"slug":"the-cost-of-owning-a-model","title":"The cost of owning a model","url":"https://addaly.com/learn/fine-tuning/the-cost-of-owning-a-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Distillation, the case that usually survives', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/distillation-as-the-real-case (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Distillation, the case that usually survives","courseTitle":"Fine-Tuning, and When Not To","module":{"index":1,"slug":"deciding-whether-to-fine-tune","title":"Deciding whether to fine-tune at all"},"answer":null,"keyIdea":"Distillation is the fine-tuning case with no cheaper rival, but the student is capped by the teacher on the traffic you sampled, so the filter between them (verifiers, rejection sampling, a hundred pairs read by a person) is where the quality actually comes from.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-one-argument-that-keeps-working","runs":[{"text":"The one argument that keeps working"}]},{"t":"p","runs":[{"text":"Every other reason to fine-tune has a cheaper rival. Format has constrained decoding, knowledge has retrieval, style has few-shot examples. Distillation has none: you want the behaviour of an expensive model at the price of a cheap one, and there is no prompt that makes a 3B model cost what a 3B model costs while answering like a 200B one, other than training it."}]},{"t":"p","runs":[{"text":"The shape is simple. Run the large model over inputs drawn from your real traffic. Keep the good outputs. Train a small model on those pairs. Serve the small model."}]},{"t":"h2","id":"the-arithmetic-that-makes-it-worth-it","runs":[{"text":"The arithmetic that makes it worth it"}]},{"t":"p","runs":[{"text":"Suppose you handle two million requests a month, each about 1,200 input tokens and 300 output tokens."}]},{"t":"ul","items":[[{"text":"Hosted frontier model","b":true},{"text":", at roughly $3 per million input and $15 per million output tokens: about $7,200 input plus $9,000 output, so on the order of $16,000 a month."}],[{"text":"A 7B model you serve yourself","b":true},{"text":", on a single rented L4 or A10G at roughly $0.60 to $1.00 an hour: about $430 to $720 a month, plus the engineering to keep it up. Batched, one such card handles that volume with room to spare for 300-token replies."}]]},{"t":"p","runs":[{"text":"The gap is twenty to thirty times, and it is recurring. Against it you have a one-off cost: generating the training data (say 50,000 teacher outputs, which at the same prices is around $300), the training run itself (single-digit dollars for a LoRA), and the evaluation, which is the part that actually costs you: days of someone’s attention."}]},{"t":"p","runs":[{"text":"That is the honest shape of the deal: a four-figure one-off and a headcount cost, against a five-figure monthly saving. It is a good deal at two million requests and a terrible one at twenty thousand."}]},{"t":"h2","id":"where-the-ceiling-is","runs":[{"text":"Where the ceiling is"}]},{"t":"p","runs":[{"text":"A distilled student is bounded by its teacher on the distribution you sampled. Three consequences follow, and each has bitten real teams:"}]},{"t":"ol","start":1,"items":[[{"text":"The teacher’s mistakes become the student’s convictions.","b":true},{"text":" If the teacher is wrong on 6% of your inputs and you do not filter, the student learns those errors as correct answers, and it produces them with less hedging than the teacher did, because it never learned the teacher’s uncertainty: only its words."}],[{"text":"Off-distribution, the student collapses faster.","b":true},{"text":" The teacher degrades gracefully on inputs unlike anything it saw; the student, having been trained narrowly, often produces confident nonsense. Sample your training inputs from real traffic, including the weird tail, not from a synthetic list of nice questions."}],[{"text":"The student does not inherit reasoning it cannot represent.","b":true},{"text":" Distilling a chain-of-thought model into a 0.5B model gives you a model that writes something that looks like reasoning and gets the arithmetic wrong. There is a floor of capability below which imitation is mimicry."}]]},{"t":"h2","id":"filtering-is-the-whole-craft","runs":[{"text":"Filtering is the whole craft"}]},{"t":"p","runs":[{"text":"The difference between a distillation that works and one that does not is almost entirely the filter between teacher and student."}]},{"t":"ul","items":[[{"text":"Verifier filtering.","b":true},{"text":" Where correctness is checkable (code that must compile, JSON that must validate, arithmetic you can recompute, a SQL query you can run) keep only outputs that pass. This is the strongest filter in the field and it costs nothing."}],[{"text":"Rejection sampling.","b":true},{"text":" Generate four teacher outputs per input, score them, keep the best one. Roughly quadruples the generation bill and reliably improves the student."}],[{"text":"Human spot-check.","b":true},{"text":" Read 100 of the kept pairs yourself. You will find a systematic teacher habit you did not want (a preamble, a hedge, a recurring wrong assumption) and you will find it in ten minutes."}]]},{"t":"h2","id":"the-part-people-skip","runs":[{"text":"The part people skip"}]},{"t":"p","runs":[{"text":"Check the licence. Some model providers’ terms restrict using their outputs to train other models, and some open-weight licences pass obligations down to anything trained on the model’s outputs. The terms differ by provider and have changed more than once; several have relaxed, some have tightened. Read the current terms for the specific model you are calling, and if the answer is commercially important, put it in front of a lawyer rather than a forum post. This is also an area where the underlying law (what rights, if any, attach to model outputs at all) is unsettled and differs between jurisdictions. Nobody can currently tell you the answer with confidence, and anybody who does is telling you their opinion."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"None of this requires a big budget. You can distil from an open-weight teacher you run yourself (a 70B model rented by the hour for a day generates a large dataset) and the licence question becomes much simpler, because permissive open-weight licences such as Apache-2.0 impose no downstream restriction on outputs at all. The student trains on a free Colab or Kaggle T4 if it is small enough, and "},{"text":"llama.cpp","code":true},{"text":" serves it on a laptop CPU afterwards."}]}],"check":{"question":[{"text":"A team distils a frontier model into a 7B student using 50,000 teacher outputs with no filtering. The teacher is correct on 94% of their traffic. The student scores 91% but its wrong answers sound far more certain than the teacher’s did. What explains the increased confidence?"}],"options":[{"text":[{"text":"Fifty thousand examples is too many, and the student memorised the teacher’s phrasing rather than generalising from it."}],"correct":false,"tempting":[{"text":"Blames dataset size for an effect caused by what the data contains; the same confidence appears with 5,000 unfiltered examples."}]},{"text":[{"text":"Smaller models are inherently more confident, because fewer parameters produce sharper probability distributions."}],"correct":false,"tempting":[{"text":"States a property of model size that does not hold; sharpness comes from the training objective and data, not from parameter count."}]},{"text":[{"text":"The 3-point accuracy drop caused the confidence rise, since accuracy and calibration always move together."}],"correct":false,"tempting":[{"text":"Assumes calibration tracks accuracy; a model can lose accuracy and gain or lose confidence independently, which is why they are measured separately."}]},{"text":[{"text":"It learned the teacher’s wording, but the teacher’s internal uncertainty was never in the training signal."}],"correct":true,"tempting":null}],"order":[1,3,2,0],"why":null},"prev":{"slug":"classifiers-and-encoders","title":"When the answer is one of six labels","minutes":9},"next":{"slug":"the-cost-of-owning-a-model","title":"The cost of owning a model","minutes":9}}}