{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"distillation-and-small-models","title":"Distillation: how small models got good","url":"https://addaly.com/learn/how-llms-work/distillation-and-small-models","position":66,"of":78,"minutes":9,"answer":"A small model trained to match a large model's full next-token distribution learns from the ranking of every wrong answer as well as the right one.","keyIdea":"A small model trained to match a large model's full next-token distribution learns from the ranking of every wrong answer as well as the right one, which transfers behaviour and reasoning habits well and long-tail knowledge poorly."},"body":{"format":"markdown","text":"## The observation\n\nAn 8-billion-parameter model released in 2025 matches or beats, on most public benchmarks, a 175-billion model from 2023. Module five's scaling-laws lesson gave half the reason: the later model was trained on far more data than the old compute-optimal rule suggested. The other half is that small models are no longer trained only on raw text. They are trained to imitate large ones, and the technique has a name and two forms.\n\n## Soft labels: matching the whole distribution\n\nOrdinary training says: given this context, the next token was \"cat\"; raise the probability of \"cat\". Distillation with soft labels says: given this context, the **teacher** (a larger, better model) assigned 0.70 to \"cat\", 0.20 to \"kitten\", 0.05 to \"dog\" and small amounts to everything else; make your distribution match that one.\n\nThe reason this teaches more per example is visible in the numbers. The one-hot label carries one fact: the answer was \"cat\". The teacher's distribution carries a ranking of every wrong answer too: \"kitten\" was nearly as good, \"dog\" was plausible, \"carburettor\" was not. That ranking is a compressed summary of what the teacher knows about the context, and the student receives it on every single token. The loss is the divergence between the two distributions, and its gradient carries signal across the whole vocabulary rather than on one entry. Small models trained this way reach a given quality with a fraction of the data of one trained on labels alone.\n\nThe catch is access. You need the teacher's logits at every position, which means running the teacher yourself. Labs do this with their own large models (several well-known small open models were trained this way), and anyone with an open teacher and a GPU can do it with a few dozen lines of custom loss.\n\n## Sequence level: learning from the teacher's text\n\nThe second form needs only the teacher's outputs. Generate answers from the large model, then fine-tune the small one on them as ordinary supervised data. This works through an API, requires no logits, and is how a great many small open models were post-trained. It is also the form that has produced disputes, which come at the end.\n\nSequence-level distillation carries less information per example than soft labels (the sampled text discarded the distribution that produced it) and it inherits the teacher's errors verbatim. It compensates with volume, because generating a million examples from an API is cheap.\n\n## What is lost, and what is kept\n\nA small model has fewer parameters and, by the reversal-curse lesson's arithmetic, holds fewer facts. Distillation cannot change that. What it transfers well is **behaviour**: formats, instruction-following, tone, refusal patterns, the shape of a good answer, and, recently, the habit of working through a problem before answering. What it transfers poorly is long-tail knowledge, robustness under the perturbations of module six, and strength in weakly represented languages.\n\nA 2023 study made the gap uncomfortable. Small models fine-tuned on a large model's outputs were rated by humans as nearly as good as the teacher, because they had learned its style, while automated tests showed them factually much weaker. The mimicry fooled the evaluation. Any comparison between a distilled model and its teacher has to test content, not tone, which is a point the evaluation course makes at length.\n\n## Reasoning distillation\n\nIn 2025 the sequence-level method was applied to the long working traces of reasoning models. Small models fine-tuned on hundreds of thousands of such traces gained a large share of the reasoning improvement (they learned to check, backtrack and try a second approach) at sizes that fit on a laptop. The traces teach the *how* rather than the *what*, which is exactly the kind of thing distillation transfers well. The perturbation gap from module six narrows and does not close, for the same reason as before: the facts and the margins are still those of a small model.\n\n## Choosing a small model\n\nTest on your task and take the smallest that clears your bar. Rough expectations, to be confirmed by measurement: one to four billion parameters for classification, extraction and formatting; seven to fourteen billion for a general assistant; larger for knowledge-heavy or many-step work. The cascade lesson that follows is about using two sizes together.\n\n## The dispute, in shape only\n\nMost API providers' terms forbid using outputs to train a competing model. Whether such terms are enforceable, and against whom, differs by country and has not been settled in court. Open-weight licences vary: some permit distillation freely, some restrict it, some require attribution. In 2025 there were public accusations between labs about exactly this. Underneath sits an older unsettled question (whether a model's output is anyone's copyright at all) which module eight's memorisation lesson returns to and which no jurisdiction has fully answered.\n\nThis lesson resolves none of it and gives no legal advice. The practical rule is short: read the licence of the model you learn from, and if you are distilling from a commercial API for a commercial product, ask a lawyer in your own country before you ship.\n\n## The free path\n\nRun an open teacher locally, as this module's fourth lesson showed. Generate a few thousand examples on your task, or capture logits if you want soft labels. Train the student with `trl` or Unsloth on Kaggle's free GPU hours. A working distilled model for a narrow task is an afternoon, and it will run on the phone the fourth lesson described."},"check":{"question":"Why does training a student model on the teacher's full next-token distribution teach more per example than training it on text the teacher generated?","options":[{"text":"The full distribution includes the teacher's hidden reasoning tokens, which sampled text omits.","correct":false,"tempting":"Confuses a distribution over the next token with a sequence of thinking tokens; the distribution is a set of probabilities at one position and contains no hidden text."},{"text":"The relative probabilities of the wrong tokens carry information that a single sampled token discards.","correct":true},{"text":"Soft targets have lower variance than sampled tokens, which lets the student train at a higher learning rate for the same stability.","correct":false,"tempting":"There is a real variance-reduction effect, which makes this sound technical and right, but it is a side benefit; the gain in information per example comes from what the target contains, not from how smooth it is."},{"text":"Sampled text can contain the teacher's mistakes, whereas the distribution is the teacher's true belief before sampling and so is always correct.","correct":false,"tempting":"The distribution contains exactly the same mistakes (a wrong token the teacher would sample is a wrong token it assigned high probability to), so matching it inherits them just as faithfully."}]},"prev":{"slug":"the-price-of-a-token","title":"The price of a token, from both sides","url":"https://addaly.com/learn/how-llms-work/the-price-of-a-token"},"next":{"slug":"cascades-and-routing","title":"Cascades and routers: paying for the hard cases only","url":"https://addaly.com/learn/how-llms-work/cascades-and-routing"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Distillation: how small models got good', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/distillation-and-small-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Distillation: how small models got good","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"A small model trained to match a large model’s full next-token distribution learns from the ranking of every wrong answer as well as the right one, which transfers behaviour and reasoning habits well and long-tail knowledge poorly.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-observation","runs":[{"text":"The observation"}]},{"t":"p","runs":[{"text":"An 8-billion-parameter model released in 2025 matches or beats, on most public benchmarks, a 175-billion model from 2023. Module five’s scaling-laws lesson gave half the reason: the later model was trained on far more data than the old compute-optimal rule suggested. The other half is that small models are no longer trained only on raw text. They are trained to imitate large ones, and the technique has a name and two forms."}]},{"t":"h2","id":"soft-labels-matching-the-whole-distribution","runs":[{"text":"Soft labels: matching the whole distribution"}]},{"t":"p","runs":[{"text":"Ordinary training says: given this context, the next token was “cat”; raise the probability of “cat”. Distillation with soft labels says: given this context, the "},{"text":"teacher","b":true},{"text":" (a larger, better model) assigned 0.70 to “cat”, 0.20 to “kitten”, 0.05 to “dog” and small amounts to everything else; make your distribution match that one."}]},{"t":"p","runs":[{"text":"The reason this teaches more per example is visible in the numbers. The one-hot label carries one fact: the answer was “cat”. The teacher’s distribution carries a ranking of every wrong answer too: “kitten” was nearly as good, “dog” was plausible, “carburettor” was not. That ranking is a compressed summary of what the teacher knows about the context, and the student receives it on every single token. The loss is the divergence between the two distributions, and its gradient carries signal across the whole vocabulary rather than on one entry. Small models trained this way reach a given quality with a fraction of the data of one trained on labels alone."}]},{"t":"p","runs":[{"text":"The catch is access. You need the teacher’s logits at every position, which means running the teacher yourself. Labs do this with their own large models (several well-known small open models were trained this way), and anyone with an open teacher and a GPU can do it with a few dozen lines of custom loss."}]},{"t":"h2","id":"sequence-level-learning-from-the-teacher-s-text","runs":[{"text":"Sequence level: learning from the teacher’s text"}]},{"t":"p","runs":[{"text":"The second form needs only the teacher’s outputs. Generate answers from the large model, then fine-tune the small one on them as ordinary supervised data. This works through an API, requires no logits, and is how a great many small open models were post-trained. It is also the form that has produced disputes, which come at the end."}]},{"t":"p","runs":[{"text":"Sequence-level distillation carries less information per example than soft labels (the sampled text discarded the distribution that produced it) and it inherits the teacher’s errors verbatim. It compensates with volume, because generating a million examples from an API is cheap."}]},{"t":"h2","id":"what-is-lost-and-what-is-kept","runs":[{"text":"What is lost, and what is kept"}]},{"t":"p","runs":[{"text":"A small model has fewer parameters and, by the reversal-curse lesson’s arithmetic, holds fewer facts. Distillation cannot change that. What it transfers well is "},{"text":"behaviour","b":true},{"text":": formats, instruction-following, tone, refusal patterns, the shape of a good answer, and, recently, the habit of working through a problem before answering. What it transfers poorly is long-tail knowledge, robustness under the perturbations of module six, and strength in weakly represented languages."}]},{"t":"p","runs":[{"text":"A 2023 study made the gap uncomfortable. Small models fine-tuned on a large model’s outputs were rated by humans as nearly as good as the teacher, because they had learned its style, while automated tests showed them factually much weaker. The mimicry fooled the evaluation. Any comparison between a distilled model and its teacher has to test content, not tone, which is a point the evaluation course makes at length."}]},{"t":"h2","id":"reasoning-distillation","runs":[{"text":"Reasoning distillation"}]},{"t":"p","runs":[{"text":"In 2025 the sequence-level method was applied to the long working traces of reasoning models. Small models fine-tuned on hundreds of thousands of such traces gained a large share of the reasoning improvement (they learned to check, backtrack and try a second approach) at sizes that fit on a laptop. The traces teach the "},{"text":"how","i":true},{"text":" rather than the "},{"text":"what","i":true},{"text":", which is exactly the kind of thing distillation transfers well. The perturbation gap from module six narrows and does not close, for the same reason as before: the facts and the margins are still those of a small model."}]},{"t":"h2","id":"choosing-a-small-model","runs":[{"text":"Choosing a small model"}]},{"t":"p","runs":[{"text":"Test on your task and take the smallest that clears your bar. Rough expectations, to be confirmed by measurement: one to four billion parameters for classification, extraction and formatting; seven to fourteen billion for a general assistant; larger for knowledge-heavy or many-step work. The cascade lesson that follows is about using two sizes together."}]},{"t":"h2","id":"the-dispute-in-shape-only","runs":[{"text":"The dispute, in shape only"}]},{"t":"p","runs":[{"text":"Most API providers’ terms forbid using outputs to train a competing model. Whether such terms are enforceable, and against whom, differs by country and has not been settled in court. Open-weight licences vary: some permit distillation freely, some restrict it, some require attribution. In 2025 there were public accusations between labs about exactly this. Underneath sits an older unsettled question (whether a model’s output is anyone’s copyright at all) which module eight’s memorisation lesson returns to and which no jurisdiction has fully answered."}]},{"t":"p","runs":[{"text":"This lesson resolves none of it and gives no legal advice. The practical rule is short: read the licence of the model you learn from, and if you are distilling from a commercial API for a commercial product, ask a lawyer in your own country before you ship."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Run an open teacher locally, as this module’s fourth lesson showed. Generate a few thousand examples on your task, or capture logits if you want soft labels. Train the student with "},{"text":"trl","code":true},{"text":" or Unsloth on Kaggle’s free GPU hours. A working distilled model for a narrow task is an afternoon, and it will run on the phone the fourth lesson described."}]}],"check":{"question":[{"text":"Why does training a student model on the teacher’s full next-token distribution teach more per example than training it on text the teacher generated?"}],"options":[{"text":[{"text":"The full distribution includes the teacher’s hidden reasoning tokens, which sampled text omits."}],"correct":false,"tempting":[{"text":"Confuses a distribution over the next token with a sequence of thinking tokens; the distribution is a set of probabilities at one position and contains no hidden text."}]},{"text":[{"text":"The relative probabilities of the wrong tokens carry information that a single sampled token discards."}],"correct":true,"tempting":null},{"text":[{"text":"Soft targets have lower variance than sampled tokens, which lets the student train at a higher learning rate for the same stability."}],"correct":false,"tempting":[{"text":"There is a real variance-reduction effect, which makes this sound technical and right, but it is a side benefit; the gain in information per example comes from what the target contains, not from how smooth it is."}]},{"text":[{"text":"Sampled text can contain the teacher’s mistakes, whereas the distribution is the teacher’s true belief before sampling and so is always correct."}],"correct":false,"tempting":[{"text":"The distribution contains exactly the same mistakes (a wrong token the teacher would sample is a wrong token it assigned high probability to), so matching it inherits them just as faithfully."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"the-price-of-a-token","title":"The price of a token, from both sides","minutes":9},"next":{"slug":"cascades-and-routing","title":"Cascades and routers: paying for the hard cases only","minutes":8}}}