Distillation: how small models got good

The observation

An 8-billion-parameter model released in 2025 matches or beats, on most public benchmarks, a 175-billion model from 2023. Module five’s scaling-laws lesson gave half the reason: the later model was trained on far more data than the old compute-optimal rule suggested. The other half is that small models are no longer trained only on raw text. They are trained to imitate large ones, and the technique has a name and two forms.

Soft labels: matching the whole distribution

Ordinary training says: given this context, the next token was “cat”; raise the probability of “cat”. Distillation with soft labels says: given this context, the teacher (a larger, better model) assigned 0.70 to “cat”, 0.20 to “kitten”, 0.05 to “dog” and small amounts to everything else; make your distribution match that one.

The reason this teaches more per example is visible in the numbers. The one-hot label carries one fact: the answer was “cat”. The teacher’s distribution carries a ranking of every wrong answer too: “kitten” was nearly as good, “dog” was plausible, “carburettor” was not. That ranking is a compressed summary of what the teacher knows about the context, and the student receives it on every single token. The loss is the divergence between the two distributions, and its gradient carries signal across the whole vocabulary rather than on one entry. Small models trained this way reach a given quality with a fraction of the data of one trained on labels alone.

The catch is access. You need the teacher’s logits at every position, which means running the teacher yourself. Labs do this with their own large models (several well-known small open models were trained this way), and anyone with an open teacher and a GPU can do it with a few dozen lines of custom loss.

Sequence level: learning from the teacher’s text

The second form needs only the teacher’s outputs. Generate answers from the large model, then fine-tune the small one on them as ordinary supervised data. This works through an API, requires no logits, and is how a great many small open models were post-trained. It is also the form that has produced disputes, which come at the end.

Sequence-level distillation carries less information per example than soft labels (the sampled text discarded the distribution that produced it) and it inherits the teacher’s errors verbatim. It compensates with volume, because generating a million examples from an API is cheap.

What is lost, and what is kept

A small model has fewer parameters and, by the reversal-curse lesson’s arithmetic, holds fewer facts. Distillation cannot change that. What it transfers well is behaviour: formats, instruction-following, tone, refusal patterns, the shape of a good answer, and, recently, the habit of working through a problem before answering. What it transfers poorly is long-tail knowledge, robustness under the perturbations of module six, and strength in weakly represented languages.

A 2023 study made the gap uncomfortable. Small models fine-tuned on a large model’s outputs were rated by humans as nearly as good as the teacher, because they had learned its style, while automated tests showed them factually much weaker. The mimicry fooled the evaluation. Any comparison between a distilled model and its teacher has to test content, not tone, which is a point the evaluation course makes at length.

Reasoning distillation

In 2025 the sequence-level method was applied to the long working traces of reasoning models. Small models fine-tuned on hundreds of thousands of such traces gained a large share of the reasoning improvement (they learned to check, backtrack and try a second approach) at sizes that fit on a laptop. The traces teach the how rather than the what, which is exactly the kind of thing distillation transfers well. The perturbation gap from module six narrows and does not close, for the same reason as before: the facts and the margins are still those of a small model.

Choosing a small model

Test on your task and take the smallest that clears your bar. Rough expectations, to be confirmed by measurement: one to four billion parameters for classification, extraction and formatting; seven to fourteen billion for a general assistant; larger for knowledge-heavy or many-step work. The cascade lesson that follows is about using two sizes together.

The dispute, in shape only

Most API providers’ terms forbid using outputs to train a competing model. Whether such terms are enforceable, and against whom, differs by country and has not been settled in court. Open-weight licences vary: some permit distillation freely, some restrict it, some require attribution. In 2025 there were public accusations between labs about exactly this. Underneath sits an older unsettled question (whether a model’s output is anyone’s copyright at all) which module eight’s memorisation lesson returns to and which no jurisdiction has fully answered.

This lesson resolves none of it and gives no legal advice. The practical rule is short: read the licence of the model you learn from, and if you are distilling from a commercial API for a commercial product, ask a lawyer in your own country before you ship.

The free path

Run an open teacher locally, as this module’s fourth lesson showed. Generate a few thousand examples on your task, or capture logits if you want soft labels. Train the student with trl or Unsloth on Kaggle’s free GPU hours. A working distilled model for a narrow task is an afternoon, and it will run on the phone the fourth lesson described.