Distilling a larger model into a smaller one

The idea, and the two versions of it

You have a large model that does the task well and is too expensive to serve. Distillation trains a small model to imitate it.

Response distillation is what almost everybody does. Run a set of prompts through the teacher, keep the outputs, fine-tune the student on those pairs. It is ordinary supervised fine-tuning where the labels came from a model instead of a person.

Logit distillation is stronger and rarer. Instead of the teacher’s chosen token, you train against the teacher’s full probability distribution at each step, minimising the divergence between student and teacher. The student learns not just what the teacher said but how uncertain it was, which carries much more information per example. It requires the same tokenizer for both models and access to the teacher’s logits, which rules out any closed API and most convenient setups.

For most people, response distillation with a good filter on the outputs is the practical choice.

The legal part, which is not optional

Closed commercial APIs. Terms of service typically prohibit using outputs to train competing models. A great many popular community fine-tunes did exactly this, which is one reason their provenance sections are empty. Read the terms of the specific service; do not assume, and do not assume that because others did it, it is fine.

Open-weights teachers. Generally permitted, subject to the base licence. Apache 2.0 and MIT teachers impose nothing. Llama’s terms permit training on outputs while requiring the resulting model’s name to begin with Llama, plus the attribution obligations. Some licences restrict it outright. This is a five-minute check with a real answer, unlike most of the copyright material in this course.

Your own outputs. If the teacher is a model you are entitled to use commercially, distilling it is a straightforwardly clean route, and it is why open-weight teachers are worth the small quality gap.

A recipe that works

1. Collect 5,000 to 50,000 real prompts. From logs, ideally. Coverage matters more than count.
2. Generate with the teacher at low temperature, with the prompt you would actually ship.
3. Filter. This step is what separates a good distillation from a mediocre one. Drop outputs that fail schema validation, that are empty or truncated, that are refusals when a refusal was wrong, or that a cheap check finds inconsistent with the input. Expect to discard 10 to 30%.
4. Have a person read 100 of them. You are about to train on this; if 15% are wrong, your student learns to be wrong 15% of the time with total confidence.
5. Fine-tune the student with the previous lessons’ settings.
6. Evaluate against ground truth, not against the teacher. Agreement with the teacher measures imitation. Only your labelled held-out set measures whether the student is any good.

The ceiling, and the inheritance

The student will not exceed the teacher on the distilled task, and normally lands somewhere below it. What it gains is cost: a 3B student serving at a fraction of a 70B’s price, running where the teacher cannot.

It also inherits the teacher’s errors and its style, and the second one is worth watching. Distilled models pick up the teacher’s structure (the headings, the hedging, the closing summary) which makes them read as more capable than they are. Somebody evaluating by skimming five outputs will overrate a distilled model, which is exactly the failure the earlier vetting lesson described from the other side.

Reasoning distillation, which is a special case

Training a small model on a reasoning model’s chains of thought produces a student that reasons in the same style. The well-known distills of DeepSeek-R1 into Qwen and Llama bases are the public examples, and they are better at multi-step problems than the bases they came from.

Two honest caveats. The gain is largest on problems resembling the distilled traces, mostly mathematics and code. And the student produces long chains, so it inherits the teacher’s token cost per answer even though it is a smaller model, which removes part of the reason you wanted a smaller model.