Distillation, the case that usually survives

The one argument that keeps working

Every other reason to fine-tune has a cheaper rival. Format has constrained decoding, knowledge has retrieval, style has few-shot examples. Distillation has none: you want the behaviour of an expensive model at the price of a cheap one, and there is no prompt that makes a 3B model cost what a 3B model costs while answering like a 200B one, other than training it.

The shape is simple. Run the large model over inputs drawn from your real traffic. Keep the good outputs. Train a small model on those pairs. Serve the small model.

The arithmetic that makes it worth it

Suppose you handle two million requests a month, each about 1,200 input tokens and 300 output tokens.

• Hosted frontier model, at roughly $3 per million input and $15 per million output tokens: about $7,200 input plus $9,000 output, so on the order of $16,000 a month.
• A 7B model you serve yourself, on a single rented L4 or A10G at roughly $0.60 to $1.00 an hour: about $430 to $720 a month, plus the engineering to keep it up. Batched, one such card handles that volume with room to spare for 300-token replies.

The gap is twenty to thirty times, and it is recurring. Against it you have a one-off cost: generating the training data (say 50,000 teacher outputs, which at the same prices is around $300), the training run itself (single-digit dollars for a LoRA), and the evaluation, which is the part that actually costs you: days of someone’s attention.

That is the honest shape of the deal: a four-figure one-off and a headcount cost, against a five-figure monthly saving. It is a good deal at two million requests and a terrible one at twenty thousand.

Where the ceiling is

A distilled student is bounded by its teacher on the distribution you sampled. Three consequences follow, and each has bitten real teams:

1. The teacher’s mistakes become the student’s convictions. If the teacher is wrong on 6% of your inputs and you do not filter, the student learns those errors as correct answers, and it produces them with less hedging than the teacher did, because it never learned the teacher’s uncertainty: only its words.
2. Off-distribution, the student collapses faster. The teacher degrades gracefully on inputs unlike anything it saw; the student, having been trained narrowly, often produces confident nonsense. Sample your training inputs from real traffic, including the weird tail, not from a synthetic list of nice questions.
3. The student does not inherit reasoning it cannot represent. Distilling a chain-of-thought model into a 0.5B model gives you a model that writes something that looks like reasoning and gets the arithmetic wrong. There is a floor of capability below which imitation is mimicry.

Filtering is the whole craft

The difference between a distillation that works and one that does not is almost entirely the filter between teacher and student.

• Verifier filtering. Where correctness is checkable (code that must compile, JSON that must validate, arithmetic you can recompute, a SQL query you can run) keep only outputs that pass. This is the strongest filter in the field and it costs nothing.
• Rejection sampling. Generate four teacher outputs per input, score them, keep the best one. Roughly quadruples the generation bill and reliably improves the student.
• Human spot-check. Read 100 of the kept pairs yourself. You will find a systematic teacher habit you did not want (a preamble, a hedge, a recurring wrong assumption) and you will find it in ten minutes.

The part people skip

Check the licence. Some model providers’ terms restrict using their outputs to train other models, and some open-weight licences pass obligations down to anything trained on the model’s outputs. The terms differ by provider and have changed more than once; several have relaxed, some have tightened. Read the current terms for the specific model you are calling, and if the answer is commercially important, put it in front of a lawyer rather than a forum post. This is also an area where the underlying law (what rights, if any, attach to model outputs at all) is unsettled and differs between jurisdictions. Nobody can currently tell you the answer with confidence, and anybody who does is telling you their opinion.

The free path

None of this requires a big budget. You can distil from an open-weight teacher you run yourself (a 70B model rented by the hour for a day generates a large dataset) and the licence question becomes much simpler, because permissive open-weight licences such as Apache-2.0 impose no downstream restriction on outputs at all. The student trains on a free Colab or Kaggle T4 if it is small enough, and llama.cpp serves it on a laptop CPU afterwards.