Judging a model you distilled from the judge

The circularity

You distilled a 7B student from a large teacher. To evaluate the student cheaply, you ask that same teacher which of two answers is better.

The teacher prefers the student.

Of course it does. The student was trained to produce text that resembles the teacher’s, and the teacher’s judgement of quality is substantially a judgement of resemblance to its own output: its phrasing, its structure, its characteristic hedges, its idea of an appropriate length. You have built a measurement that rewards the thing you trained for, independently of whether that thing is good.

This is self-preference bias, it has been measured in several studies of model judges, and it is at its strongest in exactly this configuration.

What to do instead

In descending order of trustworthiness:

1. A verifier, where one exists. Unarguable and free.
2. Human pairwise comparison on a sample. Expensive, and the standard the others are calibrated against.
3. A judge from a different model family than the teacher. Not perfect (it has its own preferences) but it is not the model whose output you trained to imitate.
4. A panel of two or three different judges, with disagreement flagged for human review. More expensive, and it makes the bias visible rather than removing it.

Using the teacher as judge is not always wrong; it is wrong to do it without saying so, and without an agreement figure against human labels on a sample of the same items.

Establish agreement before you believe any of it

Whatever judge you use, the number that licenses trusting it is its agreement with human judgement on your own task:

200 items judged by both a person and the model judge
agreement 0.83   (people agree with each other 0.87 on the same items)

That second figure is the ceiling. A judge cannot be more reliable than the task’s own definiteness. A judge at 0.83 against a human ceiling of 0.87 is doing well and its numbers are usable with a stated caveat. A judge at 0.62 against a ceiling of 0.85 is not measuring your task.

Re-check this when the model changes. A judge calibrated against your pre-tuning outputs may behave differently on post-tuning ones, precisely because those outputs have changed in the direction the judge rewards.

The biases, and the ones that can be reduced

Position bias. Judges favour the first option, sometimes strongly. The fix is complete: evaluate every pair twice with the order swapped and count a win only if both orders agree. Disagreement becomes a tie. It doubles the cost and it removes the bias rather than mitigating it, which is a good trade.

Verbosity bias. Judges prefer longer answers. Report mean length alongside the win rate, always, and check the win rate on length-matched items.

Style over substance. Confident, well-formatted, fluent answers beat correct hedged ones. A rubric that asks about correctness explicitly, as a separate judgement from quality, reduces this. Asking for the reasoning before the verdict generally improves judgement quality, at the cost of more output tokens.

The rubric matters more than the model. A vague “which is better” gets a vague answer. A rubric that names three criteria and asks for a verdict on each is a different and much more reliable instrument, and the improvement from writing one usually exceeds the improvement from using a larger judge.

The trap specific to distillation

One more, and it is subtle. A distilled student can beat its teacher on a teacher-judged comparison while being worse for users.

The mechanism: the student learned the teacher’s surface style strongly and its substance partially. On a judgement that weighs style heavily (and most do) the student’s more consistent, more formulaic output can score above the teacher’s more varied one. The student has become a caricature of the teacher, and caricatures are recognisable.

When you see a student beating its teacher, treat it as a finding about your measurement rather than about your model, until a human comparison says otherwise.

The rule

Write down, in the evaluation report, what judged the model and what established that the judge was trustworthy. If the answer to the first is “the model we distilled from” and the answer to the second is nothing, the result is not evidence. It is the training objective, measured again.