{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"judging-a-model-you-distilled","title":"Judging a model you distilled from the judge","url":"https://addaly.com/learn/fine-tuning/judging-a-model-you-distilled","position":71,"of":84,"minutes":9,"answer":"A teacher judging its own student rewards resemblance rather than quality, so use a verifier, a human sample or a judge from a different family.","keyIdea":"A teacher judging its own student rewards resemblance rather than quality, so use a verifier, a human sample or a judge from a different family; and a student that beats its teacher on teacher-judged comparison is a finding about the measurement, not the model."},"body":{"format":"markdown","text":"## The circularity\n\nYou distilled a 7B student from a large teacher. To evaluate the student cheaply, you ask that same teacher which of two answers is better.\n\nThe teacher prefers the student.\n\nOf course it does. The student was trained to produce text that resembles the teacher's, and the teacher's judgement of quality is substantially a judgement of resemblance to its own output: its phrasing, its structure, its characteristic hedges, its idea of an appropriate length. You have built a measurement that rewards the thing you trained for, independently of whether that thing is good.\n\nThis is self-preference bias, it has been measured in several studies of model judges, and it is at its strongest in exactly this configuration.\n\n## What to do instead\n\nIn descending order of trustworthiness:\n\n1. **A verifier**, where one exists. Unarguable and free.\n2. **Human pairwise comparison** on a sample. Expensive, and the standard the others are calibrated against.\n3. **A judge from a different model family** than the teacher. Not perfect (it has its own preferences) but it is not the model whose output you trained to imitate.\n4. **A panel** of two or three different judges, with disagreement flagged for human review. More expensive, and it makes the bias visible rather than removing it.\n\nUsing the teacher as judge is not always wrong; it is wrong to do it without saying so, and without an agreement figure against human labels on a sample of the same items.\n\n## Establish agreement before you believe any of it\n\nWhatever judge you use, the number that licenses trusting it is its agreement with human judgement on your own task:\n\n```\n200 items judged by both a person and the model judge\nagreement 0.83   (people agree with each other 0.87 on the same items)\n```\n\nThat second figure is the ceiling. A judge cannot be more reliable than the task's own definiteness. A judge at 0.83 against a human ceiling of 0.87 is doing well and its numbers are usable with a stated caveat. A judge at 0.62 against a ceiling of 0.85 is not measuring your task.\n\nRe-check this when the model changes. A judge calibrated against your pre-tuning outputs may behave differently on post-tuning ones, precisely because those outputs have changed in the direction the judge rewards.\n\n## The biases, and the ones that can be reduced\n\n**Position bias.** Judges favour the first option, sometimes strongly. The fix is complete: evaluate every pair twice with the order swapped and count a win only if both orders agree. Disagreement becomes a tie. It doubles the cost and it removes the bias rather than mitigating it, which is a good trade.\n\n**Verbosity bias.** Judges prefer longer answers. Report mean length alongside the win rate, always, and check the win rate on length-matched items.\n\n**Style over substance.** Confident, well-formatted, fluent answers beat correct hedged ones. A rubric that asks about correctness explicitly, as a separate judgement from quality, reduces this. Asking for the reasoning before the verdict generally improves judgement quality, at the cost of more output tokens.\n\n**The rubric matters more than the model.** A vague \"which is better\" gets a vague answer. A rubric that names three criteria and asks for a verdict on each is a different and much more reliable instrument, and the improvement from writing one usually exceeds the improvement from using a larger judge.\n\n## The trap specific to distillation\n\nOne more, and it is subtle. A distilled student can *beat its teacher* on a teacher-judged comparison while being worse for users.\n\nThe mechanism: the student learned the teacher's surface style strongly and its substance partially. On a judgement that weighs style heavily (and most do) the student's more consistent, more formulaic output can score above the teacher's more varied one. The student has become a caricature of the teacher, and caricatures are recognisable.\n\nWhen you see a student beating its teacher, treat it as a finding about your measurement rather than about your model, until a human comparison says otherwise.\n\n## The rule\n\nWrite down, in the evaluation report, what judged the model and what established that the judge was trustworthy. If the answer to the first is \"the model we distilled from\" and the answer to the second is nothing, the result is not evidence. It is the training objective, measured again."},"check":{"question":"A 7B student distilled from a large teacher scores a 58% win rate against that teacher, judged by the teacher itself. What is the most likely explanation?","options":[{"text":"Distillation compresses the teacher's knowledge into a form that generalises better, so exceeding the teacher is a known effect of the method.","correct":false,"tempting":"Invokes a compression-improves-generalisation story that does not hold for a student capped by its teacher on the sampled distribution."},{"text":"The student was trained on filtered teacher outputs, so it inherited only the teacher's best behaviour and is genuinely better.","correct":false,"tempting":"Would be a reasonable explanation if filtering had been described, and even then the teacher-as-judge makes the measurement unable to establish it."},{"text":"Position bias favoured the student, which was presented first in most comparisons.","correct":false,"tempting":"Names a real bias that randomising order would have removed, and which does not explain a systematic preference for one system's characteristics."},{"text":"The judge rewards resemblance to its own style, which is what the student was trained to maximise.","correct":true}]},"prev":{"slug":"pairwise-human-comparison","title":"Pairwise human comparison, done properly","url":"https://addaly.com/learn/fine-tuning/pairwise-human-comparison"},"next":{"slug":"contamination-in-your-own-data","title":"Contamination you can check, and contamination you cannot","url":"https://addaly.com/learn/fine-tuning/contamination-in-your-own-data"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Judging a model you distilled from the judge', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/judging-a-model-you-distilled (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Judging a model you distilled from the judge","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"A teacher judging its own student rewards resemblance rather than quality, so use a verifier, a human sample or a judge from a different family; and a student that beats its teacher on teacher-judged comparison is a finding about the measurement, not the model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-circularity","runs":[{"text":"The circularity"}]},{"t":"p","runs":[{"text":"You distilled a 7B student from a large teacher. To evaluate the student cheaply, you ask that same teacher which of two answers is better."}]},{"t":"p","runs":[{"text":"The teacher prefers the student."}]},{"t":"p","runs":[{"text":"Of course it does. The student was trained to produce text that resembles the teacher’s, and the teacher’s judgement of quality is substantially a judgement of resemblance to its own output: its phrasing, its structure, its characteristic hedges, its idea of an appropriate length. You have built a measurement that rewards the thing you trained for, independently of whether that thing is good."}]},{"t":"p","runs":[{"text":"This is self-preference bias, it has been measured in several studies of model judges, and it is at its strongest in exactly this configuration."}]},{"t":"h2","id":"what-to-do-instead","runs":[{"text":"What to do instead"}]},{"t":"p","runs":[{"text":"In descending order of trustworthiness:"}]},{"t":"ol","start":1,"items":[[{"text":"A verifier","b":true},{"text":", where one exists. Unarguable and free."}],[{"text":"Human pairwise comparison","b":true},{"text":" on a sample. Expensive, and the standard the others are calibrated against."}],[{"text":"A judge from a different model family","b":true},{"text":" than the teacher. Not perfect (it has its own preferences) but it is not the model whose output you trained to imitate."}],[{"text":"A panel","b":true},{"text":" of two or three different judges, with disagreement flagged for human review. More expensive, and it makes the bias visible rather than removing it."}]]},{"t":"p","runs":[{"text":"Using the teacher as judge is not always wrong; it is wrong to do it without saying so, and without an agreement figure against human labels on a sample of the same items."}]},{"t":"h2","id":"establish-agreement-before-you-believe-any-of-it","runs":[{"text":"Establish agreement before you believe any of it"}]},{"t":"p","runs":[{"text":"Whatever judge you use, the number that licenses trusting it is its agreement with human judgement on your own task:"}]},{"t":"code","lang":"","label":"Text","text":"200 items judged by both a person and the model judge\nagreement 0.83   (people agree with each other 0.87 on the same items)"},{"t":"p","runs":[{"text":"That second figure is the ceiling. A judge cannot be more reliable than the task’s own definiteness. A judge at 0.83 against a human ceiling of 0.87 is doing well and its numbers are usable with a stated caveat. A judge at 0.62 against a ceiling of 0.85 is not measuring your task."}]},{"t":"p","runs":[{"text":"Re-check this when the model changes. A judge calibrated against your pre-tuning outputs may behave differently on post-tuning ones, precisely because those outputs have changed in the direction the judge rewards."}]},{"t":"h2","id":"the-biases-and-the-ones-that-can-be-reduced","runs":[{"text":"The biases, and the ones that can be reduced"}]},{"t":"p","runs":[{"text":"Position bias.","b":true},{"text":" Judges favour the first option, sometimes strongly. The fix is complete: evaluate every pair twice with the order swapped and count a win only if both orders agree. Disagreement becomes a tie. It doubles the cost and it removes the bias rather than mitigating it, which is a good trade."}]},{"t":"p","runs":[{"text":"Verbosity bias.","b":true},{"text":" Judges prefer longer answers. Report mean length alongside the win rate, always, and check the win rate on length-matched items."}]},{"t":"p","runs":[{"text":"Style over substance.","b":true},{"text":" Confident, well-formatted, fluent answers beat correct hedged ones. A rubric that asks about correctness explicitly, as a separate judgement from quality, reduces this. Asking for the reasoning before the verdict generally improves judgement quality, at the cost of more output tokens."}]},{"t":"p","runs":[{"text":"The rubric matters more than the model.","b":true},{"text":" A vague “which is better” gets a vague answer. A rubric that names three criteria and asks for a verdict on each is a different and much more reliable instrument, and the improvement from writing one usually exceeds the improvement from using a larger judge."}]},{"t":"h2","id":"the-trap-specific-to-distillation","runs":[{"text":"The trap specific to distillation"}]},{"t":"p","runs":[{"text":"One more, and it is subtle. A distilled student can "},{"text":"beat its teacher","i":true},{"text":" on a teacher-judged comparison while being worse for users."}]},{"t":"p","runs":[{"text":"The mechanism: the student learned the teacher’s surface style strongly and its substance partially. On a judgement that weighs style heavily (and most do) the student’s more consistent, more formulaic output can score above the teacher’s more varied one. The student has become a caricature of the teacher, and caricatures are recognisable."}]},{"t":"p","runs":[{"text":"When you see a student beating its teacher, treat it as a finding about your measurement rather than about your model, until a human comparison says otherwise."}]},{"t":"h2","id":"the-rule","runs":[{"text":"The rule"}]},{"t":"p","runs":[{"text":"Write down, in the evaluation report, what judged the model and what established that the judge was trustworthy. If the answer to the first is “the model we distilled from” and the answer to the second is nothing, the result is not evidence. It is the training objective, measured again."}]}],"check":{"question":[{"text":"A 7B student distilled from a large teacher scores a 58% win rate against that teacher, judged by the teacher itself. What is the most likely explanation?"}],"options":[{"text":[{"text":"Distillation compresses the teacher’s knowledge into a form that generalises better, so exceeding the teacher is a known effect of the method."}],"correct":false,"tempting":[{"text":"Invokes a compression-improves-generalisation story that does not hold for a student capped by its teacher on the sampled distribution."}]},{"text":[{"text":"The student was trained on filtered teacher outputs, so it inherited only the teacher’s best behaviour and is genuinely better."}],"correct":false,"tempting":[{"text":"Would be a reasonable explanation if filtering had been described, and even then the teacher-as-judge makes the measurement unable to establish it."}]},{"text":[{"text":"Position bias favoured the student, which was presented first in most comparisons."}],"correct":false,"tempting":[{"text":"Names a real bias that randomising order would have removed, and which does not explain a systematic preference for one system’s characteristics."}]},{"text":[{"text":"The judge rewards resemblance to its own style, which is what the student was trained to maximise."}],"correct":true,"tempting":null}],"order":[0,3,1,2],"why":null},"prev":{"slug":"pairwise-human-comparison","title":"Pairwise human comparison, done properly","minutes":9},"next":{"slug":"contamination-in-your-own-data","title":"Contamination you can check, and contamination you cannot","minutes":9}}}