Using a model to mark the homework

When you actually need one

Most of what a golden set checks needs no judge. A cited identifier, a required substring, a refusal, a number, a schema field: all mechanical, all free, all exact. Push everything you can into that category first, because a check that costs nothing runs on every commit and a check that costs money runs monthly.

What is left is open-ended: is this summary faithful to the source, is this explanation clear, does this answer address what was asked. Human labelling is the gold standard and does not scale past a few hundred items a week. A model judge is the practical instrument, and it is a biased one.

The three biases, and the fixes

Position bias. Shown two answers and asked which is better, judges systematically favour one position: usually the first. The effect is large enough to reverse verdicts on close pairs.

The fix is complete and cheap: run every comparison twice, with the order swapped, and count a win only when both orders agree. When they disagree, score it a tie. This doubles the cost and it is not optional; a pairwise judge run in one order is measuring position as much as quality.

Verbosity bias. Longer answers score higher, independent of content. This one is dangerous because it interacts with what you are testing: a prompt change that makes answers longer will look like an improvement. Control for it by checking whether the length distributions differ between variants, and be suspicious of any win that arrives alongside a length increase. Instructing the judge to disregard length helps somewhat and does not remove it.

Self-preference. Models tend to rate their own outputs above others’. Where it matters, judge with a different model family from the one being evaluated.

Calibrate, or the number is decoration

Before trusting a judge, find out whether it agrees with you.

Label 100 items by hand. Run the judge on the same 100. Compute agreement. Raw percentage is fine to start with, though Cohen’s kappa is better because it accounts for agreement by chance: with two categories and an unbalanced set, 80 percent raw agreement can be close to worthless.

Then read the number for what it implies about resolution. A judge agreeing with humans 70 percent of the time is making a wrong call on three items in ten, and it cannot reliably detect a three-point difference between two variants, because its own error is larger than the effect. Knowing that saves you from a quarter of decisions made on noise.

If agreement is poor, the usual cause is the rubric rather than the model, and the usual cure is specificity: replace “rate helpfulness from 1 to 10” with a small number of concrete, checkable questions.

Pairwise beats scoring

Asking for a 1-to-10 score produces a distribution clustered at 7 and 8, with almost no discrimination in the region you care about, and the scale drifts between runs so yesterday’s 7 is not today’s 7.

Asking “which of these two is better, and why” is a much easier judgement and a much more stable one. It also fits the actual decision: you are comparing two variants, not certifying an absolute quality level.

Where you do need absolute scores, make them binary and specific. “Is every factual claim supported by the provided passages: yes or no” is answerable. “Rate faithfulness out of 10” is not.

Make the judge explain first

Put the reasoning field before the verdict field in the schema, for the reason the constrained-decoding lesson gave: a verdict generated first is a guess the explanation then rationalises.

The explanations have a second use that is worth more than the scores. Reading thirty judge explanations for the cases your system failed is the fastest qualitative diagnosis available, and it regularly surfaces failure modes nobody had named, which then become their own mechanical check, and stop needing a judge at all.

The free path

Judging can be expensive: two orders, several hundred items, every run. Three things keep it affordable.

Route judging through a batch endpoint, since nobody is waiting. Roughly half price for work that runs overnight.

Use a small model where the rubric is simple. Faithfulness against a provided passage is close to an entailment task, and small open models handle it respectably. Calibrate it against your human labels the same way; if a 7-billion-parameter model running locally agrees with you as often as a frontier model does, it is the correct choice and it costs nothing per item.

Judge a sample. You do not need a judge on all 300 items on every commit. Mechanical checks on everything, judged scores on a fixed 100, full runs before a release.

The limit to state out loud

A judge measures agreement with a rubric, not truth. If the rubric is wrong, the judge is confidently and consistently wrong, and consistency reads as reliability on a dashboard.

So keep a small human-labelled sample running alongside, permanently, and watch the agreement rate as its own metric. When it drifts, the judge has stopped measuring what you think it measures. And that drift will not announce itself in the scores, which will carry on looking perfectly steady.