Gold answers, and asking whether two answers agree

Two protocols, and the easier question

There are two ways to judge an output. Reference-free: give the judge a rubric and the material, and ask whether the output satisfies it. Reference-based: give the judge a gold answer, and ask whether the candidate agrees with it.

The second is a much easier question for a model, and much more stable. “Is this a good answer” is an aesthetic judgement that drifts with phrasing, length and mood. “Do these two answers give the same answer to this question” is a comparison, and comparisons are what judges are reliably good at.

So whenever a question has an answer you can write down, write it down.

QUESTION: {question}
REFERENCE ANSWER: {reference}
CANDIDATE ANSWER: {candidate}

Ignore wording, length, tone and ordering. Consider only whether the
candidate conveys the same answer to the question as the reference.

Quote the part of the candidate that decides this, then output JSON:
{"quote": "...", "verdict": "same" | "different" | "partial"}

Write references as facts, not prose

A paragraph-shaped reference invites the judge to compare paragraphs, and the judge then punishes an answer that is correct and phrased differently. The mechanism is anchoring: a fluent reference gives the model a template, and deviation from a template reads as error.

Write the reference as structure instead:

{ "required": ["refund window is 14 days", "counted from delivery"],
  "forbidden": ["30 days", "counted from order date"],
  "acceptable_variants": ["two weeks"] }

Now the judge checks membership of facts rather than resemblance of prose, which is a question with a defensible right answer. It is also directly checkable by code for the parts that are literal strings, which means many items never need a model call at all.

Some of your gold answers are wrong

Every gold set carries errors: a policy that changed, a mislabelled item, an annotator’s misreading. In practice five to ten per cent is normal, and it distorts your measurement in a specific direction: a model that is right where the reference is wrong is scored as failing.

The fix is a targeted audit, and it is the same trick as in the multi-label lesson. Take fifty items where the candidate disagrees with the reference, and have a person judge the item, not the model: is the reference actually correct here. Typically a third of the disagreements turn out to be reference errors. Correct them, note the correction rate in the report, and re-run.

Do this before concluding anything from a score in the nineties, because at that level most of the remaining failures may be your file rather than the model.

When there is no single right answer

Plenty of tasks have many valid outputs: a support reply, a code review comment, a summary’s phrasing, a translation into a language with several registers. Forcing a reference on those creates false negatives at scale, and the rubric approach from earlier in this block is the right tool.

The most useful design is usually a hybrid. Split the output into the part with a right answer and the part without:

• Reference-checked core: the refund window, the amount, the date, the decision, the API name. Exact or normalised match, no model needed.
• Rubric-judged remainder: tone, completeness, whether it answered the question asked, whether it invented a policy.

Reporting these separately also survives contact with a product meeting, because “the facts are right 97% of the time and the tone check passes 84%” supports different actions than a blended 91%.

References need versions

A reference answer encodes a policy at a point in time. When the refund window changes from 14 days to 30, every affected reference becomes wrong, silently, and the eval starts penalising the correct behaviour.

Attach a policy version to each item and fail loudly when the versions do not match the current policy document. It sounds bureaucratic until the first time an eval blocks a correct change for a fortnight because nobody remembered which items mentioned the old window.

Multiple references help too (two or three acceptable answers per question reduce false negatives) but they multiply the maintenance burden, so spend them on the items where the variation is real rather than on the whole set.