{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"reference-answers-and-equivalence","title":"Gold answers, and asking whether two answers agree","url":"https://addaly.com/learn/evaluating-ai/reference-answers-and-equivalence","position":42,"of":76,"minutes":9,"answer":"Ask a judge whether the candidate says the same thing as the reference rather than whether it is good, write references as required and forbidden facts instead of prose.","keyIdea":"Ask a judge whether the candidate says the same thing as the reference rather than whether it is good, write references as required and forbidden facts instead of prose, and audit the disagreements because a share of them are errors in your gold answers."},"body":{"format":"markdown","text":"## Two protocols, and the easier question\n\nThere are two ways to judge an output. **Reference-free**: give the judge a rubric and the material, and ask whether the output satisfies it. **Reference-based**: give the judge a gold answer, and ask whether the candidate agrees with it.\n\nThe second is a much easier question for a model, and much more stable. \"Is this a good answer\" is an aesthetic judgement that drifts with phrasing, length and mood. \"Do these two answers give the same answer to this question\" is a comparison, and comparisons are what judges are reliably good at.\n\nSo whenever a question has an answer you can write down, write it down.\n\n```\nQUESTION: {question}\nREFERENCE ANSWER: {reference}\nCANDIDATE ANSWER: {candidate}\n\nIgnore wording, length, tone and ordering. Consider only whether the\ncandidate conveys the same answer to the question as the reference.\n\nQuote the part of the candidate that decides this, then output JSON:\n{\"quote\": \"...\", \"verdict\": \"same\" | \"different\" | \"partial\"}\n```\n\n## Write references as facts, not prose\n\nA paragraph-shaped reference invites the judge to compare paragraphs, and the judge then punishes an answer that is correct and phrased differently. The mechanism is anchoring: a fluent reference gives the model a template, and deviation from a template reads as error.\n\nWrite the reference as structure instead:\n\n```json\n{ \"required\": [\"refund window is 14 days\", \"counted from delivery\"],\n  \"forbidden\": [\"30 days\", \"counted from order date\"],\n  \"acceptable_variants\": [\"two weeks\"] }\n```\n\nNow the judge checks membership of facts rather than resemblance of prose, which is a question with a defensible right answer. It is also directly checkable by code for the parts that are literal strings, which means many items never need a model call at all.\n\n## Some of your gold answers are wrong\n\nEvery gold set carries errors: a policy that changed, a mislabelled item, an annotator's misreading. In practice five to ten per cent is normal, and it distorts your measurement in a specific direction: a model that is right where the reference is wrong is scored as failing.\n\nThe fix is a targeted audit, and it is the same trick as in the multi-label lesson. Take fifty items where the candidate disagrees with the reference, and have a person judge the *item*, not the model: is the reference actually correct here. Typically a third of the disagreements turn out to be reference errors. Correct them, note the correction rate in the report, and re-run.\n\nDo this before concluding anything from a score in the nineties, because at that level most of the remaining failures may be your file rather than the model.\n\n## When there is no single right answer\n\nPlenty of tasks have many valid outputs: a support reply, a code review comment, a summary's phrasing, a translation into a language with several registers. Forcing a reference on those creates false negatives at scale, and the rubric approach from earlier in this block is the right tool.\n\nThe most useful design is usually a hybrid. Split the output into the part with a right answer and the part without:\n\n- **Reference-checked core**: the refund window, the amount, the date, the decision, the API name. Exact or normalised match, no model needed.\n- **Rubric-judged remainder**: tone, completeness, whether it answered the question asked, whether it invented a policy.\n\nReporting these separately also survives contact with a product meeting, because \"the facts are right 97% of the time and the tone check passes 84%\" supports different actions than a blended 91%.\n\n## References need versions\n\nA reference answer encodes a policy at a point in time. When the refund window changes from 14 days to 30, every affected reference becomes wrong, silently, and the eval starts penalising the correct behaviour.\n\nAttach a policy version to each item and fail loudly when the versions do not match the current policy document. It sounds bureaucratic until the first time an eval blocks a correct change for a fortnight because nobody remembered which items mentioned the old window.\n\nMultiple references help too (two or three acceptable answers per question reduce false negatives) but they multiply the maintenance burden, so spend them on the items where the variation is real rather than on the whole set."},"check":{"question":"A judge is given a fluent paragraph as the reference answer and asked whether the candidate is correct. Correct-but-differently-worded answers are frequently marked wrong. What is the mechanism?","options":[{"text":"The judge model is too small to handle long references and truncates them.","correct":false,"tempting":"Attributes a systematic bias to capacity; the effect appears with capable models and short references alike."},{"text":"A prose reference gives the judge a template to compare against, so deviation in wording reads as deviation in content.","correct":true},{"text":"The judge has not been given the original question, so it can only compare the two texts.","correct":false,"tempting":"A real design error when it occurs, but the anchoring effect persists even when the question is supplied."},{"text":"Reference-based judging is inherently less reliable than reference-free rubric judging.","correct":false,"tempting":"Reverses the usual finding: comparison against a reference is the more stable task, provided the reference is expressed as facts rather than prose."}]},"prev":{"slug":"writing-a-rubric-a-machine-can-follow","title":"Turning a quality you can feel into rules a machine can apply","url":"https://addaly.com/learn/evaluating-ai/writing-a-rubric-a-machine-can-follow"},"next":{"slug":"calibrating-and-monitoring-the-judge","title":"Proving the judge agrees with you, and re-proving it","url":"https://addaly.com/learn/evaluating-ai/calibrating-and-monitoring-the-judge"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Gold answers, and asking whether two answers agree', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/reference-answers-and-equivalence (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Gold answers, and asking whether two answers agree","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Ask a judge whether the candidate says the same thing as the reference rather than whether it is good, write references as required and forbidden facts instead of prose, and audit the disagreements because a share of them are errors in your gold answers.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-protocols-and-the-easier-question","runs":[{"text":"Two protocols, and the easier question"}]},{"t":"p","runs":[{"text":"There are two ways to judge an output. "},{"text":"Reference-free","b":true},{"text":": give the judge a rubric and the material, and ask whether the output satisfies it. "},{"text":"Reference-based","b":true},{"text":": give the judge a gold answer, and ask whether the candidate agrees with it."}]},{"t":"p","runs":[{"text":"The second is a much easier question for a model, and much more stable. “Is this a good answer” is an aesthetic judgement that drifts with phrasing, length and mood. “Do these two answers give the same answer to this question” is a comparison, and comparisons are what judges are reliably good at."}]},{"t":"p","runs":[{"text":"So whenever a question has an answer you can write down, write it down."}]},{"t":"code","lang":"","label":"Text","text":"QUESTION: {question}\nREFERENCE ANSWER: {reference}\nCANDIDATE ANSWER: {candidate}\n\nIgnore wording, length, tone and ordering. Consider only whether the\ncandidate conveys the same answer to the question as the reference.\n\nQuote the part of the candidate that decides this, then output JSON:\n{\"quote\": \"...\", \"verdict\": \"same\" | \"different\" | \"partial\"}"},{"t":"h2","id":"write-references-as-facts-not-prose","runs":[{"text":"Write references as facts, not prose"}]},{"t":"p","runs":[{"text":"A paragraph-shaped reference invites the judge to compare paragraphs, and the judge then punishes an answer that is correct and phrased differently. The mechanism is anchoring: a fluent reference gives the model a template, and deviation from a template reads as error."}]},{"t":"p","runs":[{"text":"Write the reference as structure instead:"}]},{"t":"code","lang":"json","label":"JSON","text":"{ \"required\": [\"refund window is 14 days\", \"counted from delivery\"],\n  \"forbidden\": [\"30 days\", \"counted from order date\"],\n  \"acceptable_variants\": [\"two weeks\"] }"},{"t":"p","runs":[{"text":"Now the judge checks membership of facts rather than resemblance of prose, which is a question with a defensible right answer. It is also directly checkable by code for the parts that are literal strings, which means many items never need a model call at all."}]},{"t":"h2","id":"some-of-your-gold-answers-are-wrong","runs":[{"text":"Some of your gold answers are wrong"}]},{"t":"p","runs":[{"text":"Every gold set carries errors: a policy that changed, a mislabelled item, an annotator’s misreading. In practice five to ten per cent is normal, and it distorts your measurement in a specific direction: a model that is right where the reference is wrong is scored as failing."}]},{"t":"p","runs":[{"text":"The fix is a targeted audit, and it is the same trick as in the multi-label lesson. Take fifty items where the candidate disagrees with the reference, and have a person judge the "},{"text":"item","i":true},{"text":", not the model: is the reference actually correct here. Typically a third of the disagreements turn out to be reference errors. Correct them, note the correction rate in the report, and re-run."}]},{"t":"p","runs":[{"text":"Do this before concluding anything from a score in the nineties, because at that level most of the remaining failures may be your file rather than the model."}]},{"t":"h2","id":"when-there-is-no-single-right-answer","runs":[{"text":"When there is no single right answer"}]},{"t":"p","runs":[{"text":"Plenty of tasks have many valid outputs: a support reply, a code review comment, a summary’s phrasing, a translation into a language with several registers. Forcing a reference on those creates false negatives at scale, and the rubric approach from earlier in this block is the right tool."}]},{"t":"p","runs":[{"text":"The most useful design is usually a hybrid. Split the output into the part with a right answer and the part without:"}]},{"t":"ul","items":[[{"text":"Reference-checked core","b":true},{"text":": the refund window, the amount, the date, the decision, the API name. Exact or normalised match, no model needed."}],[{"text":"Rubric-judged remainder","b":true},{"text":": tone, completeness, whether it answered the question asked, whether it invented a policy."}]]},{"t":"p","runs":[{"text":"Reporting these separately also survives contact with a product meeting, because “the facts are right 97% of the time and the tone check passes 84%” supports different actions than a blended 91%."}]},{"t":"h2","id":"references-need-versions","runs":[{"text":"References need versions"}]},{"t":"p","runs":[{"text":"A reference answer encodes a policy at a point in time. When the refund window changes from 14 days to 30, every affected reference becomes wrong, silently, and the eval starts penalising the correct behaviour."}]},{"t":"p","runs":[{"text":"Attach a policy version to each item and fail loudly when the versions do not match the current policy document. It sounds bureaucratic until the first time an eval blocks a correct change for a fortnight because nobody remembered which items mentioned the old window."}]},{"t":"p","runs":[{"text":"Multiple references help too (two or three acceptable answers per question reduce false negatives) but they multiply the maintenance burden, so spend them on the items where the variation is real rather than on the whole set."}]}],"check":{"question":[{"text":"A judge is given a fluent paragraph as the reference answer and asked whether the candidate is correct. Correct-but-differently-worded answers are frequently marked wrong. What is the mechanism?"}],"options":[{"text":[{"text":"The judge model is too small to handle long references and truncates them."}],"correct":false,"tempting":[{"text":"Attributes a systematic bias to capacity; the effect appears with capable models and short references alike."}]},{"text":[{"text":"A prose reference gives the judge a template to compare against, so deviation in wording reads as deviation in content."}],"correct":true,"tempting":null},{"text":[{"text":"The judge has not been given the original question, so it can only compare the two texts."}],"correct":false,"tempting":[{"text":"A real design error when it occurs, but the anchoring effect persists even when the question is supplied."}]},{"text":[{"text":"Reference-based judging is inherently less reliable than reference-free rubric judging."}],"correct":false,"tempting":[{"text":"Reverses the usual finding: comparison against a reference is the more stable task, provided the reference is expressed as facts rather than prose."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"writing-a-rubric-a-machine-can-follow","title":"Turning a quality you can feel into rules a machine can apply","minutes":9},"next":{"slug":"calibrating-and-monitoring-the-judge","title":"Proving the judge agrees with you, and re-proving it","minutes":9}}}