{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"writing-a-rubric-a-machine-can-follow","title":"Turning a quality you can feel into rules a machine can apply","url":"https://addaly.com/learn/evaluating-ai/writing-a-rubric-a-machine-can-follow","position":41,"of":76,"minutes":9,"answer":"A usable rubric is a short list of single-property binary rules written from real failures, judged one per call, with the evidence quoted before the verdict.","keyIdea":"A usable rubric is a short list of single-property binary rules written from real failures, judged one per call, with the evidence quoted before the verdict."},"body":{"format":"markdown","text":"## The gap between a standard and a rubric\n\nYou know what a good reply looks like. The problem is that knowing is stored in your head as a pattern, and a judge (human or model) needs it as a list.\n\nA rubric is that list. Writing one is the same exercise as the labelling guide in module one, pushed a step further, because a model is a more literal reader than a colleague. Your colleague fills gaps with common sense and shared context. The model fills them with whatever is statistically likely, which is usually generous.\n\nThe test for a finished rubric: hand it to somebody who has never seen your product, along with twenty outputs. If they come back with a question, the model was going to guess at that same point, and you will never see it guess.\n\n## Anatomy of a rule\n\nEvery rule has the same four properties, and a rule missing one of them will produce inconsistent scores:\n\n1. **One property.** \"Clear and complete\" is two rules pretending to be one. Split it.\n2. **Observable in the output.** \"The customer will feel reassured\" is not checkable; \"the reply states what happens next and by when\" is.\n3. **Binary.** True or false, with no middle. If a rule needs three levels, it is two rules stacked.\n4. **Evidence-bearing.** The judge must be able to point at the span that decides it.\n\nA worked rubric for a support reply, ordered hard constraints first:\n\n> **R1 (gate).** The reply makes no commitment about money, dates or eligibility that is absent from the supplied policy text.\n>\n> **R2 (gate).** The reply does not state a fact about the customer's account that is absent from the supplied account data.\n>\n> **R3.** The reply answers the question asked, not an adjacent one.\n>\n> **R4.** Where required information is missing, the reply asks for it rather than assuming a value.\n>\n> **R5.** The reply is in the same language as the customer's message.\n>\n> **R6.** The reply states the next step and who takes it.\n\nSix rules. Each separately true or false. R1 and R2 are gates: fail either and the item fails regardless of the rest, because a fabricated refund date is not compensated for by good structure.\n\nThat gate distinction is worth building into the score explicitly. An average across six rules would let a system fail R1 five per cent of the time and still report 0.95. Report gates as a separate count, always as raw numbers: \"R1 violated in 3 of 200\".\n\n## One rule per call\n\nGive a model all six rules in one prompt and ask for six verdicts, and the verdicts correlate. It forms an overall impression and then produces six judgements consistent with that impression. You can watch this happen: outputs cluster on all-pass and all-fail far more than independent rules would produce.\n\nSix separate calls cost six times as much and are worth it, because each judgement becomes independent evidence. At typical small-model prices, 200 items times six rules is a few tens of thousands of tokens per rule and cents in total. Cheaper than being wrong.\n\nIf the cost matters, group only rules that are logically unrelated, and never group two rules that a lazy reader would conflate.\n\n## Reasoning first, verdict last\n\nAsk for the verdict first and the model commits, then writes a justification for a decision already made. Ask for the evidence first and the verdict is conditioned on it.\n\n```\nRULE: The reply makes no commitment about money, dates or\n      eligibility that is absent from POLICY.\n\nPOLICY: {policy_text}\nREPLY:  {reply_text}\n\n1. Quote every span of REPLY that makes such a commitment,\n   or write NONE.\n2. For each quoted span, quote the sentence of POLICY that\n   authorises it, or write NOT FOUND.\n3. Output JSON: {\"spans\": [...], \"violates\": true|false}\n```\n\nAnd then verify mechanically that every quoted span is a substring of the reply. It is one line of code, it is free, and it catches the judge inventing evidence, which it does, at a low but non-zero rate that you would otherwise never see.\n\n## Write the rubric from failures, not from first principles\n\nThe rubric people write at a desk is neat and wrong. The rubric that works comes from reading 50 real failures and asking, for each, \"what rule would have caught this?\"\n\nDo it in that order and you get rules that fire. Do it the other way and you get a beautiful eight-rule rubric where six rules pass on every item ever produced and contribute nothing but cost.\n\nA rule that has never failed in 500 items is not a safety net. It is either the wrong rule or a solved problem, and either way it should be retired or moved to a cheap deterministic check.\n\n## Version it\n\nPut the rubric in the repository with a version number, and record which rubric version produced every score. When you change a rule, and you will, roughly every time you read a new batch of failures, old numbers stop being comparable to new ones.\n\nTeams lose weeks to this. A metric drops four points, everybody investigates the system, and the cause is a rubric edit somebody made on Tuesday. The scores were measuring a different thing, and nothing in the dashboard said so."},"check":{"question":"A judge prompt asks a model to rate a reply from 1 to 5 on \"clarity, accuracy and tone\" and return a single score with a short explanation. What is the most damaging flaw?","options":[{"text":"Five points is too coarse a scale; ten would give more resolution to detect small improvements.","correct":false,"tempting":"Assumes more scale points mean more information, when models cluster their ratings regardless of how many points you offer."},{"text":"Three separate properties are fused into one number, so a failure of accuracy can be offset by good tone and the score cannot be acted on.","correct":true},{"text":"The explanation comes after the score, so the model has already committed before reasoning.","correct":false,"tempting":"Names a real ordering problem that is worth fixing, but it is secondary to a score that could not be interpreted even if perfectly reasoned."},{"text":"No reference answer is supplied, so the model has nothing to compare the reply against.","correct":false,"tempting":"Assumes judging always needs a gold answer, when rule-based checks against a policy or context work without one."}]},"prev":{"slug":"the-judge-needs-judging","title":"LLM-as-judge, and where it misleads you","url":"https://addaly.com/learn/evaluating-ai/the-judge-needs-judging"},"next":{"slug":"reference-answers-and-equivalence","title":"Gold answers, and asking whether two answers agree","url":"https://addaly.com/learn/evaluating-ai/reference-answers-and-equivalence"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Turning a quality you can feel into rules a machine can apply', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/writing-a-rubric-a-machine-can-follow (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Turning a quality you can feel into rules a machine can apply","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"A usable rubric is a short list of single-property binary rules written from real failures, judged one per call, with the evidence quoted before the verdict.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-gap-between-a-standard-and-a-rubric","runs":[{"text":"The gap between a standard and a rubric"}]},{"t":"p","runs":[{"text":"You know what a good reply looks like. The problem is that knowing is stored in your head as a pattern, and a judge (human or model) needs it as a list."}]},{"t":"p","runs":[{"text":"A rubric is that list. Writing one is the same exercise as the labelling guide in module one, pushed a step further, because a model is a more literal reader than a colleague. Your colleague fills gaps with common sense and shared context. The model fills them with whatever is statistically likely, which is usually generous."}]},{"t":"p","runs":[{"text":"The test for a finished rubric: hand it to somebody who has never seen your product, along with twenty outputs. If they come back with a question, the model was going to guess at that same point, and you will never see it guess."}]},{"t":"h2","id":"anatomy-of-a-rule","runs":[{"text":"Anatomy of a rule"}]},{"t":"p","runs":[{"text":"Every rule has the same four properties, and a rule missing one of them will produce inconsistent scores:"}]},{"t":"ol","start":1,"items":[[{"text":"One property.","b":true},{"text":" “Clear and complete” is two rules pretending to be one. Split it."}],[{"text":"Observable in the output.","b":true},{"text":" “The customer will feel reassured” is not checkable; “the reply states what happens next and by when” is."}],[{"text":"Binary.","b":true},{"text":" True or false, with no middle. If a rule needs three levels, it is two rules stacked."}],[{"text":"Evidence-bearing.","b":true},{"text":" The judge must be able to point at the span that decides it."}]]},{"t":"p","runs":[{"text":"A worked rubric for a support reply, ordered hard constraints first:"}]},{"t":"quote","runs":[{"text":"R1 (gate).","b":true},{"text":" The reply makes no commitment about money, dates or eligibility that is absent from the supplied policy text."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"R2 (gate).","b":true},{"text":" The reply does not state a fact about the customer’s account that is absent from the supplied account data."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"R3.","b":true},{"text":" The reply answers the question asked, not an adjacent one."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"R4.","b":true},{"text":" Where required information is missing, the reply asks for it rather than assuming a value."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"R5.","b":true},{"text":" The reply is in the same language as the customer’s message."}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"R6.","b":true},{"text":" The reply states the next step and who takes it."}]},{"t":"p","runs":[{"text":"Six rules. Each separately true or false. R1 and R2 are gates: fail either and the item fails regardless of the rest, because a fabricated refund date is not compensated for by good structure."}]},{"t":"p","runs":[{"text":"That gate distinction is worth building into the score explicitly. An average across six rules would let a system fail R1 five per cent of the time and still report 0.95. Report gates as a separate count, always as raw numbers: “R1 violated in 3 of 200”."}]},{"t":"h2","id":"one-rule-per-call","runs":[{"text":"One rule per call"}]},{"t":"p","runs":[{"text":"Give a model all six rules in one prompt and ask for six verdicts, and the verdicts correlate. It forms an overall impression and then produces six judgements consistent with that impression. You can watch this happen: outputs cluster on all-pass and all-fail far more than independent rules would produce."}]},{"t":"p","runs":[{"text":"Six separate calls cost six times as much and are worth it, because each judgement becomes independent evidence. At typical small-model prices, 200 items times six rules is a few tens of thousands of tokens per rule and cents in total. Cheaper than being wrong."}]},{"t":"p","runs":[{"text":"If the cost matters, group only rules that are logically unrelated, and never group two rules that a lazy reader would conflate."}]},{"t":"h2","id":"reasoning-first-verdict-last","runs":[{"text":"Reasoning first, verdict last"}]},{"t":"p","runs":[{"text":"Ask for the verdict first and the model commits, then writes a justification for a decision already made. Ask for the evidence first and the verdict is conditioned on it."}]},{"t":"code","lang":"","label":"Text","text":"RULE: The reply makes no commitment about money, dates or\n      eligibility that is absent from POLICY.\n\nPOLICY: {policy_text}\nREPLY:  {reply_text}\n\n1. Quote every span of REPLY that makes such a commitment,\n   or write NONE.\n2. For each quoted span, quote the sentence of POLICY that\n   authorises it, or write NOT FOUND.\n3. Output JSON: {\"spans\": [...], \"violates\": true|false}"},{"t":"p","runs":[{"text":"And then verify mechanically that every quoted span is a substring of the reply. It is one line of code, it is free, and it catches the judge inventing evidence, which it does, at a low but non-zero rate that you would otherwise never see."}]},{"t":"h2","id":"write-the-rubric-from-failures-not-from-first-principles","runs":[{"text":"Write the rubric from failures, not from first principles"}]},{"t":"p","runs":[{"text":"The rubric people write at a desk is neat and wrong. The rubric that works comes from reading 50 real failures and asking, for each, “what rule would have caught this?”"}]},{"t":"p","runs":[{"text":"Do it in that order and you get rules that fire. Do it the other way and you get a beautiful eight-rule rubric where six rules pass on every item ever produced and contribute nothing but cost."}]},{"t":"p","runs":[{"text":"A rule that has never failed in 500 items is not a safety net. It is either the wrong rule or a solved problem, and either way it should be retired or moved to a cheap deterministic check."}]},{"t":"h2","id":"version-it","runs":[{"text":"Version it"}]},{"t":"p","runs":[{"text":"Put the rubric in the repository with a version number, and record which rubric version produced every score. When you change a rule, and you will, roughly every time you read a new batch of failures, old numbers stop being comparable to new ones."}]},{"t":"p","runs":[{"text":"Teams lose weeks to this. A metric drops four points, everybody investigates the system, and the cause is a rubric edit somebody made on Tuesday. The scores were measuring a different thing, and nothing in the dashboard said so."}]}],"check":{"question":[{"text":"A judge prompt asks a model to rate a reply from 1 to 5 on “clarity, accuracy and tone” and return a single score with a short explanation. What is the most damaging flaw?"}],"options":[{"text":[{"text":"Five points is too coarse a scale; ten would give more resolution to detect small improvements."}],"correct":false,"tempting":[{"text":"Assumes more scale points mean more information, when models cluster their ratings regardless of how many points you offer."}]},{"text":[{"text":"Three separate properties are fused into one number, so a failure of accuracy can be offset by good tone and the score cannot be acted on."}],"correct":true,"tempting":null},{"text":[{"text":"The explanation comes after the score, so the model has already committed before reasoning."}],"correct":false,"tempting":[{"text":"Names a real ordering problem that is worth fixing, but it is secondary to a score that could not be interpreted even if perfectly reasoned."}]},{"text":[{"text":"No reference answer is supplied, so the model has nothing to compare the reply against."}],"correct":false,"tempting":[{"text":"Assumes judging always needs a gold answer, when rule-based checks against a policy or context work without one."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"the-judge-needs-judging","title":"LLM-as-judge, and where it misleads you","minutes":9},"next":{"slug":"reference-answers-and-equivalence","title":"Gold answers, and asking whether two answers agree","minutes":9}}}