Turning a quality you can feel into rules a machine can apply

The gap between a standard and a rubric

You know what a good reply looks like. The problem is that knowing is stored in your head as a pattern, and a judge (human or model) needs it as a list.

A rubric is that list. Writing one is the same exercise as the labelling guide in module one, pushed a step further, because a model is a more literal reader than a colleague. Your colleague fills gaps with common sense and shared context. The model fills them with whatever is statistically likely, which is usually generous.

The test for a finished rubric: hand it to somebody who has never seen your product, along with twenty outputs. If they come back with a question, the model was going to guess at that same point, and you will never see it guess.

Anatomy of a rule

Every rule has the same four properties, and a rule missing one of them will produce inconsistent scores:

1. One property. “Clear and complete” is two rules pretending to be one. Split it.
2. Observable in the output. “The customer will feel reassured” is not checkable; “the reply states what happens next and by when” is.
3. Binary. True or false, with no middle. If a rule needs three levels, it is two rules stacked.
4. Evidence-bearing. The judge must be able to point at the span that decides it.

A worked rubric for a support reply, ordered hard constraints first:

R1 (gate). The reply makes no commitment about money, dates or eligibility that is absent from the supplied policy text.

R2 (gate). The reply does not state a fact about the customer’s account that is absent from the supplied account data.

R3. The reply answers the question asked, not an adjacent one.

R4. Where required information is missing, the reply asks for it rather than assuming a value.

R5. The reply is in the same language as the customer’s message.

R6. The reply states the next step and who takes it.

Six rules. Each separately true or false. R1 and R2 are gates: fail either and the item fails regardless of the rest, because a fabricated refund date is not compensated for by good structure.

That gate distinction is worth building into the score explicitly. An average across six rules would let a system fail R1 five per cent of the time and still report 0.95. Report gates as a separate count, always as raw numbers: “R1 violated in 3 of 200”.

One rule per call

Give a model all six rules in one prompt and ask for six verdicts, and the verdicts correlate. It forms an overall impression and then produces six judgements consistent with that impression. You can watch this happen: outputs cluster on all-pass and all-fail far more than independent rules would produce.

Six separate calls cost six times as much and are worth it, because each judgement becomes independent evidence. At typical small-model prices, 200 items times six rules is a few tens of thousands of tokens per rule and cents in total. Cheaper than being wrong.

If the cost matters, group only rules that are logically unrelated, and never group two rules that a lazy reader would conflate.

Reasoning first, verdict last

Ask for the verdict first and the model commits, then writes a justification for a decision already made. Ask for the evidence first and the verdict is conditioned on it.

RULE: The reply makes no commitment about money, dates or
      eligibility that is absent from POLICY.

POLICY: {policy_text}
REPLY:  {reply_text}

1. Quote every span of REPLY that makes such a commitment,
   or write NONE.
2. For each quoted span, quote the sentence of POLICY that
   authorises it, or write NOT FOUND.
3. Output JSON: {"spans": [...], "violates": true|false}

And then verify mechanically that every quoted span is a substring of the reply. It is one line of code, it is free, and it catches the judge inventing evidence, which it does, at a low but non-zero rate that you would otherwise never see.

Write the rubric from failures, not from first principles

The rubric people write at a desk is neat and wrong. The rubric that works comes from reading 50 real failures and asking, for each, “what rule would have caught this?”

Do it in that order and you get rules that fire. Do it the other way and you get a beautiful eight-rule rubric where six rules pass on every item ever produced and contribute nothing but cost.

A rule that has never failed in 500 items is not a safety net. It is either the wrong rule or a solved problem, and either way it should be retired or moved to a cheap deterministic check.

Version it

Put the rubric in the repository with a version number, and record which rubric version produced every score. When you change a rule, and you will, roughly every time you read a new batch of failures, old numbers stop being comparable to new ones.

Teams lose weeks to this. A metric drops four points, everybody investigates the system, and the cause is a rubric edit somebody made on Tuesday. The scores were measuring a different thing, and nothing in the dashboard said so.