What to measure when the output is prose

Stop scoring "quality"

The instinct is to rate each output 1 to 5 for quality. Resist it. A 5-point score means something different to each labeller, something different to the same labeller on a Friday, and nothing at all to the engineer who has to act on it. "We went from 3.4 to 3.6" points at no change you can make.

Split the thing you care about into checks that are separately true or false. Five binary checks beat one 5-point score, every time, for the same labelling effort.

Three families of check, best first

Extract and compare exactly. Most "open-ended" tasks have a right answer buried in the prose. A summary of an invoice contains an amount. A reply about a refund contains a policy window. A support answer names a product that either exists or does not. Pull that value out and compare it with ==. This feels crude and it is the strongest signal you will get — it does not drift, it does not need a judge, and it is impossible to argue with.

Rubric checks, binary, written as questions. "Does it reply in the language the customer wrote in?" "Does it avoid promising a refund?" "Is it under 120 words?" "Does every link it gives resolve?" Each is yes or no, each is cheap to check, and when the score drops you know which one moved.

Behavioural checks — did the next thing work. Did the JSON parse. Did the SQL run without error. Did the tool call have valid arguments. Did the customer's next message say "thank you" rather than "no, I meant the other order". These cost nothing to collect and they are the closest thing to ground truth you will ever have.

def grade(item, output):
    return {
        "parses":        is_json(output),
        "amount_exact":  extract_amount(output) == item["expected"]["amount"],
        "currency_exact": extract_currency(output) == item["expected"]["currency"],
        "same_language": detect_lang(output) == detect_lang(item["input"]),
        "no_refund_promise": not PROMISE_RE.search(output),
    }
    # store the dict, per item. Never collapse to one number here.

Keep every check separately for every item. Report the pass rate of each check and the pass rate within each tag. An overall number is for the meeting; the per-check table is for the work.

The metrics that will waste your week

BLEU, ROUGE, and embedding-similarity-to-a-reference are widely used and mostly wrong for this. They were built for machine translation and for summarisation scored against reference corpora, where many valid outputs share vocabulary with the reference. Your task is not that. A support reply can share almost no words with your reference answer and be perfect; it can share most of them and be confidently wrong about the refund window.

When a similarity score moves, the honest reading is "the wording moved", not "the quality moved". Occasionally that is what you want — detecting that a prompt change made outputs drift stylistically is a real use. Just do not report it as accuracy.

A note on the ones you cannot check

Some things genuinely resist binary checks: tone, whether an explanation is at the right level, whether a piece of writing is any good. You have two honest options. Have humans rate them, in pairwise comparisons rather than absolute scores, on a small sample. Or use a model as a judge — which works, and misleads in specific ways that the next lesson is entirely about.

What you should not do is invent a number for them and put it on a dashboard, because a number on a dashboard gets optimised, and a fake number gets faithfully optimised into a worse product.