Evaluation: deciding what “correct” means, in advance

Why this is the skill worth having

Work that is well specified, abundantly exampled and quickly checkable is the work that gets automated first. Deciding what “good enough” means is none of those things. It is a judgement someone has to be accountable for, and organisations do not hand accountability to a tool.

That is the strategic argument. The practical one is simpler: almost no beginner shows evidence of having evaluated their own work, so doing it makes your portfolio unusual in the one way that matters.

Build the set by hand, and build it small

Thirty to fifty cases. Not three thousand. You are going to read every one of them personally, several times, and a set you will not read is a set you will not learn from.

Where the cases come from:

• Real questions people actually asked. Support tickets, search logs, the questions in a team’s chat channel. These are worth ten invented ones.
• The edge cases the team argues about. Every team has three examples they disagree on. Those are the most valuable cases you own.
• Three cases that should fail. A question the system has no information to answer, a request it should refuse, and a question in another language or dialect. What a system does when it should say “I don’t know” is where most of the risk lives.

The rubric, and why three points beats five

For each case, write down what a correct answer must contain, before you see any output. Then grade.

Use binary correct/incorrect wherever you can. Where you cannot, use three levels: correct, partly correct, wrong. Resist the five-point scale: graders do not agree on the difference between a 3 and a 4, agreement between them collapses, and once two people scoring the same output produce different numbers, the average stops meaning anything. Three coarse levels that people agree on beat five fine ones they do not.

What a record looks like

case_id: 017
question: "What is the refund window for orders shipped to Nigeria?"
must_contain: ["30 days", "counted from delivery, not order date"]
answer: "You can request a refund within 14 days of ordering."
verdict: wrong
note: retrieved the UK policy page; the Nigeria page exists
      but is titled 'West Africa returns' and did not match

The note field is where the value is. “Wrong” is a number. “Retrieved the wrong document because the title uses a regional name” is a fix.

The two numbers to report, and the one to lead with

Report accuracy on the set (33 of 40 correct) and report the worst failure by name. A single named failure moves a decision more than any average, because a stakeholder can picture it. “It confidently gave a customer in Lagos the UK refund window” is a sentence that gets a bug fixed. “82 per cent accuracy” is a sentence people nod at.

The baseline almost nobody builds

Before you report that your system scores 82 per cent, score the dumb version on the same forty cases. Keyword search over the documents. Or a fixed reply saying “contact support”. Or the previous process.

If plain keyword search scores 74, your honest headline is eight points above search, and that changes what the thing is worth. Reporting 82 with no baseline is not lying, but it lets everyone in the room assume the baseline was zero, which it never is.

Using a model as a judge, carefully

Grading by hand does not scale past a few hundred cases, so people have models grade outputs. It works, with two known biases: judges prefer longer answers, and they prefer text that reads like their own output. Both inflate scores in ways unrelated to correctness.

The discipline: hand-label about fifty cases, run the judge on the same fifty, and measure how often they agree. If agreement is poor, fix the rubric before you trust any large run. Re-check whenever the judge’s model version changes, because a provider updating a model silently changes your measuring instrument.

The free path

You do not need an evaluation platform. A CSV of cases, a script that runs each one, and a column for the verdict is a complete evaluation harness, and it is what most teams actually use for the first year.

The named paid products (LangSmith, Braintrust, Weights & Biases Weave) buy you tracing, dashboards and collaboration when many people grade together. The free equivalents are promptfoo (open source, config-file driven), the open evaluation harnesses on Hugging Face, or sixty lines of Python. Start with the sixty lines; you will understand what the product is doing when you eventually need one.

The limitation that catches everyone

An evaluation set built once goes stale. You fix the cases in it, the score climbs to 95 per cent, and the system is no better in production, because you have optimised against forty fixed examples rather than against reality. This is overfitting to your test set, and it happens quietly.

The fix is a refresh habit: every month, sample fresh cases from real traffic and add them, retiring some old ones. Keep a small held-out set you never look at while developing. And treat any score above about 95 per cent on a hand-built set as a sign that the set has become too easy, not that the work is done.