The evaluation that decides it

Why thirty, and why yours

A leaderboard narrows the field to three candidates. Nothing public can pick the winner, because no public benchmark contains your text, your languages, your formats or your users’ phrasing. The thing that decides it is a small set of your own examples, and it takes an afternoon to build.

Thirty is chosen deliberately. It is few enough to read every failure, which is where the information is, and enough to detect the differences worth acting on: a model that cannot handle your language, a configuration that broke structured output, a prompt change that lost a capability. It will not detect a two-point difference, and treating a 26-out-of-30 against a 27-out-of-30 as a result is how people convince themselves of noise.

Building it

{"id": "t001", "input": "मेरा ऑर्डर कैंसिल करना है", "expect_label": "cancellation", "note": "hindi"}
{"id": "t002", "input": "where's my stuff??", "expect_label": "delivery", "note": "informal"}
{"id": "t003", "input": "invoice INV-2024-8823 wrong amount", "expect_label": "billing", "note": "has id"}

Four rules for choosing the thirty:

1. Take them from real inputs, not invented ones. Invented examples are always cleaner than reality.
2. Cover the shape of your traffic, including the rare classes you care about. This is stratified sampling, and it means you cannot quote a single headline rate as if it were production performance.
3. Include the hard cases deliberately: the other language, the misspelling, the empty input, the extremely long input, the one with an identifier in it, the ambiguous one.
4. Include three where the right answer is “I do not know”, so you can see whether the model invents something.

Redact personal data before this file exists anywhere. It is a copy of real user text, and it will end up in a repository.

Scoring what can be scored

import evaluate, json

rows = [json.loads(l) for l in open("eval.jsonl")]
acc = evaluate.load("accuracy")

def run(model_fn):
    preds, refs, fails = [], [], []
    for r in rows:
        got = model_fn(r["input"])
        preds.append(got); refs.append(r["expect_label"])
        if got != r["expect_label"]:
            fails.append((r["id"], r["note"], r["input"][:60], got))
    return acc.compute(predictions=preds, references=refs), fails

More of your task is exactly scorable than you expect: is the JSON valid, is the label right, is the number correct, is the date in the right format, is the answer in the right language, is it under the length limit. Score all of those automatically and cheaply.

For open-ended output, read it. Thirty is a deliberate size because thirty is readable. A model scoring another model is a real technique with real biases (position effects, a preference for its own style, a preference for longer answers) and at thirty items it buys you nothing you cannot get by reading.

Comparing two models honestly

Run both on the identical set with identical prompts and identical generation settings. Differences of ten points or more at thirty items are worth acting on. Smaller ones are not.

Two confounders that ruin more comparisons than anything else:

• Different chat templates or prompts. You are then comparing prompts, not models.
• Inherited generation defaults. Module two’s point: two models can appear to differ in quality when their generation_config.json files differ in temperature.

And always include a baseline that is not a large model: a keyword rule, a small fine-tuned classifier, the current system. A great many projects discover that a 40-line rule scores within a few points of the model, which is a finding worth having in week one rather than month three.

The tools, when you want more than a script

evaluate loads standard metric implementations so everybody’s F1 is the same F1. lighteval runs standard benchmark suites against a model, locally or through a provider, if you want to reproduce a leaderboard number yourself rather than trust it, which is occasionally illuminating, because reproducing a published score is harder than it sounds and the gap is informative.

Neither replaces the thirty. They are for the questions that are about the field; the thirty is for the question that is about you.

Keep it, and grow it

Store the file in your repository, versioned. Add a row every time production surprises you: an input that produced a wrong answer becomes a permanent test. Over a year the set becomes a record of everything you have learned about your own problem, and it is the single most valuable artefact a small team builds.

When you change model, prompt, quantization or library version, run it. That is the whole discipline.

Do this now

Open a file and write ten rows from real inputs in the next fifteen minutes. Not thirty: ten. Run your current system against them and read every failure. You will find something in the first ten, and that is the point.