Evaluate a Model on Your Task in an Afternoon

Four hours, budgeted: sixty minutes building the dataset, thirty on the harness, ninety running candidates, sixty reading failures. The last hour is the one that pays.

1. The dataset is the actual work

Collect forty real inputs from your own system: logs, tickets, documents, transcripts. Not invented examples. Invented examples are always cleaner, better punctuated and more polite than what real users send, and they will rank models wrongly for exactly that reason.

For each one, write the expected output, or a one-line rubric where there is no single right answer.

Deliberately include: five hard cases, three things your current system got wrong in production, two inputs that should be refused or escalated, and two in your weakest language.

Split it: 15 dev examples you iterate prompts against, 25 held-out you touch exactly once at the end. If you tune on all forty, you have measured your own tuning.

2. One API, several backends

All of these speak the OpenAI chat format, so one script tests everything:

# llama.cpp on CPU or Apple Metal
llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M -c 8192 --port 8080

# Ollama
ollama serve & ollama pull gemma3:4b

# a GPU box, batched
vllm serve Qwen/Qwen3-8B --max-model-len 8192

# or rent, and download nothing
export BASE_URL=https://openrouter.ai/api/v1

Start with a hosted aggregator. You can compare six models in an hour without a single download, then pull only the winner locally. Downloading first is how the afternoon becomes a week.

3. Freeze the harness

Same prompt, temperature 0, same max_tokens, same system message, fixed seed where supported. One JSONL line per call, written to disk immediately:

import json, time, httpx

def run(model, item, out_file):
    t0 = time.time()
    r = httpx.post(f"{BASE}/chat/completions", timeout=120, json={
        "model": model, "temperature": 0, "max_tokens": 512,
        "messages": [{"role": "system", "content": SYS},
                     {"role": "user", "content": item["input"]}],
    }).json()
    row = {"model": model, "id": item["id"],
           "out": r["choices"][0]["message"]["content"],
           "ms": int((time.time() - t0) * 1000),
           "tok": r["usage"]["completion_tokens"]}
    with open(out_file, "a") as f:
        f.write(json.dumps(row) + "\n")
    return row

Never keep results only in memory. You will want to re-score them tomorrow with a different rubric.

4. Scoring, in order of preference

• Exact or normalised match where there is a right answer. Cheapest and most trustworthy.
• A rubric scored by a strong model, then hand-check twenty of its judgements yourself. If the judge disagrees with you more than twice in twenty, the rubric is broken, not the models.
• Pairwise comparison for writing. Show two anonymised outputs, pick one. Forty comparisons take an hour and tell you more than any score.

5. Know the noise floor before you celebrate

With forty examples, the 95% interval around 80% accuracy is roughly plus or minus twelve points. Comparing two models on forty examples, the interval on the difference is wider still.

So: a five-point gap on forty examples is nothing. Act on gaps larger than about ten points, or collect more examples. Write this sentence into your own report, because whoever reads it will otherwise treat 82.5% versus 77.5% as a decision.

6. Report in six lines per model

model | accuracy | p50 ms | p95 ms | tokens/req | licence | failure mode

The last column is the most valuable thing in the document. “Drops the currency symbol on European formats.” “Refuses anything mentioning self-harm, including the support cases.” “Fine until the document exceeds about 12k tokens, then invents section numbers.”

The part everyone skips

Sort failures by how bad they are and read twenty of them, output next to input. Most afternoons end with the same finding: the fix is your prompt, your chunking, or your output schema, not the model. That is not a wasted afternoon. That is the afternoon paying for itself, because you now know something no leaderboard could have told you.