{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"evaluate-in-an-afternoon","title":"Evaluate a Model on Your Task in an Afternoon","url":"https://addaly.com/learn/open-models/evaluate-in-an-afternoon","position":71,"of":71,"minutes":10,"answer":"Forty real examples and a fixed harness settle a model choice better than any leaderboard, in about four hours.","keyIdea":"Forty real examples and a fixed harness settle a model choice better than any leaderboard, in about four hours."},"body":{"format":"markdown","text":"Four hours, budgeted: sixty minutes building the dataset, thirty on the harness, ninety running candidates, sixty reading failures. The last hour is the one that pays.\n\n> Figure: Four hours, and the hour that pays. Timeline: 0:00, Collect forty real inputs from your own logs and write the expected output, or a one-line rubric, for each; 1:00, Freeze the harness: one prompt, temperature 0, one max_tokens, one JSONL line written to disk per call; 1:30, Run three to six candidates against a hosted aggregator, downloading nothing; 3:00, Sort the failures by how bad they are and read twenty of them, output beside input; 4:00, Six lines per model: accuracy, p50, p95, tokens per request, licence, failure mode. Fifteen of the forty examples are for iterating prompts and twenty-five are touched once at the end; tune on all forty and you have measured your own tuning. With forty examples a five-point gap is nothing, so act on gaps above about ten points or collect more examples.\n\n## 1. The dataset is the actual work\n\nCollect **forty real inputs** from your own system: logs, tickets, documents, transcripts. Not invented examples. Invented examples are always cleaner, better punctuated and more polite than what real users send, and they will rank models wrongly for exactly that reason.\n\nFor each one, write the expected output, or a one-line rubric where there is no single right answer.\n\nDeliberately include: five hard cases, three things your current system got wrong in production, two inputs that should be refused or escalated, and two in your weakest language.\n\nSplit it: **15 dev** examples you iterate prompts against, **25 held-out** you touch exactly once at the end. If you tune on all forty, you have measured your own tuning.\n\n## 2. One API, several backends\n\nAll of these speak the OpenAI chat format, so one script tests everything:\n\n```bash\n# llama.cpp on CPU or Apple Metal\nllama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M -c 8192 --port 8080\n\n# Ollama\nollama serve & ollama pull gemma3:4b\n\n# a GPU box, batched\nvllm serve Qwen/Qwen3-8B --max-model-len 8192\n\n# or rent, and download nothing\nexport BASE_URL=https://openrouter.ai/api/v1\n```\n\nStart with a hosted aggregator. You can compare six models in an hour without a single download, then pull only the winner locally. Downloading first is how the afternoon becomes a week.\n\n## 3. Freeze the harness\n\nSame prompt, temperature 0, same `max_tokens`, same system message, fixed seed where supported. One JSONL line per call, written to disk immediately:\n\n```python\nimport json, time, httpx\n\ndef run(model, item, out_file):\n    t0 = time.time()\n    r = httpx.post(f\"{BASE}/chat/completions\", timeout=120, json={\n        \"model\": model, \"temperature\": 0, \"max_tokens\": 512,\n        \"messages\": [{\"role\": \"system\", \"content\": SYS},\n                     {\"role\": \"user\", \"content\": item[\"input\"]}],\n    }).json()\n    row = {\"model\": model, \"id\": item[\"id\"],\n           \"out\": r[\"choices\"][0][\"message\"][\"content\"],\n           \"ms\": int((time.time() - t0) * 1000),\n           \"tok\": r[\"usage\"][\"completion_tokens\"]}\n    with open(out_file, \"a\") as f:\n        f.write(json.dumps(row) + \"\\n\")\n    return row\n```\n\nNever keep results only in memory. You will want to re-score them tomorrow with a different rubric.\n\n## 4. Scoring, in order of preference\n\n- **Exact or normalised match** where there is a right answer. Cheapest and most trustworthy.\n- **A rubric scored by a strong model**, then hand-check twenty of its judgements yourself. If the judge disagrees with you more than twice in twenty, the rubric is broken, not the models.\n- **Pairwise comparison** for writing. Show two anonymised outputs, pick one. Forty comparisons take an hour and tell you more than any score.\n\n## 5. Know the noise floor before you celebrate\n\nWith forty examples, the 95% interval around 80% accuracy is roughly plus or minus twelve points. Comparing two models on forty examples, the interval on the *difference* is wider still.\n\nSo: a five-point gap on forty examples is nothing. Act on gaps larger than about ten points, or collect more examples. Write this sentence into your own report, because whoever reads it will otherwise treat 82.5% versus 77.5% as a decision.\n\n## 6. Report in six lines per model\n\n```\nmodel | accuracy | p50 ms | p95 ms | tokens/req | licence | failure mode\n```\n\nThe last column is the most valuable thing in the document. \"Drops the currency symbol on European formats.\" \"Refuses anything mentioning self-harm, including the support cases.\" \"Fine until the document exceeds about 12k tokens, then invents section numbers.\"\n\n## The part everyone skips\n\nSort failures by how bad they are and read twenty of them, output next to input. Most afternoons end with the same finding: the fix is your prompt, your chunking, or your output schema, not the model. That is not a wasted afternoon. That is the afternoon paying for itself, because you now know something no leaderboard could have told you."},"check":{"question":"On 25 held-out examples plus 15 dev, Model A scores 82.5% and Model B 77.5%. A costs three times as much and is twice as slow. What is the best next step?","options":[{"text":"Treat the gap as unresolved: it sits inside the noise at this sample size, so add examples or read the failures before paying three times more.","correct":true},{"text":"Ship A. Five points is five points, and accuracy matters more than cost in almost every product.","correct":false,"tempting":"Treats a point estimate as a measurement, ignoring how wide the interval is at forty examples."},{"text":"Ship B and close the gap with prompt work, since a better prompt is usually worth about five points.","correct":false,"tempting":"Uses a plausible rule of thumb to skip the statistics; the gap may not exist to close."},{"text":"Rerun both at temperature 0.7 three times and take each model's best run to break the tie.","correct":false,"tempting":"Mines variance instead of reducing it, which is exactly the aggregation trick that inflates published benchmarks."}]},"prev":{"slug":"swapping-the-model","title":"Swapping the model under a live product","url":"https://addaly.com/learn/open-models/swapping-the-model"},"next":null,"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Evaluate a Model on Your Task in an Afternoon', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/evaluate-in-an-afternoon (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Evaluate a Model on Your Task in an Afternoon","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Forty real examples and a fixed harness settle a model choice better than any leaderboard, in about four hours.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"Four hours, and the hour that pays","events":[{"when":"0:00","what":"Collect forty real inputs from your own logs and write the expected output, or a one-line rubric, for each"},{"when":"1:00","what":"Freeze the harness: one prompt, temperature 0, one max_tokens, one JSONL line written to disk per call"},{"when":"1:30","what":"Run three to six candidates against a hosted aggregator, downloading nothing"},{"when":"3:00","what":"Sort the failures by how bad they are and read twenty of them, output beside input"},{"when":"4:00","what":"Six lines per model: accuracy, p50, p95, tokens per request, licence, failure mode"}],"caption":"Fifteen of the forty examples are for iterating prompts and twenty-five are touched once at the end; tune on all forty and you have measured your own tuning. With forty examples a five-point gap is nothing, so act on gaps above about ten points or collect more examples."},"alt":"Figure: Four hours, and the hour that pays. Timeline: 0:00, Collect forty real inputs from your own logs and write the expected output, or a one-line rubric, for each; 1:00, Freeze the harness: one prompt, temperature 0, one max_tokens, one JSONL line written to disk per call; 1:30, Run three to six candidates against a hosted aggregator, downloading nothing; 3:00, Sort the failures by how bad they are and read twenty of them, output beside input; 4:00, Six lines per model: accuracy, p50, p95, tokens per request, licence, failure mode. Fifteen of the forty examples are for iterating prompts and twenty-five are touched once at the end; tune on all forty and you have measured your own tuning. With forty examples a five-point gap is nothing, so act on gaps above about ten points or collect more examples."}},"blocks":[{"t":"p","runs":[{"text":"Four hours, budgeted: sixty minutes building the dataset, thirty on the harness, ninety running candidates, sixty reading failures. The last hour is the one that pays."}],"lede":true},{"t":"h2","id":"1-the-dataset-is-the-actual-work","runs":[{"text":"1. The dataset is the actual work"}]},{"t":"p","runs":[{"text":"Collect "},{"text":"forty real inputs","b":true},{"text":" from your own system: logs, tickets, documents, transcripts. Not invented examples. Invented examples are always cleaner, better punctuated and more polite than what real users send, and they will rank models wrongly for exactly that reason."}]},{"t":"p","runs":[{"text":"For each one, write the expected output, or a one-line rubric where there is no single right answer."}]},{"t":"p","runs":[{"text":"Deliberately include: five hard cases, three things your current system got wrong in production, two inputs that should be refused or escalated, and two in your weakest language."}]},{"t":"p","runs":[{"text":"Split it: "},{"text":"15 dev","b":true},{"text":" examples you iterate prompts against, "},{"text":"25 held-out","b":true},{"text":" you touch exactly once at the end. If you tune on all forty, you have measured your own tuning."}]},{"t":"h2","id":"2-one-api-several-backends","runs":[{"text":"2. One API, several backends"}]},{"t":"p","runs":[{"text":"All of these speak the OpenAI chat format, so one script tests everything:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# llama.cpp on CPU or Apple Metal\nllama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M -c 8192 --port 8080\n\n# Ollama\nollama serve & ollama pull gemma3:4b\n\n# a GPU box, batched\nvllm serve Qwen/Qwen3-8B --max-model-len 8192\n\n# or rent, and download nothing\nexport BASE_URL=https://openrouter.ai/api/v1"},{"t":"p","runs":[{"text":"Start with a hosted aggregator. You can compare six models in an hour without a single download, then pull only the winner locally. Downloading first is how the afternoon becomes a week."}]},{"t":"h2","id":"3-freeze-the-harness","runs":[{"text":"3. Freeze the harness"}]},{"t":"p","runs":[{"text":"Same prompt, temperature 0, same "},{"text":"max_tokens","code":true},{"text":", same system message, fixed seed where supported. One JSONL line per call, written to disk immediately:"}]},{"t":"code","lang":"python","label":"Python","text":"import json, time, httpx\n\ndef run(model, item, out_file):\n    t0 = time.time()\n    r = httpx.post(f\"{BASE}/chat/completions\", timeout=120, json={\n        \"model\": model, \"temperature\": 0, \"max_tokens\": 512,\n        \"messages\": [{\"role\": \"system\", \"content\": SYS},\n                     {\"role\": \"user\", \"content\": item[\"input\"]}],\n    }).json()\n    row = {\"model\": model, \"id\": item[\"id\"],\n           \"out\": r[\"choices\"][0][\"message\"][\"content\"],\n           \"ms\": int((time.time() - t0) * 1000),\n           \"tok\": r[\"usage\"][\"completion_tokens\"]}\n    with open(out_file, \"a\") as f:\n        f.write(json.dumps(row) + \"\\n\")\n    return row"},{"t":"p","runs":[{"text":"Never keep results only in memory. You will want to re-score them tomorrow with a different rubric."}]},{"t":"h2","id":"4-scoring-in-order-of-preference","runs":[{"text":"4. Scoring, in order of preference"}]},{"t":"ul","items":[[{"text":"Exact or normalised match","b":true},{"text":" where there is a right answer. Cheapest and most trustworthy."}],[{"text":"A rubric scored by a strong model","b":true},{"text":", then hand-check twenty of its judgements yourself. If the judge disagrees with you more than twice in twenty, the rubric is broken, not the models."}],[{"text":"Pairwise comparison","b":true},{"text":" for writing. Show two anonymised outputs, pick one. Forty comparisons take an hour and tell you more than any score."}]]},{"t":"h2","id":"5-know-the-noise-floor-before-you-celebrate","runs":[{"text":"5. Know the noise floor before you celebrate"}]},{"t":"p","runs":[{"text":"With forty examples, the 95% interval around 80% accuracy is roughly plus or minus twelve points. Comparing two models on forty examples, the interval on the "},{"text":"difference","i":true},{"text":" is wider still."}]},{"t":"p","runs":[{"text":"So: a five-point gap on forty examples is nothing. Act on gaps larger than about ten points, or collect more examples. Write this sentence into your own report, because whoever reads it will otherwise treat 82.5% versus 77.5% as a decision."}]},{"t":"h2","id":"6-report-in-six-lines-per-model","runs":[{"text":"6. Report in six lines per model"}]},{"t":"code","lang":"","label":"Text","text":"model | accuracy | p50 ms | p95 ms | tokens/req | licence | failure mode"},{"t":"p","runs":[{"text":"The last column is the most valuable thing in the document. “Drops the currency symbol on European formats.” “Refuses anything mentioning self-harm, including the support cases.” “Fine until the document exceeds about 12k tokens, then invents section numbers.”"}]},{"t":"h2","id":"the-part-everyone-skips","runs":[{"text":"The part everyone skips"}]},{"t":"p","runs":[{"text":"Sort failures by how bad they are and read twenty of them, output next to input. Most afternoons end with the same finding: the fix is your prompt, your chunking, or your output schema, not the model. That is not a wasted afternoon. That is the afternoon paying for itself, because you now know something no leaderboard could have told you."}]}],"check":{"question":[{"text":"On 25 held-out examples plus 15 dev, Model A scores 82.5% and Model B 77.5%. A costs three times as much and is twice as slow. What is the best next step?"}],"options":[{"text":[{"text":"Treat the gap as unresolved: it sits inside the noise at this sample size, so add examples or read the failures before paying three times more."}],"correct":true,"tempting":null},{"text":[{"text":"Ship A. Five points is five points, and accuracy matters more than cost in almost every product."}],"correct":false,"tempting":[{"text":"Treats a point estimate as a measurement, ignoring how wide the interval is at forty examples."}]},{"text":[{"text":"Ship B and close the gap with prompt work, since a better prompt is usually worth about five points."}],"correct":false,"tempting":[{"text":"Uses a plausible rule of thumb to skip the statistics; the gap may not exist to close."}]},{"text":[{"text":"Rerun both at temperature 0.7 three times and take each model’s best run to break the tie."}],"correct":false,"tempting":[{"text":"Mines variance instead of reducing it, which is exactly the aggregation trick that inflates published benchmarks."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"swapping-the-model","title":"Swapping the model under a live product","minutes":9},"next":null}}