{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"running-the-eval-harness","title":"Running the standard harness yourself","url":"https://addaly.com/learn/open-models/running-the-eval-harness","position":65,"of":71,"minutes":9,"answer":"Running lm-evaluation-harness yourself against any OpenAI-compatible endpoint is how you calibrate published claims and catch regressions, and your number will differ from a paper's mostly through shot count, chat template and harness version.","keyIdea":"Running lm-evaluation-harness yourself against any OpenAI-compatible endpoint is how you calibrate published claims and catch regressions, and your number will differ from a paper's mostly through shot count, chat template and harness version, so pin the commit and log the samples."},"body":{"format":"markdown","text":"## Why run it at all\n\nYou have been told repeatedly in this course that public benchmarks are weak evidence for your task. They remain useful for two things: **calibration**, so you know whether the number a card reports is reproducible at all, and **regression checking**, so you know whether a quantisation, a fine-tune or a provider change broke something general.\n\nThe community standard is EleutherAI's `lm-evaluation-harness`, which is what most reported numbers come from and what you should therefore run.\n\n## The basic run\n\n```bash\npip install lm-eval\n\nlm_eval --model hf   --model_args pretrained=Qwen/Qwen3-8B,revision=9a3f21c,dtype=bfloat16   --tasks arc_challenge,hellaswag,gsm8k   --num_fewshot 5   --batch_size auto   --output_path ./results   --log_samples\n```\n\n`--log_samples` writes every prompt and completion to disk. Turn it on always. The aggregate number tells you very little; reading twenty failures tells you what the model actually did.\n\nFor a quick check, `--limit 200` runs a subset. Useful for iteration, not for a number you report.\n\n## Evaluating anything with an endpoint\n\nThe important capability, and the one people miss: the harness can run against any OpenAI-compatible server. That means you can evaluate a GGUF served by llama.cpp, a quantised model in vLLM, a fine-tune, or a rented endpoint: all with the identical harness:\n\n```bash\nlm_eval --model local-chat-completions   --model_args model=qwen3-8b,base_url=http://localhost:8080/v1,num_concurrent=4   --tasks gsm8k --apply_chat_template --num_fewshot 5\n```\n\nThis is how you compare a quantisation against its reference, or a provider against another, without changing anything else in the pipeline.\n\n`--apply_chat_template` matters. Benchmarks were historically run on base models with raw prompts. Running an instruct model without its template gives a lower number and running it with the template gives a different one; both appear in the wild. Whichever you choose, be consistent and say which you did.\n\n## Why your number differs from the paper's\n\nExpect a difference, and expect it to be a few points. The causes, in rough order of size:\n\n- **Shot count.** Five-shot and zero-shot differ substantially.\n- **Chat template applied or not.** Several points on instruct models.\n- **Answer extraction.** How the harness pulls the answer out of free text: a regular expression, a letter match, a normalisation step. Task implementations vary and change between harness versions.\n- **Harness version.** Task definitions get fixed. A score from last year's version is not comparable with this year's.\n- **Precision and quantisation.** Small, but real.\n- **Batch size.** Non-determinism from batched floating-point reductions occasionally flips a near-tie.\n\nWhich is why the discipline is to **pin the harness commit** alongside the model revision, and to report both.\n\n## Your own task, in the same harness\n\nThe harness accepts custom tasks as YAML, so your private evaluation can live next to the public ones and be run with one command:\n\n```yaml\ntask: support_triage\ndataset_path: json\ndataset_kwargs:\n  data_files: {test: ./data/triage_test.jsonl}\noutput_type: generate_until\ndoc_to_text: \"Ticket: {{ticket}}\\nLabel:\"\ndoc_to_target: \"{{label}}\"\ngeneration_kwargs: {until: [\"\\n\"], temperature: 0.0}\nmetric_list:\n  - metric: exact_match\n```\n\nThis is worth the hour it takes. It gives you one command that produces both the public numbers and yours, at the same settings, logged the same way, which is what makes a comparison across models honest rather than assembled.\n\n## What to actually run\n\nFor a model swap or a quantisation change, a compact set is enough: one general knowledge task, one reasoning task, one instruction-following task, one in each language you serve, and your own task. Six numbers, half an hour on a small GPU, and it catches the large regressions.\n\nDo not run forty tasks. The time cost is real and the extra numbers do not change decisions, because you already know that the decision belongs to your own held-out set."},"check":{"question":"A team's MMLU score for an instruct model comes out four points below the published figure, using the same harness and the same shot count. What is the most likely cause?","options":[{"text":"Their download is a different quantisation from the one the publisher evaluated.","correct":false,"tempting":"Quantisation does cost something, but rarely four points on MMLU, and the team would usually know which file they loaded."},{"text":"The harness is sampling rather than using greedy decoding, adding variance.","correct":false,"tempting":"Sampling adds variance around the mean rather than a consistent four-point deficit."},{"text":"One run applied the model's chat template and the other did not, which shifts instruct-model scores by several points.","correct":true},{"text":"MMLU contamination affects the publisher's copy of the model but not theirs.","correct":false,"tempting":"Contamination is a property of the training data, so it is identical in both copies of the same weights."}]},"prev":{"slug":"leaderboards-that-still-work","title":"The leaderboards that still carry information","url":"https://addaly.com/learn/open-models/leaderboards-that-still-work"},"next":{"slug":"comparing-quantisations","title":"Proving what a quantisation cost","url":"https://addaly.com/learn/open-models/comparing-quantisations"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Running the standard harness yourself', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/running-the-eval-harness (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Running the standard harness yourself","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Running lm-evaluation-harness yourself against any OpenAI-compatible endpoint is how you calibrate published claims and catch regressions, and your number will differ from a paper’s mostly through shot count, chat template and harness version, so pin the commit and log the samples.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-run-it-at-all","runs":[{"text":"Why run it at all"}]},{"t":"p","runs":[{"text":"You have been told repeatedly in this course that public benchmarks are weak evidence for your task. They remain useful for two things: "},{"text":"calibration","b":true},{"text":", so you know whether the number a card reports is reproducible at all, and "},{"text":"regression checking","b":true},{"text":", so you know whether a quantisation, a fine-tune or a provider change broke something general."}]},{"t":"p","runs":[{"text":"The community standard is EleutherAI’s "},{"text":"lm-evaluation-harness","code":true},{"text":", which is what most reported numbers come from and what you should therefore run."}]},{"t":"h2","id":"the-basic-run","runs":[{"text":"The basic run"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install lm-eval\n\nlm_eval --model hf   --model_args pretrained=Qwen/Qwen3-8B,revision=9a3f21c,dtype=bfloat16   --tasks arc_challenge,hellaswag,gsm8k   --num_fewshot 5   --batch_size auto   --output_path ./results   --log_samples"},{"t":"p","runs":[{"text":"--log_samples","code":true},{"text":" writes every prompt and completion to disk. Turn it on always. The aggregate number tells you very little; reading twenty failures tells you what the model actually did."}]},{"t":"p","runs":[{"text":"For a quick check, "},{"text":"--limit 200","code":true},{"text":" runs a subset. Useful for iteration, not for a number you report."}]},{"t":"h2","id":"evaluating-anything-with-an-endpoint","runs":[{"text":"Evaluating anything with an endpoint"}]},{"t":"p","runs":[{"text":"The important capability, and the one people miss: the harness can run against any OpenAI-compatible server. That means you can evaluate a GGUF served by llama.cpp, a quantised model in vLLM, a fine-tune, or a rented endpoint: all with the identical harness:"}]},{"t":"code","lang":"bash","label":"Bash","text":"lm_eval --model local-chat-completions   --model_args model=qwen3-8b,base_url=http://localhost:8080/v1,num_concurrent=4   --tasks gsm8k --apply_chat_template --num_fewshot 5"},{"t":"p","runs":[{"text":"This is how you compare a quantisation against its reference, or a provider against another, without changing anything else in the pipeline."}]},{"t":"p","runs":[{"text":"--apply_chat_template","code":true},{"text":" matters. Benchmarks were historically run on base models with raw prompts. Running an instruct model without its template gives a lower number and running it with the template gives a different one; both appear in the wild. Whichever you choose, be consistent and say which you did."}]},{"t":"h2","id":"why-your-number-differs-from-the-paper-s","runs":[{"text":"Why your number differs from the paper’s"}]},{"t":"p","runs":[{"text":"Expect a difference, and expect it to be a few points. The causes, in rough order of size:"}]},{"t":"ul","items":[[{"text":"Shot count.","b":true},{"text":" Five-shot and zero-shot differ substantially."}],[{"text":"Chat template applied or not.","b":true},{"text":" Several points on instruct models."}],[{"text":"Answer extraction.","b":true},{"text":" How the harness pulls the answer out of free text: a regular expression, a letter match, a normalisation step. Task implementations vary and change between harness versions."}],[{"text":"Harness version.","b":true},{"text":" Task definitions get fixed. A score from last year’s version is not comparable with this year’s."}],[{"text":"Precision and quantisation.","b":true},{"text":" Small, but real."}],[{"text":"Batch size.","b":true},{"text":" Non-determinism from batched floating-point reductions occasionally flips a near-tie."}]]},{"t":"p","runs":[{"text":"Which is why the discipline is to "},{"text":"pin the harness commit","b":true},{"text":" alongside the model revision, and to report both."}]},{"t":"h2","id":"your-own-task-in-the-same-harness","runs":[{"text":"Your own task, in the same harness"}]},{"t":"p","runs":[{"text":"The harness accepts custom tasks as YAML, so your private evaluation can live next to the public ones and be run with one command:"}]},{"t":"code","lang":"yaml","label":"YAML","text":"task: support_triage\ndataset_path: json\ndataset_kwargs:\n  data_files: {test: ./data/triage_test.jsonl}\noutput_type: generate_until\ndoc_to_text: \"Ticket: {{ticket}}\\nLabel:\"\ndoc_to_target: \"{{label}}\"\ngeneration_kwargs: {until: [\"\\n\"], temperature: 0.0}\nmetric_list:\n  - metric: exact_match"},{"t":"p","runs":[{"text":"This is worth the hour it takes. It gives you one command that produces both the public numbers and yours, at the same settings, logged the same way, which is what makes a comparison across models honest rather than assembled."}]},{"t":"h2","id":"what-to-actually-run","runs":[{"text":"What to actually run"}]},{"t":"p","runs":[{"text":"For a model swap or a quantisation change, a compact set is enough: one general knowledge task, one reasoning task, one instruction-following task, one in each language you serve, and your own task. Six numbers, half an hour on a small GPU, and it catches the large regressions."}]},{"t":"p","runs":[{"text":"Do not run forty tasks. The time cost is real and the extra numbers do not change decisions, because you already know that the decision belongs to your own held-out set."}]}],"check":{"question":[{"text":"A team’s MMLU score for an instruct model comes out four points below the published figure, using the same harness and the same shot count. What is the most likely cause?"}],"options":[{"text":[{"text":"Their download is a different quantisation from the one the publisher evaluated."}],"correct":false,"tempting":[{"text":"Quantisation does cost something, but rarely four points on MMLU, and the team would usually know which file they loaded."}]},{"text":[{"text":"The harness is sampling rather than using greedy decoding, adding variance."}],"correct":false,"tempting":[{"text":"Sampling adds variance around the mean rather than a consistent four-point deficit."}]},{"text":[{"text":"One run applied the model’s chat template and the other did not, which shifts instruct-model scores by several points."}],"correct":true,"tempting":null},{"text":[{"text":"MMLU contamination affects the publisher’s copy of the model but not theirs."}],"correct":false,"tempting":[{"text":"Contamination is a property of the training data, so it is identical in both copies of the same weights."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"leaderboards-that-still-work","title":"The leaderboards that still carry information","minutes":9},"next":{"slug":"comparing-quantisations","title":"Proving what a quantisation cost","minutes":9}}}