Running the standard harness yourself

Why run it at all

You have been told repeatedly in this course that public benchmarks are weak evidence for your task. They remain useful for two things: calibration, so you know whether the number a card reports is reproducible at all, and regression checking, so you know whether a quantisation, a fine-tune or a provider change broke something general.

The community standard is EleutherAI’s lm-evaluation-harness, which is what most reported numbers come from and what you should therefore run.

The basic run

pip install lm-eval

lm_eval --model hf   --model_args pretrained=Qwen/Qwen3-8B,revision=9a3f21c,dtype=bfloat16   --tasks arc_challenge,hellaswag,gsm8k   --num_fewshot 5   --batch_size auto   --output_path ./results   --log_samples

--log_samples writes every prompt and completion to disk. Turn it on always. The aggregate number tells you very little; reading twenty failures tells you what the model actually did.

For a quick check, --limit 200 runs a subset. Useful for iteration, not for a number you report.

Evaluating anything with an endpoint

The important capability, and the one people miss: the harness can run against any OpenAI-compatible server. That means you can evaluate a GGUF served by llama.cpp, a quantised model in vLLM, a fine-tune, or a rented endpoint: all with the identical harness:

lm_eval --model local-chat-completions   --model_args model=qwen3-8b,base_url=http://localhost:8080/v1,num_concurrent=4   --tasks gsm8k --apply_chat_template --num_fewshot 5

This is how you compare a quantisation against its reference, or a provider against another, without changing anything else in the pipeline.

--apply_chat_template matters. Benchmarks were historically run on base models with raw prompts. Running an instruct model without its template gives a lower number and running it with the template gives a different one; both appear in the wild. Whichever you choose, be consistent and say which you did.

Why your number differs from the paper’s

Expect a difference, and expect it to be a few points. The causes, in rough order of size:

• Shot count. Five-shot and zero-shot differ substantially.
• Chat template applied or not. Several points on instruct models.
• Answer extraction. How the harness pulls the answer out of free text: a regular expression, a letter match, a normalisation step. Task implementations vary and change between harness versions.
• Harness version. Task definitions get fixed. A score from last year’s version is not comparable with this year’s.
• Precision and quantisation. Small, but real.
• Batch size. Non-determinism from batched floating-point reductions occasionally flips a near-tie.

Which is why the discipline is to pin the harness commit alongside the model revision, and to report both.

Your own task, in the same harness

The harness accepts custom tasks as YAML, so your private evaluation can live next to the public ones and be run with one command:

task: support_triage
dataset_path: json
dataset_kwargs:
  data_files: {test: ./data/triage_test.jsonl}
output_type: generate_until
doc_to_text: "Ticket: {{ticket}}\nLabel:"
doc_to_target: "{{label}}"
generation_kwargs: {until: ["\n"], temperature: 0.0}
metric_list:
  - metric: exact_match

This is worth the hour it takes. It gives you one command that produces both the public numbers and yours, at the same settings, logged the same way, which is what makes a comparison across models honest rather than assembled.

What to actually run

For a model swap or a quantisation change, a compact set is enough: one general knowledge task, one reasoning task, one instruction-following task, one in each language you serve, and your own task. Six numbers, half an hour on a small GPU, and it catches the large regressions.

Do not run forty tasks. The time cost is real and the extra numbers do not change decisions, because you already know that the decision belongs to your own held-out set.