{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"a-local-eval-harness","title":"A small harness that tells you whether a change helped","url":"https://addaly.com/learn/running-models-yourself/a-local-eval-harness","position":55,"of":82,"minutes":9,"answer":"Thirty frozen items with structural and property checks, run at temperature 0 and logged with latency and token counts, converts every question in this course from an argument into a ten-minute run.","keyIdea":"Thirty frozen items with structural and property checks, run at temperature 0 and logged with latency and token counts, converts every question in this course from an argument into a ten-minute run."},"body":{"format":"markdown","text":"## Impressions do not survive a week\n\nEvery lesson in this block ends with \"measure it\". This one is the measuring apparatus: thirty items, a script, and a file of results. It takes an afternoon to build and it is the difference between knowing whether the new model is better and believing it is.\n\nWithout it, every question in this course becomes an argument: is Q4 worse than Q5, did the prompt rewrite help, is the new release better, did the fine-tune work. With it, each is a ten-minute run.\n\n## The shape\n\nA JSON Lines file, one item per line:\n\n```json\n{\"id\": \"ext-001\", \"kind\": \"extract\", \"prompt\": \"...\", \"expect\": {\"total\": 4500}}\n{\"id\": \"cls-014\", \"kind\": \"classify\", \"prompt\": \"...\", \"expect\": \"billing\"}\n{\"id\": \"sum-007\", \"kind\": \"summary\", \"prompt\": \"...\", \"must_mention\": [\"refund\", \"14 days\"]}\n```\n\nThree kinds of check, in descending order of how much you should rely on them:\n\n**Exact or structural.** Does the JSON parse, does the field equal the expected value, does the code compile, does the label match. Objective, free to run, and where most of your items should live.\n\n**Property-based.** Does the summary mention these two facts, is the output under 100 words, does it avoid a forbidden phrase, is the number in range. Cheap, robust to wording, and the right tool for prose.\n\n**Human graded.** Three-point scale: usable, usable after a small edit, not usable. Reserve for the items where the first two cannot say anything, because this is the expensive one.\n\nResist the urge to use a model to grade the outputs at this scale. With thirty items you can read them yourself in fifteen minutes, and a judge model on a local setup adds a second unreliable component to a measurement you are trying to make trustworthy.\n\n## The runner\n\n```python\nimport json, time\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"x\")\nCONFIG = {\"model\": \"local\", \"temperature\": 0, \"max_tokens\": 512}\n\nrows = [json.loads(l) for l in open(\"evalset.jsonl\")]\nout = []\nfor r in rows:\n    t0 = time.time()\n    resp = client.chat.completions.create(\n        messages=[{\"role\": \"user\", \"content\": r[\"prompt\"]}], **CONFIG)\n    out.append({**r,\n                \"got\": resp.choices[0].message.content,\n                \"secs\": round(time.time() - t0, 2),\n                \"in_tok\": resp.usage.prompt_tokens,\n                \"out_tok\": resp.usage.completion_tokens})\n\njson.dump(out, open(\"run-2026-09-08-q4.json\", \"w\"), indent=1)\n```\n\nThirty lines including the scorer. Name the output file after the configuration, because in three weeks you will not remember which run was which.\n\nRecord latency and token counts alongside quality. Half the decisions in this course are quality-against-speed, and a harness that measures only one of them cannot answer them.\n\n## Reading a small sample honestly\n\nThirty items is a small number and it is worth being clear about what it can detect.\n\nA change of one or two items is noise. A change of eight is real. In between, you have not learned much; and the honest response is either to add items or to accept that the difference is below what you can measure and choose on another basis, such as speed or memory.\n\nWhen two configurations come out within a couple of items of each other, that is a genuine result: it says the difference does not matter for your work. Take the cheaper one.\n\n## Keeping it useful\n\n**Add every real failure.** When something goes wrong in production, that input becomes item thirty-one. The set grows in exactly the places where your system is weak, which is the correct growth pattern.\n\n**Freeze the items.** Do not edit an item because a model failed it. If the expected answer was wrong, fix it and note that you did; if the model was wrong, leave it.\n\n**Never tune against the whole set.** Keep ten items you do not look at while iterating. Otherwise you will optimise a prompt for thirty specific inputs and learn nothing about the thirty-first.\n\n**Re-run after every engine or driver upgrade.** Regressions happen, and this is how you find out in ten minutes rather than through a user report a fortnight later.\n\n## What this connects to\n\nThis is the smallest useful version of a much larger subject (evaluation design, statistical significance, judge models, per-group reporting) which the evaluation course on this site covers properly. What matters here is that you have *something*, run consistently, written down. Thirty items and a text file beat any amount of careful reasoning about which quant ought to be better."},"check":{"question":"A team's 30-item harness scores 24 for their current setup and 26 for a new quantisation. They conclude the new quant is better and roll it out. What is wrong with that reasoning?","options":[{"text":"The comparison is invalid unless both runs used the same seed, since sampling randomness dominates at this sample size.","correct":false,"tempting":"Reaches for a seed when a harness of this kind runs at temperature 0, where the sampler is not the source of variation."},{"text":"A two-item difference on thirty items is within noise, so the run supports at most that the two are indistinguishable; the decision should turn on speed, memory or another measurable difference.","correct":true},{"text":"Thirty items is never enough to compare quantisations, and a set of at least 300 is required before any conclusion.","correct":false,"tempting":"Overstates the limit into a rule; thirty items can detect a large difference perfectly well, and it is the size of this difference that is the problem."},{"text":"The harness should have used a judge model, since human-defined expectations cannot capture quality differences between quants.","correct":false,"tempting":"Adds an unreliable component to fix a sample-size problem, and structural checks are precisely what detects quantisation damage best."}]},"prev":{"slug":"debugging-a-bad-output","title":"The diagnostic ladder for a bad output","url":"https://addaly.com/learn/running-models-yourself/debugging-a-bad-output"},"next":{"slug":"serving-it-to-others","title":"Serving it to other people","url":"https://addaly.com/learn/running-models-yourself/serving-it-to-others"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A small harness that tells you whether a change helped', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/a-local-eval-harness (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A small harness that tells you whether a change helped","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Thirty frozen items with structural and property checks, run at temperature 0 and logged with latency and token counts, converts every question in this course from an argument into a ten-minute run.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"impressions-do-not-survive-a-week","runs":[{"text":"Impressions do not survive a week"}]},{"t":"p","runs":[{"text":"Every lesson in this block ends with “measure it”. This one is the measuring apparatus: thirty items, a script, and a file of results. It takes an afternoon to build and it is the difference between knowing whether the new model is better and believing it is."}]},{"t":"p","runs":[{"text":"Without it, every question in this course becomes an argument: is Q4 worse than Q5, did the prompt rewrite help, is the new release better, did the fine-tune work. With it, each is a ten-minute run."}]},{"t":"h2","id":"the-shape","runs":[{"text":"The shape"}]},{"t":"p","runs":[{"text":"A JSON Lines file, one item per line:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"id\": \"ext-001\", \"kind\": \"extract\", \"prompt\": \"...\", \"expect\": {\"total\": 4500}}\n{\"id\": \"cls-014\", \"kind\": \"classify\", \"prompt\": \"...\", \"expect\": \"billing\"}\n{\"id\": \"sum-007\", \"kind\": \"summary\", \"prompt\": \"...\", \"must_mention\": [\"refund\", \"14 days\"]}"},{"t":"p","runs":[{"text":"Three kinds of check, in descending order of how much you should rely on them:"}]},{"t":"p","runs":[{"text":"Exact or structural.","b":true},{"text":" Does the JSON parse, does the field equal the expected value, does the code compile, does the label match. Objective, free to run, and where most of your items should live."}]},{"t":"p","runs":[{"text":"Property-based.","b":true},{"text":" Does the summary mention these two facts, is the output under 100 words, does it avoid a forbidden phrase, is the number in range. Cheap, robust to wording, and the right tool for prose."}]},{"t":"p","runs":[{"text":"Human graded.","b":true},{"text":" Three-point scale: usable, usable after a small edit, not usable. Reserve for the items where the first two cannot say anything, because this is the expensive one."}]},{"t":"p","runs":[{"text":"Resist the urge to use a model to grade the outputs at this scale. With thirty items you can read them yourself in fifteen minutes, and a judge model on a local setup adds a second unreliable component to a measurement you are trying to make trustworthy."}]},{"t":"h2","id":"the-runner","runs":[{"text":"The runner"}]},{"t":"code","lang":"python","label":"Python","text":"import json, time\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"x\")\nCONFIG = {\"model\": \"local\", \"temperature\": 0, \"max_tokens\": 512}\n\nrows = [json.loads(l) for l in open(\"evalset.jsonl\")]\nout = []\nfor r in rows:\n    t0 = time.time()\n    resp = client.chat.completions.create(\n        messages=[{\"role\": \"user\", \"content\": r[\"prompt\"]}], **CONFIG)\n    out.append({**r,\n                \"got\": resp.choices[0].message.content,\n                \"secs\": round(time.time() - t0, 2),\n                \"in_tok\": resp.usage.prompt_tokens,\n                \"out_tok\": resp.usage.completion_tokens})\n\njson.dump(out, open(\"run-2026-09-08-q4.json\", \"w\"), indent=1)"},{"t":"p","runs":[{"text":"Thirty lines including the scorer. Name the output file after the configuration, because in three weeks you will not remember which run was which."}]},{"t":"p","runs":[{"text":"Record latency and token counts alongside quality. Half the decisions in this course are quality-against-speed, and a harness that measures only one of them cannot answer them."}]},{"t":"h2","id":"reading-a-small-sample-honestly","runs":[{"text":"Reading a small sample honestly"}]},{"t":"p","runs":[{"text":"Thirty items is a small number and it is worth being clear about what it can detect."}]},{"t":"p","runs":[{"text":"A change of one or two items is noise. A change of eight is real. In between, you have not learned much; and the honest response is either to add items or to accept that the difference is below what you can measure and choose on another basis, such as speed or memory."}]},{"t":"p","runs":[{"text":"When two configurations come out within a couple of items of each other, that is a genuine result: it says the difference does not matter for your work. Take the cheaper one."}]},{"t":"h2","id":"keeping-it-useful","runs":[{"text":"Keeping it useful"}]},{"t":"p","runs":[{"text":"Add every real failure.","b":true},{"text":" When something goes wrong in production, that input becomes item thirty-one. The set grows in exactly the places where your system is weak, which is the correct growth pattern."}]},{"t":"p","runs":[{"text":"Freeze the items.","b":true},{"text":" Do not edit an item because a model failed it. If the expected answer was wrong, fix it and note that you did; if the model was wrong, leave it."}]},{"t":"p","runs":[{"text":"Never tune against the whole set.","b":true},{"text":" Keep ten items you do not look at while iterating. Otherwise you will optimise a prompt for thirty specific inputs and learn nothing about the thirty-first."}]},{"t":"p","runs":[{"text":"Re-run after every engine or driver upgrade.","b":true},{"text":" Regressions happen, and this is how you find out in ten minutes rather than through a user report a fortnight later."}]},{"t":"h2","id":"what-this-connects-to","runs":[{"text":"What this connects to"}]},{"t":"p","runs":[{"text":"This is the smallest useful version of a much larger subject (evaluation design, statistical significance, judge models, per-group reporting) which the evaluation course on this site covers properly. What matters here is that you have "},{"text":"something","i":true},{"text":", run consistently, written down. Thirty items and a text file beat any amount of careful reasoning about which quant ought to be better."}]}],"check":{"question":[{"text":"A team’s 30-item harness scores 24 for their current setup and 26 for a new quantisation. They conclude the new quant is better and roll it out. What is wrong with that reasoning?"}],"options":[{"text":[{"text":"The comparison is invalid unless both runs used the same seed, since sampling randomness dominates at this sample size."}],"correct":false,"tempting":[{"text":"Reaches for a seed when a harness of this kind runs at temperature 0, where the sampler is not the source of variation."}]},{"text":[{"text":"A two-item difference on thirty items is within noise, so the run supports at most that the two are indistinguishable; the decision should turn on speed, memory or another measurable difference."}],"correct":true,"tempting":null},{"text":[{"text":"Thirty items is never enough to compare quantisations, and a set of at least 300 is required before any conclusion."}],"correct":false,"tempting":[{"text":"Overstates the limit into a rule; thirty items can detect a large difference perfectly well, and it is the size of this difference that is the problem."}]},{"text":[{"text":"The harness should have used a judge model, since human-defined expectations cannot capture quality differences between quants."}],"correct":false,"tempting":[{"text":"Adds an unreliable component to fix a sample-size problem, and structural checks are precisely what detects quantisation damage best."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"debugging-a-bad-output","title":"The diagnostic ladder for a bad output","minutes":9},"next":{"slug":"serving-it-to-others","title":"Serving it to other people","minutes":9}}}