{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"judging-on-a-budget","title":"What judging costs, and how to pay a tenth of it","url":"https://addaly.com/learn/evaluating-ai/judging-on-a-budget","position":47,"of":76,"minutes":9,"answer":"Cascade the checks (free code first, a small or local judge next, the expensive judge only on the uncertain remainder) and cache by output hash, because a rerun that changed no outputs should cost nothing.","keyIdea":"Cascade the checks (free code first, a small or local judge next, the expensive judge only on the uncertain remainder) and cache by output hash, because a rerun that changed no outputs should cost nothing."},"body":{"format":"markdown","text":"## Do the arithmetic once\n\nA realistic harness: 2,000 items, four rubric checks each, run in both orders for the pairwise part. That is 16,000 judge calls. At roughly 1,500 tokens per call, 24 million tokens per run.\n\nWhatever today's prices are, the ratio between a small model and a frontier one is roughly ten to one, so the same run costs a few dollars or several dozen. Run it on every pull request, twenty times a day, and the annual figure is the kind that appears in a cost review with your name beside it. Teams do not usually abandon evaluation because it stopped working; they abandon it because it became the third-largest line in the bill and nobody could say what it bought.\n\n## The cascade\n\nOrder the checks by cost and stop as early as possible.\n\n1. **Deterministic code.** Schema validity, forbidden strings, numbers grounded in the source, citations resolving, length bands. Free, instant, and on a typical set this decides 20 to 40% of items outright.\n2. **A small or local judge** for the rule-shaped checks. An 8-billion-parameter open-weights model through Ollama or llama.cpp runs on a laptop, costs nothing per call, and is entirely adequate for questions like \"does this reply state a deadline that is absent from the policy\".\n3. **The expensive judge**, only for items where the cheap layer is uncertain, or where the rubric requires nuance.\n\nThen measure what the cascade cost you. Take 300 items, judge them with the full expensive judge and with the cascade, and compare verdicts. A typical result (small judge confident on 80% of items and agreeing with the large one on 97% of those) means you pay about a fifth of the bill for roughly half a point of accuracy, and you can state that trade rather than assume it.\n\n## Cache on the thing that determines the answer\n\nThe judge's verdict depends on the input, the output, the rubric and the judge version. So cache on a hash of exactly those:\n\n```python\nkey = sha256(f\"{item_id}|{sha256(output)}|{rubric_version}|{judge_model}\".encode())\n```\n\nA rerun after a change that touched documentation, or a retry after a network failure, should cost nothing. In practice a well-keyed cache removes 60 to 90% of calls in day-to-day development, because most edits change a few outputs rather than all of them.\n\nCache the claim decomposition separately from the verdict, since claims survive rubric changes.\n\n## Judge less often, on purpose\n\nNot every run needs every item.\n\n- **A core set** of 150 to 300 items, judged on every run, chosen to cover the strata. This is the CI signal.\n- **The full set**, judged nightly or weekly, which is the number you report.\n- **A random sample** of live traffic, judged continuously at a fixed rate, which is the production signal.\n\nThe sizing arithmetic from the statistics block tells you what the core set can detect (usually large regressions only) and that is exactly what a CI gate should be for.\n\n## The other levers\n\n- **Batch endpoints** where the provider offers them: typically half price for results within a day, which suits a nightly full run perfectly.\n- **Shorter prompts.** Do not paste an entire document into a check about one paragraph. Halving prompt length halves the bill, and it usually improves the judge, because the relevant text is no longer buried.\n- **One rubric item per call** stays right even under budget pressure. Bundling five questions into one call saves tokens and correlates the errors, which costs you more in reliability than it saves in money.\n- **Skip judging what code can decide.** Every check moved down into deterministic code is a permanent saving and a permanent gain in reproducibility.\n\n## The free path, stated plainly\n\nA judge running on your own machine costs electricity. An 8-billion-parameter model quantised to four bits needs about 6 GB of memory and produces a verdict in a few seconds on an ordinary CPU, faster with any recent GPU. `ollama run llama3.1:8b` or a `llama.cpp` build is the whole setup.\n\nWhere it is adequate: binary rule checks, schema and format questions, obvious contradictions, routing decisions in a cascade. Where it is not: fine distinctions of tone, multi-step reasoning about a policy, anything where the expensive judge itself only just agrees with your labels. Measure which of those you are in with the same calibration set you built earlier, and you will find that a meaningful share of a typical rubric runs locally for nothing.\n\n## Put the cost in the report\n\nPrint cost per run, cost per item, and the cache hit rate alongside the quality numbers, in the same table the team already reads. A harness whose cost is visible gets tuned. A harness whose cost is invisible gets switched off in a quarter when somebody is asked to cut 20% from the bill, and the first anybody hears of it is when quality drops with no measurement to detect it."},"check":{"question":"A cascade sends only uncertain items to the expensive judge. What must be measured before trusting its numbers?","options":[{"text":"The share of items each layer resolves, so the cost saving can be reported accurately.","correct":false,"tempting":"Useful for the cost story and silent on whether the verdicts are still correct, which is what trusting the numbers requires."},{"text":"Agreement between the cascade's verdicts and the expensive judge's verdicts on a sample judged by both.","correct":true},{"text":"The small judge's agreement with human labels on the full evaluation set.","correct":false,"tempting":"Human agreement is the right check for the judge you finally rely on, and the specific question here is what the routing rule loses relative to the judge it is standing in for."},{"text":"The latency of each layer, since the cascade adds round trips for escalated items.","correct":false,"tempting":"A practical consideration for run time, not evidence that the cheaper path reaches the same conclusions."}]},"prev":{"slug":"groundedness-checks","title":"Checking an answer against the passages it was given","url":"https://addaly.com/learn/evaluating-ai/groundedness-checks"},"next":{"slug":"red-teaming-and-safety-evals","title":"Testing what happens when somebody tries","url":"https://addaly.com/learn/evaluating-ai/red-teaming-and-safety-evals"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What judging costs, and how to pay a tenth of it', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/judging-on-a-budget (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What judging costs, and how to pay a tenth of it","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Cascade the checks (free code first, a small or local judge next, the expensive judge only on the uncertain remainder) and cache by output hash, because a rerun that changed no outputs should cost nothing.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"do-the-arithmetic-once","runs":[{"text":"Do the arithmetic once"}]},{"t":"p","runs":[{"text":"A realistic harness: 2,000 items, four rubric checks each, run in both orders for the pairwise part. That is 16,000 judge calls. At roughly 1,500 tokens per call, 24 million tokens per run."}]},{"t":"p","runs":[{"text":"Whatever today’s prices are, the ratio between a small model and a frontier one is roughly ten to one, so the same run costs a few dollars or several dozen. Run it on every pull request, twenty times a day, and the annual figure is the kind that appears in a cost review with your name beside it. Teams do not usually abandon evaluation because it stopped working; they abandon it because it became the third-largest line in the bill and nobody could say what it bought."}]},{"t":"h2","id":"the-cascade","runs":[{"text":"The cascade"}]},{"t":"p","runs":[{"text":"Order the checks by cost and stop as early as possible."}]},{"t":"ol","start":1,"items":[[{"text":"Deterministic code.","b":true},{"text":" Schema validity, forbidden strings, numbers grounded in the source, citations resolving, length bands. Free, instant, and on a typical set this decides 20 to 40% of items outright."}],[{"text":"A small or local judge","b":true},{"text":" for the rule-shaped checks. An 8-billion-parameter open-weights model through Ollama or llama.cpp runs on a laptop, costs nothing per call, and is entirely adequate for questions like “does this reply state a deadline that is absent from the policy”."}],[{"text":"The expensive judge","b":true},{"text":", only for items where the cheap layer is uncertain, or where the rubric requires nuance."}]]},{"t":"p","runs":[{"text":"Then measure what the cascade cost you. Take 300 items, judge them with the full expensive judge and with the cascade, and compare verdicts. A typical result (small judge confident on 80% of items and agreeing with the large one on 97% of those) means you pay about a fifth of the bill for roughly half a point of accuracy, and you can state that trade rather than assume it."}]},{"t":"h2","id":"cache-on-the-thing-that-determines-the-answer","runs":[{"text":"Cache on the thing that determines the answer"}]},{"t":"p","runs":[{"text":"The judge’s verdict depends on the input, the output, the rubric and the judge version. So cache on a hash of exactly those:"}]},{"t":"code","lang":"python","label":"Python","text":"key = sha256(f\"{item_id}|{sha256(output)}|{rubric_version}|{judge_model}\".encode())"},{"t":"p","runs":[{"text":"A rerun after a change that touched documentation, or a retry after a network failure, should cost nothing. In practice a well-keyed cache removes 60 to 90% of calls in day-to-day development, because most edits change a few outputs rather than all of them."}]},{"t":"p","runs":[{"text":"Cache the claim decomposition separately from the verdict, since claims survive rubric changes."}]},{"t":"h2","id":"judge-less-often-on-purpose","runs":[{"text":"Judge less often, on purpose"}]},{"t":"p","runs":[{"text":"Not every run needs every item."}]},{"t":"ul","items":[[{"text":"A core set","b":true},{"text":" of 150 to 300 items, judged on every run, chosen to cover the strata. This is the CI signal."}],[{"text":"The full set","b":true},{"text":", judged nightly or weekly, which is the number you report."}],[{"text":"A random sample","b":true},{"text":" of live traffic, judged continuously at a fixed rate, which is the production signal."}]]},{"t":"p","runs":[{"text":"The sizing arithmetic from the statistics block tells you what the core set can detect (usually large regressions only) and that is exactly what a CI gate should be for."}]},{"t":"h2","id":"the-other-levers","runs":[{"text":"The other levers"}]},{"t":"ul","items":[[{"text":"Batch endpoints","b":true},{"text":" where the provider offers them: typically half price for results within a day, which suits a nightly full run perfectly."}],[{"text":"Shorter prompts.","b":true},{"text":" Do not paste an entire document into a check about one paragraph. Halving prompt length halves the bill, and it usually improves the judge, because the relevant text is no longer buried."}],[{"text":"One rubric item per call","b":true},{"text":" stays right even under budget pressure. Bundling five questions into one call saves tokens and correlates the errors, which costs you more in reliability than it saves in money."}],[{"text":"Skip judging what code can decide.","b":true},{"text":" Every check moved down into deterministic code is a permanent saving and a permanent gain in reproducibility."}]]},{"t":"h2","id":"the-free-path-stated-plainly","runs":[{"text":"The free path, stated plainly"}],"role":"free"},{"t":"p","runs":[{"text":"A judge running on your own machine costs electricity. An 8-billion-parameter model quantised to four bits needs about 6 GB of memory and produces a verdict in a few seconds on an ordinary CPU, faster with any recent GPU. "},{"text":"ollama run llama3.1:8b","code":true},{"text":" or a "},{"text":"llama.cpp","code":true},{"text":" build is the whole setup."}]},{"t":"p","runs":[{"text":"Where it is adequate: binary rule checks, schema and format questions, obvious contradictions, routing decisions in a cascade. Where it is not: fine distinctions of tone, multi-step reasoning about a policy, anything where the expensive judge itself only just agrees with your labels. Measure which of those you are in with the same calibration set you built earlier, and you will find that a meaningful share of a typical rubric runs locally for nothing."}]},{"t":"h2","id":"put-the-cost-in-the-report","runs":[{"text":"Put the cost in the report"}]},{"t":"p","runs":[{"text":"Print cost per run, cost per item, and the cache hit rate alongside the quality numbers, in the same table the team already reads. A harness whose cost is visible gets tuned. A harness whose cost is invisible gets switched off in a quarter when somebody is asked to cut 20% from the bill, and the first anybody hears of it is when quality drops with no measurement to detect it."}]}],"check":{"question":[{"text":"A cascade sends only uncertain items to the expensive judge. What must be measured before trusting its numbers?"}],"options":[{"text":[{"text":"The share of items each layer resolves, so the cost saving can be reported accurately."}],"correct":false,"tempting":[{"text":"Useful for the cost story and silent on whether the verdicts are still correct, which is what trusting the numbers requires."}]},{"text":[{"text":"Agreement between the cascade’s verdicts and the expensive judge’s verdicts on a sample judged by both."}],"correct":true,"tempting":null},{"text":[{"text":"The small judge’s agreement with human labels on the full evaluation set."}],"correct":false,"tempting":[{"text":"Human agreement is the right check for the judge you finally rely on, and the specific question here is what the routing rule loses relative to the judge it is standing in for."}]},{"text":[{"text":"The latency of each layer, since the cascade adds round trips for escalated items."}],"correct":false,"tempting":[{"text":"A practical consideration for run time, not evidence that the cheaper path reaches the same conclusions."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"groundedness-checks","title":"Checking an answer against the passages it was given","minutes":9},"next":{"slug":"red-teaming-and-safety-evals","title":"Testing what happens when somebody tries","minutes":10}}}