What judging costs, and how to pay a tenth of it

Do the arithmetic once

A realistic harness: 2,000 items, four rubric checks each, run in both orders for the pairwise part. That is 16,000 judge calls. At roughly 1,500 tokens per call, 24 million tokens per run.

Whatever today’s prices are, the ratio between a small model and a frontier one is roughly ten to one, so the same run costs a few dollars or several dozen. Run it on every pull request, twenty times a day, and the annual figure is the kind that appears in a cost review with your name beside it. Teams do not usually abandon evaluation because it stopped working; they abandon it because it became the third-largest line in the bill and nobody could say what it bought.

The cascade

Order the checks by cost and stop as early as possible.

1. Deterministic code. Schema validity, forbidden strings, numbers grounded in the source, citations resolving, length bands. Free, instant, and on a typical set this decides 20 to 40% of items outright.
2. A small or local judge for the rule-shaped checks. An 8-billion-parameter open-weights model through Ollama or llama.cpp runs on a laptop, costs nothing per call, and is entirely adequate for questions like “does this reply state a deadline that is absent from the policy”.
3. The expensive judge, only for items where the cheap layer is uncertain, or where the rubric requires nuance.

Then measure what the cascade cost you. Take 300 items, judge them with the full expensive judge and with the cascade, and compare verdicts. A typical result (small judge confident on 80% of items and agreeing with the large one on 97% of those) means you pay about a fifth of the bill for roughly half a point of accuracy, and you can state that trade rather than assume it.

Cache on the thing that determines the answer

The judge’s verdict depends on the input, the output, the rubric and the judge version. So cache on a hash of exactly those:

key = sha256(f"{item_id}|{sha256(output)}|{rubric_version}|{judge_model}".encode())

A rerun after a change that touched documentation, or a retry after a network failure, should cost nothing. In practice a well-keyed cache removes 60 to 90% of calls in day-to-day development, because most edits change a few outputs rather than all of them.

Cache the claim decomposition separately from the verdict, since claims survive rubric changes.

Judge less often, on purpose

Not every run needs every item.

• A core set of 150 to 300 items, judged on every run, chosen to cover the strata. This is the CI signal.
• The full set, judged nightly or weekly, which is the number you report.
• A random sample of live traffic, judged continuously at a fixed rate, which is the production signal.

The sizing arithmetic from the statistics block tells you what the core set can detect (usually large regressions only) and that is exactly what a CI gate should be for.

The other levers

• Batch endpoints where the provider offers them: typically half price for results within a day, which suits a nightly full run perfectly.
• Shorter prompts. Do not paste an entire document into a check about one paragraph. Halving prompt length halves the bill, and it usually improves the judge, because the relevant text is no longer buried.
• One rubric item per call stays right even under budget pressure. Bundling five questions into one call saves tokens and correlates the errors, which costs you more in reliability than it saves in money.
• Skip judging what code can decide. Every check moved down into deterministic code is a permanent saving and a permanent gain in reproducibility.

The free path, stated plainly

A judge running on your own machine costs electricity. An 8-billion-parameter model quantised to four bits needs about 6 GB of memory and produces a verdict in a few seconds on an ordinary CPU, faster with any recent GPU. ollama run llama3.1:8b or a llama.cpp build is the whole setup.

Where it is adequate: binary rule checks, schema and format questions, obvious contradictions, routing decisions in a cascade. Where it is not: fine distinctions of tone, multi-step reasoning about a policy, anything where the expensive judge itself only just agrees with your labels. Measure which of those you are in with the same calibration set you built earlier, and you will find that a meaningful share of a typical rubric runs locally for nothing.

Put the cost in the report

Print cost per run, cost per item, and the cache hit rate alongside the quality numbers, in the same table the team already reads. A harness whose cost is visible gets tuned. A harness whose cost is invisible gets switched off in a quarter when somebody is asked to cut 20% from the bill, and the first anybody hears of it is when quality drops with no measurement to detect it.