Thirty tasks you run before every change

Assert on the world, not the words

Evaluating a chatbot is hard because the output is prose and prose has no correct answer. Evaluating an agent is easier than people expect, because agents change things, and things are checkable.

After the run, ask the environment:

• Does the row exist in the database, with these three field values?
• Does the file at this path contain this string?
• Was the draft email addressed to this person, and does it contain the correct amount?
• Was the refund tool called with an amount under the cap?
• Was the delete tool called at all?

These are ordinary assertions. They are objective, cheap and stable across model versions, and they do not require a judge model or a rubric. This single move, score the end state, not the final message, makes agent evaluation tractable and is the thing most teams have not done.

Build the set from real work

Do not invent tasks. Harvest them.

Take twenty real inputs from the last month: the ordinary ones, the two that went wrong, the one with the attachment nobody expected. Add every incident as it happens: a bad run becomes a permanent task the same day, which is the habit that stops the same failure recurring twice a year.

Then deliberately include two categories most sets are missing:

Tasks that should fail. Around a fifth of the set should be impossible: the order does not exist, the policy does not cover this case, the file is corrupt. You are testing whether the agent stops and says so, or invents an answer. This is where systems that look excellent on the happy path fall apart, and you will not find it any other way.

One injection task. A document or message containing an instruction aimed at the agent. Pass means the instruction was not followed. Once this is in the set, a prompt change that quietly weakens your defences shows up the same afternoon.

Thirty to fifty tasks is a good size. Beyond that the run gets slow and people stop running it, which is worse than a smaller set.

Run each task three times

Agents are stochastic; a single run tells you almost nothing about a borderline task. Run each three times and record the per-task pass count: 3/3, 2/3, 0/3.

Then read the per-task table, not the aggregate. A task that moves from 3/3 to 0/3 after a prompt change is a real signal and you can go and look at it. A task sitting at 2/3 for weeks is telling you something specific about instability that a headline number cannot express.

The cost is small. Forty tasks at three runs each, at a few pence a run, is a few pounds per change, cheaper than one person spending an hour deciding whether things feel better. Time is the real constraint, so run them in parallel; a set that takes forty minutes is a set nobody runs before lunch.

Where the numbers stop meaning things

Suppose your pass rate goes from 82 per cent to 85 per cent after a prompt change. On forty tasks, three percentage points is 1.2 tasks. One task changed. On a stochastic system, with three runs each, that is well within what re-running the same prompt would produce.

So the aggregate is a dashboard number and the per-task diff is the evidence. When somebody claims an improvement, the question is which tasks changed, and does the change make sense given what you altered. If the tasks that moved have nothing to do with the edit, you are looking at noise and about to ship a superstition.

Genuine improvements on a set this size look like five or six tasks moving together in a direction you can explain.

Two harnesses, two schedules

The replay harness (recorded tool results, no network, from the testing lesson) is deterministic apart from the model, runs in seconds, and belongs in CI on every commit.

The live harness runs against a real sandbox environment with a seeded database that resets before each run. It is slower, occasionally flaky, and the only thing that tests your actual integrations. Run it nightly and before every deploy.

Both matter. The replay set catches prompt and parsing regressions; the live set catches the afternoon the vendor changed an endpoint. A team with only the first has a suite that stays green while production is broken.

Keep the environment frozen

An eval set is only comparable over time if the world it runs against does not move. Seed the database from a fixture, pin the model id, pin the prompt version, and record all three with the result. The day somebody upgrades the fixture without saying so, every historical comparison you have becomes meaningless, so version the fixture too, and put its hash in the report.