When the output is a sequence of actions

Two things to score, and you need both

An agent takes actions: it calls tools, reads results, decides again. Evaluating it means scoring two different things.

Outcome. Did the world end up in the right state? The ticket exists, with the right customer and priority. The file contains the right rows. The refund of ₹2,400 was issued to the right order. This is checkable in code against the system’s final state, and it is the number that matters.

Trajectory. How did it get there? Which tools, in what order, with what arguments, how many times. This does not decide pass or fail on its own, but it is where every diagnosis lives, and it catches the agent that got the right answer by an unrepeatable accident: nine searches, three of them identical, and a lucky guess.

Score outcome, record trajectory, and read the trajectory whenever the outcome is wrong.

Partial credit, done carefully

A six-step task that fails at step five scores zero on outcome, and zero tells you nothing about whether you are close. So define checkpoints in advance, per task:

{"task": "refund-042",
 "checkpoints": [
   {"id": "found_order",   "check": "tool_called('lookup_order', order_id='A-8891')"},
   {"id": "checked_policy","check": "tool_called('get_policy')"},
   {"id": "refund_issued", "check": "db.refunds.has(order='A-8891', amount=2400)"},
   {"id": "customer_told", "check": "tool_called('send_message')"}],
 "gate": ["refund_issued"]}

The checkpoint profile across many runs is diagnostic: if 80% of failures stop at checked_policy, that tool is the problem, and no prompt change is required to know it.

One discipline: checkpoints are defined before the run. Written afterwards, they describe the path your agent happened to take, and every future agent is scored on resembling it. Several correct paths usually exist, so express checkpoints as states reached rather than as an exact sequence wherever you can.

Non-determinism is not a nuisance, it is the measurement

Run each task once and you have learned almost nothing. Agents branch: a slightly different tool result leads to a different plan.

Run each task five times and report pass rate per task. The distribution is the finding:

• 5/5: reliable
• 3/5: this is not a passing task, and reporting it as one is how agent demos become production incidents
• 0/5: reliably broken, which is at least easy to debug

A suite of 40 tasks run 5 times is 200 runs, which is affordable and which changes the conversation from “it worked when I tried it” to a number with a spread. Report the mean pass rate and the count of tasks that passed all five times. The second number is usually the one a business would actually care about.

Cost, steps and time are part of the score

Two agents both pass 82% of tasks. One averages 8 tool calls and ₹1.60 per task; the other averages 34 calls and ₹9.20. These are not the same system, and only one of them can be turned on for a hundred thousand users.

Record per run: tool calls, total tokens in and out, money, wall-clock seconds, and whether it hit the step limit. Hitting the step limit deserves its own count: it is the signature of an agent that is looping, and it looks identical to ordinary failure in a pass-rate table.

Watch for the pattern where a change raises pass rate by two points and doubles cost. That is usually a longer step budget rather than better reasoning, and it is a trade you should make deliberately rather than discover in the invoice.

The environment is the hard part

An agent evaluation that sends real emails, charges real cards or writes to the real database can be run once. To run it two hundred times you need an environment that resets:

• Fake the side-effecting tools. A record of what would have been sent is enough to check the outcome, and it is much faster.
• Seed a fresh database per run and tear it down. Docker or an in-memory SQLite is plenty.
• Freeze time and randomness where the task depends on them, or “tomorrow” makes half your tasks fail in March.
• Snapshot the tool results for a fixed suite so retrieval or API flakiness does not show up as agent failure. Keep a smaller live suite as well, because a fully mocked evaluation stops noticing when a real API changes.

Building this environment is usually more work than building the agent. That is not a sign you have gone wrong; it is the normal ratio, and it is why teams skip it and ship on demos instead.

Tools

Inspect (UK AISI, MIT) has first-class support for tool-using tasks and scoring. SWE-bench and τ-bench are worth reading as designs even if you never run them: τ-bench in particular formalises the multi-run idea with a pass^k metric, the fraction of tasks solved on all k attempts, which is the honest thing to report and a number that is always lower than the one in the announcement.