{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-agents","title":"When the output is a sequence of actions","url":"https://addaly.com/learn/evaluating-ai/evaluating-agents","position":49,"of":76,"minutes":10,"answer":"Score an agent on the final state, record the trajectory for diagnosis, run every task several times because a task that passes three times in five has not passed.","keyIdea":"Score an agent on the final state, record the trajectory for diagnosis, run every task several times because a task that passes three times in five has not passed, and count cost and steps as part of the result."},"body":{"format":"markdown","text":"## Two things to score, and you need both\n\nAn agent takes actions: it calls tools, reads results, decides again. Evaluating it means scoring two different things.\n\n**Outcome.** Did the world end up in the right state? The ticket exists, with the right customer and priority. The file contains the right rows. The refund of ₹2,400 was issued to the right order. This is checkable in code against the system's final state, and it is the number that matters.\n\n**Trajectory.** How did it get there? Which tools, in what order, with what arguments, how many times. This does not decide pass or fail on its own, but it is where every diagnosis lives, and it catches the agent that got the right answer by an unrepeatable accident: nine searches, three of them identical, and a lucky guess.\n\nScore outcome, record trajectory, and read the trajectory whenever the outcome is wrong.\n\n## Partial credit, done carefully\n\nA six-step task that fails at step five scores zero on outcome, and zero tells you nothing about whether you are close. So define checkpoints in advance, per task:\n\n```json\n{\"task\": \"refund-042\",\n \"checkpoints\": [\n   {\"id\": \"found_order\",   \"check\": \"tool_called('lookup_order', order_id='A-8891')\"},\n   {\"id\": \"checked_policy\",\"check\": \"tool_called('get_policy')\"},\n   {\"id\": \"refund_issued\", \"check\": \"db.refunds.has(order='A-8891', amount=2400)\"},\n   {\"id\": \"customer_told\", \"check\": \"tool_called('send_message')\"}],\n \"gate\": [\"refund_issued\"]}\n```\n\nThe checkpoint profile across many runs is diagnostic: if 80% of failures stop at `checked_policy`, that tool is the problem, and no prompt change is required to know it.\n\nOne discipline: checkpoints are defined before the run. Written afterwards, they describe the path your agent happened to take, and every future agent is scored on resembling it. Several correct paths usually exist, so express checkpoints as states reached rather than as an exact sequence wherever you can.\n\n## Non-determinism is not a nuisance, it is the measurement\n\nRun each task once and you have learned almost nothing. Agents branch: a slightly different tool result leads to a different plan.\n\nRun each task **five times** and report pass rate per task. The distribution is the finding:\n\n- 5/5: reliable\n- 3/5: this is not a passing task, and reporting it as one is how agent demos become production incidents\n- 0/5: reliably broken, which is at least easy to debug\n\nA suite of 40 tasks run 5 times is 200 runs, which is affordable and which changes the conversation from \"it worked when I tried it\" to a number with a spread. Report the mean pass rate *and* the count of tasks that passed all five times. The second number is usually the one a business would actually care about.\n\n> Figure: Forty tasks, each run five times. Bars (tasks): Passed 5 of 5 22; Passed 3 or 4 of 5 11; Passed 1 or 2 of 5 4; Passed 0 of 5 3. The mean pass rate is about 77%, which is the number in the announcement. Twenty-two tasks pass every time, which is the number a business can rely on. A task at three of five has not passed.\n\n## Cost, steps and time are part of the score\n\nTwo agents both pass 82% of tasks. One averages 8 tool calls and ₹1.60 per task; the other averages 34 calls and ₹9.20. These are not the same system, and only one of them can be turned on for a hundred thousand users.\n\nRecord per run: tool calls, total tokens in and out, money, wall-clock seconds, and whether it hit the step limit. Hitting the step limit deserves its own count: it is the signature of an agent that is looping, and it looks identical to ordinary failure in a pass-rate table.\n\nWatch for the pattern where a change raises pass rate by two points and doubles cost. That is usually a longer step budget rather than better reasoning, and it is a trade you should make deliberately rather than discover in the invoice.\n\n## The environment is the hard part\n\nAn agent evaluation that sends real emails, charges real cards or writes to the real database can be run once. To run it two hundred times you need an environment that resets:\n\n- **Fake the side-effecting tools.** A record of what would have been sent is enough to check the outcome, and it is much faster.\n- **Seed a fresh database per run** and tear it down. Docker or an in-memory SQLite is plenty.\n- **Freeze time and randomness** where the task depends on them, or \"tomorrow\" makes half your tasks fail in March.\n- **Snapshot the tool results** for a fixed suite so retrieval or API flakiness does not show up as agent failure. Keep a smaller live suite as well, because a fully mocked evaluation stops noticing when a real API changes.\n\nBuilding this environment is usually more work than building the agent. That is not a sign you have gone wrong; it is the normal ratio, and it is why teams skip it and ship on demos instead.\n\n## Tools\n\n**Inspect** (UK AISI, MIT) has first-class support for tool-using tasks and scoring. **SWE-bench** and **τ-bench** are worth reading as designs even if you never run them: τ-bench in particular formalises the multi-run idea with a pass^k metric, the fraction of tasks solved on *all* k attempts, which is the honest thing to report and a number that is always lower than the one in the announcement."},"check":{"question":"An agent suite of 30 tasks is run once each; 25 pass, and the team reports 83%. What is the most important correction?","options":[{"text":"Each task should be run several times and the pass rate reported per task, because a single run cannot distinguish a reliable task from one that succeeds two times in five.","correct":true},{"text":"30 tasks give a confidence interval of about ±13 points, so the suite needs to be several times larger.","correct":false,"tempting":"Applies the sample-size correction correctly but treats run-to-run variance in the agent itself as though it did not exist."},{"text":"The trajectories of the 5 failures should be read before any number is reported.","correct":false,"tempting":"Prescribes a genuinely useful debugging step, but reading five trajectories does not make the headline figure any more trustworthy."},{"text":"Pass rate should be replaced by average checkpoint completion, which is more informative than a binary outcome.","correct":false,"tempting":"Swaps the headline for a partial-credit measure that flatters an agent which reliably gets most of the way and never finishes."}]},"prev":{"slug":"red-teaming-and-safety-evals","title":"Testing what happens when somebody tries","url":"https://addaly.com/learn/evaluating-ai/red-teaming-and-safety-evals"},"next":{"slug":"the-eval-harness","title":"The harness, and why you should build the small one first","url":"https://addaly.com/learn/evaluating-ai/the-eval-harness"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the output is a sequence of actions', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-agents (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the output is a sequence of actions","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Score an agent on the final state, record the trajectory for diagnosis, run every task several times because a task that passes three times in five has not passed, and count cost and steps as part of the result.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tasks","title":"Forty tasks, each run five times","series":[{"label":"Passed 5 of 5","value":22,"highlight":true},{"label":"Passed 3 or 4 of 5","value":11},{"label":"Passed 1 or 2 of 5","value":4},{"label":"Passed 0 of 5","value":3}],"caption":"The mean pass rate is about 77%, which is the number in the announcement. Twenty-two tasks pass every time, which is the number a business can rely on. A task at three of five has not passed."},"alt":"Figure: Forty tasks, each run five times. Bars (tasks): Passed 5 of 5 22; Passed 3 or 4 of 5 11; Passed 1 or 2 of 5 4; Passed 0 of 5 3. The mean pass rate is about 77%, which is the number in the announcement. Twenty-two tasks pass every time, which is the number a business can rely on. A task at three of five has not passed."}},"blocks":[{"t":"h2","id":"two-things-to-score-and-you-need-both","runs":[{"text":"Two things to score, and you need both"}]},{"t":"p","runs":[{"text":"An agent takes actions: it calls tools, reads results, decides again. Evaluating it means scoring two different things."}]},{"t":"p","runs":[{"text":"Outcome.","b":true},{"text":" Did the world end up in the right state? The ticket exists, with the right customer and priority. The file contains the right rows. The refund of ₹2,400 was issued to the right order. This is checkable in code against the system’s final state, and it is the number that matters."}]},{"t":"p","runs":[{"text":"Trajectory.","b":true},{"text":" How did it get there? Which tools, in what order, with what arguments, how many times. This does not decide pass or fail on its own, but it is where every diagnosis lives, and it catches the agent that got the right answer by an unrepeatable accident: nine searches, three of them identical, and a lucky guess."}]},{"t":"p","runs":[{"text":"Score outcome, record trajectory, and read the trajectory whenever the outcome is wrong."}]},{"t":"h2","id":"partial-credit-done-carefully","runs":[{"text":"Partial credit, done carefully"}]},{"t":"p","runs":[{"text":"A six-step task that fails at step five scores zero on outcome, and zero tells you nothing about whether you are close. So define checkpoints in advance, per task:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"task\": \"refund-042\",\n \"checkpoints\": [\n   {\"id\": \"found_order\",   \"check\": \"tool_called('lookup_order', order_id='A-8891')\"},\n   {\"id\": \"checked_policy\",\"check\": \"tool_called('get_policy')\"},\n   {\"id\": \"refund_issued\", \"check\": \"db.refunds.has(order='A-8891', amount=2400)\"},\n   {\"id\": \"customer_told\", \"check\": \"tool_called('send_message')\"}],\n \"gate\": [\"refund_issued\"]}"},{"t":"p","runs":[{"text":"The checkpoint profile across many runs is diagnostic: if 80% of failures stop at "},{"text":"checked_policy","code":true},{"text":", that tool is the problem, and no prompt change is required to know it."}]},{"t":"p","runs":[{"text":"One discipline: checkpoints are defined before the run. Written afterwards, they describe the path your agent happened to take, and every future agent is scored on resembling it. Several correct paths usually exist, so express checkpoints as states reached rather than as an exact sequence wherever you can."}]},{"t":"h2","id":"non-determinism-is-not-a-nuisance-it-is-the-measurement","runs":[{"text":"Non-determinism is not a nuisance, it is the measurement"}]},{"t":"p","runs":[{"text":"Run each task once and you have learned almost nothing. Agents branch: a slightly different tool result leads to a different plan."}]},{"t":"p","runs":[{"text":"Run each task "},{"text":"five times","b":true},{"text":" and report pass rate per task. The distribution is the finding:"}]},{"t":"ul","items":[[{"text":"5/5: reliable"}],[{"text":"3/5: this is not a passing task, and reporting it as one is how agent demos become production incidents"}],[{"text":"0/5: reliably broken, which is at least easy to debug"}]]},{"t":"p","runs":[{"text":"A suite of 40 tasks run 5 times is 200 runs, which is affordable and which changes the conversation from “it worked when I tried it” to a number with a spread. Report the mean pass rate "},{"text":"and","i":true},{"text":" the count of tasks that passed all five times. The second number is usually the one a business would actually care about."}]},{"t":"h2","id":"cost-steps-and-time-are-part-of-the-score","runs":[{"text":"Cost, steps and time are part of the score"}]},{"t":"p","runs":[{"text":"Two agents both pass 82% of tasks. One averages 8 tool calls and ₹1.60 per task; the other averages 34 calls and ₹9.20. These are not the same system, and only one of them can be turned on for a hundred thousand users."}]},{"t":"p","runs":[{"text":"Record per run: tool calls, total tokens in and out, money, wall-clock seconds, and whether it hit the step limit. Hitting the step limit deserves its own count: it is the signature of an agent that is looping, and it looks identical to ordinary failure in a pass-rate table."}]},{"t":"p","runs":[{"text":"Watch for the pattern where a change raises pass rate by two points and doubles cost. That is usually a longer step budget rather than better reasoning, and it is a trade you should make deliberately rather than discover in the invoice."}]},{"t":"h2","id":"the-environment-is-the-hard-part","runs":[{"text":"The environment is the hard part"}]},{"t":"p","runs":[{"text":"An agent evaluation that sends real emails, charges real cards or writes to the real database can be run once. To run it two hundred times you need an environment that resets:"}]},{"t":"ul","items":[[{"text":"Fake the side-effecting tools.","b":true},{"text":" A record of what would have been sent is enough to check the outcome, and it is much faster."}],[{"text":"Seed a fresh database per run","b":true},{"text":" and tear it down. Docker or an in-memory SQLite is plenty."}],[{"text":"Freeze time and randomness","b":true},{"text":" where the task depends on them, or “tomorrow” makes half your tasks fail in March."}],[{"text":"Snapshot the tool results","b":true},{"text":" for a fixed suite so retrieval or API flakiness does not show up as agent failure. Keep a smaller live suite as well, because a fully mocked evaluation stops noticing when a real API changes."}]]},{"t":"p","runs":[{"text":"Building this environment is usually more work than building the agent. That is not a sign you have gone wrong; it is the normal ratio, and it is why teams skip it and ship on demos instead."}]},{"t":"h2","id":"tools","runs":[{"text":"Tools"}]},{"t":"p","runs":[{"text":"Inspect","b":true},{"text":" (UK AISI, MIT) has first-class support for tool-using tasks and scoring. "},{"text":"SWE-bench","b":true},{"text":" and "},{"text":"τ-bench","b":true},{"text":" are worth reading as designs even if you never run them: τ-bench in particular formalises the multi-run idea with a pass^k metric, the fraction of tasks solved on "},{"text":"all","i":true},{"text":" k attempts, which is the honest thing to report and a number that is always lower than the one in the announcement."}]}],"check":{"question":[{"text":"An agent suite of 30 tasks is run once each; 25 pass, and the team reports 83%. What is the most important correction?"}],"options":[{"text":[{"text":"Each task should be run several times and the pass rate reported per task, because a single run cannot distinguish a reliable task from one that succeeds two times in five."}],"correct":true,"tempting":null},{"text":[{"text":"30 tasks give a confidence interval of about ±13 points, so the suite needs to be several times larger."}],"correct":false,"tempting":[{"text":"Applies the sample-size correction correctly but treats run-to-run variance in the agent itself as though it did not exist."}]},{"text":[{"text":"The trajectories of the 5 failures should be read before any number is reported."}],"correct":false,"tempting":[{"text":"Prescribes a genuinely useful debugging step, but reading five trajectories does not make the headline figure any more trustworthy."}]},{"text":[{"text":"Pass rate should be replaced by average checkpoint completion, which is more informative than a binary outcome."}],"correct":false,"tempting":[{"text":"Swaps the headline for a partial-credit measure that flatters an agent which reliably gets most of the way and never finishes."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"red-teaming-and-safety-evals","title":"Testing what happens when somebody tries","minutes":10},"next":{"slug":"the-eval-harness","title":"The harness, and why you should build the small one first","minutes":9}}}