{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"golden-tasks-for-agents","title":"Thirty tasks you run before every change","url":"https://addaly.com/learn/agents-and-automation/golden-tasks-for-agents","position":37,"of":72,"minutes":10,"answer":"Score an agent on the state of the world after the run rather than on what it said, and compare per-task results, because a three-point move on forty tasks is about one task.","keyIdea":"Score an agent on the state of the world after the run rather than on what it said, and compare per-task results, because a three-point move on forty tasks is about one task."},"body":{"format":"markdown","text":"## Assert on the world, not the words\n\nEvaluating a chatbot is hard because the output is prose and prose has no correct answer. Evaluating an agent is easier than people expect, because agents change things, and things are checkable.\n\nAfter the run, ask the environment:\n\n- Does the row exist in the database, with these three field values?\n- Does the file at this path contain this string?\n- Was the draft email addressed to this person, and does it contain the correct amount?\n- Was the refund tool called with an amount under the cap?\n- Was the `delete` tool called at all?\n\nThese are ordinary assertions. They are objective, cheap and stable across model versions, and they do not require a judge model or a rubric. This single move, score the end state, not the final message, makes agent evaluation tractable and is the thing most teams have not done.\n\n## Build the set from real work\n\nDo not invent tasks. Harvest them.\n\nTake twenty real inputs from the last month: the ordinary ones, the two that went wrong, the one with the attachment nobody expected. Add every incident as it happens: a bad run becomes a permanent task the same day, which is the habit that stops the same failure recurring twice a year.\n\nThen deliberately include two categories most sets are missing:\n\n**Tasks that should fail.** Around a fifth of the set should be impossible: the order does not exist, the policy does not cover this case, the file is corrupt. You are testing whether the agent stops and says so, or invents an answer. This is where systems that look excellent on the happy path fall apart, and you will not find it any other way.\n\n**One injection task.** A document or message containing an instruction aimed at the agent. Pass means the instruction was not followed. Once this is in the set, a prompt change that quietly weakens your defences shows up the same afternoon.\n\nThirty to fifty tasks is a good size. Beyond that the run gets slow and people stop running it, which is worse than a smaller set.\n\n## Run each task three times\n\nAgents are stochastic; a single run tells you almost nothing about a borderline task. Run each three times and record the per-task pass count: 3/3, 2/3, 0/3.\n\nThen read the per-task table, not the aggregate. A task that moves from 3/3 to 0/3 after a prompt change is a real signal and you can go and look at it. A task sitting at 2/3 for weeks is telling you something specific about instability that a headline number cannot express.\n\nThe cost is small. Forty tasks at three runs each, at a few pence a run, is a few pounds per change, cheaper than one person spending an hour deciding whether things feel better. Time is the real constraint, so run them in parallel; a set that takes forty minutes is a set nobody runs before lunch.\n\n## Where the numbers stop meaning things\n\nSuppose your pass rate goes from 82 per cent to 85 per cent after a prompt change. On forty tasks, three percentage points is **1.2 tasks**. One task changed. On a stochastic system, with three runs each, that is well within what re-running the same prompt would produce.\n\nSo the aggregate is a dashboard number and the per-task diff is the evidence. When somebody claims an improvement, the question is *which tasks changed, and does the change make sense given what you altered*. If the tasks that moved have nothing to do with the edit, you are looking at noise and about to ship a superstition.\n\nGenuine improvements on a set this size look like five or six tasks moving together in a direction you can explain.\n\n## Two harnesses, two schedules\n\nThe replay harness (recorded tool results, no network, from the testing lesson) is deterministic apart from the model, runs in seconds, and belongs in CI on every commit.\n\nThe live harness runs against a real sandbox environment with a seeded database that resets before each run. It is slower, occasionally flaky, and the only thing that tests your actual integrations. Run it nightly and before every deploy.\n\nBoth matter. The replay set catches prompt and parsing regressions; the live set catches the afternoon the vendor changed an endpoint. A team with only the first has a suite that stays green while production is broken.\n\n## Keep the environment frozen\n\nAn eval set is only comparable over time if the world it runs against does not move. Seed the database from a fixture, pin the model id, pin the prompt version, and record all three with the result. The day somebody upgrades the fixture without saying so, every historical comparison you have becomes meaningless, so version the fixture too, and put its hash in the report."},"check":{"question":"After a prompt change, an agent’s golden-task pass rate moves from 82 per cent to 85 per cent across 40 tasks run once each. What is the right conclusion?","options":[{"text":"It is a genuine improvement, because the task set was held fixed between the two runs.","correct":false,"tempting":"Fixing the set removes variation in the tasks but not the run-to-run variation of a stochastic agent, which is the dominant source here."},{"text":"Three points on 40 tasks is roughly one task; repeat the runs and look at which specific tasks changed before concluding anything.","correct":true},{"text":"The set is too small to detect real changes and should be replaced with a larger public benchmark.","correct":false,"tempting":"Overcorrects: a small harvested set is highly informative about which tasks moved, just not about a three-point aggregate."},{"text":"The average rose and nothing regressed, so ship it and watch production metrics.","correct":false,"tempting":"Treats the aggregate as the unit of evidence, which hides the case where two tasks improved and one important one broke."}]},"prev":{"slug":"testing-with-replay","title":"Testing something that is not deterministic","url":"https://addaly.com/learn/agents-and-automation/testing-with-replay"},"next":{"slug":"prompt-injection-in-agents","title":"The email that tells your agent what to do","url":"https://addaly.com/learn/agents-and-automation/prompt-injection-in-agents"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Thirty tasks you run before every change', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/golden-tasks-for-agents (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Thirty tasks you run before every change","courseTitle":"Agents and Automation","module":{"index":4,"slug":"making-it-reliable","title":"Surviving real data"},"answer":null,"keyIdea":"Score an agent on the state of the world after the run rather than on what it said, and compare per-task results, because a three-point move on forty tasks is about one task.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"assert-on-the-world-not-the-words","runs":[{"text":"Assert on the world, not the words"}]},{"t":"p","runs":[{"text":"Evaluating a chatbot is hard because the output is prose and prose has no correct answer. Evaluating an agent is easier than people expect, because agents change things, and things are checkable."}]},{"t":"p","runs":[{"text":"After the run, ask the environment:"}]},{"t":"ul","items":[[{"text":"Does the row exist in the database, with these three field values?"}],[{"text":"Does the file at this path contain this string?"}],[{"text":"Was the draft email addressed to this person, and does it contain the correct amount?"}],[{"text":"Was the refund tool called with an amount under the cap?"}],[{"text":"Was the "},{"text":"delete","code":true},{"text":" tool called at all?"}]]},{"t":"p","runs":[{"text":"These are ordinary assertions. They are objective, cheap and stable across model versions, and they do not require a judge model or a rubric. This single move, score the end state, not the final message, makes agent evaluation tractable and is the thing most teams have not done."}]},{"t":"h2","id":"build-the-set-from-real-work","runs":[{"text":"Build the set from real work"}]},{"t":"p","runs":[{"text":"Do not invent tasks. Harvest them."}]},{"t":"p","runs":[{"text":"Take twenty real inputs from the last month: the ordinary ones, the two that went wrong, the one with the attachment nobody expected. Add every incident as it happens: a bad run becomes a permanent task the same day, which is the habit that stops the same failure recurring twice a year."}]},{"t":"p","runs":[{"text":"Then deliberately include two categories most sets are missing:"}]},{"t":"p","runs":[{"text":"Tasks that should fail.","b":true},{"text":" Around a fifth of the set should be impossible: the order does not exist, the policy does not cover this case, the file is corrupt. You are testing whether the agent stops and says so, or invents an answer. This is where systems that look excellent on the happy path fall apart, and you will not find it any other way."}]},{"t":"p","runs":[{"text":"One injection task.","b":true},{"text":" A document or message containing an instruction aimed at the agent. Pass means the instruction was not followed. Once this is in the set, a prompt change that quietly weakens your defences shows up the same afternoon."}]},{"t":"p","runs":[{"text":"Thirty to fifty tasks is a good size. Beyond that the run gets slow and people stop running it, which is worse than a smaller set."}]},{"t":"h2","id":"run-each-task-three-times","runs":[{"text":"Run each task three times"}]},{"t":"p","runs":[{"text":"Agents are stochastic; a single run tells you almost nothing about a borderline task. Run each three times and record the per-task pass count: 3/3, 2/3, 0/3."}]},{"t":"p","runs":[{"text":"Then read the per-task table, not the aggregate. A task that moves from 3/3 to 0/3 after a prompt change is a real signal and you can go and look at it. A task sitting at 2/3 for weeks is telling you something specific about instability that a headline number cannot express."}]},{"t":"p","runs":[{"text":"The cost is small. Forty tasks at three runs each, at a few pence a run, is a few pounds per change, cheaper than one person spending an hour deciding whether things feel better. Time is the real constraint, so run them in parallel; a set that takes forty minutes is a set nobody runs before lunch."}]},{"t":"h2","id":"where-the-numbers-stop-meaning-things","runs":[{"text":"Where the numbers stop meaning things"}]},{"t":"p","runs":[{"text":"Suppose your pass rate goes from 82 per cent to 85 per cent after a prompt change. On forty tasks, three percentage points is "},{"text":"1.2 tasks","b":true},{"text":". One task changed. On a stochastic system, with three runs each, that is well within what re-running the same prompt would produce."}]},{"t":"p","runs":[{"text":"So the aggregate is a dashboard number and the per-task diff is the evidence. When somebody claims an improvement, the question is "},{"text":"which tasks changed, and does the change make sense given what you altered","i":true},{"text":". If the tasks that moved have nothing to do with the edit, you are looking at noise and about to ship a superstition."}]},{"t":"p","runs":[{"text":"Genuine improvements on a set this size look like five or six tasks moving together in a direction you can explain."}]},{"t":"h2","id":"two-harnesses-two-schedules","runs":[{"text":"Two harnesses, two schedules"}]},{"t":"p","runs":[{"text":"The replay harness (recorded tool results, no network, from the testing lesson) is deterministic apart from the model, runs in seconds, and belongs in CI on every commit."}]},{"t":"p","runs":[{"text":"The live harness runs against a real sandbox environment with a seeded database that resets before each run. It is slower, occasionally flaky, and the only thing that tests your actual integrations. Run it nightly and before every deploy."}]},{"t":"p","runs":[{"text":"Both matter. The replay set catches prompt and parsing regressions; the live set catches the afternoon the vendor changed an endpoint. A team with only the first has a suite that stays green while production is broken."}]},{"t":"h2","id":"keep-the-environment-frozen","runs":[{"text":"Keep the environment frozen"}]},{"t":"p","runs":[{"text":"An eval set is only comparable over time if the world it runs against does not move. Seed the database from a fixture, pin the model id, pin the prompt version, and record all three with the result. The day somebody upgrades the fixture without saying so, every historical comparison you have becomes meaningless, so version the fixture too, and put its hash in the report."}]}],"check":{"question":[{"text":"After a prompt change, an agent’s golden-task pass rate moves from 82 per cent to 85 per cent across 40 tasks run once each. What is the right conclusion?"}],"options":[{"text":[{"text":"It is a genuine improvement, because the task set was held fixed between the two runs."}],"correct":false,"tempting":[{"text":"Fixing the set removes variation in the tasks but not the run-to-run variation of a stochastic agent, which is the dominant source here."}]},{"text":[{"text":"Three points on 40 tasks is roughly one task; repeat the runs and look at which specific tasks changed before concluding anything."}],"correct":true,"tempting":null},{"text":[{"text":"The set is too small to detect real changes and should be replaced with a larger public benchmark."}],"correct":false,"tempting":[{"text":"Overcorrects: a small harvested set is highly informative about which tasks moved, just not about a three-point aggregate."}]},{"text":[{"text":"The average rose and nothing regressed, so ship it and watch production metrics."}],"correct":false,"tempting":[{"text":"Treats the aggregate as the unit of evidence, which hides the case where two tasks improved and one important one broke."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"testing-with-replay","title":"Testing something that is not deterministic","minutes":9},"next":{"slug":"prompt-injection-in-agents","title":"The email that tells your agent what to do","minutes":10}}}