{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"testing-with-replay","title":"Testing something that is not deterministic","url":"https://addaly.com/learn/agents-and-automation/testing-with-replay","position":36,"of":72,"minutes":9,"answer":"Record tool results once and replay them: with the tools held fixed, almost the whole harness becomes an ordinary deterministic test.","keyIdea":"Record tool results once and replay them: with the tools held fixed, almost the whole harness becomes an ordinary deterministic test."},"body":{"format":"markdown","text":"## The objection, and the answer\n\n\"You cannot unit-test an agent, the model is non-deterministic.\"\n\nYou cannot unit-test the *model*. You can absolutely unit-test the harness, and the harness is where nearly all of your bugs are. The trick is to hold the other variable still.\n\n## Record and replay\n\nRun the agent once, for real, with recording on. Every tool call and its result is written to a file keyed by tool name and canonical arguments.\n\n```\ntests/fixtures/refund_happy_path.jsonl\n  {\"tool\":\"get_order\",\"args\":{\"order_id\":\"ORD-48213\"},\"result\":\"{\\\"status\\\":\\\"shipped\\\"...}\"}\n  {\"tool\":\"refund_order\",\"args\":{\"order_id\":\"ORD-48213\",\"amount\":2499},\"result\":\"{\\\"ok\\\":true...}\"}\n```\n\nOn replay, the harness serves results from the file and never touches the network. Now the run is fast, free, offline, and repeatable: this is the same technique HTTP-testing libraries have called cassettes for years, applied one level up.\n\nTwo things to get right: **redact secrets and personal data at record time**, and **record the failures too**. A fixture directory of only happy paths tests the least interesting third of your code.\n\n## Three layers of test\n\n**Layer 1: the harness, with a fake model.** Replace the model with a scripted sequence: \"return a call to `get_order`, then a call to `refund_order`, then a final message.\" Now test everything that is yours: budgets, stopping rules, retries, the idempotency key, validators, compaction, the permission gate. Fully deterministic, runs in CI, no API key, milliseconds.\n\nThis layer is the most valuable and the most skipped. It is where you test that the stopping rule stops.\n\n**Layer 2: single steps, real model, fixed context.** \"Given this observation, does it pick `search_orders`?\" Run it ten times and assert a rate: eight or more out of ten. A single pass tells you nothing about a sampled system.\n\n**Layer 3: golden runs.** Twenty to forty real tasks with known-good outcomes, run nightly against the real model. Slow, costs money, and it is the only thing that catches a provider changing under you.\n\n## Assert on outcomes, never on prose\n\n```python\nassert refunded_amount(\"ORD-48213\") == 2499\nassert tool_call_count(\"refund_order\") == 1\nassert \"ORD-48214\" not in touched_orders()\n```\n\nNot `assert \"successfully refunded\" in output`. Wording changes between model versions and means nothing; state does not and means everything.\n\n## Flakiness is a measurement, not a nuisance\n\nRecord the pass rate of each test over time instead of deleting the ones that wobble. A layer-2 test that passes seven times in ten is not broken: it is reporting that this step is 70% reliable, which is the number you needed for the compounding arithmetic in module one. Set thresholds, alert on drops, and treat a fall from 9/10 to 6/10 as the regression it is.\n\n## The free path\n\n`pytest` plus a directory of JSON fixtures. No framework needed. If you want a harness for prompt-level tests, **promptfoo** and **DeepEval** are open source and worth a look. The recorder is about forty lines.\n\n## Test this first\n\nThe stopping rule. Before the prompt, before the tools, before anything. It is the cheapest test to write and the most expensive failure to have in production at 3am."},"check":{"question":"A team wants CI coverage for their agent but has no budget for API calls on every commit. Which test layer gives the most protection per rupee spent?","options":[{"text":"Harness tests driven by a fake model returning a scripted sequence of tool calls, with tool results replayed from recorded fixtures.","correct":true},{"text":"End-to-end golden runs against the real model on every commit.","correct":false,"tempting":"The most expensive and slowest layer, run at the highest frequency; it belongs on a nightly schedule, not in the commit loop."},{"text":"Assertions that the final message contains expected phrases.","correct":false,"tempting":"Tests the wording of a sampled system, so it breaks on harmless rewording and passes on runs that did nothing."},{"text":"A judge model scoring each run's output out of ten, failing the build below eight.","correct":false,"tempting":"Adds a second non-deterministic system as the arbiter, so the build now fails for two unrelated reasons and costs money each time."}]},"prev":{"slug":"validators-not-vibes","title":"The validator is the product","url":"https://addaly.com/learn/agents-and-automation/validators-not-vibes"},"next":{"slug":"golden-tasks-for-agents","title":"Thirty tasks you run before every change","url":"https://addaly.com/learn/agents-and-automation/golden-tasks-for-agents"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Testing something that is not deterministic', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/testing-with-replay (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Testing something that is not deterministic","courseTitle":"Agents and Automation","module":{"index":4,"slug":"making-it-reliable","title":"Surviving real data"},"answer":null,"keyIdea":"Record tool results once and replay them: with the tools held fixed, almost the whole harness becomes an ordinary deterministic test.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-objection-and-the-answer","runs":[{"text":"The objection, and the answer"}]},{"t":"p","runs":[{"text":"“You cannot unit-test an agent, the model is non-deterministic.”"}]},{"t":"p","runs":[{"text":"You cannot unit-test the "},{"text":"model","i":true},{"text":". You can absolutely unit-test the harness, and the harness is where nearly all of your bugs are. The trick is to hold the other variable still."}]},{"t":"h2","id":"record-and-replay","runs":[{"text":"Record and replay"}]},{"t":"p","runs":[{"text":"Run the agent once, for real, with recording on. Every tool call and its result is written to a file keyed by tool name and canonical arguments."}]},{"t":"code","lang":"","label":"Text","text":"tests/fixtures/refund_happy_path.jsonl\n  {\"tool\":\"get_order\",\"args\":{\"order_id\":\"ORD-48213\"},\"result\":\"{\\\"status\\\":\\\"shipped\\\"...}\"}\n  {\"tool\":\"refund_order\",\"args\":{\"order_id\":\"ORD-48213\",\"amount\":2499},\"result\":\"{\\\"ok\\\":true...}\"}"},{"t":"p","runs":[{"text":"On replay, the harness serves results from the file and never touches the network. Now the run is fast, free, offline, and repeatable: this is the same technique HTTP-testing libraries have called cassettes for years, applied one level up."}]},{"t":"p","runs":[{"text":"Two things to get right: "},{"text":"redact secrets and personal data at record time","b":true},{"text":", and "},{"text":"record the failures too","b":true},{"text":". A fixture directory of only happy paths tests the least interesting third of your code."}]},{"t":"h2","id":"three-layers-of-test","runs":[{"text":"Three layers of test"}]},{"t":"p","runs":[{"text":"Layer 1: the harness, with a fake model.","b":true},{"text":" Replace the model with a scripted sequence: “return a call to "},{"text":"get_order","code":true},{"text":", then a call to "},{"text":"refund_order","code":true},{"text":", then a final message.” Now test everything that is yours: budgets, stopping rules, retries, the idempotency key, validators, compaction, the permission gate. Fully deterministic, runs in CI, no API key, milliseconds."}]},{"t":"p","runs":[{"text":"This layer is the most valuable and the most skipped. It is where you test that the stopping rule stops."}]},{"t":"p","runs":[{"text":"Layer 2: single steps, real model, fixed context.","b":true},{"text":" “Given this observation, does it pick "},{"text":"search_orders","code":true},{"text":"?” Run it ten times and assert a rate: eight or more out of ten. A single pass tells you nothing about a sampled system."}]},{"t":"p","runs":[{"text":"Layer 3: golden runs.","b":true},{"text":" Twenty to forty real tasks with known-good outcomes, run nightly against the real model. Slow, costs money, and it is the only thing that catches a provider changing under you."}]},{"t":"h2","id":"assert-on-outcomes-never-on-prose","runs":[{"text":"Assert on outcomes, never on prose"}]},{"t":"code","lang":"python","label":"Python","text":"assert refunded_amount(\"ORD-48213\") == 2499\nassert tool_call_count(\"refund_order\") == 1\nassert \"ORD-48214\" not in touched_orders()"},{"t":"p","runs":[{"text":"Not "},{"text":"assert \"successfully refunded\" in output","code":true},{"text":". Wording changes between model versions and means nothing; state does not and means everything."}]},{"t":"h2","id":"flakiness-is-a-measurement-not-a-nuisance","runs":[{"text":"Flakiness is a measurement, not a nuisance"}]},{"t":"p","runs":[{"text":"Record the pass rate of each test over time instead of deleting the ones that wobble. A layer-2 test that passes seven times in ten is not broken: it is reporting that this step is 70% reliable, which is the number you needed for the compounding arithmetic in module one. Set thresholds, alert on drops, and treat a fall from 9/10 to 6/10 as the regression it is."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"pytest","code":true},{"text":" plus a directory of JSON fixtures. No framework needed. If you want a harness for prompt-level tests, "},{"text":"promptfoo","b":true},{"text":" and "},{"text":"DeepEval","b":true},{"text":" are open source and worth a look. The recorder is about forty lines."}]},{"t":"h2","id":"test-this-first","runs":[{"text":"Test this first"}]},{"t":"p","runs":[{"text":"The stopping rule. Before the prompt, before the tools, before anything. It is the cheapest test to write and the most expensive failure to have in production at 3am."}]}],"check":{"question":[{"text":"A team wants CI coverage for their agent but has no budget for API calls on every commit. Which test layer gives the most protection per rupee spent?"}],"options":[{"text":[{"text":"Harness tests driven by a fake model returning a scripted sequence of tool calls, with tool results replayed from recorded fixtures."}],"correct":true,"tempting":null},{"text":[{"text":"End-to-end golden runs against the real model on every commit."}],"correct":false,"tempting":[{"text":"The most expensive and slowest layer, run at the highest frequency; it belongs on a nightly schedule, not in the commit loop."}]},{"text":[{"text":"Assertions that the final message contains expected phrases."}],"correct":false,"tempting":[{"text":"Tests the wording of a sampled system, so it breaks on harmless rewording and passes on runs that did nothing."}]},{"text":[{"text":"A judge model scoring each run’s output out of ten, failing the build below eight."}],"correct":false,"tempting":[{"text":"Adds a second non-deterministic system as the arbiter, so the build now fails for two unrelated reasons and costs money each time."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"validators-not-vibes","title":"The validator is the product","minutes":9},"next":{"slug":"golden-tasks-for-agents","title":"Thirty tasks you run before every change","minutes":10}}}