Testing something that is not deterministic

The objection, and the answer

“You cannot unit-test an agent, the model is non-deterministic.”

You cannot unit-test the model. You can absolutely unit-test the harness, and the harness is where nearly all of your bugs are. The trick is to hold the other variable still.

Record and replay

Run the agent once, for real, with recording on. Every tool call and its result is written to a file keyed by tool name and canonical arguments.

tests/fixtures/refund_happy_path.jsonl
  {"tool":"get_order","args":{"order_id":"ORD-48213"},"result":"{\"status\":\"shipped\"...}"}
  {"tool":"refund_order","args":{"order_id":"ORD-48213","amount":2499},"result":"{\"ok\":true...}"}

On replay, the harness serves results from the file and never touches the network. Now the run is fast, free, offline, and repeatable: this is the same technique HTTP-testing libraries have called cassettes for years, applied one level up.

Two things to get right: redact secrets and personal data at record time, and record the failures too. A fixture directory of only happy paths tests the least interesting third of your code.

Three layers of test

Layer 1: the harness, with a fake model. Replace the model with a scripted sequence: “return a call to get_order, then a call to refund_order, then a final message.” Now test everything that is yours: budgets, stopping rules, retries, the idempotency key, validators, compaction, the permission gate. Fully deterministic, runs in CI, no API key, milliseconds.

This layer is the most valuable and the most skipped. It is where you test that the stopping rule stops.

Layer 2: single steps, real model, fixed context. “Given this observation, does it pick search_orders?” Run it ten times and assert a rate: eight or more out of ten. A single pass tells you nothing about a sampled system.

Layer 3: golden runs. Twenty to forty real tasks with known-good outcomes, run nightly against the real model. Slow, costs money, and it is the only thing that catches a provider changing under you.

Assert on outcomes, never on prose

assert refunded_amount("ORD-48213") == 2499
assert tool_call_count("refund_order") == 1
assert "ORD-48214" not in touched_orders()

Not assert "successfully refunded" in output. Wording changes between model versions and means nothing; state does not and means everything.

Flakiness is a measurement, not a nuisance

Record the pass rate of each test over time instead of deleting the ones that wobble. A layer-2 test that passes seven times in ten is not broken: it is reporting that this step is 70% reliable, which is the number you needed for the compounding arithmetic in module one. Set thresholds, alert on drops, and treat a fall from 9/10 to 6/10 as the regression it is.

The free path

pytest plus a directory of JSON fixtures. No framework needed. If you want a harness for prompt-level tests, promptfoo and DeepEval are open source and worth a look. The recorder is about forty lines.

Test this first

The stopping rule. Before the prompt, before the tools, before anything. It is the cheapest test to write and the most expensive failure to have in production at 3am.