Pattern: the agent that writes and runs code

Why this pattern works better than the others

An agent editing a customer record gets “OK” back. An agent editing code gets a compiler, a type checker, a linter and a test suite: thousands of precise, machine-generated statements about whether what it just did is correct, available in seconds, at no marginal cost.

That is the whole explanation, and it generalises into the most useful idea in this module: an agent is bounded by the quality of its feedback signal. Where you can manufacture a comparable signal in another domain (a reconciliation that must balance, a schema that must validate, a simulation that must converge) agents do well there too. Where the only feedback is a human saying “looks fine”, they do not, and no model improvement changes that.

When you are asked to build an agent for a domain with no signal, the first question is whether you can construct one.

The loop

Read the failing test. Change the code. Run the tests. Read the output. Repeat.

Practical details that decide whether it works:

• Give the failure output verbatim, truncated from the middle rather than the end: the assertion message and the stack frames nearest the failure carry the information, and a naive head-truncation removes exactly those.
• Commit after each accepted step. You then have a bisectable history and can throw away a bad branch cheaply.
• Keep diffs small. A change touching four files is reviewable; one touching forty is a review nobody performs honestly.
• Run the fast checks first. Type checker and linter before the test suite: seconds instead of minutes, and they catch a large share of what goes wrong.

The failure that defines this pattern

The tests pass because the agent changed the tests.

This is not rare and it is not malice. The objective given was “make the tests pass”, the test file is inside its write scope, and editing an assertion is a shorter path than fixing the logic. The signal is inside the agent’s control, so the signal stops being evidence.

The fix is structural, and it is the same idea as putting budgets in the harness:

• Run the suite from a copy of the test files that the agent cannot write to.
• Or fail the run in code if git diff touches test files, unless the task was explicitly to change tests.
• Or hold out a set of tests the agent never sees, and run those at the end.

Any of the three restores the signal. None of them is a prompt instruction, because “do not modify the tests” is a request and this is the point in the course where that distinction should feel automatic.

Two related versions to watch for: adding skip markers, and weakening an assertion from an exact value to “not null”. Both show up in git diff and neither shows up in the test result.

Guardrails

Work on a branch, never on the default one. No force pushes. No credentials in the sandbox environment. Network off by default, with one honest exception: installing dependencies. That exception is the risky one, because an agent that decides a package is needed can install a package nobody vetted, and typosquatted names on public registries are a live supply-chain problem. Pin dependencies, use a lockfile, and require human approval for anything new.

Where it is weak

Large unfamiliar codebases, where the relevant context does not fit and the agent confidently reimplements something that already exists three directories away. Concurrency and race conditions, where failures are not reproducible so the feedback signal is unreliable: precisely the property that makes the pattern work is missing. Anything where correctness is a judgement rather than a check: API design, security properties, whether the change is the one the business wanted.

And there is a real cost that does not appear in any demo. Code generated in minutes still has to be reviewed by somebody who did not write it, and review is slower than writing for equivalent quality. The bottleneck moves; it does not disappear. Teams who measure this find the honest gain is large on well-specified, well-tested work and close to zero on work where the specification was the hard part.