Code: run it, in a box, against tests it has not seen

The rare task with a real oracle

Almost everything else in this course needs a judgement. Code has an oracle: run it and see. Any evaluation of code generation that reads the code instead of executing it is throwing away its main advantage: plausible-looking code that does not run is exactly the failure mode.

pass@k, and which k your product needs

Sample n completions for a problem, count how many pass the tests, and estimate the probability that at least one of k samples works:

pass@k = 1 - C(n - c, k) / C(n, k)

With n = 10 samples of which c = 3 pass:

• pass@1 = 3/10 = 0.30
• pass@5 = 1 - C(7,5)/C(10,5) = 1 - 21/252 = 0.92

Both are true and they describe different products. If a person sees one suggestion and accepts or rejects it, pass@1 is your number. If the tool shows five options, or an agent retries against a failing test, pass@k is. Papers quote the flattering one; decide yours from the interface and state which you used.

The sandbox is not optional

Generated code will, sooner or later, delete files, open a socket, or spin forever. Run it isolated, always:

docker run --rm --network=none --memory=512m --pids-limit=64 \
  --read-only -v "$PWD/work:/work:ro" -u 1000:1000 \
  python:3.12-slim timeout 10 python /work/candidate.py

No network, a memory cap, a process cap, a read-only mount, a non-root user, and a hard timeout. Podman works identically and needs no daemon. Treat a timeout as a failure and record it separately, because a spike in timeouts usually means the model started writing loops rather than logic.

Tests decide what you are measuring

Hidden tests must be adversarial or you are measuring happy paths. For each problem, include empty input, a boundary value, a large input, and a wrong-type input. Property-based testing with hypothesis is the cheapest way to get past your own imagination: state the invariant, let it generate hundreds of inputs.

Then check the tests themselves. If a naive stub implementation passes them, they are too weak. Running the reference solution and a deliberately broken one through the suite takes minutes and tells you whether the suite discriminates.

One more detail decides whether the numbers move at all: flaky tests. A suite that depends on the clock, on network access, on dictionary ordering, or on a temporary directory will fail some candidates for reasons the model did not cause. Run the reference solution through the suite twenty times before you trust it; any test that is not deterministic across those runs comes out of the set, or your comparison inherits its noise.

Public benchmarks tell you about the training data

HumanEval and MBPP have been in public repositories for years and are in the training corpora of every current model. A high score on them is evidence about memorisation as much as about capability, and it says nothing about your codebase, your framework versions or your internal conventions.

Build a private set from your own repository. A workable recipe: take 100 merged pull requests that touched a single function and had tests, hide the implementation, keep the tests and the docstring, and ask the model to fill it in. It takes a day, it cannot leak, and it measures the task you actually have.

Correctness is not the whole of it

Passing tests is necessary and not sufficient. Add checks that run free on every candidate:

• Does it typecheck and lint? mypy, ruff, tsc: a suggestion that fails CI wastes the developer’s time even if it is logically right.
• Does every import resolve? Models invent package names, and an invented name is a supply-chain risk the moment somebody publishes a package under it. Compare imports against the lockfile.
• Does a static scanner flag it? semgrep and bandit catch injected SQL, shelling out with user input, and disabled certificate checks.
• How large is the diff? A patch that rewrites the file to fix one line is technically passing and practically rejected.

What none of this measures

The eventual value is developer time, and it is not the same as pass rate. Measure the product outcome separately: acceptance rate of suggestions, time to a merged pull request, and how often a review finds a defect that the tests did not. A model with a lower pass rate that produces small, readable, obviously-wrong-when-wrong patches can be worth more than one that produces large patches which pass and nobody can review.