{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-code-generation","title":"Code: run it, in a box, against tests it has not seen","url":"https://addaly.com/learn/evaluating-ai/evaluating-code-generation","position":32,"of":76,"minutes":9,"answer":"Generated code is the one output you can grade by execution, so evaluate with hidden tests in a sandbox, quote pass@1 when a person will accept one answer.","keyIdea":"Generated code is the one output you can grade by execution, so evaluate with hidden tests in a sandbox, quote pass@1 when a person will accept one answer, and build the set from your own repository because the public ones are in the training data."},"body":{"format":"markdown","text":"## The rare task with a real oracle\n\nAlmost everything else in this course needs a judgement. Code has an oracle: run it and see. Any evaluation of code generation that reads the code instead of executing it is throwing away its main advantage: plausible-looking code that does not run is exactly the failure mode.\n\n## pass@k, and which k your product needs\n\nSample `n` completions for a problem, count how many pass the tests, and estimate the probability that at least one of `k` samples works:\n\n```\npass@k = 1 - C(n - c, k) / C(n, k)\n```\n\nWith `n = 10` samples of which `c = 3` pass:\n\n- `pass@1 = 3/10 = 0.30`\n- `pass@5 = 1 - C(7,5)/C(10,5) = 1 - 21/252 = 0.92`\n\nBoth are true and they describe different products. If a person sees one suggestion and accepts or rejects it, `pass@1` is your number. If the tool shows five options, or an agent retries against a failing test, `pass@k` is. Papers quote the flattering one; decide yours from the interface and state which you used.\n\n## The sandbox is not optional\n\nGenerated code will, sooner or later, delete files, open a socket, or spin forever. Run it isolated, always:\n\n```bash\ndocker run --rm --network=none --memory=512m --pids-limit=64 \\\n  --read-only -v \"$PWD/work:/work:ro\" -u 1000:1000 \\\n  python:3.12-slim timeout 10 python /work/candidate.py\n```\n\nNo network, a memory cap, a process cap, a read-only mount, a non-root user, and a hard timeout. Podman works identically and needs no daemon. Treat a timeout as a failure and record it separately, because a spike in timeouts usually means the model started writing loops rather than logic.\n\n> Figure: Grading generated code by running it. Steps: 1. Sample n completions (Ten per problem is typical.); 2. Run each in a box (No network, memory cap, process cap, read-only mount, hard timeout. A timeout is a failure, counted separately.); 3. Hidden adversarial tests (Empty, boundary, large and wrong-type inputs. A naive stub must fail them.); 4. pass@k (pass@1 if a person sees one suggestion; pass@k if the tool shows k or retries.); 5. Free static checks (Lint, imports against the lockfile, a scanner, the size of the diff.). With n = 10 samples and 3 passing, pass@1 is 0.30 and pass@5 is 0.92. Both are true; the interface decides which one is yours.\n\n## Tests decide what you are measuring\n\nHidden tests must be adversarial or you are measuring happy paths. For each problem, include empty input, a boundary value, a large input, and a wrong-type input. Property-based testing with `hypothesis` is the cheapest way to get past your own imagination: state the invariant, let it generate hundreds of inputs.\n\nThen check the tests themselves. If a naive stub implementation passes them, they are too weak. Running the reference solution and a deliberately broken one through the suite takes minutes and tells you whether the suite discriminates.\n\nOne more detail decides whether the numbers move at all: **flaky tests**. A suite that depends on the clock, on network access, on dictionary ordering, or on a temporary directory will fail some candidates for reasons the model did not cause. Run the reference solution through the suite twenty times before you trust it; any test that is not deterministic across those runs comes out of the set, or your comparison inherits its noise.\n\n## Public benchmarks tell you about the training data\n\nHumanEval and MBPP have been in public repositories for years and are in the training corpora of every current model. A high score on them is evidence about memorisation as much as about capability, and it says nothing about your codebase, your framework versions or your internal conventions.\n\nBuild a private set from your own repository. A workable recipe: take 100 merged pull requests that touched a single function and had tests, hide the implementation, keep the tests and the docstring, and ask the model to fill it in. It takes a day, it cannot leak, and it measures the task you actually have.\n\n## Correctness is not the whole of it\n\nPassing tests is necessary and not sufficient. Add checks that run free on every candidate:\n\n- **Does it typecheck and lint?** `mypy`, `ruff`, `tsc`: a suggestion that fails CI wastes the developer's time even if it is logically right.\n- **Does every import resolve?** Models invent package names, and an invented name is a supply-chain risk the moment somebody publishes a package under it. Compare imports against the lockfile.\n- **Does a static scanner flag it?** `semgrep` and `bandit` catch injected SQL, shelling out with user input, and disabled certificate checks.\n- **How large is the diff?** A patch that rewrites the file to fix one line is technically passing and practically rejected.\n\n## What none of this measures\n\nThe eventual value is developer time, and it is not the same as pass rate. Measure the product outcome separately: acceptance rate of suggestions, time to a merged pull request, and how often a review finds a defect that the tests did not. A model with a lower pass rate that produces small, readable, obviously-wrong-when-wrong patches can be worth more than one that produces large patches which pass and nobody can review."},"check":{"question":"A team samples 10 completions per problem, finds 3 pass on average, and reports \"92% pass@5\" for an editor plug-in that shows one suggestion inline. What is wrong?","options":[{"text":"Nothing, provided the plug-in retries automatically when the user rejects the first suggestion.","correct":false,"tempting":"Would justify a k above one only if the interface really samples repeatedly; the stated product shows one suggestion, so k = 5 does not describe it."},{"text":"pass@5 requires at least 50 samples per problem to be estimated without bias.","correct":false,"tempting":"Confuses estimator variance with validity; the standard estimator is unbiased for any n greater than or equal to k, and 10 samples is normal practice."},{"text":"The product shows one suggestion, so the relevant figure is pass@1 at 30%, and pass@5 describes a system nobody is shipping.","correct":true},{"text":"The number should have been computed on a public benchmark so it can be compared with published results.","correct":false,"tempting":"Reaches for exactly the comparison this lesson warns against, since public benchmarks sit in the training data and describe a different codebase."}]},"prev":{"slug":"evaluating-translation","title":"Translation, and the languages nobody measures","url":"https://addaly.com/learn/evaluating-ai/evaluating-translation"},"next":{"slug":"evaluating-extraction","title":"Structured extraction, where normalisation is most of the work","url":"https://addaly.com/learn/evaluating-ai/evaluating-extraction"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Code: run it, in a box, against tests it has not seen', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-code-generation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Code: run it, in a box, against tests it has not seen","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"Generated code is the one output you can grade by execution, so evaluate with hidden tests in a sandbox, quote pass@1 when a person will accept one answer, and build the set from your own repository because the public ones are in the training data.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Sample n completions","detail":"Ten per problem is typical."},{"label":"Run each in a box","detail":"No network, memory cap, process cap, read-only mount, hard timeout. A timeout is a failure, counted separately."},{"label":"Hidden adversarial tests","detail":"Empty, boundary, large and wrong-type inputs. A naive stub must fail them."},{"label":"pass@k","detail":"pass@1 if a person sees one suggestion; pass@k if the tool shows k or retries."},{"label":"Free static checks","detail":"Lint, imports against the lockfile, a scanner, the size of the diff."}],"title":"Grading generated code by running it","caption":"With n = 10 samples and 3 passing, pass@1 is 0.30 and pass@5 is 0.92. Both are true; the interface decides which one is yours."},"alt":"Figure: Grading generated code by running it. Steps: 1. Sample n completions (Ten per problem is typical.); 2. Run each in a box (No network, memory cap, process cap, read-only mount, hard timeout. A timeout is a failure, counted separately.); 3. Hidden adversarial tests (Empty, boundary, large and wrong-type inputs. A naive stub must fail them.); 4. pass@k (pass@1 if a person sees one suggestion; pass@k if the tool shows k or retries.); 5. Free static checks (Lint, imports against the lockfile, a scanner, the size of the diff.). With n = 10 samples and 3 passing, pass@1 is 0.30 and pass@5 is 0.92. Both are true; the interface decides which one is yours."}},"blocks":[{"t":"h2","id":"the-rare-task-with-a-real-oracle","runs":[{"text":"The rare task with a real oracle"}]},{"t":"p","runs":[{"text":"Almost everything else in this course needs a judgement. Code has an oracle: run it and see. Any evaluation of code generation that reads the code instead of executing it is throwing away its main advantage: plausible-looking code that does not run is exactly the failure mode."}]},{"t":"h2","id":"pass-k-and-which-k-your-product-needs","runs":[{"text":"pass@k, and which k your product needs"}]},{"t":"p","runs":[{"text":"Sample "},{"text":"n","code":true},{"text":" completions for a problem, count how many pass the tests, and estimate the probability that at least one of "},{"text":"k","code":true},{"text":" samples works:"}]},{"t":"code","lang":"","label":"Text","text":"pass@k = 1 - C(n - c, k) / C(n, k)"},{"t":"p","runs":[{"text":"With "},{"text":"n = 10","code":true},{"text":" samples of which "},{"text":"c = 3","code":true},{"text":" pass:"}]},{"t":"ul","items":[[{"text":"pass@1 = 3/10 = 0.30","code":true}],[{"text":"pass@5 = 1 - C(7,5)/C(10,5) = 1 - 21/252 = 0.92","code":true}]]},{"t":"p","runs":[{"text":"Both are true and they describe different products. If a person sees one suggestion and accepts or rejects it, "},{"text":"pass@1","code":true},{"text":" is your number. If the tool shows five options, or an agent retries against a failing test, "},{"text":"pass@k","code":true},{"text":" is. Papers quote the flattering one; decide yours from the interface and state which you used."}]},{"t":"h2","id":"the-sandbox-is-not-optional","runs":[{"text":"The sandbox is not optional"}]},{"t":"p","runs":[{"text":"Generated code will, sooner or later, delete files, open a socket, or spin forever. Run it isolated, always:"}]},{"t":"code","lang":"bash","label":"Bash","text":"docker run --rm --network=none --memory=512m --pids-limit=64 \\\n  --read-only -v \"$PWD/work:/work:ro\" -u 1000:1000 \\\n  python:3.12-slim timeout 10 python /work/candidate.py"},{"t":"p","runs":[{"text":"No network, a memory cap, a process cap, a read-only mount, a non-root user, and a hard timeout. Podman works identically and needs no daemon. Treat a timeout as a failure and record it separately, because a spike in timeouts usually means the model started writing loops rather than logic."}]},{"t":"h2","id":"tests-decide-what-you-are-measuring","runs":[{"text":"Tests decide what you are measuring"}]},{"t":"p","runs":[{"text":"Hidden tests must be adversarial or you are measuring happy paths. For each problem, include empty input, a boundary value, a large input, and a wrong-type input. Property-based testing with "},{"text":"hypothesis","code":true},{"text":" is the cheapest way to get past your own imagination: state the invariant, let it generate hundreds of inputs."}]},{"t":"p","runs":[{"text":"Then check the tests themselves. If a naive stub implementation passes them, they are too weak. Running the reference solution and a deliberately broken one through the suite takes minutes and tells you whether the suite discriminates."}]},{"t":"p","runs":[{"text":"One more detail decides whether the numbers move at all: "},{"text":"flaky tests","b":true},{"text":". A suite that depends on the clock, on network access, on dictionary ordering, or on a temporary directory will fail some candidates for reasons the model did not cause. Run the reference solution through the suite twenty times before you trust it; any test that is not deterministic across those runs comes out of the set, or your comparison inherits its noise."}]},{"t":"h2","id":"public-benchmarks-tell-you-about-the-training-data","runs":[{"text":"Public benchmarks tell you about the training data"}]},{"t":"p","runs":[{"text":"HumanEval and MBPP have been in public repositories for years and are in the training corpora of every current model. A high score on them is evidence about memorisation as much as about capability, and it says nothing about your codebase, your framework versions or your internal conventions."}]},{"t":"p","runs":[{"text":"Build a private set from your own repository. A workable recipe: take 100 merged pull requests that touched a single function and had tests, hide the implementation, keep the tests and the docstring, and ask the model to fill it in. It takes a day, it cannot leak, and it measures the task you actually have."}]},{"t":"h2","id":"correctness-is-not-the-whole-of-it","runs":[{"text":"Correctness is not the whole of it"}]},{"t":"p","runs":[{"text":"Passing tests is necessary and not sufficient. Add checks that run free on every candidate:"}]},{"t":"ul","items":[[{"text":"Does it typecheck and lint?","b":true},{"text":" "},{"text":"mypy","code":true},{"text":", "},{"text":"ruff","code":true},{"text":", "},{"text":"tsc","code":true},{"text":": a suggestion that fails CI wastes the developer’s time even if it is logically right."}],[{"text":"Does every import resolve?","b":true},{"text":" Models invent package names, and an invented name is a supply-chain risk the moment somebody publishes a package under it. Compare imports against the lockfile."}],[{"text":"Does a static scanner flag it?","b":true},{"text":" "},{"text":"semgrep","code":true},{"text":" and "},{"text":"bandit","code":true},{"text":" catch injected SQL, shelling out with user input, and disabled certificate checks."}],[{"text":"How large is the diff?","b":true},{"text":" A patch that rewrites the file to fix one line is technically passing and practically rejected."}]]},{"t":"h2","id":"what-none-of-this-measures","runs":[{"text":"What none of this measures"}]},{"t":"p","runs":[{"text":"The eventual value is developer time, and it is not the same as pass rate. Measure the product outcome separately: acceptance rate of suggestions, time to a merged pull request, and how often a review finds a defect that the tests did not. A model with a lower pass rate that produces small, readable, obviously-wrong-when-wrong patches can be worth more than one that produces large patches which pass and nobody can review."}]}],"check":{"question":[{"text":"A team samples 10 completions per problem, finds 3 pass on average, and reports “92% pass@5” for an editor plug-in that shows one suggestion inline. What is wrong?"}],"options":[{"text":[{"text":"Nothing, provided the plug-in retries automatically when the user rejects the first suggestion."}],"correct":false,"tempting":[{"text":"Would justify a k above one only if the interface really samples repeatedly; the stated product shows one suggestion, so k = 5 does not describe it."}]},{"text":[{"text":"pass@5 requires at least 50 samples per problem to be estimated without bias."}],"correct":false,"tempting":[{"text":"Confuses estimator variance with validity; the standard estimator is unbiased for any n greater than or equal to k, and 10 samples is normal practice."}]},{"text":[{"text":"The product shows one suggestion, so the relevant figure is pass@1 at 30%, and pass@5 describes a system nobody is shipping."}],"correct":true,"tempting":null},{"text":[{"text":"The number should have been computed on a public benchmark so it can be compared with published results."}],"correct":false,"tempting":[{"text":"Reaches for exactly the comparison this lesson warns against, since public benchmarks sit in the training data and describe a different codebase."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"evaluating-translation","title":"Translation, and the languages nobody measures","minutes":10},"next":{"slug":"evaluating-extraction","title":"Structured extraction, where normalisation is most of the work","minutes":9}}}