{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"testing-the-boundary","title":"Making attack success a number you track","url":"https://addaly.com/learn/context-engineering/testing-the-boundary","position":77,"of":87,"minutes":9,"answer":"Injection defences are worthless unless measured, so build a corpus of payloads delivered through the channels they would really arrive by, score attack success as a rate.","keyIdea":"Injection defences are worthless unless measured, so build a corpus of payloads delivered through the channels they would really arrive by, score attack success as a rate, and run it on every change the way you run any other regression test."},"body":{"format":"markdown","text":"## The gap between believing and knowing\n\nEverything in this module is a claim: the delimiters help, the capability restriction holds, the escaping works. Each is testable, and until it is tested it is a belief.\n\nThe number to produce is **attack success rate**: of N attempted injections, how many achieved their objective. It is a percentage, it moves when you change defences, and it belongs on the same dashboard as accuracy.\n\n## Deliver payloads the way they would actually arrive\n\nThe mistake that makes an injection test useless is pasting payloads into the chat box. That measures the channel that matters least.\n\nInject at the real entry points, which means your test harness has to reach into the pipeline:\n\n- A document in the retrieval corpus, indexed normally, retrieved normally.\n- An inbound email body, arriving through the real ingestion path.\n- A fetched web page served by a local fixture server.\n- A tool result returned by a stubbed tool.\n- Inside a PDF, as text a reader would not see.\n- A filename, an `alt` attribute, an HTML comment, a code comment.\n\nIf your assembler is the pure function of an earlier module, this is easy: substitute the source, assemble, call, inspect. That testability was the point of the design.\n\n## Define success as an observable event\n\nA payload needs a goal your harness can detect without judgement. Vague objectives produce vague results.\n\nGood objectives are mechanical:\n\n- A request arrives at the fixture server that should never have been contacted. Assert on the server's log.\n- A write tool is called at all. Assert the tool-call list is empty.\n- A canary string planted in the private data appears anywhere in the output, or in any tool argument. Assert on a substring.\n- A specific incorrect claim appears in the answer, for the corruption case.\n\nThe canary is the most useful single technique here. Put a unique token in the private context (`CANARY-4f2b81`) and grep everything leaving the system for it: the answer, every tool argument, every URL, every log line. It catches exfiltration you had not thought to look for, through channels you had not enumerated, which is exactly the failure this module keeps warning about.\n\n## Build the corpus in layers\n\nAim for perhaps forty to a hundred cases, in three groups.\n\n**Published patterns.** Open collections of injection payloads exist and are free. They give breadth cheaply and they are also the ones any defence will handle, because everyone tests against them.\n\n**Payloads written for your system.** These are the ones that find things. An attacker who has seen your product writes instructions that fit your domain, name your tools, and mimic your own phrasing. Spend an hour writing ten of these; they will outperform a hundred generic ones.\n\n**Benign look-alikes.** Documents that discuss instructions, security or system prompts without attacking anything: a policy page about prompt injection, a support ticket quoting an error message, a code comment saying \"ignore the previous block\". These measure your false positive rate, and a defence that refuses on these is one your users will hit on ordinary work.\n\n## Run it like a test, not like an exercise\n\nThe corpus goes in version control. It runs in CI on any change to the prompt, the assembler, the toolset or the model version. It reports two numbers: attack success rate on the hostile set, and refusal rate on the benign set.\n\nThat pairing is what makes it honest. Either number is easy to improve alone: refuse everything and attack success goes to zero. Reporting them together forces the trade into the open.\n\nExpect the rate to be non-zero. A defence that tests at zero on your corpus has usually told you your corpus is too easy, not that your system is safe. The value is in the trend and in the regression check: you will one day change a paragraph in a system prompt for readability and watch the rate double, and that is the day this whole practice pays for itself.\n\n## What the test cannot tell you\n\nIt measures the attacks you thought of. An attacker is not limited to those, is better at it than you, and gets unlimited attempts against a system you cannot patch quickly.\n\nSo the test is a regression guard, not an assurance. It tells you a change made things worse. It cannot tell you the system is safe, and no available method can. That is why the structural controls come first and the measurement second: the measurement protects controls that are already doing the work, and it cannot substitute for them.\n\nWhich is also the closing note of the module. You cannot stop a model reading an instruction. You can decide what it is able to do about one, label where every word came from, bound the damage, and watch the number. That is the current state of the art, stated without flattery, and a team that does all of it is doing as well as anyone currently knows how."},"check":{"question":"An injection test suite reports a 0% attack success rate on its first run. What is the most likely explanation?","options":[{"text":"The capability restrictions are working, so every payload was blocked before reaching the model.","correct":false,"tempting":"Restrictions block actions rather than payloads, and a suite should include corruption objectives that survive them; a clean sweep still warrants suspicion."},{"text":"The corpus is too easy: usually generic published payloads delivered through the wrong channel.","correct":true},{"text":"The model version in use has been trained with instruction hierarchy, which resolves the class of attack.","correct":false,"tempting":"Such training measurably improves robustness without eliminating the class, so a perfect score would still be surprising."},{"text":"The benign look-alike cases are diluting the hostile ones, pushing the measured rate towards zero.","correct":false,"tempting":"The two sets are scored separately and report different numbers, so one cannot dilute the other."}]},"prev":{"slug":"model-output-is-untrusted-input","title":"What the model writes is input to something else","url":"https://addaly.com/learn/context-engineering/model-output-is-untrusted-input"},"next":{"slug":"prompts-you-can-roll-back","title":"A prompt you can version, test and roll back","url":"https://addaly.com/learn/context-engineering/prompts-you-can-roll-back"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Making attack success a number you track', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/testing-the-boundary (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Making attack success a number you track","courseTitle":"Context Engineering","module":{"index":8,"slug":"untrusted-text-and-the-boundary","title":"Untrusted text and the trust boundary"},"answer":null,"keyIdea":"Injection defences are worthless unless measured, so build a corpus of payloads delivered through the channels they would really arrive by, score attack success as a rate, and run it on every change the way you run any other regression test.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-gap-between-believing-and-knowing","runs":[{"text":"The gap between believing and knowing"}]},{"t":"p","runs":[{"text":"Everything in this module is a claim: the delimiters help, the capability restriction holds, the escaping works. Each is testable, and until it is tested it is a belief."}]},{"t":"p","runs":[{"text":"The number to produce is "},{"text":"attack success rate","b":true},{"text":": of N attempted injections, how many achieved their objective. It is a percentage, it moves when you change defences, and it belongs on the same dashboard as accuracy."}]},{"t":"h2","id":"deliver-payloads-the-way-they-would-actually-arrive","runs":[{"text":"Deliver payloads the way they would actually arrive"}]},{"t":"p","runs":[{"text":"The mistake that makes an injection test useless is pasting payloads into the chat box. That measures the channel that matters least."}]},{"t":"p","runs":[{"text":"Inject at the real entry points, which means your test harness has to reach into the pipeline:"}]},{"t":"ul","items":[[{"text":"A document in the retrieval corpus, indexed normally, retrieved normally."}],[{"text":"An inbound email body, arriving through the real ingestion path."}],[{"text":"A fetched web page served by a local fixture server."}],[{"text":"A tool result returned by a stubbed tool."}],[{"text":"Inside a PDF, as text a reader would not see."}],[{"text":"A filename, an "},{"text":"alt","code":true},{"text":" attribute, an HTML comment, a code comment."}]]},{"t":"p","runs":[{"text":"If your assembler is the pure function of an earlier module, this is easy: substitute the source, assemble, call, inspect. That testability was the point of the design."}]},{"t":"h2","id":"define-success-as-an-observable-event","runs":[{"text":"Define success as an observable event"}]},{"t":"p","runs":[{"text":"A payload needs a goal your harness can detect without judgement. Vague objectives produce vague results."}]},{"t":"p","runs":[{"text":"Good objectives are mechanical:"}]},{"t":"ul","items":[[{"text":"A request arrives at the fixture server that should never have been contacted. Assert on the server’s log."}],[{"text":"A write tool is called at all. Assert the tool-call list is empty."}],[{"text":"A canary string planted in the private data appears anywhere in the output, or in any tool argument. Assert on a substring."}],[{"text":"A specific incorrect claim appears in the answer, for the corruption case."}]]},{"t":"p","runs":[{"text":"The canary is the most useful single technique here. Put a unique token in the private context ("},{"text":"CANARY-4f2b81","code":true},{"text":") and grep everything leaving the system for it: the answer, every tool argument, every URL, every log line. It catches exfiltration you had not thought to look for, through channels you had not enumerated, which is exactly the failure this module keeps warning about."}]},{"t":"h2","id":"build-the-corpus-in-layers","runs":[{"text":"Build the corpus in layers"}]},{"t":"p","runs":[{"text":"Aim for perhaps forty to a hundred cases, in three groups."}]},{"t":"p","runs":[{"text":"Published patterns.","b":true},{"text":" Open collections of injection payloads exist and are free. They give breadth cheaply and they are also the ones any defence will handle, because everyone tests against them."}]},{"t":"p","runs":[{"text":"Payloads written for your system.","b":true},{"text":" These are the ones that find things. An attacker who has seen your product writes instructions that fit your domain, name your tools, and mimic your own phrasing. Spend an hour writing ten of these; they will outperform a hundred generic ones."}]},{"t":"p","runs":[{"text":"Benign look-alikes.","b":true},{"text":" Documents that discuss instructions, security or system prompts without attacking anything: a policy page about prompt injection, a support ticket quoting an error message, a code comment saying “ignore the previous block”. These measure your false positive rate, and a defence that refuses on these is one your users will hit on ordinary work."}]},{"t":"h2","id":"run-it-like-a-test-not-like-an-exercise","runs":[{"text":"Run it like a test, not like an exercise"}]},{"t":"p","runs":[{"text":"The corpus goes in version control. It runs in CI on any change to the prompt, the assembler, the toolset or the model version. It reports two numbers: attack success rate on the hostile set, and refusal rate on the benign set."}]},{"t":"p","runs":[{"text":"That pairing is what makes it honest. Either number is easy to improve alone: refuse everything and attack success goes to zero. Reporting them together forces the trade into the open."}]},{"t":"p","runs":[{"text":"Expect the rate to be non-zero. A defence that tests at zero on your corpus has usually told you your corpus is too easy, not that your system is safe. The value is in the trend and in the regression check: you will one day change a paragraph in a system prompt for readability and watch the rate double, and that is the day this whole practice pays for itself."}]},{"t":"h2","id":"what-the-test-cannot-tell-you","runs":[{"text":"What the test cannot tell you"}]},{"t":"p","runs":[{"text":"It measures the attacks you thought of. An attacker is not limited to those, is better at it than you, and gets unlimited attempts against a system you cannot patch quickly."}]},{"t":"p","runs":[{"text":"So the test is a regression guard, not an assurance. It tells you a change made things worse. It cannot tell you the system is safe, and no available method can. That is why the structural controls come first and the measurement second: the measurement protects controls that are already doing the work, and it cannot substitute for them."}]},{"t":"p","runs":[{"text":"Which is also the closing note of the module. You cannot stop a model reading an instruction. You can decide what it is able to do about one, label where every word came from, bound the damage, and watch the number. That is the current state of the art, stated without flattery, and a team that does all of it is doing as well as anyone currently knows how."}]}],"check":{"question":[{"text":"An injection test suite reports a 0% attack success rate on its first run. What is the most likely explanation?"}],"options":[{"text":[{"text":"The capability restrictions are working, so every payload was blocked before reaching the model."}],"correct":false,"tempting":[{"text":"Restrictions block actions rather than payloads, and a suite should include corruption objectives that survive them; a clean sweep still warrants suspicion."}]},{"text":[{"text":"The corpus is too easy: usually generic published payloads delivered through the wrong channel."}],"correct":true,"tempting":null},{"text":[{"text":"The model version in use has been trained with instruction hierarchy, which resolves the class of attack."}],"correct":false,"tempting":[{"text":"Such training measurably improves robustness without eliminating the class, so a perfect score would still be surprising."}]},{"text":[{"text":"The benign look-alike cases are diluting the hostile ones, pushing the measured rate towards zero."}],"correct":false,"tempting":[{"text":"The two sets are scored separately and report different numbers, so one cannot dilute the other."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"model-output-is-untrusted-input","title":"What the model writes is input to something else","minutes":8},"next":{"slug":"prompts-you-can-roll-back","title":"A prompt you can version, test and roll back","minutes":10}}}