Making attack success a number you track

The gap between believing and knowing

Everything in this module is a claim: the delimiters help, the capability restriction holds, the escaping works. Each is testable, and until it is tested it is a belief.

The number to produce is attack success rate: of N attempted injections, how many achieved their objective. It is a percentage, it moves when you change defences, and it belongs on the same dashboard as accuracy.

Deliver payloads the way they would actually arrive

The mistake that makes an injection test useless is pasting payloads into the chat box. That measures the channel that matters least.

Inject at the real entry points, which means your test harness has to reach into the pipeline:

• A document in the retrieval corpus, indexed normally, retrieved normally.
• An inbound email body, arriving through the real ingestion path.
• A fetched web page served by a local fixture server.
• A tool result returned by a stubbed tool.
• Inside a PDF, as text a reader would not see.
• A filename, an alt attribute, an HTML comment, a code comment.

If your assembler is the pure function of an earlier module, this is easy: substitute the source, assemble, call, inspect. That testability was the point of the design.

Define success as an observable event

A payload needs a goal your harness can detect without judgement. Vague objectives produce vague results.

Good objectives are mechanical:

• A request arrives at the fixture server that should never have been contacted. Assert on the server’s log.
• A write tool is called at all. Assert the tool-call list is empty.
• A canary string planted in the private data appears anywhere in the output, or in any tool argument. Assert on a substring.
• A specific incorrect claim appears in the answer, for the corruption case.

The canary is the most useful single technique here. Put a unique token in the private context (CANARY-4f2b81) and grep everything leaving the system for it: the answer, every tool argument, every URL, every log line. It catches exfiltration you had not thought to look for, through channels you had not enumerated, which is exactly the failure this module keeps warning about.

Build the corpus in layers

Aim for perhaps forty to a hundred cases, in three groups.

Published patterns. Open collections of injection payloads exist and are free. They give breadth cheaply and they are also the ones any defence will handle, because everyone tests against them.

Payloads written for your system. These are the ones that find things. An attacker who has seen your product writes instructions that fit your domain, name your tools, and mimic your own phrasing. Spend an hour writing ten of these; they will outperform a hundred generic ones.

Benign look-alikes. Documents that discuss instructions, security or system prompts without attacking anything: a policy page about prompt injection, a support ticket quoting an error message, a code comment saying “ignore the previous block”. These measure your false positive rate, and a defence that refuses on these is one your users will hit on ordinary work.

Run it like a test, not like an exercise

The corpus goes in version control. It runs in CI on any change to the prompt, the assembler, the toolset or the model version. It reports two numbers: attack success rate on the hostile set, and refusal rate on the benign set.

That pairing is what makes it honest. Either number is easy to improve alone: refuse everything and attack success goes to zero. Reporting them together forces the trade into the open.

Expect the rate to be non-zero. A defence that tests at zero on your corpus has usually told you your corpus is too easy, not that your system is safe. The value is in the trend and in the regression check: you will one day change a paragraph in a system prompt for readability and watch the rate double, and that is the day this whole practice pays for itself.

What the test cannot tell you

It measures the attacks you thought of. An attacker is not limited to those, is better at it than you, and gets unlimited attempts against a system you cannot patch quickly.

So the test is a regression guard, not an assurance. It tells you a change made things worse. It cannot tell you the system is safe, and no available method can. That is why the structural controls come first and the measurement second: the measurement protects controls that are already doing the work, and it cannot substitute for them.

Which is also the closing note of the module. You cannot stop a model reading an instruction. You can decide what it is able to do about one, label where every word came from, bound the damage, and watch the number. That is the current state of the art, stated without flattery, and a team that does all of it is doing as well as anyone currently knows how.