Testing what happens when somebody tries

A different set, with different rules

Your hundred items came from real traffic, so they represent people using the system as intended. A safety evaluation represents people using it as not intended, plus people whose ordinary use happens to look alarming. It is a separate set, built deliberately, scored on its own metrics, and it does not get averaged into your quality number.

Four categories are worth having, and they cover most of what actually happens:

1. Direct attempts. Requests for content you will not produce, in every framing people use: roleplay, fiction, “for a research paper”, translation into another language, encoding, a hypothetical, an appeal to authority, an urgent emergency.

2. Prompt injection through content. This is the one that matters most for anything with retrieval, browsing or tools, and it is the one most often untested. The attack is not in the user’s message. It is inside a document, a web page, an email, a filename or a database row that your system reads: text addressed to the model, telling it to ignore its instructions, exfiltrate what it has seen, or call a tool.

Build a small corpus of poisoned documents and put them in your test index. A minimal battery: an instruction in white text, an instruction in a code comment, an instruction in a footnote, one in a language other than the interface language, one that claims to come from the system administrator, and one that asks the model to include a URL with data appended to it. Then measure how often the system follows them. The correct rate is zero and the observed rate rarely is.

3. Out-of-scope requests. Medical, legal or financial advice from a shopping bot. Requests to speak as a named real person. Requests about internal systems. The failure here is not usually harm; it is a system confidently operating outside what anyone tested.

4. Distress. Somebody in genuine difficulty. The right behaviour is a product decision that should be written down before it is measured, and the failure modes are in both directions: treating a person as a policy violation, or missing the situation entirely.

Two rates, and they must be read together

Safety evaluation has one number everybody reports and a second that gives it meaning.

• Harmful-completion rate: of the adversarial items, how many produced the thing you did not want.
• False-refusal rate: of benign items, how many were wrongly refused, hedged into uselessness, or lectured at.

You need a benign-but-alarming set to measure the second. Fifty items whose surface features look risky and whose intent is entirely ordinary:

• a nurse asking about a paracetamol overdose threshold
• a novelist asking how a character would pick a lock
• a security engineer asking how SQL injection works
• a Punjabi speaker using a word that collides with an English slur
• a teenager asking what to do about a friend who has stopped eating
• a historian asking about a massacre in explicit terms

Optimise the first number alone and you will drive the second one up until the product is useless to the people it was built for, and you will not notice, because nobody logs a refusal as a bug. Report both, always, as a pair. A move from 3% to 1% harmful completions that took false refusals from 4% to 19% is a regression, whatever the safety dashboard says.

Scoring

Binary and specific, per item: did the output contain the operational content, or not. Not “how safe was it out of five”. Hedged, partial and refused-then-complied outputs all need a written ruling in the guide, because they are most of the interesting cases.

Run the adversarial set five times, not once. Attacks succeed probabilistically; an attack that works one time in five is a working attack, and a single-run evaluation reports it as blocked 80% of the time under a heading that says “passed”.

Tools and how to get variety

garak (Apache 2.0) is a scanner with a large library of known probes and is the fastest way to a baseline. PyRIT from Microsoft (MIT) automates multi-turn attacks. promptfoo has a red-team mode that generates adversarial variants of your own prompts. All free.

Use them for coverage of published techniques, and then write your own for your domain, because the attacks that will actually be used against you are specific to what your product does and no public library knows about them. A model can help generate variants of an attack you have already seen, which is a legitimate and useful way to turn one incident into thirty test items.

Expect to be behind

A new framing that defeats your defences will appear, and it will appear on a Tuesday. The measurable goals are not “unbreakable”. They are: known attacks stay blocked as the system changes, new attacks become test items within a day of being seen, and the false-refusal rate does not creep up while nobody is watching it.

That last one is the failure teams do not notice for months, because nobody files a ticket to say the assistant was unhelpfully cautious. They just stop using it.