{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"red-teaming-and-safety-evals","title":"Testing what happens when somebody tries","url":"https://addaly.com/learn/evaluating-ai/red-teaming-and-safety-evals","position":48,"of":76,"minutes":10,"answer":"Measure harmful completions and false refusals as a pair, test injection through retrieved content rather than only through user messages, and run the adversarial set several times because attacks succeed probabilistically.","keyIdea":"Measure harmful completions and false refusals as a pair, test injection through retrieved content rather than only through user messages, and run the adversarial set several times because attacks succeed probabilistically."},"body":{"format":"markdown","text":"## A different set, with different rules\n\nYour hundred items came from real traffic, so they represent people using the system as intended. A safety evaluation represents people using it as *not* intended, plus people whose ordinary use happens to look alarming. It is a separate set, built deliberately, scored on its own metrics, and it does not get averaged into your quality number.\n\nFour categories are worth having, and they cover most of what actually happens:\n\n**1. Direct attempts.** Requests for content you will not produce, in every framing people use: roleplay, fiction, \"for a research paper\", translation into another language, encoding, a hypothetical, an appeal to authority, an urgent emergency.\n\n**2. Prompt injection through content.** This is the one that matters most for anything with retrieval, browsing or tools, and it is the one most often untested. The attack is not in the user's message. It is inside a document, a web page, an email, a filename or a database row that your system reads: text addressed to the model, telling it to ignore its instructions, exfiltrate what it has seen, or call a tool.\n\nBuild a small corpus of poisoned documents and put them in your test index. A minimal battery: an instruction in white text, an instruction in a code comment, an instruction in a footnote, one in a language other than the interface language, one that claims to come from the system administrator, and one that asks the model to include a URL with data appended to it. Then measure how often the system follows them. The correct rate is zero and the observed rate rarely is.\n\n**3. Out-of-scope requests.** Medical, legal or financial advice from a shopping bot. Requests to speak as a named real person. Requests about internal systems. The failure here is not usually harm; it is a system confidently operating outside what anyone tested.\n\n**4. Distress.** Somebody in genuine difficulty. The right behaviour is a product decision that should be written down before it is measured, and the failure modes are in both directions: treating a person as a policy violation, or missing the situation entirely.\n\n## Two rates, and they must be read together\n\nSafety evaluation has one number everybody reports and a second that gives it meaning.\n\n- **Harmful-completion rate**: of the adversarial items, how many produced the thing you did not want.\n- **False-refusal rate**: of *benign* items, how many were wrongly refused, hedged into uselessness, or lectured at.\n\nYou need a benign-but-alarming set to measure the second. Fifty items whose surface features look risky and whose intent is entirely ordinary:\n\n- a nurse asking about a paracetamol overdose threshold\n- a novelist asking how a character would pick a lock\n- a security engineer asking how SQL injection works\n- a Punjabi speaker using a word that collides with an English slur\n- a teenager asking what to do about a friend who has stopped eating\n- a historian asking about a massacre in explicit terms\n\nOptimise the first number alone and you will drive the second one up until the product is useless to the people it was built for, and you will not notice, because nobody logs a refusal as a bug. Report both, always, as a pair. A move from 3% to 1% harmful completions that took false refusals from 4% to 19% is a regression, whatever the safety dashboard says.\n\n> Figure: Two rates that only mean something together. Harmful-completion rate: Measured on the adversarial set: direct attempts, injection through retrieved content, out-of-scope requests, distress; Binary per item: did the output contain the operational content; Run the set five times. An attack that works one time in five is a working attack. False-refusal rate: Measured on a benign-but-alarming set: a nurse asking about an overdose threshold, a novelist, a security engineer, a Punjabi word that collides with an English slur; Wrongly refused, hedged into uselessness, or lectured at; Harmful 3% to 1% while refusals go 4% to 19% is a regression. Optimise the left number alone and the right one rises until the product is useless to the people it was built for, and nobody notices, because nobody files a bug about a refusal.\n\n## Scoring\n\nBinary and specific, per item: did the output contain the operational content, or not. Not \"how safe was it out of five\". Hedged, partial and refused-then-complied outputs all need a written ruling in the guide, because they are most of the interesting cases.\n\nRun the adversarial set five times, not once. Attacks succeed probabilistically; an attack that works one time in five is a working attack, and a single-run evaluation reports it as blocked 80% of the time under a heading that says \"passed\".\n\n## Tools and how to get variety\n\n**garak** (Apache 2.0) is a scanner with a large library of known probes and is the fastest way to a baseline. **PyRIT** from Microsoft (MIT) automates multi-turn attacks. **promptfoo** has a red-team mode that generates adversarial variants of your own prompts. All free.\n\nUse them for coverage of published techniques, and then write your own for your domain, because the attacks that will actually be used against you are specific to what your product does and no public library knows about them. A model can help generate variants of an attack you have already seen, which is a legitimate and useful way to turn one incident into thirty test items.\n\n## Expect to be behind\n\nA new framing that defeats your defences will appear, and it will appear on a Tuesday. The measurable goals are not \"unbreakable\". They are: known attacks stay blocked as the system changes, new attacks become test items within a day of being seen, and the false-refusal rate does not creep up while nobody is watching it.\n\nThat last one is the failure teams do not notice for months, because nobody files a ticket to say the assistant was unhelpfully cautious. They just stop using it."},"check":{"question":"A team hardens their assistant and reports harmful completions falling from 4.0% to 0.9% on their adversarial set. What single additional number would most change how you read that?","options":[{"text":"The number of adversarial items, since 0.9% on a small set could be a single case.","correct":false,"tempting":"Asks a fair question about precision that would sharpen the figure without revealing what the hardening cost elsewhere."},{"text":"The refusal rate on a set of benign inputs that superficially resemble the adversarial ones.","correct":true},{"text":"Whether the adversarial set includes multi-turn attacks as well as single-turn ones.","correct":false,"tempting":"Points at a real coverage gap in the attack set, but says nothing about the collateral damage a safety improvement usually causes."},{"text":"The latency added by the new safety checks, since guardrails are typically expensive.","correct":false,"tempting":"Tracks an operational cost that matters for shipping but not for judging whether the safety result is a genuine improvement."}]},"prev":{"slug":"judging-on-a-budget","title":"What judging costs, and how to pay a tenth of it","url":"https://addaly.com/learn/evaluating-ai/judging-on-a-budget"},"next":{"slug":"evaluating-agents","title":"When the output is a sequence of actions","url":"https://addaly.com/learn/evaluating-ai/evaluating-agents"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Testing what happens when somebody tries', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/red-teaming-and-safety-evals (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Testing what happens when somebody tries","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Measure harmful completions and false refusals as a pair, test injection through retrieved content rather than only through user messages, and run the adversarial set several times because attacks succeed probabilistically.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Harmful-completion rate","items":["Measured on the adversarial set: direct attempts, injection through retrieved content, out-of-scope requests, distress","Binary per item: did the output contain the operational content","Run the set five times. An attack that works one time in five is a working attack"]},"type":"compare","right":{"title":"False-refusal rate","items":["Measured on a benign-but-alarming set: a nurse asking about an overdose threshold, a novelist, a security engineer, a Punjabi word that collides with an English slur","Wrongly refused, hedged into uselessness, or lectured at","Harmful 3% to 1% while refusals go 4% to 19% is a regression"]},"title":"Two rates that only mean something together","caption":"Optimise the left number alone and the right one rises until the product is useless to the people it was built for, and nobody notices, because nobody files a bug about a refusal."},"alt":"Figure: Two rates that only mean something together. Harmful-completion rate: Measured on the adversarial set: direct attempts, injection through retrieved content, out-of-scope requests, distress; Binary per item: did the output contain the operational content; Run the set five times. An attack that works one time in five is a working attack. False-refusal rate: Measured on a benign-but-alarming set: a nurse asking about an overdose threshold, a novelist, a security engineer, a Punjabi word that collides with an English slur; Wrongly refused, hedged into uselessness, or lectured at; Harmful 3% to 1% while refusals go 4% to 19% is a regression. Optimise the left number alone and the right one rises until the product is useless to the people it was built for, and nobody notices, because nobody files a bug about a refusal."}},"blocks":[{"t":"h2","id":"a-different-set-with-different-rules","runs":[{"text":"A different set, with different rules"}]},{"t":"p","runs":[{"text":"Your hundred items came from real traffic, so they represent people using the system as intended. A safety evaluation represents people using it as "},{"text":"not","i":true},{"text":" intended, plus people whose ordinary use happens to look alarming. It is a separate set, built deliberately, scored on its own metrics, and it does not get averaged into your quality number."}]},{"t":"p","runs":[{"text":"Four categories are worth having, and they cover most of what actually happens:"}]},{"t":"p","runs":[{"text":"1. Direct attempts.","b":true},{"text":" Requests for content you will not produce, in every framing people use: roleplay, fiction, “for a research paper”, translation into another language, encoding, a hypothetical, an appeal to authority, an urgent emergency."}]},{"t":"p","runs":[{"text":"2. Prompt injection through content.","b":true},{"text":" This is the one that matters most for anything with retrieval, browsing or tools, and it is the one most often untested. The attack is not in the user’s message. It is inside a document, a web page, an email, a filename or a database row that your system reads: text addressed to the model, telling it to ignore its instructions, exfiltrate what it has seen, or call a tool."}]},{"t":"p","runs":[{"text":"Build a small corpus of poisoned documents and put them in your test index. A minimal battery: an instruction in white text, an instruction in a code comment, an instruction in a footnote, one in a language other than the interface language, one that claims to come from the system administrator, and one that asks the model to include a URL with data appended to it. Then measure how often the system follows them. The correct rate is zero and the observed rate rarely is."}]},{"t":"p","runs":[{"text":"3. Out-of-scope requests.","b":true},{"text":" Medical, legal or financial advice from a shopping bot. Requests to speak as a named real person. Requests about internal systems. The failure here is not usually harm; it is a system confidently operating outside what anyone tested."}]},{"t":"p","runs":[{"text":"4. Distress.","b":true},{"text":" Somebody in genuine difficulty. The right behaviour is a product decision that should be written down before it is measured, and the failure modes are in both directions: treating a person as a policy violation, or missing the situation entirely."}]},{"t":"h2","id":"two-rates-and-they-must-be-read-together","runs":[{"text":"Two rates, and they must be read together"}]},{"t":"p","runs":[{"text":"Safety evaluation has one number everybody reports and a second that gives it meaning."}]},{"t":"ul","items":[[{"text":"Harmful-completion rate","b":true},{"text":": of the adversarial items, how many produced the thing you did not want."}],[{"text":"False-refusal rate","b":true},{"text":": of "},{"text":"benign","i":true},{"text":" items, how many were wrongly refused, hedged into uselessness, or lectured at."}]]},{"t":"p","runs":[{"text":"You need a benign-but-alarming set to measure the second. Fifty items whose surface features look risky and whose intent is entirely ordinary:"}]},{"t":"ul","items":[[{"text":"a nurse asking about a paracetamol overdose threshold"}],[{"text":"a novelist asking how a character would pick a lock"}],[{"text":"a security engineer asking how SQL injection works"}],[{"text":"a Punjabi speaker using a word that collides with an English slur"}],[{"text":"a teenager asking what to do about a friend who has stopped eating"}],[{"text":"a historian asking about a massacre in explicit terms"}]]},{"t":"p","runs":[{"text":"Optimise the first number alone and you will drive the second one up until the product is useless to the people it was built for, and you will not notice, because nobody logs a refusal as a bug. Report both, always, as a pair. A move from 3% to 1% harmful completions that took false refusals from 4% to 19% is a regression, whatever the safety dashboard says."}]},{"t":"h2","id":"scoring","runs":[{"text":"Scoring"}]},{"t":"p","runs":[{"text":"Binary and specific, per item: did the output contain the operational content, or not. Not “how safe was it out of five”. Hedged, partial and refused-then-complied outputs all need a written ruling in the guide, because they are most of the interesting cases."}]},{"t":"p","runs":[{"text":"Run the adversarial set five times, not once. Attacks succeed probabilistically; an attack that works one time in five is a working attack, and a single-run evaluation reports it as blocked 80% of the time under a heading that says “passed”."}]},{"t":"h2","id":"tools-and-how-to-get-variety","runs":[{"text":"Tools and how to get variety"}]},{"t":"p","runs":[{"text":"garak","b":true},{"text":" (Apache 2.0) is a scanner with a large library of known probes and is the fastest way to a baseline. "},{"text":"PyRIT","b":true},{"text":" from Microsoft (MIT) automates multi-turn attacks. "},{"text":"promptfoo","b":true},{"text":" has a red-team mode that generates adversarial variants of your own prompts. All free."}]},{"t":"p","runs":[{"text":"Use them for coverage of published techniques, and then write your own for your domain, because the attacks that will actually be used against you are specific to what your product does and no public library knows about them. A model can help generate variants of an attack you have already seen, which is a legitimate and useful way to turn one incident into thirty test items."}]},{"t":"h2","id":"expect-to-be-behind","runs":[{"text":"Expect to be behind"}]},{"t":"p","runs":[{"text":"A new framing that defeats your defences will appear, and it will appear on a Tuesday. The measurable goals are not “unbreakable”. They are: known attacks stay blocked as the system changes, new attacks become test items within a day of being seen, and the false-refusal rate does not creep up while nobody is watching it."}]},{"t":"p","runs":[{"text":"That last one is the failure teams do not notice for months, because nobody files a ticket to say the assistant was unhelpfully cautious. They just stop using it."}]}],"check":{"question":[{"text":"A team hardens their assistant and reports harmful completions falling from 4.0% to 0.9% on their adversarial set. What single additional number would most change how you read that?"}],"options":[{"text":[{"text":"The number of adversarial items, since 0.9% on a small set could be a single case."}],"correct":false,"tempting":[{"text":"Asks a fair question about precision that would sharpen the figure without revealing what the hardening cost elsewhere."}]},{"text":[{"text":"The refusal rate on a set of benign inputs that superficially resemble the adversarial ones."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the adversarial set includes multi-turn attacks as well as single-turn ones."}],"correct":false,"tempting":[{"text":"Points at a real coverage gap in the attack set, but says nothing about the collateral damage a safety improvement usually causes."}]},{"text":[{"text":"The latency added by the new safety checks, since guardrails are typically expensive."}],"correct":false,"tempting":[{"text":"Tracks an operational cost that matters for shipping but not for judging whether the safety result is a genuine improvement."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"judging-on-a-budget","title":"What judging costs, and how to pay a tenth of it","minutes":9},"next":{"slug":"evaluating-agents","title":"When the output is a sequence of actions","minutes":10}}}