{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"red-teaming-your-own-feature","title":"Red-teaming your own feature before someone else does","url":"https://addaly.com/learn/building-with-ai/red-teaming-your-own-feature","position":69,"of":80,"minutes":9,"answer":"An afternoon spent attacking your own feature through every input channel (with a fixed category list, a severity grid and free automated probes) finds the failures before users do, and the findings become harness cases.","keyIdea":"An afternoon spent attacking your own feature through every input channel (with a fixed category list, a severity grid and free automated probes) finds the failures before users do, and the findings become harness cases; assume the system prompt is public, and fix the blast radius where a refusal cannot be made perfect."},"body":{"format":"markdown","text":"## Two hours, a list and a spreadsheet\n\nBefore launch, and after every model change, someone spends an afternoon being the attacker. Not a security specialist: the person who built the feature, with a list of things to try and a spreadsheet with four columns: attempt, channel, outcome, severity. The output is a set of findings, each of which either gets fixed or gets accepted with a name attached, and every finding becomes a case in the harness so it stays fixed.\n\nThe reason to do it yourself first is that someone else will do it later, and their findings arrive as screenshots on social media.\n\n## The categories\n\nGo through each with every input channel the feature has: the message box, an uploaded document, a retrieved web page, a tool result, a stored memory, an image containing text: instructions in an image are read by vision models and get past every text filter.\n\n**Injection.** Instructions in each channel, phrased as the injection lesson described: paraphrased, in another language, framed as a story or a note to a colleague. Does the model follow them? Which control stopped it?\n\n**Exfiltration.** Can the model be made to emit a link or image with context data in it? To call a tool with an address or URL you chose?\n\n**Policy evasion.** Role-play, hypotheticals, \"for a novel I am writing\", translation into and out of another language, encoding, many examples of the desired behaviour followed by a request, gradual escalation over turns. Does the feature produce what its policy forbids?\n\n**Data extraction.** The system prompt: this usually succeeds, and the right response is below. Another user's data, through retrieval or memory. Verbatim training data.\n\n**Off-purpose use.** A support assistant used as a free general-purpose writing or coding tool. Not dangerous, but it costs money and it is embarrassing when it shows up in a screenshot.\n\n**Exhaustion.** Very long inputs, requests that cause tool storms, prompts that make an agent loop. Do the budgets hold?\n\n**Harm in your domain.** A feature that discusses medicines is asked for doses. A feature that discusses money is asked for a guaranteed return. The domain defines the dangerous question; write yours down.\n\n**Fairness probes.** The same question with different names, genders, regions, languages. Do the answers differ in quality or tone? This is a safety finding as much as a quality one.\n\n## Free tools that generate the attempts\n\n`garak`, from NVIDIA, runs hundreds of probes across these categories against any endpoint. `promptfoo` has a red-team mode that generates attacks for a stated application type. Microsoft's PyRIT orchestrates multi-turn attacks. Each is free; each produces a long report; the value is in reading the hits and putting them in the harness, not in the count.\n\n## Severity\n\nFor each finding, three questions: what leaked or what action occurred; who was affected: this user only, other users, the company; and whether any control intervened. Another user's data, or an unconfirmed external action, is critical and blocks shipping. A policy violation confined to the requesting user is high or medium depending on the domain. Off-purpose use is low, monitored. Write the grid down before the session, so severity is assigned by rule rather than by how tired you are.\n\n## What you cannot fully fix\n\nThe system prompt will be extracted. Assume it is public: no secrets, no credentials, nothing in it you would be embarrassed to see quoted. Its value is in the product, not in its secrecy.\n\nA determined user can get a general model to say things you would rather it did not. Perfect refusal is not achievable on current models. What is achievable is a bound on harm: a support assistant that can be made to be rude is an embarrassment; one that can be made to give medication doses is a danger; and the difference is scope and capability, not the refusal rate. When a finding cannot be fixed at the model, fix it at the tool surface, the permission filter, the confirmation gate, or the scope of the product. If none of those applies and the harm is real, that is a reason not to ship the feature in that form.\n\n## Cadence and outside help\n\nRe-run on every model change, every new tool, every new input channel, and quarterly regardless, because attack techniques move. Put AI features in scope for your vulnerability disclosure policy or bug bounty if you have one; people will find things you did not, and it is better that they tell you.\n\n## Write it down\n\nThe report (attempts, outcomes, fixes, accepted risks with an owner) is the evidence the next lesson's documentation asks for, and it is what you show a customer's security team when they ask whether this was tested. A red-team session that produced no document did not happen."},"check":{"question":"During a red-team session the team extracts the full system prompt in three attempts. What is the right response?","options":[{"text":"Add instructions to the prompt forbidding the model from revealing it, and re-test until extraction fails.","correct":false,"tempting":"Extraction resists such instructions reliably enough that a passing re-test proves little; it will be extracted eventually."},{"text":"Treat the prompt as public: remove any secrets or embarrassing content from it, and accept the finding.","correct":true},{"text":"Encrypt the system prompt so that it cannot be read even if the model reproduces it.","correct":false,"tempting":"The model must read the prompt to follow it; anything it can read it can reproduce."},{"text":"Move the sensitive instructions into a tool description, which the model treats as data rather than instruction.","correct":false,"tempting":"Tool descriptions are in the same context and are extracted the same way."}]},"prev":{"slug":"reducing-hallucination-by-mechanism","title":"Reducing hallucination by mechanism, not by instruction","url":"https://addaly.com/learn/building-with-ai/reducing-hallucination-by-mechanism"},"next":{"slug":"disclosure-consent-and-the-law-as-it-stands","title":"Disclosure, consent and the law as it stands","url":"https://addaly.com/learn/building-with-ai/disclosure-consent-and-the-law-as-it-stands"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Red-teaming your own feature before someone else does', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/red-teaming-your-own-feature (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Red-teaming your own feature before someone else does","courseTitle":"Building with AI","module":{"index":7,"slug":"safety-security-and-the-adversary","title":"Safety, security and the adversary"},"answer":null,"keyIdea":"An afternoon spent attacking your own feature through every input channel (with a fixed category list, a severity grid and free automated probes) finds the failures before users do, and the findings become harness cases; assume the system prompt is public, and fix the blast radius where a refusal cannot be made perfect.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-hours-a-list-and-a-spreadsheet","runs":[{"text":"Two hours, a list and a spreadsheet"}]},{"t":"p","runs":[{"text":"Before launch, and after every model change, someone spends an afternoon being the attacker. Not a security specialist: the person who built the feature, with a list of things to try and a spreadsheet with four columns: attempt, channel, outcome, severity. The output is a set of findings, each of which either gets fixed or gets accepted with a name attached, and every finding becomes a case in the harness so it stays fixed."}]},{"t":"p","runs":[{"text":"The reason to do it yourself first is that someone else will do it later, and their findings arrive as screenshots on social media."}]},{"t":"h2","id":"the-categories","runs":[{"text":"The categories"}]},{"t":"p","runs":[{"text":"Go through each with every input channel the feature has: the message box, an uploaded document, a retrieved web page, a tool result, a stored memory, an image containing text: instructions in an image are read by vision models and get past every text filter."}]},{"t":"p","runs":[{"text":"Injection.","b":true},{"text":" Instructions in each channel, phrased as the injection lesson described: paraphrased, in another language, framed as a story or a note to a colleague. Does the model follow them? Which control stopped it?"}]},{"t":"p","runs":[{"text":"Exfiltration.","b":true},{"text":" Can the model be made to emit a link or image with context data in it? To call a tool with an address or URL you chose?"}]},{"t":"p","runs":[{"text":"Policy evasion.","b":true},{"text":" Role-play, hypotheticals, “for a novel I am writing”, translation into and out of another language, encoding, many examples of the desired behaviour followed by a request, gradual escalation over turns. Does the feature produce what its policy forbids?"}]},{"t":"p","runs":[{"text":"Data extraction.","b":true},{"text":" The system prompt: this usually succeeds, and the right response is below. Another user’s data, through retrieval or memory. Verbatim training data."}]},{"t":"p","runs":[{"text":"Off-purpose use.","b":true},{"text":" A support assistant used as a free general-purpose writing or coding tool. Not dangerous, but it costs money and it is embarrassing when it shows up in a screenshot."}]},{"t":"p","runs":[{"text":"Exhaustion.","b":true},{"text":" Very long inputs, requests that cause tool storms, prompts that make an agent loop. Do the budgets hold?"}]},{"t":"p","runs":[{"text":"Harm in your domain.","b":true},{"text":" A feature that discusses medicines is asked for doses. A feature that discusses money is asked for a guaranteed return. The domain defines the dangerous question; write yours down."}]},{"t":"p","runs":[{"text":"Fairness probes.","b":true},{"text":" The same question with different names, genders, regions, languages. Do the answers differ in quality or tone? This is a safety finding as much as a quality one."}]},{"t":"h2","id":"free-tools-that-generate-the-attempts","runs":[{"text":"Free tools that generate the attempts"}]},{"t":"p","runs":[{"text":"garak","code":true},{"text":", from NVIDIA, runs hundreds of probes across these categories against any endpoint. "},{"text":"promptfoo","code":true},{"text":" has a red-team mode that generates attacks for a stated application type. Microsoft’s PyRIT orchestrates multi-turn attacks. Each is free; each produces a long report; the value is in reading the hits and putting them in the harness, not in the count."}]},{"t":"h2","id":"severity","runs":[{"text":"Severity"}]},{"t":"p","runs":[{"text":"For each finding, three questions: what leaked or what action occurred; who was affected: this user only, other users, the company; and whether any control intervened. Another user’s data, or an unconfirmed external action, is critical and blocks shipping. A policy violation confined to the requesting user is high or medium depending on the domain. Off-purpose use is low, monitored. Write the grid down before the session, so severity is assigned by rule rather than by how tired you are."}]},{"t":"h2","id":"what-you-cannot-fully-fix","runs":[{"text":"What you cannot fully fix"}]},{"t":"p","runs":[{"text":"The system prompt will be extracted. Assume it is public: no secrets, no credentials, nothing in it you would be embarrassed to see quoted. Its value is in the product, not in its secrecy."}]},{"t":"p","runs":[{"text":"A determined user can get a general model to say things you would rather it did not. Perfect refusal is not achievable on current models. What is achievable is a bound on harm: a support assistant that can be made to be rude is an embarrassment; one that can be made to give medication doses is a danger; and the difference is scope and capability, not the refusal rate. When a finding cannot be fixed at the model, fix it at the tool surface, the permission filter, the confirmation gate, or the scope of the product. If none of those applies and the harm is real, that is a reason not to ship the feature in that form."}]},{"t":"h2","id":"cadence-and-outside-help","runs":[{"text":"Cadence and outside help"}]},{"t":"p","runs":[{"text":"Re-run on every model change, every new tool, every new input channel, and quarterly regardless, because attack techniques move. Put AI features in scope for your vulnerability disclosure policy or bug bounty if you have one; people will find things you did not, and it is better that they tell you."}]},{"t":"h2","id":"write-it-down","runs":[{"text":"Write it down"}]},{"t":"p","runs":[{"text":"The report (attempts, outcomes, fixes, accepted risks with an owner) is the evidence the next lesson’s documentation asks for, and it is what you show a customer’s security team when they ask whether this was tested. A red-team session that produced no document did not happen."}]}],"check":{"question":[{"text":"During a red-team session the team extracts the full system prompt in three attempts. What is the right response?"}],"options":[{"text":[{"text":"Add instructions to the prompt forbidding the model from revealing it, and re-test until extraction fails."}],"correct":false,"tempting":[{"text":"Extraction resists such instructions reliably enough that a passing re-test proves little; it will be extracted eventually."}]},{"text":[{"text":"Treat the prompt as public: remove any secrets or embarrassing content from it, and accept the finding."}],"correct":true,"tempting":null},{"text":[{"text":"Encrypt the system prompt so that it cannot be read even if the model reproduces it."}],"correct":false,"tempting":[{"text":"The model must read the prompt to follow it; anything it can read it can reproduce."}]},{"text":[{"text":"Move the sensitive instructions into a tool description, which the model treats as data rather than instruction."}],"correct":false,"tempting":[{"text":"Tool descriptions are in the same context and are extracted the same way."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"reducing-hallucination-by-mechanism","title":"Reducing hallucination by mechanism, not by instruction","minutes":9},"next":{"slug":"disclosure-consent-and-the-law-as-it-stands","title":"Disclosure, consent and the law as it stands","minutes":10}}}