Red-teaming your own feature before someone else does

Two hours, a list and a spreadsheet

Before launch, and after every model change, someone spends an afternoon being the attacker. Not a security specialist: the person who built the feature, with a list of things to try and a spreadsheet with four columns: attempt, channel, outcome, severity. The output is a set of findings, each of which either gets fixed or gets accepted with a name attached, and every finding becomes a case in the harness so it stays fixed.

The reason to do it yourself first is that someone else will do it later, and their findings arrive as screenshots on social media.

The categories

Go through each with every input channel the feature has: the message box, an uploaded document, a retrieved web page, a tool result, a stored memory, an image containing text: instructions in an image are read by vision models and get past every text filter.

Injection. Instructions in each channel, phrased as the injection lesson described: paraphrased, in another language, framed as a story or a note to a colleague. Does the model follow them? Which control stopped it?

Exfiltration. Can the model be made to emit a link or image with context data in it? To call a tool with an address or URL you chose?

Policy evasion. Role-play, hypotheticals, “for a novel I am writing”, translation into and out of another language, encoding, many examples of the desired behaviour followed by a request, gradual escalation over turns. Does the feature produce what its policy forbids?

Data extraction. The system prompt: this usually succeeds, and the right response is below. Another user’s data, through retrieval or memory. Verbatim training data.

Off-purpose use. A support assistant used as a free general-purpose writing or coding tool. Not dangerous, but it costs money and it is embarrassing when it shows up in a screenshot.

Exhaustion. Very long inputs, requests that cause tool storms, prompts that make an agent loop. Do the budgets hold?

Harm in your domain. A feature that discusses medicines is asked for doses. A feature that discusses money is asked for a guaranteed return. The domain defines the dangerous question; write yours down.

Fairness probes. The same question with different names, genders, regions, languages. Do the answers differ in quality or tone? This is a safety finding as much as a quality one.

Free tools that generate the attempts

garak, from NVIDIA, runs hundreds of probes across these categories against any endpoint. promptfoo has a red-team mode that generates attacks for a stated application type. Microsoft’s PyRIT orchestrates multi-turn attacks. Each is free; each produces a long report; the value is in reading the hits and putting them in the harness, not in the count.

Severity

For each finding, three questions: what leaked or what action occurred; who was affected: this user only, other users, the company; and whether any control intervened. Another user’s data, or an unconfirmed external action, is critical and blocks shipping. A policy violation confined to the requesting user is high or medium depending on the domain. Off-purpose use is low, monitored. Write the grid down before the session, so severity is assigned by rule rather than by how tired you are.

What you cannot fully fix

The system prompt will be extracted. Assume it is public: no secrets, no credentials, nothing in it you would be embarrassed to see quoted. Its value is in the product, not in its secrecy.

A determined user can get a general model to say things you would rather it did not. Perfect refusal is not achievable on current models. What is achievable is a bound on harm: a support assistant that can be made to be rude is an embarrassment; one that can be made to give medication doses is a danger; and the difference is scope and capability, not the refusal rate. When a finding cannot be fixed at the model, fix it at the tool surface, the permission filter, the confirmation gate, or the scope of the product. If none of those applies and the harm is real, that is a reason not to ship the feature in that form.

Cadence and outside help

Re-run on every model change, every new tool, every new input channel, and quarterly regardless, because attack techniques move. Put AI features in scope for your vulnerability disclosure policy or bug bounty if you have one; people will find things you did not, and it is better that they tell you.

Write it down

The report (attempts, outcomes, fixes, accepted risks with an owner) is the evidence the next lesson’s documentation asks for, and it is what you show a customer’s security team when they ask whether this was tested. A red-team session that produced no document did not happen.