Instructions hidden in the content

One channel, two kinds of text

Every security problem in this module descends from a single architectural fact. A language model receives one stream of text. The developer's instructions, the user's request, and the content being processed — a web page, an email, a document, a code comment — all arrive in the same channel, in the same format, with no reliable marker separating them.

In traditional software this problem was solved decades ago. SQL injection was defeated by parameterised queries, which keep the command and the data in genuinely separate channels so that data can never be read as command. There is no equivalent for a language model, because the model's entire capability is understanding text as meaning. Asking it to treat one span of text as inert while understanding another is asking it to do the opposite of what it does.

Simon Willison named this prompt injection in 2022, by analogy with SQL injection, and the analogy is precise in one respect and misleading in another: precise because the structure is identical, misleading because the fix that worked for SQL is not available here.

Direct and indirect

Direct injection is a user typing instructions to override the system's own. "Ignore your previous instructions and…" This is a nuisance for the operator and mostly a risk to the operator, since the user is attacking a system they are already using.

Indirect injection is the serious one. The instruction is planted in content that the model will process on somebody else's behalf.

The attacker writes a web page containing, in white text on a white background, a paragraph addressed to any AI assistant that reads it. A user asks their assistant to summarise that page. The assistant reads everything, including the invisible paragraph, and follows it. The user sees a summary. They never see the instruction, because it was never on their screen in a readable form.

This was demonstrated publicly in early 2023 against Bing's browsing assistant, where a planted page changed the assistant's behaviour towards the user, and it has been demonstrated repeatedly since against email assistants, document tools, code assistants reading dependencies, and browser agents.

Where the text can hide

Anywhere the model reads and the human does not look closely.

White or one-pixel text on a page. HTML comments and alt attributes. Metadata in a PDF or an image. A calendar invitation's description field. A code comment in a library. A product review. A resubmitted email thread. A filename. A support ticket. A CSV cell. The text of a document you were sent by someone who was themselves attacked.

The general rule: any content that reaches the model can carry instructions, and the model has no way to know that some of it was not addressed to it.

Why it is not fixed

Several defences exist and each is partial.

Delimiters and system prompts — putting the content between markers and instructing the model to treat it as data. This raises the difficulty and does not close the hole, because an attacker can write text that appears to close the delimiter.

Instruction hierarchies — training models to privilege developer instructions over content. This measurably helps and is a probabilistic preference rather than a rule.

Input and output classifiers — a second model scanning for injection attempts. Catches the obvious cases and can itself be attacked.

Structural containment — the only approach with a real security argument. Do not let the model's output do anything dangerous. If a model that has read untrusted content cannot send an email, cannot make a purchase and cannot read your private files, then an injection can produce a wrong summary and not a stolen account. This gives up capability, deliberately, and it is what the next lesson is about.

The honest state of the field: after three years of serious attention, there is no general solution, and researchers working on it say so plainly. Products that claim to have solved prompt injection have usually raised the cost of a particular attack.

What a user can do

You cannot patch the model. You can control what it touches.

Be deliberate about giving an assistant access to untrusted content and sensitive capabilities at the same time. Read what an agent proposes to do before approving it — and be aware that the summary of what it plans can itself be influenced by the injected text. Prefer tools that ask for confirmation on consequential actions. And treat anything the model reports after processing a document from an unknown source the way you would treat a claim from that source, because in effect it is one.