Model output as untrusted input to everything downstream

The oldest vulnerabilities, with a new author

Cross-site scripting, SQL injection, command injection and path traversal have been the top of every web-security list for twenty years. Each is the same mistake: text from an untrusted source is handed to an interpreter (a browser, a database, a shell, a filesystem) as if it were code the developer wrote. The defences are well known and applied by default in modern frameworks, because everyone learned to distrust user input.

A language model is a new source of text that is not user input and is not developer code. It is something in between, and it inherits the danger of both: it produces whatever its context steered it towards, including text an injection put there, and it produces it fluently, in the exact syntax the downstream interpreter expects. Treat everything the model writes as untrusted input to whatever consumes it next. That is the whole lesson; the rest is the list.

Rendering: HTML and markdown

Model output rendered as HTML is a cross-site scripting vector. A script tag, an event handler in an attribute, a javascript: link, any of them can arrive in a response, whether from an injection or from a model that saw such text in a document and reproduced it. Render markdown through a sanitising pipeline, with raw HTML disabled, as the interface lesson required. Test it: put <img src=x onerror=alert(1)> in a document the model summarises, and confirm nothing runs.

Queries: SQL and its relatives

Text-to-SQL is one of the most useful things a model does and one of the easiest to make dangerous. The model writes a query; the application runs it. A query written under injection, or under a user’s ordinary but ill-advised request, can read tables the user must not see, or write, or drop.

The defences are layered and none is the model’s obedience. Run generated queries under a database role that can only read, and only the tables the feature needs; the database enforces this whatever the query says. Parse the generated SQL before executing and reject anything that is not a single SELECT: free parsers exist for every dialect. Add a row limit and a statement timeout. And for any query that filters by user, inject the user’s id from the session in your code, never from the generated text: a WHERE user_id = ... clause the model wrote is a clause the model can omit.

The same applies to any query language the model emits: a search filter, a GraphQL query, a filesystem glob.

Commands and code

A model that produces shell commands, and a system that runs them, is a remote shell with a natural-language front end. The running-code lesson in the agents module gave the answer: execute in a sandbox with no network, no credentials, a filesystem that is discarded afterwards, and resource limits. That lesson is not optional here. If the feature must run generated commands against real infrastructure, the command is shown to a person and waits, and the allowed commands are an allowlist, not a denylist.

Generated code that will be committed or deployed goes through the same review as human code, and through the same tests. It is not more trustworthy because it was fast.

Paths and names

A model asked to save a file chooses a filename. ../../etc/passwd is a filename. Resolve every generated path against a fixed base directory and reject any result that escapes it; never concatenate model output into a path. The same for object-store keys, for URLs the application will fetch, for identifiers looked up in a database: normalise, validate against the expected shape, and where possible map to an id you chose rather than a string the model wrote.

Structured output is not validation

The structured-output lessons made the point once and it bears repeating here: a schema guarantees shape. A JSON object with a recipient field that matches the email pattern is well-formed, and the address can still be the attacker’s. Schema validation checks the type; authorisation checks the value against what this user may do. Both, every time, in your code.

Downstream systems that are not yours

Model output sent to a third-party API (a payment provider, a messaging service, a CRM) is input to their system, and their validation is not your concern to rely on. Validate before sending. An assistant that writes a customer’s name into a field that a later system prints on a letter can be persuaded to write something else, and the letter goes out with your logo on it.

The habit

Every place model output is consumed, ask the question you would ask of a form field: what happens if this contains the worst possible string? If the answer involves an interpreter (browser, database, shell, filesystem, template engine) apply the interpreter’s standard defence before the text reaches it. The frameworks already know how; the only new step is remembering that the model is a source.