What the model writes is input to something else

The boundary people forget

The whole module has treated the window as the place where untrusted text arrives. There is a second boundary, on the far side, and it is regularly missed because the text coming across it was produced by your own system.

Model output is a function of the model’s weights and its window. If anything in that window was attacker-influenced, the output is attacker-influenced. Treating it as trusted because it came out of your own API call is the same error as trusting a form field because your own form rendered it.

The framing to adopt: model output is user input to whatever consumes it next. Every rule you would apply to a string from a web form applies here too.

Where it lands, and what goes wrong

Into HTML. If you render the model’s markdown into a page, you are rendering attacker-influenceable content. Markdown permits inline HTML in most parsers, so the usual cross-site scripting surface is present unless the renderer sanitises. Enable the sanitiser explicitly rather than relying on a default you have not checked, and confirm it strips event handlers and javascript: URLs, not merely <script> tags. This is the same control that closes the image exfiltration path, which is a hint about how often the rendering layer is the weak point.

Into SQL. A model producing a query, or a value interpolated into one, is a string of unknown provenance reaching your database. Parameterise, and give the connection read-only rights on named views. “The model would not write a DROP” is a claim about model behaviour under adversarial input, which is the claim we have spent the module declining to make.

Into a shell. Never build a command by string concatenation from model output. Use an argument list so there is no shell to interpret metacharacters, and allowlist the executables that may be named.

Into a filesystem. A path from model output can traverse. Resolve it and check it is still inside the working directory, then reject rather than clamp, so an attempt is visible rather than silently corrected.

Into another model. A sub-agent’s output entering a parent’s window is untrusted input to that window. This is the taint rule from earlier, seen from the other end.

Into your own logs and dashboards. Log viewers render. A payload in a log line that renders in an internal tool has reached an audience with more privilege than the user.

Validate the structure, then validate the meaning

Structured output gets you the first half. A schema guarantees the fields exist and the types parse, which removes a class of crashes.

It does not check that the values are permissible, and that is a separate pass you have to write:

action = parse(response)                      # schema: shape is right
assert action.tool in ALLOWED_FOR_THIS_TURN   # policy: this tool, now
assert action.recipient in user.contacts      # policy: this destination
assert action.amount <= user.refund_limit     # policy: this magnitude

Those three assertions are the security boundary. The schema was ergonomics. Keeping them in separate lines of code, rather than folded into the parse, keeps the distinction visible to the next person to read it.

Citations are the honest special case

A model asked to cite will produce citation-shaped output whether or not the citations are real. The identifier looks right, the format is right, and the passage may not exist.

So verify after generation, mechanically: every cited handle resolves to a passage that was actually in the window, and every quoted span appears verbatim in that passage. Both checks are string operations costing no model call. They convert citations from a stylistic feature into evidence, and they are the only thing separating those two, since a fabricated citation is indistinguishable from a real one by inspection.

The rule of thumb

Before model output crosses any boundary, ask the question you would ask of a form field: what happens if this string was written by someone who wants to hurt my user?

If the answer is uncomfortable, the fix is on your side of the boundary: a sanitiser, a parameterised query, an argument list, a path check, an allowlist. All of them are ordinary application security, all of them are well understood, and none of them is new. The only new thing is remembering that this particular string needs them, because it arrived wearing your own system’s uniform.