{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"model-output-is-untrusted-input","title":"What the model writes is input to something else","url":"https://addaly.com/learn/context-engineering/model-output-is-untrusted-input","position":76,"of":87,"minutes":8,"answer":"Model output is partly shaped by whatever was in the window, so it must be validated before it reaches a browser, a database, a shell or a filesystem.","keyIdea":"Model output is partly shaped by whatever was in the window, so it must be validated before it reaches a browser, a database, a shell or a filesystem, exactly as any other input from outside would be."},"body":{"format":"markdown","text":"## The boundary people forget\n\nThe whole module has treated the window as the place where untrusted text arrives. There is a second boundary, on the far side, and it is regularly missed because the text coming across it was produced by your own system.\n\nModel output is a function of the model's weights and its window. If anything in that window was attacker-influenced, the output is attacker-influenced. Treating it as trusted because it came out of your own API call is the same error as trusting a form field because your own form rendered it.\n\nThe framing to adopt: **model output is user input to whatever consumes it next.** Every rule you would apply to a string from a web form applies here too.\n\n## Where it lands, and what goes wrong\n\n**Into HTML.** If you render the model's markdown into a page, you are rendering attacker-influenceable content. Markdown permits inline HTML in most parsers, so the usual cross-site scripting surface is present unless the renderer sanitises. Enable the sanitiser explicitly rather than relying on a default you have not checked, and confirm it strips event handlers and `javascript:` URLs, not merely `<script>` tags. This is the same control that closes the image exfiltration path, which is a hint about how often the rendering layer is the weak point.\n\n**Into SQL.** A model producing a query, or a value interpolated into one, is a string of unknown provenance reaching your database. Parameterise, and give the connection read-only rights on named views. \"The model would not write a `DROP`\" is a claim about model behaviour under adversarial input, which is the claim we have spent the module declining to make.\n\n**Into a shell.** Never build a command by string concatenation from model output. Use an argument list so there is no shell to interpret metacharacters, and allowlist the executables that may be named.\n\n**Into a filesystem.** A path from model output can traverse. Resolve it and check it is still inside the working directory, then reject rather than clamp, so an attempt is visible rather than silently corrected.\n\n**Into another model.** A sub-agent's output entering a parent's window is untrusted input to that window. This is the taint rule from earlier, seen from the other end.\n\n**Into your own logs and dashboards.** Log viewers render. A payload in a log line that renders in an internal tool has reached an audience with more privilege than the user.\n\n## Validate the structure, then validate the meaning\n\nStructured output gets you the first half. A schema guarantees the fields exist and the types parse, which removes a class of crashes.\n\nIt does not check that the values are permissible, and that is a separate pass you have to write:\n\n```python\naction = parse(response)                      # schema: shape is right\nassert action.tool in ALLOWED_FOR_THIS_TURN   # policy: this tool, now\nassert action.recipient in user.contacts      # policy: this destination\nassert action.amount <= user.refund_limit     # policy: this magnitude\n```\n\nThose three assertions are the security boundary. The schema was ergonomics. Keeping them in separate lines of code, rather than folded into the parse, keeps the distinction visible to the next person to read it.\n\n## Citations are the honest special case\n\nA model asked to cite will produce citation-shaped output whether or not the citations are real. The identifier looks right, the format is right, and the passage may not exist.\n\nSo verify after generation, mechanically: every cited handle resolves to a passage that was actually in the window, and every quoted span appears verbatim in that passage. Both checks are string operations costing no model call. They convert citations from a stylistic feature into evidence, and they are the only thing separating those two, since a fabricated citation is indistinguishable from a real one by inspection.\n\n## The rule of thumb\n\nBefore model output crosses any boundary, ask the question you would ask of a form field: *what happens if this string was written by someone who wants to hurt my user?*\n\nIf the answer is uncomfortable, the fix is on your side of the boundary: a sanitiser, a parameterised query, an argument list, a path check, an allowlist. All of them are ordinary application security, all of them are well understood, and none of them is new. The only new thing is remembering that this particular string needs them, because it arrived wearing your own system's uniform."},"check":{"question":"Why should model output be sanitised before rendering as HTML, even when the user's own message was harmless?","options":[{"text":"Because models sometimes emit malformed markup, which can break the page layout unpredictably.","correct":false,"tempting":"A robustness concern, not a security one; sanitising is about active content rather than well-formedness."},{"text":"Because provider content filters operate on the input only, leaving generated text unchecked.","correct":false,"tempting":"Filters run on both directions at most providers, and they target policy-violating content rather than markup that is dangerous to your page."},{"text":"Because output reflects everything in the window, including text an attacker placed there.","correct":true},{"text":"Because the markdown specification permits raw HTML, so any generated document is inherently unsafe to render.","correct":false,"tempting":"True of the format and insufficient as the reason: the danger requires someone with a motive to have influenced the content."}]},"prev":{"slug":"spotlighting-and-its-limits","title":"Marking untrusted text, and how far that gets you","url":"https://addaly.com/learn/context-engineering/spotlighting-and-its-limits"},"next":{"slug":"testing-the-boundary","title":"Making attack success a number you track","url":"https://addaly.com/learn/context-engineering/testing-the-boundary"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What the model writes is input to something else', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/model-output-is-untrusted-input (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What the model writes is input to something else","courseTitle":"Context Engineering","module":{"index":8,"slug":"untrusted-text-and-the-boundary","title":"Untrusted text and the trust boundary"},"answer":null,"keyIdea":"Model output is partly shaped by whatever was in the window, so it must be validated before it reaches a browser, a database, a shell or a filesystem, exactly as any other input from outside would be.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-boundary-people-forget","runs":[{"text":"The boundary people forget"}]},{"t":"p","runs":[{"text":"The whole module has treated the window as the place where untrusted text arrives. There is a second boundary, on the far side, and it is regularly missed because the text coming across it was produced by your own system."}]},{"t":"p","runs":[{"text":"Model output is a function of the model’s weights and its window. If anything in that window was attacker-influenced, the output is attacker-influenced. Treating it as trusted because it came out of your own API call is the same error as trusting a form field because your own form rendered it."}]},{"t":"p","runs":[{"text":"The framing to adopt: "},{"text":"model output is user input to whatever consumes it next.","b":true},{"text":" Every rule you would apply to a string from a web form applies here too."}]},{"t":"h2","id":"where-it-lands-and-what-goes-wrong","runs":[{"text":"Where it lands, and what goes wrong"}]},{"t":"p","runs":[{"text":"Into HTML.","b":true},{"text":" If you render the model’s markdown into a page, you are rendering attacker-influenceable content. Markdown permits inline HTML in most parsers, so the usual cross-site scripting surface is present unless the renderer sanitises. Enable the sanitiser explicitly rather than relying on a default you have not checked, and confirm it strips event handlers and "},{"text":"javascript:","code":true},{"text":" URLs, not merely "},{"text":"<script>","code":true},{"text":" tags. This is the same control that closes the image exfiltration path, which is a hint about how often the rendering layer is the weak point."}]},{"t":"p","runs":[{"text":"Into SQL.","b":true},{"text":" A model producing a query, or a value interpolated into one, is a string of unknown provenance reaching your database. Parameterise, and give the connection read-only rights on named views. “The model would not write a "},{"text":"DROP","code":true},{"text":"” is a claim about model behaviour under adversarial input, which is the claim we have spent the module declining to make."}]},{"t":"p","runs":[{"text":"Into a shell.","b":true},{"text":" Never build a command by string concatenation from model output. Use an argument list so there is no shell to interpret metacharacters, and allowlist the executables that may be named."}]},{"t":"p","runs":[{"text":"Into a filesystem.","b":true},{"text":" A path from model output can traverse. Resolve it and check it is still inside the working directory, then reject rather than clamp, so an attempt is visible rather than silently corrected."}]},{"t":"p","runs":[{"text":"Into another model.","b":true},{"text":" A sub-agent’s output entering a parent’s window is untrusted input to that window. This is the taint rule from earlier, seen from the other end."}]},{"t":"p","runs":[{"text":"Into your own logs and dashboards.","b":true},{"text":" Log viewers render. A payload in a log line that renders in an internal tool has reached an audience with more privilege than the user."}]},{"t":"h2","id":"validate-the-structure-then-validate-the-meaning","runs":[{"text":"Validate the structure, then validate the meaning"}]},{"t":"p","runs":[{"text":"Structured output gets you the first half. A schema guarantees the fields exist and the types parse, which removes a class of crashes."}]},{"t":"p","runs":[{"text":"It does not check that the values are permissible, and that is a separate pass you have to write:"}]},{"t":"code","lang":"python","label":"Python","text":"action = parse(response)                      # schema: shape is right\nassert action.tool in ALLOWED_FOR_THIS_TURN   # policy: this tool, now\nassert action.recipient in user.contacts      # policy: this destination\nassert action.amount <= user.refund_limit     # policy: this magnitude"},{"t":"p","runs":[{"text":"Those three assertions are the security boundary. The schema was ergonomics. Keeping them in separate lines of code, rather than folded into the parse, keeps the distinction visible to the next person to read it."}]},{"t":"h2","id":"citations-are-the-honest-special-case","runs":[{"text":"Citations are the honest special case"}]},{"t":"p","runs":[{"text":"A model asked to cite will produce citation-shaped output whether or not the citations are real. The identifier looks right, the format is right, and the passage may not exist."}]},{"t":"p","runs":[{"text":"So verify after generation, mechanically: every cited handle resolves to a passage that was actually in the window, and every quoted span appears verbatim in that passage. Both checks are string operations costing no model call. They convert citations from a stylistic feature into evidence, and they are the only thing separating those two, since a fabricated citation is indistinguishable from a real one by inspection."}]},{"t":"h2","id":"the-rule-of-thumb","runs":[{"text":"The rule of thumb"}]},{"t":"p","runs":[{"text":"Before model output crosses any boundary, ask the question you would ask of a form field: "},{"text":"what happens if this string was written by someone who wants to hurt my user?","i":true}]},{"t":"p","runs":[{"text":"If the answer is uncomfortable, the fix is on your side of the boundary: a sanitiser, a parameterised query, an argument list, a path check, an allowlist. All of them are ordinary application security, all of them are well understood, and none of them is new. The only new thing is remembering that this particular string needs them, because it arrived wearing your own system’s uniform."}]}],"check":{"question":[{"text":"Why should model output be sanitised before rendering as HTML, even when the user’s own message was harmless?"}],"options":[{"text":[{"text":"Because models sometimes emit malformed markup, which can break the page layout unpredictably."}],"correct":false,"tempting":[{"text":"A robustness concern, not a security one; sanitising is about active content rather than well-formedness."}]},{"text":[{"text":"Because provider content filters operate on the input only, leaving generated text unchecked."}],"correct":false,"tempting":[{"text":"Filters run on both directions at most providers, and they target policy-violating content rather than markup that is dangerous to your page."}]},{"text":[{"text":"Because output reflects everything in the window, including text an attacker placed there."}],"correct":true,"tempting":null},{"text":[{"text":"Because the markdown specification permits raw HTML, so any generated document is inherently unsafe to render."}],"correct":false,"tempting":[{"text":"True of the format and insufficient as the reason: the danger requires someone with a motive to have influenced the content."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"spotlighting-and-its-limits","title":"Marking untrusted text, and how far that gets you","minutes":8},"next":{"slug":"testing-the-boundary","title":"Making attack success a number you track","minutes":9}}}