{"course":{"slug":"building-apps-with-ai","title":"Building Apps with AI","url":"https://addaly.com/learn/building-apps-with-ai","level":"beginner","track":"build"},"lesson":{"slug":"security-holes-ai-code-has-by-default","title":"The security holes AI code has by default","url":"https://addaly.com/learn/building-apps-with-ai/security-holes-ai-code-has-by-default","position":68,"of":87,"minutes":10,"answer":"Being logged in is not the same as being allowed; every query must be scoped to the user the session says you are.","keyIdea":"Being logged in is not the same as being allowed; every query must be scoped to the user the session says you are."},"body":{"format":"markdown","text":"## Why this keeps happening\n\nA model optimises for code that works when you try it. You try it as yourself, logged in, doing the intended thing. Security is about what happens when someone does the unintended thing, and nobody in the loop tested that.\n\nThese four are not exotic. They are in a large share of AI-built apps right now.\n\n## 1. Secrets in the browser\n\nThe fastest way to make an API call work is to put the key where the calling code is. If that code runs in the browser, the key is public. Not hidden. Not obscured. Anyone can open developer tools and read it.\n\nWatch for a key inside a `\"use client\"` component, or an environment variable whose name starts with `NEXT_PUBLIC_`, `VITE_`, or `REACT_APP_`. Those prefixes exist precisely to mean \"ship this to the browser.\" A key with that prefix is published.\n\nThe fix is structural: the browser calls your server, your server holds the key and calls the provider. \"Move this API call to a server route so the key is never sent to the browser\" is a request the model will handle well, once asked.\n\nThe other route is a `.env` file committed to a public repository. Automated scanners find those within minutes of the push, and the first sign is usually the bill. If it happens: rotate the key immediately. Deleting the file does not help, because the value is still in the history.\n\n## 2. Permission checks that only exist in the UI\n\nHiding a Delete button from users who should not delete is not a permission check. The endpoint still accepts the request from anyone who sends it, and sending it takes about fifteen seconds with the browser's own network tools.\n\nSo the question for every endpoint is: **who is allowed to call this, and which line on the server enforces it?**\n\nAnd there are two separate questions hiding inside that. *Are you logged in* is authentication. *Are you allowed to touch this particular thing* is authorization. AI code very often has the first and not the second.\n\n## 3. The one that is everywhere\n\n```js\n// What the model wrote\napp.get('/api/invoices/:id', requireLogin, async (req, res) => {\n  const invoice = await db.query(\n    'SELECT * FROM invoices WHERE id = $1',\n    [req.params.id]\n  )\n  res.json(invoice)\n})\n```\n\nThis checks that you are logged in. It never checks that the invoice is yours. Change the number in the URL and you are reading another customer's invoice. Every customer's, one at a time, with a script.\n\n```js\n// What it needed\napp.get('/api/invoices/:id', requireLogin, async (req, res) => {\n  const invoice = await db.query(\n    'SELECT * FROM invoices WHERE id = $1 AND user_id = $2',\n    [req.params.id, req.session.userId]   // from the session, never from the request\n  )\n  if (!invoice) return res.status(404).json({ error: 'Not found' })\n  res.json(invoice)\n})\n```\n\nOne extra condition. Note where the user id comes from: the session on the server. If it arrives in the URL, the body, or a header the browser set, the caller controls it, and a value the caller controls proves nothing.\n\nThis is invisible during normal use because you only ever test with your own account and your own data. Make a second account. Log in as user B and try to open user A's things by editing the URL. Do this before launch, every time.\n\n## 4. Injection\n\nUser input treated as instructions rather than data.\n\nThe classic is SQL built by gluing strings together: `\"SELECT * FROM users WHERE email = '\" + email + \"'\"`. Someone types a quote mark and the rest of their text becomes part of the query. Modern libraries mostly prevent this, so look specifically for string concatenation or a `raw(...)` call near a query.\n\nThe newer version: your app passes user text to a model that can use tools or read private data. Text in a document or a message can carry instructions, and the model may follow them. If your app has an AI feature that can act (send, delete, fetch) treat every piece of text it reads as written by someone who wants something from you.\n\n## The review that catches most of it\n\nBefore anyone else uses your app, spend twenty minutes:\n\n1. Open developer tools, Network tab, and use your app. Look at what is sent. Any key visible is public.\n2. Search the code for `sk-`, `key`, `secret`, `password`. Anything in a client file is exposed.\n3. Make a second account. Try to reach the first account's data by changing ids in URLs.\n4. Confirm `.env` is in `.gitignore` and was never committed.\n5. Ask for a specific review, not a general one: \"For every API route, tell me which line checks that the requester owns this record. List any route where there is no such line.\"\n\nGeneral requests to \"check security\" produce reassurance. Specific ones produce findings.\n\n> Figure: One extra condition. What the model wrote: requireLogin, so you must be signed in; WHERE id = $1, with the id taken from the URL; Authentication present, authorisation absent; Returns the row to whoever asked for it. What it needed: requireLogin, unchanged; WHERE id = $1 AND user_id = $2; The second value read from the session on the server; 404 when nothing matches, so an id reveals nothing. The left-hand version passes every test you will run, because you only ever open your own invoices. Change the number in the URL and it reads somebody else's, one at a time, with a script."},"check":{"question":"An endpoint at /api/invoices/:id requires a valid login, then returns whatever invoice matches that id. What is the actual defect?","options":[{"text":"The ids are sequential, so a curious user can guess neighbouring invoice numbers.","correct":false,"tempting":"Identifies a real convenience for the attacker, so switching to random UUIDs feels like a fix, but it only makes the wrong data harder to find, not harder to get."},{"text":"There is no rate limit, so someone can enumerate thousands of invoices very quickly.","correct":false,"tempting":"Focuses on the speed of the attack rather than its possibility: throttling slows the leak without preventing a single unauthorized read."},{"text":"The connection is not forced to HTTPS, so invoice data can be read in transit.","correct":false,"tempting":"Treats encryption as access control, protecting data from observers while doing nothing about a request the server itself is willing to answer."},{"text":"It never checks that the invoice belongs to the person making the request.","correct":true}]},"prev":{"slug":"telling-users-it-is-ai","title":"Telling users it is AI, and what you send about them","url":"https://addaly.com/learn/building-apps-with-ai/telling-users-it-is-ai"},"next":{"slug":"authorisation-beyond-ownership","title":"Authorisation beyond ownership","url":"https://addaly.com/learn/building-apps-with-ai/authorisation-beyond-ownership"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The security holes AI code has by default', in Building Apps with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-apps-with-ai/security-holes-ai-code-has-by-default (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The security holes AI code has by default","courseTitle":"Building Apps with AI","module":{"index":8,"slug":"the-holes-ai-code-has-by-default","title":"The security holes AI code has by default"},"answer":null,"keyIdea":"Being logged in is not the same as being allowed; every query must be scoped to the user the session says you are.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 8.1","figure":{"left":{"title":"What the model wrote","items":["requireLogin, so you must be signed in","WHERE id = $1, with the id taken from the URL","Authentication present, authorisation absent","Returns the row to whoever asked for it"]},"type":"compare","right":{"title":"What it needed","items":["requireLogin, unchanged","WHERE id = $1 AND user_id = $2","The second value read from the session on the server","404 when nothing matches, so an id reveals nothing"]},"title":"One extra condition","caption":"The left-hand version passes every test you will run, because you only ever open your own invoices. Change the number in the URL and it reads somebody else’s, one at a time, with a script."},"alt":"Figure: One extra condition. What the model wrote: requireLogin, so you must be signed in; WHERE id = $1, with the id taken from the URL; Authentication present, authorisation absent; Returns the row to whoever asked for it. What it needed: requireLogin, unchanged; WHERE id = $1 AND user_id = $2; The second value read from the session on the server; 404 when nothing matches, so an id reveals nothing. The left-hand version passes every test you will run, because you only ever open your own invoices. Change the number in the URL and it reads somebody else’s, one at a time, with a script."}},"blocks":[{"t":"h2","id":"why-this-keeps-happening","runs":[{"text":"Why this keeps happening"}]},{"t":"p","runs":[{"text":"A model optimises for code that works when you try it. You try it as yourself, logged in, doing the intended thing. Security is about what happens when someone does the unintended thing, and nobody in the loop tested that."}]},{"t":"p","runs":[{"text":"These four are not exotic. They are in a large share of AI-built apps right now."}]},{"t":"h2","id":"1-secrets-in-the-browser","runs":[{"text":"1. Secrets in the browser"}]},{"t":"p","runs":[{"text":"The fastest way to make an API call work is to put the key where the calling code is. If that code runs in the browser, the key is public. Not hidden. Not obscured. Anyone can open developer tools and read it."}]},{"t":"p","runs":[{"text":"Watch for a key inside a "},{"text":"\"use client\"","code":true},{"text":" component, or an environment variable whose name starts with "},{"text":"NEXT_PUBLIC_","code":true},{"text":", "},{"text":"VITE_","code":true},{"text":", or "},{"text":"REACT_APP_","code":true},{"text":". Those prefixes exist precisely to mean “ship this to the browser.” A key with that prefix is published."}]},{"t":"p","runs":[{"text":"The fix is structural: the browser calls your server, your server holds the key and calls the provider. “Move this API call to a server route so the key is never sent to the browser” is a request the model will handle well, once asked."}]},{"t":"p","runs":[{"text":"The other route is a "},{"text":".env","code":true},{"text":" file committed to a public repository. Automated scanners find those within minutes of the push, and the first sign is usually the bill. If it happens: rotate the key immediately. Deleting the file does not help, because the value is still in the history."}]},{"t":"h2","id":"2-permission-checks-that-only-exist-in-the-ui","runs":[{"text":"2. Permission checks that only exist in the UI"}]},{"t":"p","runs":[{"text":"Hiding a Delete button from users who should not delete is not a permission check. The endpoint still accepts the request from anyone who sends it, and sending it takes about fifteen seconds with the browser’s own network tools."}]},{"t":"p","runs":[{"text":"So the question for every endpoint is: "},{"text":"who is allowed to call this, and which line on the server enforces it?","b":true}]},{"t":"p","runs":[{"text":"And there are two separate questions hiding inside that. "},{"text":"Are you logged in","i":true},{"text":" is authentication. "},{"text":"Are you allowed to touch this particular thing","i":true},{"text":" is authorization. AI code very often has the first and not the second."}]},{"t":"h2","id":"3-the-one-that-is-everywhere","runs":[{"text":"3. The one that is everywhere"}]},{"t":"code","lang":"js","label":"JavaScript","text":"// What the model wrote\napp.get('/api/invoices/:id', requireLogin, async (req, res) => {\n  const invoice = await db.query(\n    'SELECT * FROM invoices WHERE id = $1',\n    [req.params.id]\n  )\n  res.json(invoice)\n})"},{"t":"p","runs":[{"text":"This checks that you are logged in. It never checks that the invoice is yours. Change the number in the URL and you are reading another customer’s invoice. Every customer’s, one at a time, with a script."}]},{"t":"code","lang":"js","label":"JavaScript","text":"// What it needed\napp.get('/api/invoices/:id', requireLogin, async (req, res) => {\n  const invoice = await db.query(\n    'SELECT * FROM invoices WHERE id = $1 AND user_id = $2',\n    [req.params.id, req.session.userId]   // from the session, never from the request\n  )\n  if (!invoice) return res.status(404).json({ error: 'Not found' })\n  res.json(invoice)\n})"},{"t":"p","runs":[{"text":"One extra condition. Note where the user id comes from: the session on the server. If it arrives in the URL, the body, or a header the browser set, the caller controls it, and a value the caller controls proves nothing."}]},{"t":"p","runs":[{"text":"This is invisible during normal use because you only ever test with your own account and your own data. Make a second account. Log in as user B and try to open user A’s things by editing the URL. Do this before launch, every time."}]},{"t":"h2","id":"4-injection","runs":[{"text":"4. Injection"}]},{"t":"p","runs":[{"text":"User input treated as instructions rather than data."}]},{"t":"p","runs":[{"text":"The classic is SQL built by gluing strings together: "},{"text":"\"SELECT * FROM users WHERE email = '\" + email + \"'\"","code":true},{"text":". Someone types a quote mark and the rest of their text becomes part of the query. Modern libraries mostly prevent this, so look specifically for string concatenation or a "},{"text":"raw(...)","code":true},{"text":" call near a query."}]},{"t":"p","runs":[{"text":"The newer version: your app passes user text to a model that can use tools or read private data. Text in a document or a message can carry instructions, and the model may follow them. If your app has an AI feature that can act (send, delete, fetch) treat every piece of text it reads as written by someone who wants something from you."}]},{"t":"h2","id":"the-review-that-catches-most-of-it","runs":[{"text":"The review that catches most of it"}]},{"t":"p","runs":[{"text":"Before anyone else uses your app, spend twenty minutes:"}]},{"t":"ol","start":1,"items":[[{"text":"Open developer tools, Network tab, and use your app. Look at what is sent. Any key visible is public."}],[{"text":"Search the code for "},{"text":"sk-","code":true},{"text":", "},{"text":"key","code":true},{"text":", "},{"text":"secret","code":true},{"text":", "},{"text":"password","code":true},{"text":". Anything in a client file is exposed."}],[{"text":"Make a second account. Try to reach the first account’s data by changing ids in URLs."}],[{"text":"Confirm "},{"text":".env","code":true},{"text":" is in "},{"text":".gitignore","code":true},{"text":" and was never committed."}],[{"text":"Ask for a specific review, not a general one: “For every API route, tell me which line checks that the requester owns this record. List any route where there is no such line.”"}]]},{"t":"p","runs":[{"text":"General requests to “check security” produce reassurance. Specific ones produce findings."}]}],"check":{"question":[{"text":"An endpoint at /api/invoices/:id requires a valid login, then returns whatever invoice matches that id. What is the actual defect?"}],"options":[{"text":[{"text":"The ids are sequential, so a curious user can guess neighbouring invoice numbers."}],"correct":false,"tempting":[{"text":"Identifies a real convenience for the attacker, so switching to random UUIDs feels like a fix, but it only makes the wrong data harder to find, not harder to get."}]},{"text":[{"text":"There is no rate limit, so someone can enumerate thousands of invoices very quickly."}],"correct":false,"tempting":[{"text":"Focuses on the speed of the attack rather than its possibility: throttling slows the leak without preventing a single unauthorized read."}]},{"text":[{"text":"The connection is not forced to HTTPS, so invoice data can be read in transit."}],"correct":false,"tempting":[{"text":"Treats encryption as access control, protecting data from observers while doing nothing about a request the server itself is willing to answer."}]},{"text":[{"text":"It never checks that the invoice belongs to the person making the request."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"telling-users-it-is-ai","title":"Telling users it is AI, and what you send about them","minutes":7},"next":{"slug":"authorisation-beyond-ownership","title":"Authorisation beyond ownership","minutes":9}}}