{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"permissions-freshness-and-metadata","title":"Permissions, freshness and metadata: the boring half that leaks data","url":"https://addaly.com/learn/building-with-ai/permissions-freshness-and-metadata","position":41,"of":80,"minutes":9,"answer":"An index is a copy of every document with its access controls stripped off, so permissions must travel with each chunk and be enforced by your query code, never by an instruction to the model.","keyIdea":"An index is a copy of every document with its access controls stripped off, so permissions must travel with each chunk and be enforced by your query code, never by an instruction to the model, and deletions must reach every index and cache or the document lives on."},"body":{"format":"markdown","text":"## The leak nobody tests for\n\nThe day a company indexes its shared drive, the salary spreadsheet in the HR folder becomes a chunk, in the same index as the lunch menu, with the same standing. The folder had permissions. The index does not. Any question that happens to be close to that chunk's embedding retrieves it, and the model summarises it helpfully for whoever asked.\n\nThis is not an exotic attack. It is the default behaviour of a retrieval pipeline built without thinking about it, and it has happened at companies with security teams. The mechanism is simple: an index is a copy, and a copy does not inherit the controls on the original.\n\n## Carry the permission with the chunk\n\nEvery chunk gets metadata naming who may see it: user ids, group ids, or a document-level identifier that your permission system can resolve. At query time, the search is filtered to chunks the current user may read, before ranking, using the pre-filter pattern from the vector-search lesson. Groups are the practical unit (a chunk allowed to `finance-team` rather than to forty named people) because membership changes far more often than document permissions do.\n\nWhen a document's permissions change at the source, the change must propagate. If sharing is revoked at 10:00 and your nightly re-index runs at 02:00, the document is visible for sixteen hours after it should not be. For sensitive sources, listen for permission-change events and update the metadata immediately, separately from the content re-index.\n\n## Never let the model be the filter\n\nA system prompt that says \"only use documents the user is permitted to see\" is not access control. The model has already been handed the text; the instruction asks it to pretend it has not. Under an injection, or under a perfectly innocent question that resembles the forbidden content, it will summarise what it was given. The earlier module made the same point about tools: the model can only ask, and your code decides. Retrieval is the same. Filter in the query, and treat any design where the model sees a document before the permission check as broken.\n\n## Freshness\n\nThe index holds each document as it was at ingestion. Three strategies, in order of complexity:\n\n- **Full re-index on a schedule.** Delete and rebuild nightly. Simple, obviously correct, and fine to around a hundred thousand documents, after which the rebuild takes longer than the night.\n- **Incremental with change detection.** Hash each document's content; re-chunk and re-embed only those whose hash changed. Requires that you kept the hash and the mapping from document to chunks.\n- **Event-driven.** Google Drive, Confluence, SharePoint and object stores emit change events; consume them and update within minutes.\n\nDeletions are the hard case in every strategy. A deleted document must be removed from the vector index, the keyword index, any response cache that holds an answer built from it, and any summary that quoted it. Miss one and the document lives on. Write a tombstone record when a source deletion arrives, and have every consumer honour it, with a nightly reconciliation that compares the set of live source documents against the set in each index.\n\n## The \"as of\" problem\n\nPolicies change, and \"what is the refund window?\" may need the answer from March, when the customer bought, not today's. Store effective-from and effective-to dates as metadata, keep superseded versions rather than overwriting them, and let the query filter on date. Whether your corpus needs this is a product decision; whether you can add it later without the dates is not: you cannot.\n\n## Metadata filtering in general\n\nSource, product, language, document type, date, customer: every one of these is a filter that can shrink a search from the whole corpus to the one per cent that could possibly contain the answer, and a search over the right one per cent nearly always beats a search over everything. The rewrite lesson showed how to extract these from the question. Index them as first-class fields, not as words in the text.\n\n## Chunks in, documents out\n\nRetrieve at chunk granularity, because chunks embed precisely. Return and cite at a coarser one, because a person wants a page, not a fragment. Group the top chunks by parent document, and consider returning the enclosing section rather than the chunk itself: the model reads better with the surrounding paragraphs, and the cost is a few hundred tokens.\n\n## Citations that resolve\n\nGive the model chunk identifiers and require it to cite them. Then check the citations. A model handed five chunks will, some fraction of the time, cite a sixth that does not exist. Verify every cited id against the retrieved set; strip or reject the ones that are not there; render the survivors as links to the source page and offset from the provenance you stored at ingestion. A citation the reader can click and confirm is the difference between a search engine and a rumour.\n\n## An audit log\n\nRecord who retrieved which chunks, when, for which question. When a leak is suspected, and the question will be \"did anyone see this document?\", that log is the only way to answer, and it also serves compliance and debugging. It is personal data itself; retain it under the same policy as the rest."},"check":{"question":"A team's assistant is told in its system prompt to use only documents the current user is allowed to see. An employee asks about a project and receives a summary of a document they had no access to. Why did the instruction fail?","options":[{"text":"The system prompt was too short and needed a more explicit list of every restricted folder, together with the groups that are permitted to read each of them.","correct":false,"tempting":"Treats a structural failure as a wording problem; a longer list is still an instruction to a model that already holds the text."},{"text":"The embedding model matched the document to the question too strongly for the instruction to override.","correct":false,"tempting":"Blames retrieval quality for a design in which the model was never a valid place to enforce access."},{"text":"Prompt injection in the document told the model to ignore the restriction.","correct":false,"tempting":"Possible in other cases, but not required here; an innocent question suffices when the model already has the text."},{"text":"The document reached the model before any permission check, so it summarised what it was given; access control belongs in the query, not the prompt.","correct":true}]},"prev":{"slug":"ingestion-pdfs-tables-and-scans","title":"Ingestion: PDFs, tables and scans, where most retrieval actually fails","url":"https://addaly.com/learn/building-with-ai/ingestion-pdfs-tables-and-scans"},"next":{"slug":"long-context-instead-of-retrieval","title":"Long context instead of retrieval: the arithmetic of skipping the pipeline","url":"https://addaly.com/learn/building-with-ai/long-context-instead-of-retrieval"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Permissions, freshness and metadata: the boring half that leaks data', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/permissions-freshness-and-metadata (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Permissions, freshness and metadata: the boring half that leaks data","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"An index is a copy of every document with its access controls stripped off, so permissions must travel with each chunk and be enforced by your query code, never by an instruction to the model, and deletions must reach every index and cache or the document lives on.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-leak-nobody-tests-for","runs":[{"text":"The leak nobody tests for"}]},{"t":"p","runs":[{"text":"The day a company indexes its shared drive, the salary spreadsheet in the HR folder becomes a chunk, in the same index as the lunch menu, with the same standing. The folder had permissions. The index does not. Any question that happens to be close to that chunk’s embedding retrieves it, and the model summarises it helpfully for whoever asked."}]},{"t":"p","runs":[{"text":"This is not an exotic attack. It is the default behaviour of a retrieval pipeline built without thinking about it, and it has happened at companies with security teams. The mechanism is simple: an index is a copy, and a copy does not inherit the controls on the original."}]},{"t":"h2","id":"carry-the-permission-with-the-chunk","runs":[{"text":"Carry the permission with the chunk"}]},{"t":"p","runs":[{"text":"Every chunk gets metadata naming who may see it: user ids, group ids, or a document-level identifier that your permission system can resolve. At query time, the search is filtered to chunks the current user may read, before ranking, using the pre-filter pattern from the vector-search lesson. Groups are the practical unit (a chunk allowed to "},{"text":"finance-team","code":true},{"text":" rather than to forty named people) because membership changes far more often than document permissions do."}]},{"t":"p","runs":[{"text":"When a document’s permissions change at the source, the change must propagate. If sharing is revoked at 10:00 and your nightly re-index runs at 02:00, the document is visible for sixteen hours after it should not be. For sensitive sources, listen for permission-change events and update the metadata immediately, separately from the content re-index."}]},{"t":"h2","id":"never-let-the-model-be-the-filter","runs":[{"text":"Never let the model be the filter"}]},{"t":"p","runs":[{"text":"A system prompt that says “only use documents the user is permitted to see” is not access control. The model has already been handed the text; the instruction asks it to pretend it has not. Under an injection, or under a perfectly innocent question that resembles the forbidden content, it will summarise what it was given. The earlier module made the same point about tools: the model can only ask, and your code decides. Retrieval is the same. Filter in the query, and treat any design where the model sees a document before the permission check as broken."}]},{"t":"h2","id":"freshness","runs":[{"text":"Freshness"}]},{"t":"p","runs":[{"text":"The index holds each document as it was at ingestion. Three strategies, in order of complexity:"}]},{"t":"ul","items":[[{"text":"Full re-index on a schedule.","b":true},{"text":" Delete and rebuild nightly. Simple, obviously correct, and fine to around a hundred thousand documents, after which the rebuild takes longer than the night."}],[{"text":"Incremental with change detection.","b":true},{"text":" Hash each document’s content; re-chunk and re-embed only those whose hash changed. Requires that you kept the hash and the mapping from document to chunks."}],[{"text":"Event-driven.","b":true},{"text":" Google Drive, Confluence, SharePoint and object stores emit change events; consume them and update within minutes."}]]},{"t":"p","runs":[{"text":"Deletions are the hard case in every strategy. A deleted document must be removed from the vector index, the keyword index, any response cache that holds an answer built from it, and any summary that quoted it. Miss one and the document lives on. Write a tombstone record when a source deletion arrives, and have every consumer honour it, with a nightly reconciliation that compares the set of live source documents against the set in each index."}]},{"t":"h2","id":"the-as-of-problem","runs":[{"text":"The “as of” problem"}]},{"t":"p","runs":[{"text":"Policies change, and “what is the refund window?” may need the answer from March, when the customer bought, not today’s. Store effective-from and effective-to dates as metadata, keep superseded versions rather than overwriting them, and let the query filter on date. Whether your corpus needs this is a product decision; whether you can add it later without the dates is not: you cannot."}]},{"t":"h2","id":"metadata-filtering-in-general","runs":[{"text":"Metadata filtering in general"}]},{"t":"p","runs":[{"text":"Source, product, language, document type, date, customer: every one of these is a filter that can shrink a search from the whole corpus to the one per cent that could possibly contain the answer, and a search over the right one per cent nearly always beats a search over everything. The rewrite lesson showed how to extract these from the question. Index them as first-class fields, not as words in the text."}]},{"t":"h2","id":"chunks-in-documents-out","runs":[{"text":"Chunks in, documents out"}]},{"t":"p","runs":[{"text":"Retrieve at chunk granularity, because chunks embed precisely. Return and cite at a coarser one, because a person wants a page, not a fragment. Group the top chunks by parent document, and consider returning the enclosing section rather than the chunk itself: the model reads better with the surrounding paragraphs, and the cost is a few hundred tokens."}]},{"t":"h2","id":"citations-that-resolve","runs":[{"text":"Citations that resolve"}]},{"t":"p","runs":[{"text":"Give the model chunk identifiers and require it to cite them. Then check the citations. A model handed five chunks will, some fraction of the time, cite a sixth that does not exist. Verify every cited id against the retrieved set; strip or reject the ones that are not there; render the survivors as links to the source page and offset from the provenance you stored at ingestion. A citation the reader can click and confirm is the difference between a search engine and a rumour."}]},{"t":"h2","id":"an-audit-log","runs":[{"text":"An audit log"}]},{"t":"p","runs":[{"text":"Record who retrieved which chunks, when, for which question. When a leak is suspected, and the question will be “did anyone see this document?”, that log is the only way to answer, and it also serves compliance and debugging. It is personal data itself; retain it under the same policy as the rest."}]}],"check":{"question":[{"text":"A team’s assistant is told in its system prompt to use only documents the current user is allowed to see. An employee asks about a project and receives a summary of a document they had no access to. Why did the instruction fail?"}],"options":[{"text":[{"text":"The system prompt was too short and needed a more explicit list of every restricted folder, together with the groups that are permitted to read each of them."}],"correct":false,"tempting":[{"text":"Treats a structural failure as a wording problem; a longer list is still an instruction to a model that already holds the text."}]},{"text":[{"text":"The embedding model matched the document to the question too strongly for the instruction to override."}],"correct":false,"tempting":[{"text":"Blames retrieval quality for a design in which the model was never a valid place to enforce access."}]},{"text":[{"text":"Prompt injection in the document told the model to ignore the restriction."}],"correct":false,"tempting":[{"text":"Possible in other cases, but not required here; an innocent question suffices when the model already has the text."}]},{"text":[{"text":"The document reached the model before any permission check, so it summarised what it was given; access control belongs in the query, not the prompt."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"ingestion-pdfs-tables-and-scans","title":"Ingestion: PDFs, tables and scans, where most retrieval actually fails","minutes":10},"next":{"slug":"long-context-instead-of-retrieval","title":"Long context instead of retrieval: the arithmetic of skipping the pipeline","minutes":9}}}