Permissions, freshness and metadata: the boring half that leaks data

The leak nobody tests for

The day a company indexes its shared drive, the salary spreadsheet in the HR folder becomes a chunk, in the same index as the lunch menu, with the same standing. The folder had permissions. The index does not. Any question that happens to be close to that chunk’s embedding retrieves it, and the model summarises it helpfully for whoever asked.

This is not an exotic attack. It is the default behaviour of a retrieval pipeline built without thinking about it, and it has happened at companies with security teams. The mechanism is simple: an index is a copy, and a copy does not inherit the controls on the original.

Carry the permission with the chunk

Every chunk gets metadata naming who may see it: user ids, group ids, or a document-level identifier that your permission system can resolve. At query time, the search is filtered to chunks the current user may read, before ranking, using the pre-filter pattern from the vector-search lesson. Groups are the practical unit (a chunk allowed to finance-team rather than to forty named people) because membership changes far more often than document permissions do.

When a document’s permissions change at the source, the change must propagate. If sharing is revoked at 10:00 and your nightly re-index runs at 02:00, the document is visible for sixteen hours after it should not be. For sensitive sources, listen for permission-change events and update the metadata immediately, separately from the content re-index.

Never let the model be the filter

A system prompt that says “only use documents the user is permitted to see” is not access control. The model has already been handed the text; the instruction asks it to pretend it has not. Under an injection, or under a perfectly innocent question that resembles the forbidden content, it will summarise what it was given. The earlier module made the same point about tools: the model can only ask, and your code decides. Retrieval is the same. Filter in the query, and treat any design where the model sees a document before the permission check as broken.

Freshness

The index holds each document as it was at ingestion. Three strategies, in order of complexity:

• Full re-index on a schedule. Delete and rebuild nightly. Simple, obviously correct, and fine to around a hundred thousand documents, after which the rebuild takes longer than the night.
• Incremental with change detection. Hash each document’s content; re-chunk and re-embed only those whose hash changed. Requires that you kept the hash and the mapping from document to chunks.
• Event-driven. Google Drive, Confluence, SharePoint and object stores emit change events; consume them and update within minutes.

Deletions are the hard case in every strategy. A deleted document must be removed from the vector index, the keyword index, any response cache that holds an answer built from it, and any summary that quoted it. Miss one and the document lives on. Write a tombstone record when a source deletion arrives, and have every consumer honour it, with a nightly reconciliation that compares the set of live source documents against the set in each index.

The “as of” problem

Policies change, and “what is the refund window?” may need the answer from March, when the customer bought, not today’s. Store effective-from and effective-to dates as metadata, keep superseded versions rather than overwriting them, and let the query filter on date. Whether your corpus needs this is a product decision; whether you can add it later without the dates is not: you cannot.

Metadata filtering in general

Source, product, language, document type, date, customer: every one of these is a filter that can shrink a search from the whole corpus to the one per cent that could possibly contain the answer, and a search over the right one per cent nearly always beats a search over everything. The rewrite lesson showed how to extract these from the question. Index them as first-class fields, not as words in the text.

Chunks in, documents out

Retrieve at chunk granularity, because chunks embed precisely. Return and cite at a coarser one, because a person wants a page, not a fragment. Group the top chunks by parent document, and consider returning the enclosing section rather than the chunk itself: the model reads better with the surrounding paragraphs, and the cost is a few hundred tokens.

Citations that resolve

Give the model chunk identifiers and require it to cite them. Then check the citations. A model handed five chunks will, some fraction of the time, cite a sixth that does not exist. Verify every cited id against the retrieved set; strip or reject the ones that are not there; render the survivors as links to the source page and offset from the provenance you stored at ingestion. A citation the reader can click and confirm is the difference between a search engine and a rumour.

An audit log

Record who retrieved which chunks, when, for which question. When a leak is suspected, and the question will be “did anyone see this document?”, that log is the only way to answer, and it also serves compliance and debugging. It is personal data itself; retain it under the same policy as the rest.