Filter before you rank, not after

The bug that looks like a retrieval quality problem

A system retrieves the top 10 passages, then removes the ones this user is not allowed to see, then passes the remainder to the model. On most requests eight or nine survive and everything looks fine. For a user in a restricted role, one survives, and the assistant answers from a single weak passage, or from none, and invents something.

Nothing in the metrics says “permission filter”. It looks like the retriever got worse for some users, which is exactly what happened, for a reason no retrieval tuning will fix.

Post-filtering breaks top-k because k is chosen before the filter runs. You asked for the ten best; you got however many of those ten passed. The fix is to push the filter into the query so the index searches only the permitted subset and returns ten from it.

# post-filter: wrong
hits = index.search(qvec, k=10)
hits = [h for h in hits if can_read(user, h.doc_id)]

# pre-filter: correct
hits = index.search(qvec, k=10, filter={"acl": {"$in": user.groups}})

Permissions are a context problem, not an application problem

The rule is simple and often broken: a passage the user could not open in your own product must never enter the window. Once it is in the context, the model may quote it, paraphrase it, or use it silently to shape an answer, and no instruction reliably prevents that. “Do not reveal information from documents marked confidential” is a request, not an access control.

So the access check happens at retrieval time, against the same source of truth the rest of the application uses. Two design consequences:

1. The access data must be in the index, denormalised alongside the vector, because the filter has to run inside the search. Which means it can go stale: a document whose sharing changed an hour ago is still indexed with yesterday’s permissions.
2. You need a re-check after retrieval anyway. Filter in the index for correctness of ranking, verify against the live authority before assembly for correctness of access. Belt and braces, and the second check is cheap because it runs on ten ids rather than a million.

Multi-tenant systems get a stronger version: separate namespaces or separate indexes per tenant, so a filter bug cannot leak across customers. A missing tenant_id in a WHERE clause is a familiar class of incident; the same mistake in a vector query is harder to notice because the results still look like reasonable text.

The uncomfortable part: filters degrade recall

Approximate nearest-neighbour indexes (HNSW and its relatives) work by walking a graph of neighbours. A filter that excludes most of the corpus breaks the graph’s connectivity: the walk keeps landing on excluded nodes, and the search either returns fewer results than asked for or misses close matches.

The severity depends on selectivity. Filtering to 50% of the corpus is usually fine. Filtering to 0.1% (one customer’s twelve documents out of a million) can produce recall so poor that a document you know is a perfect match does not appear.

Three practical responses:

• When the filtered set is small, do an exact search over it. A brute-force cosine over 5,000 vectors is a few milliseconds in NumPy. Set a threshold on the filtered set size and switch strategies below it.
• Partition instead of filtering for the most selective dimension you have, usually tenant. A per-tenant index has no filter problem because everything in it is already permitted.
• Check your database’s semantics. Implementations differ in whether the filter is applied during the graph walk or as a pre-computed mask, and the recall behaviour differs with it. This is worth reading in the documentation of whatever you use rather than assuming.

Hard filters versus soft preferences

Not everything that narrows results should be a filter. The distinction to hold:

• Hard filter: a correctness constraint. Permissions, tenant, language when the answer must be in that language, document type when the question is about a specific type. Wrong results here are bugs.
• Soft preference: a ranking signal. Recency, source authority, document popularity. Implement as a score adjustment, not an exclusion, because a filter here silently removes the one old document that actually contains the answer.

Teams tend to turn preferences into filters because filters are easier to write. The symptom is a system that confidently cannot find things that are definitely there: a query for a 2019 incident report returning nothing because a last_90_days filter was added to improve freshness.

The check worth adding

A single test that catches most of this: take a document, give it to a user who should not see it, and assert that its text does not appear in the assembled context, not that it is absent from the answer, but absent from the request. Assert on the assembled request, which the previous module made testable for exactly this kind of reason.