{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"filter-before-you-rank","title":"Filter before you rank, not after","url":"https://addaly.com/learn/context-engineering/filter-before-you-rank","position":13,"of":87,"minutes":8,"answer":"Apply correctness constraints inside the retrieval query rather than to its results, because filtering after top-k silently shrinks the evidence set, and remember that a highly selective filter can wreck approximate-search recall.","keyIdea":"Apply correctness constraints inside the retrieval query rather than to its results, because filtering after top-k silently shrinks the evidence set, and remember that a highly selective filter can wreck approximate-search recall, at which point exact search over the small set is the better tool."},"body":{"format":"markdown","text":"## The bug that looks like a retrieval quality problem\n\nA system retrieves the top 10 passages, then removes the ones this user is not allowed to see, then passes the remainder to the model. On most requests eight or nine survive and everything looks fine. For a user in a restricted role, one survives, and the assistant answers from a single weak passage, or from none, and invents something.\n\nNothing in the metrics says \"permission filter\". It looks like the retriever got worse for some users, which is exactly what happened, for a reason no retrieval tuning will fix.\n\nPost-filtering breaks top-k because k is chosen before the filter runs. You asked for the ten best; you got however many of those ten passed. The fix is to push the filter into the query so the index searches only the permitted subset and returns ten from it.\n\n```python\n# post-filter: wrong\nhits = index.search(qvec, k=10)\nhits = [h for h in hits if can_read(user, h.doc_id)]\n\n# pre-filter: correct\nhits = index.search(qvec, k=10, filter={\"acl\": {\"$in\": user.groups}})\n```\n\n## Permissions are a context problem, not an application problem\n\nThe rule is simple and often broken: **a passage the user could not open in your own product must never enter the window.** Once it is in the context, the model may quote it, paraphrase it, or use it silently to shape an answer, and no instruction reliably prevents that. \"Do not reveal information from documents marked confidential\" is a request, not an access control.\n\nSo the access check happens at retrieval time, against the same source of truth the rest of the application uses. Two design consequences:\n\n1. **The access data must be in the index**, denormalised alongside the vector, because the filter has to run inside the search. Which means it can go stale: a document whose sharing changed an hour ago is still indexed with yesterday's permissions.\n2. **You need a re-check after retrieval anyway.** Filter in the index for correctness of ranking, verify against the live authority before assembly for correctness of access. Belt and braces, and the second check is cheap because it runs on ten ids rather than a million.\n\nMulti-tenant systems get a stronger version: separate namespaces or separate indexes per tenant, so a filter bug cannot leak across customers. A missing `tenant_id` in a `WHERE` clause is a familiar class of incident; the same mistake in a vector query is harder to notice because the results still look like reasonable text.\n\n## The uncomfortable part: filters degrade recall\n\nApproximate nearest-neighbour indexes (HNSW and its relatives) work by walking a graph of neighbours. A filter that excludes most of the corpus breaks the graph's connectivity: the walk keeps landing on excluded nodes, and the search either returns fewer results than asked for or misses close matches.\n\nThe severity depends on selectivity. Filtering to 50% of the corpus is usually fine. Filtering to 0.1% (one customer's twelve documents out of a million) can produce recall so poor that a document you know is a perfect match does not appear.\n\nThree practical responses:\n\n- **When the filtered set is small, do an exact search over it.** A brute-force cosine over 5,000 vectors is a few milliseconds in NumPy. Set a threshold on the filtered set size and switch strategies below it.\n- **Partition instead of filtering** for the most selective dimension you have, usually tenant. A per-tenant index has no filter problem because everything in it is already permitted.\n- **Check your database's semantics.** Implementations differ in whether the filter is applied during the graph walk or as a pre-computed mask, and the recall behaviour differs with it. This is worth reading in the documentation of whatever you use rather than assuming.\n\n## Hard filters versus soft preferences\n\nNot everything that narrows results should be a filter. The distinction to hold:\n\n- **Hard filter**: a correctness constraint. Permissions, tenant, language when the answer must be in that language, document type when the question is about a specific type. Wrong results here are bugs.\n- **Soft preference**: a ranking signal. Recency, source authority, document popularity. Implement as a score adjustment, not an exclusion, because a filter here silently removes the one old document that actually contains the answer.\n\nTeams tend to turn preferences into filters because filters are easier to write. The symptom is a system that confidently cannot find things that are definitely there: a query for a 2019 incident report returning nothing because a `last_90_days` filter was added to improve freshness.\n\n## The check worth adding\n\nA single test that catches most of this: take a document, give it to a user who should not see it, and assert that its text does not appear in the assembled context, not that it is absent from the answer, but absent from the request. Assert on the assembled request, which the previous module made testable for exactly this kind of reason."},"check":{"question":"After adding a permission filter to the results of a top-10 vector search, answer quality collapses for users in narrow roles but is unchanged for administrators. Why?","options":[{"text":"Narrow-role users ask harder questions, so the corpus coverage for their queries is genuinely worse.","correct":false,"tempting":"Reaches for a content explanation when the population difference is exactly the one the filter creates."},{"text":"k is applied before the filter, so restricted users receive whatever fraction of the ten they are permitted to see, often one or two passages.","correct":true},{"text":"The embedding model was trained mostly on public documents, so restricted material embeds poorly.","correct":false,"tempting":"Invents a training-data effect; the same documents rank identically for an administrator, which rules this out."},{"text":"Approximate search degrades under any filter, so the ordering becomes unreliable for everyone with restrictions.","correct":false,"tempting":"Names a real effect of pre-filtering, but the described system filters after search, where ANN recall is unaffected."}]},"prev":{"slug":"relevance-is-not-similarity","title":"Similar is not the same as relevant","url":"https://addaly.com/learn/context-engineering/relevance-is-not-similarity"},"next":{"slug":"the-k-curve","title":"How many passages is the right number","url":"https://addaly.com/learn/context-engineering/the-k-curve"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Filter before you rank, not after', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/filter-before-you-rank (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Filter before you rank, not after","courseTitle":"Context Engineering","module":{"index":2,"slug":"what-earns-a-place","title":"Selection: what earns a place"},"answer":null,"keyIdea":"Apply correctness constraints inside the retrieval query rather than to its results, because filtering after top-k silently shrinks the evidence set, and remember that a highly selective filter can wreck approximate-search recall, at which point exact search over the small set is the better tool.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-bug-that-looks-like-a-retrieval-quality-problem","runs":[{"text":"The bug that looks like a retrieval quality problem"}]},{"t":"p","runs":[{"text":"A system retrieves the top 10 passages, then removes the ones this user is not allowed to see, then passes the remainder to the model. On most requests eight or nine survive and everything looks fine. For a user in a restricted role, one survives, and the assistant answers from a single weak passage, or from none, and invents something."}]},{"t":"p","runs":[{"text":"Nothing in the metrics says “permission filter”. It looks like the retriever got worse for some users, which is exactly what happened, for a reason no retrieval tuning will fix."}]},{"t":"p","runs":[{"text":"Post-filtering breaks top-k because k is chosen before the filter runs. You asked for the ten best; you got however many of those ten passed. The fix is to push the filter into the query so the index searches only the permitted subset and returns ten from it."}]},{"t":"code","lang":"python","label":"Python","text":"# post-filter: wrong\nhits = index.search(qvec, k=10)\nhits = [h for h in hits if can_read(user, h.doc_id)]\n\n# pre-filter: correct\nhits = index.search(qvec, k=10, filter={\"acl\": {\"$in\": user.groups}})"},{"t":"h2","id":"permissions-are-a-context-problem-not-an-application-problem","runs":[{"text":"Permissions are a context problem, not an application problem"}]},{"t":"p","runs":[{"text":"The rule is simple and often broken: "},{"text":"a passage the user could not open in your own product must never enter the window.","b":true},{"text":" Once it is in the context, the model may quote it, paraphrase it, or use it silently to shape an answer, and no instruction reliably prevents that. “Do not reveal information from documents marked confidential” is a request, not an access control."}]},{"t":"p","runs":[{"text":"So the access check happens at retrieval time, against the same source of truth the rest of the application uses. Two design consequences:"}]},{"t":"ol","start":1,"items":[[{"text":"The access data must be in the index","b":true},{"text":", denormalised alongside the vector, because the filter has to run inside the search. Which means it can go stale: a document whose sharing changed an hour ago is still indexed with yesterday’s permissions."}],[{"text":"You need a re-check after retrieval anyway.","b":true},{"text":" Filter in the index for correctness of ranking, verify against the live authority before assembly for correctness of access. Belt and braces, and the second check is cheap because it runs on ten ids rather than a million."}]]},{"t":"p","runs":[{"text":"Multi-tenant systems get a stronger version: separate namespaces or separate indexes per tenant, so a filter bug cannot leak across customers. A missing "},{"text":"tenant_id","code":true},{"text":" in a "},{"text":"WHERE","code":true},{"text":" clause is a familiar class of incident; the same mistake in a vector query is harder to notice because the results still look like reasonable text."}]},{"t":"h2","id":"the-uncomfortable-part-filters-degrade-recall","runs":[{"text":"The uncomfortable part: filters degrade recall"}]},{"t":"p","runs":[{"text":"Approximate nearest-neighbour indexes (HNSW and its relatives) work by walking a graph of neighbours. A filter that excludes most of the corpus breaks the graph’s connectivity: the walk keeps landing on excluded nodes, and the search either returns fewer results than asked for or misses close matches."}]},{"t":"p","runs":[{"text":"The severity depends on selectivity. Filtering to 50% of the corpus is usually fine. Filtering to 0.1% (one customer’s twelve documents out of a million) can produce recall so poor that a document you know is a perfect match does not appear."}]},{"t":"p","runs":[{"text":"Three practical responses:"}]},{"t":"ul","items":[[{"text":"When the filtered set is small, do an exact search over it.","b":true},{"text":" A brute-force cosine over 5,000 vectors is a few milliseconds in NumPy. Set a threshold on the filtered set size and switch strategies below it."}],[{"text":"Partition instead of filtering","b":true},{"text":" for the most selective dimension you have, usually tenant. A per-tenant index has no filter problem because everything in it is already permitted."}],[{"text":"Check your database’s semantics.","b":true},{"text":" Implementations differ in whether the filter is applied during the graph walk or as a pre-computed mask, and the recall behaviour differs with it. This is worth reading in the documentation of whatever you use rather than assuming."}]]},{"t":"h2","id":"hard-filters-versus-soft-preferences","runs":[{"text":"Hard filters versus soft preferences"}]},{"t":"p","runs":[{"text":"Not everything that narrows results should be a filter. The distinction to hold:"}]},{"t":"ul","items":[[{"text":"Hard filter","b":true},{"text":": a correctness constraint. Permissions, tenant, language when the answer must be in that language, document type when the question is about a specific type. Wrong results here are bugs."}],[{"text":"Soft preference","b":true},{"text":": a ranking signal. Recency, source authority, document popularity. Implement as a score adjustment, not an exclusion, because a filter here silently removes the one old document that actually contains the answer."}]]},{"t":"p","runs":[{"text":"Teams tend to turn preferences into filters because filters are easier to write. The symptom is a system that confidently cannot find things that are definitely there: a query for a 2019 incident report returning nothing because a "},{"text":"last_90_days","code":true},{"text":" filter was added to improve freshness."}]},{"t":"h2","id":"the-check-worth-adding","runs":[{"text":"The check worth adding"}]},{"t":"p","runs":[{"text":"A single test that catches most of this: take a document, give it to a user who should not see it, and assert that its text does not appear in the assembled context, not that it is absent from the answer, but absent from the request. Assert on the assembled request, which the previous module made testable for exactly this kind of reason."}]}],"check":{"question":[{"text":"After adding a permission filter to the results of a top-10 vector search, answer quality collapses for users in narrow roles but is unchanged for administrators. Why?"}],"options":[{"text":[{"text":"Narrow-role users ask harder questions, so the corpus coverage for their queries is genuinely worse."}],"correct":false,"tempting":[{"text":"Reaches for a content explanation when the population difference is exactly the one the filter creates."}]},{"text":[{"text":"k is applied before the filter, so restricted users receive whatever fraction of the ten they are permitted to see, often one or two passages."}],"correct":true,"tempting":null},{"text":[{"text":"The embedding model was trained mostly on public documents, so restricted material embeds poorly."}],"correct":false,"tempting":[{"text":"Invents a training-data effect; the same documents rank identically for an administrator, which rules this out."}]},{"text":[{"text":"Approximate search degrades under any filter, so the ordering becomes unreliable for everyone with restrictions."}],"correct":false,"tempting":[{"text":"Names a real effect of pre-filtering, but the described system filters after search, where ANN recall is unaffected."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"relevance-is-not-similarity","title":"Similar is not the same as relevant","minutes":9},"next":{"slug":"the-k-curve","title":"How many passages is the right number","minutes":9}}}