Permission-aware retrieval
A Socratic walk-through of permission-aware retrieval — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why must a search system know who is asking before it decides what exists?
The tidy way to build enterprise search is to keep concerns apart: the retriever finds the best-matching documents, and a filter afterwards drops the ones this user may not see. Search does search, security does security, and each part stays simple.
It sounds right, and it fails in two distinct ways — one that leaks information without ever showing a document, and one that quietly returns the wrong results to people who are authorised. Working out both is what forces identity to move from the end of the pipeline to the beginning.
Reasoning it through
REASONING #Take the leak first, since it is the one people find surprising. Ten documents match a query and the filter removes all ten, so the user sees an empty page. Now suppose the query was a colleague's name and a phrase from a confidential reorganisation memo, and compare that empty page with the one you get for a nonsense query. If they differ observably — a count computed before filtering, a "10 results hidden" notice, a facet by department, a slower response because ten documents were fetched and dropped — then the absence has told the user the document exists. That is a real disclosure with no content shown, which is why security people treat existence as needing protection in its own right.
Does every post-filter leak? Not necessarily — a filter that reports nothing derived from what it removed can be sound. The problem is that everything around search wants exactly those derived quantities: totals, pagination, facets, "did you mean," analytics. Each is computed from the unfiltered set by default, and each becomes an oracle. The leak is not in the filter; it is in everything the filter forgot to inform.
Now the second failure, subtler and more common. Retrieval returns the top k. What happens when the top ten matches are all documents this user cannot see? The filter removes ten and returns nothing — while the eleventh through fortieth results, which the user is entitled to and which answer the question well, were never fetched. That is not protection; it is a worse search than the corpus can support. Fetching more and filtering only pushes the threshold: any fixed over-fetch has a user for whom it is not enough, and typically that is the most restricted user, whose experience degrades in proportion to how little they can see.
So both failures come from the same structural mistake: ranking was computed over a corpus that is not this user's corpus. Which is the answer to the original question. "What are the best matches?" has no user-independent answer in a system with access control, because the candidate set itself depends on identity. The permission predicate is not a post-condition on results; it is part of the definition of the search space.
Then the practical question is how to push it inside. In a lexical inverted index this is easy — treat access-control terms as ordinary indexed terms and require them in the query, so unauthorised postings are never scored. In a vector index it is harder, and instructively so: approximate nearest-neighbour structures like HNSW traverse a graph of neighbours, so filtering out most nodes can fragment the region the search walks through and recall degrades. Vector databases now offer filtered search that pushes the predicate into the traversal, but how gracefully each implementation copes with a highly selective filter varies — worth measuring rather than assuming.
There is one more thing to get right, and it is where deployed systems actually fail. Permissions are copied into the index at ingestion time, and then they change: someone leaves a group, a folder is reshared, a document moves. An index holding yesterday's access list is enforcing yesterday's policy. So a permission-aware retriever needs either a live check against the source of truth for whatever survives the index filter, or a revocation path fast enough that the staleness window is acceptable — and the honest design states that window rather than pretending it is zero.
The analogy
THE ANALOGY #Consider a private archive. The wrong design lets anyone request any box, then has the clerk open it, check your pass, and say "not for you" — because you have learned the box exists, and you can enumerate the collection by asking. The right design generates the catalogue per reader: what you can browse never listed the restricted holdings at all. Your searches range over your own catalogue, and the tenth result you see is the tenth thing you were allowed to have, not the tenth thing that exists.
a paper catalogue is printed once and a reader's clearance changes afterwards without the paper knowing, which is precisely the staleness problem — so the analogy captures the design but understates the hardest part, which is keeping each reader's catalogue current in real time.
Clarifying the model
THE MODEL #Three refinements.
First, "filter before ranking" is the principle, but materialising a per-user index is almost never how it is done — the storage would multiply by the number of users. Instead, access metadata is attached to every indexed unit and the predicate is evaluated during the search. The effect is a per-user view; the implementation is one index with a predicate pushed down.
Second, the unit of permission and the unit of retrieval are usually different objects, and that mismatch causes bugs. Access is granted on a document, folder, or site; retrieval happens over chunks. Every chunk must carry and keep its parent's access facts, and when a document is reshared all of its chunks must be updated together, or a stale fragment stays findable after the document is not.
Third, filtering is necessary and not sufficient. Once a passage is retrieved, new exposure paths open: text quoted into a shared conversation, a cached summary, an embedding stored in a log, a fine-tuning set assembled from transcripts. The retrieval filter governs what enters the pipeline and says nothing about where the content goes afterwards.
A picture of it
THE PICTURE #How to readFollow the two chains that must meet. A person holds memberships in groups, and groups are named in the grants protecting a document; separately, a document is cut into chunks and each chunk becomes an index entry. The crucial relationship joins them — an index entry must inherit its document's grants, because the query only ever touches index entries. The last line is the constraint the design exists to enforce, and the inheritance edge is where deployed systems break, since a grant can change long after the chunks were indexed.
What became clearer
WHAT CLEARED #The reason identity has to arrive before ranking is that "the best matches" is not a well-defined question in a corpus with access control — there is no single corpus, only a different one per asker. Filtering afterwards gets both halves wrong: it leaks the existence of what it hides through every count, facet, and timing derived from the unfiltered set, and it silently truncates the results of the very users who can see least, because their entitled documents were ranked eleventh and the system only fetched ten. Push the predicate into the search and both problems disappear together — which leaves the genuinely hard part, keeping each indexed fragment's copy of the permissions honest as the real permissions change underneath it.
Where to go next
ONWARD #- How filtered approximate nearest-neighbour search degrades as the predicate becomes more selective.
- Whether an acceptable staleness window can be stated in policy, and how revocation reaches already-indexed chunks.
Key terms
TERMS #| Term | What it means |
|---|---|
| Post-filtering | removing unauthorised results after ranking has been computed over the full corpus. |
| Pre-filtering (predicate pushdown) | evaluating the access predicate during the search itself, so unauthorised items are never candidates. |
| Existence disclosure | learning that a document exists from counts, facets, timing, or error differences, without seeing its contents. |
| HNSW | hierarchical navigable small world, a graph-based nearest-neighbour index whose recall suffers when a selective filter removes most of the graph. |
| ACL staleness | the gap between a permission change at the source and its reflection in the index. |
Every term the collection defines is gathered in the glossary.