Grep first, embeddings later

The agent case is not the chatbot case

Retrieval for a chatbot is one shot: fetch the best passages you can and hope the answer is in them. An agent gets to search again. It can read a result, notice the part number it needed, and search for that. It can widen a date range, try a different spelling, open the file the snippet came from.

That changes what you should build. The clever part is not the index, it is the tool surface: what the agent can ask for, and what comes back. A retrieval tool with filters, a result count, snippets with line numbers and a way to fetch more of a document will outperform a better index behind a worse interface, because the agent can compensate for a mediocre first result and cannot compensate for not being able to ask a second question.

Lexical search is badly underrated

The instinct now is to reach for embeddings. For a large share of real work, plain text search is better, and it is better precisely where accuracy matters most.

Identifiers, error codes, part numbers, SKUs, invoice references, function names, legal clause numbers: these are tokens whose value lies in being exact. A vector model is trained to place similar things together, which means it deliberately blurs the difference between INV-2026-0041 and INV-2026-0141. Lexical search does not blur, and returns the one you asked for.

The free options are good and boring. ripgrep over a directory is instant on hundreds of megabytes. SQLite FTS5 gives you BM25 ranking in a single file with no server. PostgreSQL full-text search is already installed wherever your data is. All three support the filters that make retrieval useful (date ranges, document type, owner) which most vector stores bolt on awkwardly.

Give the agent both, as two tools with honest descriptions: one for “find this exact string”, one for “find passages about this idea”. Then let it choose, and watch the traces to see which it actually uses.

Hybrid, cheaply

When you want both, you do not need a framework. Run the BM25 query and the vector query, take the top twenty from each, and combine with reciprocal rank fusion: each document scores the sum of 1 / (60 + rank) over the lists it appears in. Twenty lines of code, no tuning, and it reliably beats either list alone. The constant 60 is a convention from the original work, not a magic number, and results are not sensitive to it.

What to return

Snippets, with a handle. A search result should be a few hundred words of context, a document id, a page or line number, and enough for the agent to decide whether to open the whole thing with a separate read tool.

Returning whole documents is the most common cause of a run that dies at step nine: three searches with five full documents each will fill any context window, and everything after that is degraded. Cap the result size in the tool, and say in the response how many results were omitted, so the agent knows there is more rather than assuming it has seen everything.

Permissions belong in the query

If the agent searches on behalf of a person, the permission filter goes into the query, not into a filtering step afterwards. Anything retrieved into the transcript is available to be summarised into the answer, and a post-filter that removes a document from the result list has already lost if the snippet was read.

This is the same argument as the one about injection and blast radius: the safe boundary is the one the data never crosses. Pass the caller’s identity down to the search tool and let the index enforce it.

Freshness, and knowing when the index is stale

Every index is a copy, and a copy is wrong the moment the source changes. Two things to build on day one: store an indexed_at beside each document and expose it in results, so the agent can say “as of Tuesday”; and record the count of documents indexed per run so a build that indexed nothing is visible instead of appearing to succeed.

An agent quoting a policy that was superseded last month is a failure nobody attributes to retrieval, because the answer is fluent and the citation is real. Only the timestamp gives it away.

Evaluate retrieval on its own

If the final answer is wrong, there are two candidates: the right passage was never retrieved, or it was retrieved and misused. These have completely different fixes, and you cannot tell them apart by reading answers.

Build a small set of questions with the document that ought to be found for each, and measure whether it appears in the top five. Twenty questions is enough to make the difference visible. Do this before touching the generation prompt, because most “the model got it wrong” complaints turn out to be retrieval that never delivered the evidence.