Questions no single passage answers

The assumption hiding in every retrieval pipeline

Top-k retrieval assumes the answer is in a passage. Rank the passages, take the best few, and the answer is among them.

A large class of real questions breaks that assumption outright:

• “Which of our enterprise customers have an open critical ticket?”: a join between two systems.
• “Has this clause changed since the 2023 version?”: a comparison between two documents.
• “What did we decide about the pricing model, and why did we reject the alternative?”: a chain across a decision document and the discussion that preceded it.
• “How many suppliers are affected?”: an aggregate.

No single chunk contains any of these answers. The retriever will still return five passages, the model will still produce an answer, and the answer will be built from whichever fragments looked closest. This is the failure mode that makes a demo look brilliant and a deployment look unreliable, because the questions people actually ask skew towards this shape far more than benchmark questions do.

Recognise the shape before you engineer for it

Three signatures, each with a different correct response:

It is an aggregate or a join. Counts, sums, “how many”, “which of my”, “all the”. The answer lives in a database, not in prose. The previous lesson’s second and third doors are the fix, and retrieval is the wrong tool entirely. Recognising this saves months.

It is a comparison between two known things. Two versions, two products, two time periods. Do not hope one search finds both: retrieve for each side separately with its own query, and assemble both into the window with clear labels. Deterministic, cheap, reliable.

It is a chain, where what you need to look up second depends on what you found first. “Who approved the change that caused the outage on 14 August?” You cannot write the second query until the first has answered.

Only the third needs iterative retrieval, and it is the least common of the three.

Iterative retrieval, and the cost of a hop

The loop: retrieve, let the model read and decide what it still needs, retrieve again, up to a cap.

Give it a search tool, a small budget, and an explicit instruction to state what is missing before searching again. Cap the hops at two or three.

The cap is not arbitrary. Errors compound multiplicatively: if each hop finds the right thing 80% of the time, three hops succeed 51% of the time, and the failures are invisible because the final answer is fluent. Meanwhile every hop adds its results to the context permanently, so a four-hop investigation ends with a window full of material from three abandoned lines of enquiry.

Two mitigations. Summarise each hop’s findings and drop the raw passages before the next hop, so the context grows slowly. And require the model to state, in the answer, which hop supplied which fact: attribution across hops is where the reasoning breaks and it is the only place you can see it.

Graphs

Building a knowledge graph (extracting entities and relationships, then traversing them) solves some multi-hop problems that vector search cannot touch. It is also a large, ongoing project: extraction quality caps everything, entity resolution across sources is hard, and the graph goes stale exactly as fast as your documents change.

A proportionate position: if your entities and relationships already exist in a database, use the database, and no graph construction is needed. If they exist only in prose and multi-hop questions are the core of your product, a graph may be worth the investment. If neither, decomposition plus a second retrieval will get you most of the value for a fraction of the work.

The cheapest big win

Before any of this, do the boring thing: detect the shape and route.

if looks_like_aggregate(q):      return sql_path(q)
if looks_like_comparison(q):     return multi_query_path(q)
return standard_retrieval(q)

A short classifier prompt or a handful of patterns handles the routing. Most systems that feel unreliable are running every question through one path, and the fix is not a better retriever but three paths and a router.

Say when you cannot

Where a question needs a join you cannot perform, the correct behaviour is to say so plainly: “I can look up individual orders, but I cannot count across accounts.” That is a better product than a confident number that is wrong, and it is a specific, actionable feature request from your users the moment you start counting how often it fires.