Admitting the set is partial

The model assumes it has everything

Give a model eight passages and a question and it behaves as though those eight passages are the world. It has no way to know that 341 matched, that the ninth was dropped for size, or that the retriever searched only documents in English.

That assumption is the source of a specific and common failure: a confident answer built from an incomplete set, with no signal that anything is missing. The user cannot detect it, because the answer is well-formed and cites real sources.

The fix is to tell it. One line, above the passages:

The passages below are the 8 highest-ranked of 341 matches, selected
automatically. They may not contain the answer. If they do not, say so
rather than inferring one.

This reliably increases the rate at which the model declines to answer when the evidence is absent. It also reliably increases the rate at which it declines when the evidence is present but awkward, which is the cost. Measure both: an abstention rate is only good news if the answer rate on answerable questions has not moved with it.

Abstain before you assemble

Better than telling the model the set might be weak is not sending a weak set at all.

The signal from the first lesson in this module, the gap between the top score and the fifth, is a usable trigger. So is the top reranker score, which unlike an embedding similarity is at least trained to mean relevance. Calibrate a floor on judged examples, and below it return a direct “I do not have information about this” without a model call at all.

Two benefits beyond accuracy. It is free and instant, since no tokens are spent. And it is auditable: you can count how often it fires, which is a measurement of corpus coverage that nothing else in the system gives you. A sudden rise in the abstain rate is an index problem, and you find out in an hour rather than through complaints.

Tell the user, not only the model

What the interface shows changes what a user trusts, and it is part of context engineering because it is downstream of the same decisions.

Show the sources used, with the citation handles the model was given. Show the count: “drawn from 8 of 341 matching documents”. Where the assistant declined, say what was searched, because “I found nothing about that in the product documentation” is a useful answer and “I don’t know” is not.

And where a document was excluded for permissions, the honest interface says that something exists but is not visible to this user, if your security model allows it. Silently pretending nothing exists is a design choice with real consequences: people spend hours looking for a document they were told was not there.

Ask for what is missing

The strongest version of this is a system that says what it would have needed.

If the passages do not contain the answer, respond with exactly:
INSUFFICIENT: <what information would be needed>

Those strings are worth more than the answers. Cluster a month of them and you have a ranked list of gaps in your corpus, written in the language of the questions people actually ask. It is the cheapest content roadmap available, and it comes out of the failure path you already needed to build.

One caution: models are not reliable narrators of their own gaps. The stated missing information is a plausible guess, not an introspective report. Treat the aggregate as a signal about topics and the individual line as unverified.

The manifest

For systems where an answer might later be disputed (anything touching money, health, employment or the law) record what was in the window, not just what came out:

{"request_id": "...", "assembly_version": "a91f3c2d0b4e",
 "passages": [{"id": "doc_8831#4", "rank": 1, "tokens": 412}, ...],
 "candidates_considered": 341, "dropped_for_size": 3,
 "filters": {"acl": ["support"], "lang": "en"}}

Small, cheap, and it converts “why did it say that” from an investigation into a query. It is also the artefact that makes the measurement module later in this course possible at all: you cannot attribute a regression to selection unless you recorded what was selected.

The habit

Once a week, read ten manifests from failed or complained-about requests. Not the answers: the manifests. The question you are asking is only ever: was the necessary passage in this list? That single question splits every failure into a retrieval problem and a generation problem, and it takes about fifteen minutes.