{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"saying-what-was-left-out","title":"Admitting the set is partial","url":"https://addaly.com/learn/context-engineering/saying-what-was-left-out","position":20,"of":87,"minutes":8,"answer":"Tell the model its evidence set is partial, abstain in code when the retrieval signal is too weak to be worth a call, show the user what was searched and what was excluded.","keyIdea":"Tell the model its evidence set is partial, abstain in code when the retrieval signal is too weak to be worth a call, show the user what was searched and what was excluded, and record a manifest of what actually entered the window so a disputed answer can be explained."},"body":{"format":"markdown","text":"## The model assumes it has everything\n\nGive a model eight passages and a question and it behaves as though those eight passages are the world. It has no way to know that 341 matched, that the ninth was dropped for size, or that the retriever searched only documents in English.\n\nThat assumption is the source of a specific and common failure: a confident answer built from an incomplete set, with no signal that anything is missing. The user cannot detect it, because the answer is well-formed and cites real sources.\n\nThe fix is to tell it. One line, above the passages:\n\n```\nThe passages below are the 8 highest-ranked of 341 matches, selected\nautomatically. They may not contain the answer. If they do not, say so\nrather than inferring one.\n```\n\nThis reliably increases the rate at which the model declines to answer when the evidence is absent. It also reliably increases the rate at which it declines when the evidence is present but awkward, which is the cost. Measure both: an abstention rate is only good news if the answer rate on answerable questions has not moved with it.\n\n## Abstain before you assemble\n\nBetter than telling the model the set might be weak is not sending a weak set at all.\n\nThe signal from the first lesson in this module, the gap between the top score and the fifth, is a usable trigger. So is the top reranker score, which unlike an embedding similarity is at least trained to mean relevance. Calibrate a floor on judged examples, and below it return a direct \"I do not have information about this\" without a model call at all.\n\nTwo benefits beyond accuracy. It is free and instant, since no tokens are spent. And it is auditable: you can count how often it fires, which is a measurement of corpus coverage that nothing else in the system gives you. A sudden rise in the abstain rate is an index problem, and you find out in an hour rather than through complaints.\n\n## Tell the user, not only the model\n\nWhat the interface shows changes what a user trusts, and it is part of context engineering because it is downstream of the same decisions.\n\nShow the sources used, with the citation handles the model was given. Show the count: \"drawn from 8 of 341 matching documents\". Where the assistant declined, say what was searched, because \"I found nothing about that in the product documentation\" is a useful answer and \"I don't know\" is not.\n\nAnd where a document was excluded for permissions, the honest interface says that something exists but is not visible to this user, if your security model allows it. Silently pretending nothing exists is a design choice with real consequences: people spend hours looking for a document they were told was not there.\n\n## Ask for what is missing\n\nThe strongest version of this is a system that says what it would have needed.\n\n```\nIf the passages do not contain the answer, respond with exactly:\nINSUFFICIENT: <what information would be needed>\n```\n\nThose strings are worth more than the answers. Cluster a month of them and you have a ranked list of gaps in your corpus, written in the language of the questions people actually ask. It is the cheapest content roadmap available, and it comes out of the failure path you already needed to build.\n\nOne caution: models are not reliable narrators of their own gaps. The stated missing information is a plausible guess, not an introspective report. Treat the aggregate as a signal about topics and the individual line as unverified.\n\n## The manifest\n\nFor systems where an answer might later be disputed (anything touching money, health, employment or the law) record what was in the window, not just what came out:\n\n```json\n{\"request_id\": \"...\", \"assembly_version\": \"a91f3c2d0b4e\",\n \"passages\": [{\"id\": \"doc_8831#4\", \"rank\": 1, \"tokens\": 412}, ...],\n \"candidates_considered\": 341, \"dropped_for_size\": 3,\n \"filters\": {\"acl\": [\"support\"], \"lang\": \"en\"}}\n```\n\nSmall, cheap, and it converts \"why did it say that\" from an investigation into a query. It is also the artefact that makes the measurement module later in this course possible at all: you cannot attribute a regression to selection unless you recorded what was selected.\n\n## The habit\n\nOnce a week, read ten manifests from failed or complained-about requests. Not the answers: the manifests. The question you are asking is only ever: *was the necessary passage in this list?* That single question splits every failure into a retrieval problem and a generation problem, and it takes about fifteen minutes."},"check":{"question":"Adding \"these passages may not contain the answer; say so if they do not\" raises the abstention rate from 4% to 19%. What must be checked before calling this an improvement?","options":[{"text":"Whether the token cost of the added instruction is justified by the change in abstention.","correct":false,"tempting":"Weighs a negligible cost (one sentence) against a change in behaviour that could be worth far more or far less than any token saving."},{"text":"Whether the retrieval recall changed, since abstention should track the quality of the retrieved set.","correct":false,"tempting":"Retrieval is unchanged by an instruction added at assembly time, so recall cannot explain the movement."},{"text":"Whether accuracy on questions the evidence does answer stayed the same, since the instruction can also suppress correct answers.","correct":true},{"text":"Whether the model is abstaining for the stated reason rather than for an unrelated one, by reading its explanations.","correct":false,"tempting":"Treats the model's stated reasons as evidence about its own processing, which they are not reliable as."}]},"prev":{"slug":"packing-to-a-budget","title":"Packing the budget","url":"https://addaly.com/learn/context-engineering/packing-to-a-budget"},"next":{"slug":"documents-that-fight-back","title":"Getting text out of documents that resist","url":"https://addaly.com/learn/context-engineering/documents-that-fight-back"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Admitting the set is partial', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/saying-what-was-left-out (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Admitting the set is partial","courseTitle":"Context Engineering","module":{"index":2,"slug":"what-earns-a-place","title":"Selection: what earns a place"},"answer":null,"keyIdea":"Tell the model its evidence set is partial, abstain in code when the retrieval signal is too weak to be worth a call, show the user what was searched and what was excluded, and record a manifest of what actually entered the window so a disputed answer can be explained.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-model-assumes-it-has-everything","runs":[{"text":"The model assumes it has everything"}]},{"t":"p","runs":[{"text":"Give a model eight passages and a question and it behaves as though those eight passages are the world. It has no way to know that 341 matched, that the ninth was dropped for size, or that the retriever searched only documents in English."}]},{"t":"p","runs":[{"text":"That assumption is the source of a specific and common failure: a confident answer built from an incomplete set, with no signal that anything is missing. The user cannot detect it, because the answer is well-formed and cites real sources."}]},{"t":"p","runs":[{"text":"The fix is to tell it. One line, above the passages:"}]},{"t":"code","lang":"","label":"Text","text":"The passages below are the 8 highest-ranked of 341 matches, selected\nautomatically. They may not contain the answer. If they do not, say so\nrather than inferring one."},{"t":"p","runs":[{"text":"This reliably increases the rate at which the model declines to answer when the evidence is absent. It also reliably increases the rate at which it declines when the evidence is present but awkward, which is the cost. Measure both: an abstention rate is only good news if the answer rate on answerable questions has not moved with it."}]},{"t":"h2","id":"abstain-before-you-assemble","runs":[{"text":"Abstain before you assemble"}]},{"t":"p","runs":[{"text":"Better than telling the model the set might be weak is not sending a weak set at all."}]},{"t":"p","runs":[{"text":"The signal from the first lesson in this module, the gap between the top score and the fifth, is a usable trigger. So is the top reranker score, which unlike an embedding similarity is at least trained to mean relevance. Calibrate a floor on judged examples, and below it return a direct “I do not have information about this” without a model call at all."}]},{"t":"p","runs":[{"text":"Two benefits beyond accuracy. It is free and instant, since no tokens are spent. And it is auditable: you can count how often it fires, which is a measurement of corpus coverage that nothing else in the system gives you. A sudden rise in the abstain rate is an index problem, and you find out in an hour rather than through complaints."}]},{"t":"h2","id":"tell-the-user-not-only-the-model","runs":[{"text":"Tell the user, not only the model"}]},{"t":"p","runs":[{"text":"What the interface shows changes what a user trusts, and it is part of context engineering because it is downstream of the same decisions."}]},{"t":"p","runs":[{"text":"Show the sources used, with the citation handles the model was given. Show the count: “drawn from 8 of 341 matching documents”. Where the assistant declined, say what was searched, because “I found nothing about that in the product documentation” is a useful answer and “I don’t know” is not."}]},{"t":"p","runs":[{"text":"And where a document was excluded for permissions, the honest interface says that something exists but is not visible to this user, if your security model allows it. Silently pretending nothing exists is a design choice with real consequences: people spend hours looking for a document they were told was not there."}]},{"t":"h2","id":"ask-for-what-is-missing","runs":[{"text":"Ask for what is missing"}]},{"t":"p","runs":[{"text":"The strongest version of this is a system that says what it would have needed."}]},{"t":"code","lang":"","label":"Text","text":"If the passages do not contain the answer, respond with exactly:\nINSUFFICIENT: <what information would be needed>"},{"t":"p","runs":[{"text":"Those strings are worth more than the answers. Cluster a month of them and you have a ranked list of gaps in your corpus, written in the language of the questions people actually ask. It is the cheapest content roadmap available, and it comes out of the failure path you already needed to build."}]},{"t":"p","runs":[{"text":"One caution: models are not reliable narrators of their own gaps. The stated missing information is a plausible guess, not an introspective report. Treat the aggregate as a signal about topics and the individual line as unverified."}]},{"t":"h2","id":"the-manifest","runs":[{"text":"The manifest"}]},{"t":"p","runs":[{"text":"For systems where an answer might later be disputed (anything touching money, health, employment or the law) record what was in the window, not just what came out:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"request_id\": \"...\", \"assembly_version\": \"a91f3c2d0b4e\",\n \"passages\": [{\"id\": \"doc_8831#4\", \"rank\": 1, \"tokens\": 412}, ...],\n \"candidates_considered\": 341, \"dropped_for_size\": 3,\n \"filters\": {\"acl\": [\"support\"], \"lang\": \"en\"}}"},{"t":"p","runs":[{"text":"Small, cheap, and it converts “why did it say that” from an investigation into a query. It is also the artefact that makes the measurement module later in this course possible at all: you cannot attribute a regression to selection unless you recorded what was selected."}]},{"t":"h2","id":"the-habit","runs":[{"text":"The habit"}]},{"t":"p","runs":[{"text":"Once a week, read ten manifests from failed or complained-about requests. Not the answers: the manifests. The question you are asking is only ever: "},{"text":"was the necessary passage in this list?","i":true},{"text":" That single question splits every failure into a retrieval problem and a generation problem, and it takes about fifteen minutes."}]}],"check":{"question":[{"text":"Adding “these passages may not contain the answer; say so if they do not” raises the abstention rate from 4% to 19%. What must be checked before calling this an improvement?"}],"options":[{"text":[{"text":"Whether the token cost of the added instruction is justified by the change in abstention."}],"correct":false,"tempting":[{"text":"Weighs a negligible cost (one sentence) against a change in behaviour that could be worth far more or far less than any token saving."}]},{"text":[{"text":"Whether the retrieval recall changed, since abstention should track the quality of the retrieved set."}],"correct":false,"tempting":[{"text":"Retrieval is unchanged by an instruction added at assembly time, so recall cannot explain the movement."}]},{"text":[{"text":"Whether accuracy on questions the evidence does answer stayed the same, since the instruction can also suppress correct answers."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the model is abstaining for the stated reason rather than for an unrelated one, by reading its explanations."}],"correct":false,"tempting":[{"text":"Treats the model’s stated reasons as evidence about its own processing, which they are not reliable as."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"packing-to-a-budget","title":"Packing the budget","minutes":9},"next":{"slug":"documents-that-fight-back","title":"Getting text out of documents that resist","minutes":10}}}