{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"diagnosing-a-bad-answer","title":"From a complaint to the stage that caused it","url":"https://addaly.com/learn/context-engineering/diagnosing-a-bad-answer","position":87,"of":87,"minutes":10,"answer":"A bad answer is produced by one identifiable stage of the pipeline, and with a logged manifest and a replay command the diagnosis is a short sequence of lookups rather than a guess.","keyIdea":"A bad answer is produced by one identifiable stage of the pipeline, and with a logged manifest and a replay command the diagnosis is a short sequence of lookups rather than a guess, which is the whole reason for the instrumentation this course has been building."},"body":{"format":"markdown","text":"## The capstone is a procedure, not an idea\n\nA user reports a wrong answer. You have the request id. Everything in this course exists so that the next twenty minutes are a lookup rather than a debate.\n\nWork the stages in order. Each step is a question with a mechanical answer.\n\n## 1. Does it still happen?\n\n```bash\nreplay r_8812 --config ctx-v38     # the config it ran under\nreplay r_8812 --config current\n```\n\nIf it does not reproduce under the original configuration, you are looking at sampling variance or a provider-side change, and the first thing to check is whether the served model version in the log matches what is served now.\n\nIf it reproduces under the original but not under current, it is already fixed. Say so in the ticket, note which version fixed it, and add the case to the golden set so it stays fixed.\n\n## 2. Was the evidence in the window at all?\n\nThis is the highest-yield question and the manifest answers it in one glance. Compare `retrieved` with `packed`.\n\n**Not retrieved.** A retrieval failure, and the sub-causes are distinguishable. Does the passage exist in the index: if not, it is an ingestion problem: parsing dropped it, or the crawl never reached it. Does keyword search find it but vector search not, which points at a similarity failure the hybrid fusion should have caught. Did a filter exclude it, which is the permission and freshness stage. Did the rewritten query wander from the real question: compare `query_raw` with `query_rewritten`, which is where a surprising share of these end.\n\n**Retrieved but not packed.** It ranked too low and the budget cut it, or dedup removed it as a near-duplicate of something worse. Check its rank. Rank 7 with k=6 is a reranker problem; rank 1 dropped at packing is a bug.\n\n**Packed.** Move on. Retrieval did its job and the problem is downstream, which is worth establishing firmly because retrieval gets blamed by default.\n\n> Figure: From a request id to the stage that caused it. Steps: 1. Replay it on its own config (No reproduction means sampling noise or a provider change; check the served model version); 2. Compare retrieved with packed (Never retrieved is ingestion or retrieval; retrieved and dropped is rank, budget or dedup); 3. If packed, ask where it sat (Middle of a long window: move it to the front and see whether the answer appears); 4. Read what else was packed (Passages ranked just below the answer are the most effective distractors there are); 5. Find the instruction, and its twin (Far from the question and gating: restate it. Then look for one contradicting it); 6. Add the case to the golden set (Then ask whether an offline assertion could have caught it, as most of these could). Almost none of this is insight. It is the manifest, the replay command, the configuration version, the trust labels, the retrieval scores and the snapshot test, which is the argument of the whole course standing in one place.\n\n## 3. It was there and was not used\n\nThree usual causes.\n\n**Position.** Where did it sit? If it was in the middle of a long window, run the position probe from module four on this case: hold everything constant and move the passage to the front. If the answer becomes correct, you have a position problem, and the fix is ordering or a lower k rather than anything about the prompt.\n\n**Distractors.** Look at what else was packed. Passages ranked just below a correct answer are on-topic and wrong, which is the most effective distractor there is. Try the case at k=3. If it becomes correct with less evidence, your k is too high: a counter-intuitive result that is common enough to check early.\n\n**Contradiction.** Two passages disagree, usually because one is an old version. Check the dates you attached. If the model picked the stale one, the fix is at ingestion and dedup, not in the prompt.\n\n## 4. The instruction was not followed\n\nFind the instruction in the rendered request and note where it is. If it is in a long stable prefix, far from the question, and it is a gating constraint, this is the case for a short restatement near the question.\n\nThen check for a conflicting instruction. Long prompts accumulate them, and two paragraphs written eight months apart can contradict each other in a way nobody has read end to end since. The snapshot file makes this readable; reading your own assembled prompt in full is an underrated diagnostic and most people have never done it once.\n\n## 5. The format was wrong\n\nIf a schema was enforced, the output parsed, so this is a semantic failure rather than a formatting one: usually a constrained enum with no escape value, forcing a choice where none applied. If no schema was enforced, that is the fix.\n\n## 6. A citation was fabricated\n\nCheck the handle against the manifest. If it names nothing that was in the window, your post-generation verification step is missing or disabled. This is mechanical and should never reach a user.\n\n## 7. It did something unexpected\n\nCheck `has_untrusted` and the tool calls. An action taken in a turn containing untrusted material is the injection signature from module eight, and it should have alerted rather than arriving as a support ticket. If it did not alert, the gap is in your monitoring and that is the more important finding.\n\n## 8. It contradicted itself, or forgot\n\nA long conversation. Find the compaction point in the log. Was the lost fact in a state field or only in prose that got summarised? If only in prose, the fix is to promote it to a field, not to write a better summariser.\n\n## Then write it down\n\nThe diagnosis is worth more than the fix, because diagnoses repeat.\n\nFor each one, do three things: add the case to the golden set so the fix is protected; ask whether an offline check could have caught it: a great many of these are mechanical and belong in continuous integration rather than in a ticket; and if the same stage appears three times in a quarter, stop fixing instances and fix the stage.\n\n## What this procedure actually depends on\n\nRead back over the eight steps and notice how little of it is insight. It is the manifest, the replay command, the config version, the trust labels, the retrieval scores, the compaction log and the snapshot file: all of it instrumentation decided long before this ticket existed.\n\nThat is the argument of the whole course in one place. The window is a budget, and it is also a system, and a system you can see into is one you can fix on a Tuesday afternoon. A team without this instrumentation is not worse at prompting than you are. It is guessing, because guessing is the only thing available to it, and no amount of skill at writing instructions substitutes for being able to see what was in the window."},"check":{"question":"A manifest shows the correct passage was retrieved at rank 1 and packed, yet the answer is wrong. What does that rule out?","options":[{"text":"A contradiction between passages, since the correct one ranked highest and would therefore be preferred.","correct":false,"tempting":"Rank does not settle which passage the model believes; a stale passage packed alongside can still win on the model's reading."},{"text":"Retrieval as the cause, directing the investigation to position, distractors or the instructions.","correct":true},{"text":"A schema failure, because a packed passage means the output contract was satisfied.","correct":false,"tempting":"Packing concerns the input side entirely and says nothing about how the output was shaped or constrained."},{"text":"An injection, since attacker-controlled text would have displaced the correct passage from rank 1.","correct":false,"tempting":"Injected content usually rides along with genuine results rather than displacing them, so ranking tells you nothing about it."}]},"prev":{"slug":"the-model-changed-under-you","title":"The dependency you do not control","url":"https://addaly.com/learn/context-engineering/the-model-changed-under-you"},"next":null,"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'From a complaint to the stage that caused it', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/diagnosing-a-bad-answer (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"From a complaint to the stage that caused it","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"A bad answer is produced by one identifiable stage of the pipeline, and with a logged manifest and a replay command the diagnosis is a short sequence of lookups rather than a guess, which is the whole reason for the instrumentation this course has been building.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Replay it on its own config","detail":"No reproduction means sampling noise or a provider change; check the served model version"},{"label":"Compare retrieved with packed","detail":"Never retrieved is ingestion or retrieval; retrieved and dropped is rank, budget or dedup"},{"label":"If packed, ask where it sat","detail":"Middle of a long window: move it to the front and see whether the answer appears"},{"label":"Read what else was packed","detail":"Passages ranked just below the answer are the most effective distractors there are"},{"label":"Find the instruction, and its twin","detail":"Far from the question and gating: restate it. Then look for one contradicting it"},{"label":"Add the case to the golden set","detail":"Then ask whether an offline assertion could have caught it, as most of these could"}],"title":"From a request id to the stage that caused it","caption":"Almost none of this is insight. It is the manifest, the replay command, the configuration version, the trust labels, the retrieval scores and the snapshot test, which is the argument of the whole course standing in one place."},"alt":"Figure: From a request id to the stage that caused it. Steps: 1. Replay it on its own config (No reproduction means sampling noise or a provider change; check the served model version); 2. Compare retrieved with packed (Never retrieved is ingestion or retrieval; retrieved and dropped is rank, budget or dedup); 3. If packed, ask where it sat (Middle of a long window: move it to the front and see whether the answer appears); 4. Read what else was packed (Passages ranked just below the answer are the most effective distractors there are); 5. Find the instruction, and its twin (Far from the question and gating: restate it. Then look for one contradicting it); 6. Add the case to the golden set (Then ask whether an offline assertion could have caught it, as most of these could). Almost none of this is insight. It is the manifest, the replay command, the configuration version, the trust labels, the retrieval scores and the snapshot test, which is the argument of the whole course standing in one place."}},"blocks":[{"t":"h2","id":"the-capstone-is-a-procedure-not-an-idea","runs":[{"text":"The capstone is a procedure, not an idea"}]},{"t":"p","runs":[{"text":"A user reports a wrong answer. You have the request id. Everything in this course exists so that the next twenty minutes are a lookup rather than a debate."}]},{"t":"p","runs":[{"text":"Work the stages in order. Each step is a question with a mechanical answer."}]},{"t":"h2","id":"1-does-it-still-happen","runs":[{"text":"1. Does it still happen?"}]},{"t":"code","lang":"bash","label":"Bash","text":"replay r_8812 --config ctx-v38     # the config it ran under\nreplay r_8812 --config current"},{"t":"p","runs":[{"text":"If it does not reproduce under the original configuration, you are looking at sampling variance or a provider-side change, and the first thing to check is whether the served model version in the log matches what is served now."}]},{"t":"p","runs":[{"text":"If it reproduces under the original but not under current, it is already fixed. Say so in the ticket, note which version fixed it, and add the case to the golden set so it stays fixed."}]},{"t":"h2","id":"2-was-the-evidence-in-the-window-at-all","runs":[{"text":"2. Was the evidence in the window at all?"}]},{"t":"p","runs":[{"text":"This is the highest-yield question and the manifest answers it in one glance. Compare "},{"text":"retrieved","code":true},{"text":" with "},{"text":"packed","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Not retrieved.","b":true},{"text":" A retrieval failure, and the sub-causes are distinguishable. Does the passage exist in the index: if not, it is an ingestion problem: parsing dropped it, or the crawl never reached it. Does keyword search find it but vector search not, which points at a similarity failure the hybrid fusion should have caught. Did a filter exclude it, which is the permission and freshness stage. Did the rewritten query wander from the real question: compare "},{"text":"query_raw","code":true},{"text":" with "},{"text":"query_rewritten","code":true},{"text":", which is where a surprising share of these end."}]},{"t":"p","runs":[{"text":"Retrieved but not packed.","b":true},{"text":" It ranked too low and the budget cut it, or dedup removed it as a near-duplicate of something worse. Check its rank. Rank 7 with k=6 is a reranker problem; rank 1 dropped at packing is a bug."}]},{"t":"p","runs":[{"text":"Packed.","b":true},{"text":" Move on. Retrieval did its job and the problem is downstream, which is worth establishing firmly because retrieval gets blamed by default."}]},{"t":"h2","id":"3-it-was-there-and-was-not-used","runs":[{"text":"3. It was there and was not used"}]},{"t":"p","runs":[{"text":"Three usual causes."}]},{"t":"p","runs":[{"text":"Position.","b":true},{"text":" Where did it sit? If it was in the middle of a long window, run the position probe from module four on this case: hold everything constant and move the passage to the front. If the answer becomes correct, you have a position problem, and the fix is ordering or a lower k rather than anything about the prompt."}]},{"t":"p","runs":[{"text":"Distractors.","b":true},{"text":" Look at what else was packed. Passages ranked just below a correct answer are on-topic and wrong, which is the most effective distractor there is. Try the case at k=3. If it becomes correct with less evidence, your k is too high: a counter-intuitive result that is common enough to check early."}]},{"t":"p","runs":[{"text":"Contradiction.","b":true},{"text":" Two passages disagree, usually because one is an old version. Check the dates you attached. If the model picked the stale one, the fix is at ingestion and dedup, not in the prompt."}]},{"t":"h2","id":"4-the-instruction-was-not-followed","runs":[{"text":"4. The instruction was not followed"}]},{"t":"p","runs":[{"text":"Find the instruction in the rendered request and note where it is. If it is in a long stable prefix, far from the question, and it is a gating constraint, this is the case for a short restatement near the question."}]},{"t":"p","runs":[{"text":"Then check for a conflicting instruction. Long prompts accumulate them, and two paragraphs written eight months apart can contradict each other in a way nobody has read end to end since. The snapshot file makes this readable; reading your own assembled prompt in full is an underrated diagnostic and most people have never done it once."}]},{"t":"h2","id":"5-the-format-was-wrong","runs":[{"text":"5. The format was wrong"}]},{"t":"p","runs":[{"text":"If a schema was enforced, the output parsed, so this is a semantic failure rather than a formatting one: usually a constrained enum with no escape value, forcing a choice where none applied. If no schema was enforced, that is the fix."}]},{"t":"h2","id":"6-a-citation-was-fabricated","runs":[{"text":"6. A citation was fabricated"}]},{"t":"p","runs":[{"text":"Check the handle against the manifest. If it names nothing that was in the window, your post-generation verification step is missing or disabled. This is mechanical and should never reach a user."}]},{"t":"h2","id":"7-it-did-something-unexpected","runs":[{"text":"7. It did something unexpected"}]},{"t":"p","runs":[{"text":"Check "},{"text":"has_untrusted","code":true},{"text":" and the tool calls. An action taken in a turn containing untrusted material is the injection signature from module eight, and it should have alerted rather than arriving as a support ticket. If it did not alert, the gap is in your monitoring and that is the more important finding."}]},{"t":"h2","id":"8-it-contradicted-itself-or-forgot","runs":[{"text":"8. It contradicted itself, or forgot"}]},{"t":"p","runs":[{"text":"A long conversation. Find the compaction point in the log. Was the lost fact in a state field or only in prose that got summarised? If only in prose, the fix is to promote it to a field, not to write a better summariser."}]},{"t":"h2","id":"then-write-it-down","runs":[{"text":"Then write it down"}]},{"t":"p","runs":[{"text":"The diagnosis is worth more than the fix, because diagnoses repeat."}]},{"t":"p","runs":[{"text":"For each one, do three things: add the case to the golden set so the fix is protected; ask whether an offline check could have caught it: a great many of these are mechanical and belong in continuous integration rather than in a ticket; and if the same stage appears three times in a quarter, stop fixing instances and fix the stage."}]},{"t":"h2","id":"what-this-procedure-actually-depends-on","runs":[{"text":"What this procedure actually depends on"}]},{"t":"p","runs":[{"text":"Read back over the eight steps and notice how little of it is insight. It is the manifest, the replay command, the config version, the trust labels, the retrieval scores, the compaction log and the snapshot file: all of it instrumentation decided long before this ticket existed."}]},{"t":"p","runs":[{"text":"That is the argument of the whole course in one place. The window is a budget, and it is also a system, and a system you can see into is one you can fix on a Tuesday afternoon. A team without this instrumentation is not worse at prompting than you are. It is guessing, because guessing is the only thing available to it, and no amount of skill at writing instructions substitutes for being able to see what was in the window."}]}],"check":{"question":[{"text":"A manifest shows the correct passage was retrieved at rank 1 and packed, yet the answer is wrong. What does that rule out?"}],"options":[{"text":[{"text":"A contradiction between passages, since the correct one ranked highest and would therefore be preferred."}],"correct":false,"tempting":[{"text":"Rank does not settle which passage the model believes; a stale passage packed alongside can still win on the model’s reading."}]},{"text":[{"text":"Retrieval as the cause, directing the investigation to position, distractors or the instructions."}],"correct":true,"tempting":null},{"text":[{"text":"A schema failure, because a packed passage means the output contract was satisfied."}],"correct":false,"tempting":[{"text":"Packing concerns the input side entirely and says nothing about how the output was shaped or constrained."}]},{"text":[{"text":"An injection, since attacker-controlled text would have displaced the correct passage from rank 1."}],"correct":false,"tempting":[{"text":"Injected content usually rides along with genuine results rather than displacing them, so ranking tells you nothing about it."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"the-model-changed-under-you","title":"The dependency you do not control","minutes":8},"next":null}}