From a complaint to the stage that caused it

The capstone is a procedure, not an idea

A user reports a wrong answer. You have the request id. Everything in this course exists so that the next twenty minutes are a lookup rather than a debate.

Work the stages in order. Each step is a question with a mechanical answer.

1. Does it still happen?

replay r_8812 --config ctx-v38     # the config it ran under
replay r_8812 --config current

If it does not reproduce under the original configuration, you are looking at sampling variance or a provider-side change, and the first thing to check is whether the served model version in the log matches what is served now.

If it reproduces under the original but not under current, it is already fixed. Say so in the ticket, note which version fixed it, and add the case to the golden set so it stays fixed.

2. Was the evidence in the window at all?

This is the highest-yield question and the manifest answers it in one glance. Compare retrieved with packed.

Not retrieved. A retrieval failure, and the sub-causes are distinguishable. Does the passage exist in the index: if not, it is an ingestion problem: parsing dropped it, or the crawl never reached it. Does keyword search find it but vector search not, which points at a similarity failure the hybrid fusion should have caught. Did a filter exclude it, which is the permission and freshness stage. Did the rewritten query wander from the real question: compare query_raw with query_rewritten, which is where a surprising share of these end.

Retrieved but not packed. It ranked too low and the budget cut it, or dedup removed it as a near-duplicate of something worse. Check its rank. Rank 7 with k=6 is a reranker problem; rank 1 dropped at packing is a bug.

Packed. Move on. Retrieval did its job and the problem is downstream, which is worth establishing firmly because retrieval gets blamed by default.

3. It was there and was not used

Three usual causes.

Position. Where did it sit? If it was in the middle of a long window, run the position probe from module four on this case: hold everything constant and move the passage to the front. If the answer becomes correct, you have a position problem, and the fix is ordering or a lower k rather than anything about the prompt.

Distractors. Look at what else was packed. Passages ranked just below a correct answer are on-topic and wrong, which is the most effective distractor there is. Try the case at k=3. If it becomes correct with less evidence, your k is too high: a counter-intuitive result that is common enough to check early.

Contradiction. Two passages disagree, usually because one is an old version. Check the dates you attached. If the model picked the stale one, the fix is at ingestion and dedup, not in the prompt.

4. The instruction was not followed

Find the instruction in the rendered request and note where it is. If it is in a long stable prefix, far from the question, and it is a gating constraint, this is the case for a short restatement near the question.

Then check for a conflicting instruction. Long prompts accumulate them, and two paragraphs written eight months apart can contradict each other in a way nobody has read end to end since. The snapshot file makes this readable; reading your own assembled prompt in full is an underrated diagnostic and most people have never done it once.

5. The format was wrong

If a schema was enforced, the output parsed, so this is a semantic failure rather than a formatting one: usually a constrained enum with no escape value, forcing a choice where none applied. If no schema was enforced, that is the fix.

6. A citation was fabricated

Check the handle against the manifest. If it names nothing that was in the window, your post-generation verification step is missing or disabled. This is mechanical and should never reach a user.

7. It did something unexpected

Check has_untrusted and the tool calls. An action taken in a turn containing untrusted material is the injection signature from module eight, and it should have alerted rather than arriving as a support ticket. If it did not alert, the gap is in your monitoring and that is the more important finding.

8. It contradicted itself, or forgot

A long conversation. Find the compaction point in the log. Was the lost fact in a state field or only in prose that got summarised? If only in prose, the fix is to promote it to a field, not to write a better summariser.

Then write it down

The diagnosis is worth more than the fix, because diagnoses repeat.

For each one, do three things: add the case to the golden set so the fix is protected; ask whether an offline check could have caught it: a great many of these are mechanical and belong in continuous integration rather than in a ticket; and if the same stage appears three times in a quarter, stop fixing instances and fix the stage.

What this procedure actually depends on

Read back over the eight steps and notice how little of it is insight. It is the manifest, the replay command, the config version, the trust labels, the retrieval scores, the compaction log and the snapshot file: all of it instrumentation decided long before this ticket existed.

That is the argument of the whole course in one place. The window is a budget, and it is also a system, and a system you can see into is one you can fix on a Tuesday afternoon. A team without this instrumentation is not worse at prompting than you are. It is guessing, because guessing is the only thing available to it, and no amount of skill at writing instructions substitutes for being able to see what was in the window.