{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"ablation-earns-the-block","title":"Take it out and see what happens","url":"https://addaly.com/learn/context-engineering/ablation-earns-the-block","position":81,"of":87,"minutes":9,"answer":"The only honest way to know whether a block of context earns its tokens is to remove it and re-measure.","keyIdea":"The only honest way to know whether a block of context earns its tokens is to remove it and re-measure, which routinely reveals paragraphs that have been paid for on every request for a year and change nothing."},"body":{"format":"markdown","text":"## Prompts accumulate and nothing removes them\n\nA prompt in production is a sedimentary record. Someone added a paragraph after a bad demo. Someone added three sentences after an incident. Someone added an example. Someone added a warning about a failure mode that was actually a retrieval bug, fixed since.\n\nNothing in the process removes any of it. Each addition was plausible, each was made under pressure, and none was ever tested alone. Two years later the system prompt is 2,400 tokens and nobody in the building can tell you what any individual paragraph does.\n\nAblation is the only instrument that answers the question. Remove one block. Re-run the golden set. Look at the difference.\n\n## The procedure\n\nList the removable units: each paragraph of the system prompt, each few-shot example, the repeated instruction near the question, the chunk header fields, the reranker, the query rewriter, and each numeric setting worth testing at a lower value.\n\nThen run the set once per variant, changing exactly one thing.\n\n```python\nvariants = {\n    \"baseline\":        CONFIG,\n    \"no_tone_para\":    CONFIG.without(\"system.tone\"),\n    \"no_examples\":     CONFIG.with_(few_shot=[]),\n    \"no_reranker\":     CONFIG.with_(rerank=False),\n    \"k=5\":             CONFIG.with_(k=5),\n    \"no_chunk_dates\":  CONFIG.with_(header_fields=[\"path\", \"handle\"]),\n}\n```\n\nFor each, record accuracy by group, tokens per request, and latency. Three columns, one row per variant, and the row that matters is the one where accuracy did not move and tokens fell.\n\n## What you will find\n\nTwo results are close to universal.\n\n**Something large does nothing.** Typically a paragraph of 200 to 500 tokens whose removal moves no metric at all. It has been billed on every request since it was written. Teams are always slightly embarrassed by this and should not be: it is the expected outcome of a process with additions and no deletions, and finding it is the point.\n\n**Something small is load-bearing.** A single sentence whose removal drops one group by fifteen points. Usually a constraint: the refusal condition, the instruction to cite, the rule about not inventing prices. Now you know, and, more usefully, it is now written down, which protects it from the future tidy-up where someone shortens the prompt for readability.\n\nThat second outcome is the underrated half. Ablation produces documentation nobody would otherwise write: a list of which parts of your context are actually doing the work, with a number beside each.\n\n## Ablate more than the prompt\n\n**k.** Almost always lower than it is set to. The k-curve is flat near its left edge, so k=5 frequently matches k=12 at less than half the tokens.\n\n**Few-shot examples.** The most expensive instruction format there is. Test whether two work as well as five, and whether the instruction alone works as well as either. On capable models it increasingly does, and the examples stay because nobody re-tested them after the model changed.\n\n**Chunk header fields.** Paid k times per request. A date field costing eight tokens with k=10 is 80 tokens a request; if versions never conflict in your corpus, it buys nothing.\n\n**The reranker.** Real latency and real cost. Sometimes a large accuracy gain and sometimes, on a small clean corpus, none.\n\n**The repeated instruction.** Test whether repeating it near the question beats stating it once at the top, on your model rather than on the model the advice was written for.\n\n## The trap\n\nAblation on a small set will hand you noise dressed as signal. Remove a paragraph, see accuracy rise by two points, and conclude the paragraph was harmful, when re-running the identical configuration would have moved it by two points anyway.\n\nSo: never act on a single small difference. Anything under a few points on a set of a couple of hundred cases should be treated as \"no detectable effect\", and the correct response to *no detectable effect* on an expensive block is still to remove it, because you are removing cost you can measure in exchange for a benefit you cannot.\n\nThe next lesson is how to tell the two apart properly.\n\n## Make it a habit, not a project\n\nAn ablation sweep of a dozen variants is one script and one evening of compute, and the results go stale as the model, corpus and prompt change.\n\nRun it quarterly, and run it whenever you change model. The second one is the more valuable: a new model changes which blocks are load-bearing, often dramatically, and the examples or warnings that a previous generation needed are frequently the first things a newer one no longer requires. Prompts get longer over time and models get better at not needing them, and only a sweep will tell you that the two have crossed."},"check":{"question":"An ablation removes a 400-token paragraph and accuracy is unchanged within noise. What is the right conclusion?","options":[{"text":"Keep it, since an effect too small to measure may still be protecting against a rare failure the set omits.","correct":false,"tempting":"A real risk, and the remedy is to add that case to the set rather than to pay indefinitely for an untested belief."},{"text":"Re-run with a higher temperature to see whether the paragraph matters under more variable sampling.","correct":false,"tempting":"Added variance makes any difference harder to detect, so this reduces the power of the test rather than sharpening it."},{"text":"Remove it: the cost is measurable and the benefit is not.","correct":true},{"text":"The result is inconclusive until the paragraph is tested in combination with the other removable blocks.","correct":false,"tempting":"Interaction effects exist, but waiting for a full combinatorial sweep is how a block nobody can justify stays in the prompt forever."}]},"prev":{"slug":"offline-checks-on-the-assembler","title":"The tests that need no model at all","url":"https://addaly.com/learn/context-engineering/offline-checks-on-the-assembler"},"next":{"slug":"is-the-difference-real","title":"Three points better, or three points of noise","url":"https://addaly.com/learn/context-engineering/is-the-difference-real"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Take it out and see what happens', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/ablation-earns-the-block (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Take it out and see what happens","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"The only honest way to know whether a block of context earns its tokens is to remove it and re-measure, which routinely reveals paragraphs that have been paid for on every request for a year and change nothing.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"prompts-accumulate-and-nothing-removes-them","runs":[{"text":"Prompts accumulate and nothing removes them"}]},{"t":"p","runs":[{"text":"A prompt in production is a sedimentary record. Someone added a paragraph after a bad demo. Someone added three sentences after an incident. Someone added an example. Someone added a warning about a failure mode that was actually a retrieval bug, fixed since."}]},{"t":"p","runs":[{"text":"Nothing in the process removes any of it. Each addition was plausible, each was made under pressure, and none was ever tested alone. Two years later the system prompt is 2,400 tokens and nobody in the building can tell you what any individual paragraph does."}]},{"t":"p","runs":[{"text":"Ablation is the only instrument that answers the question. Remove one block. Re-run the golden set. Look at the difference."}]},{"t":"h2","id":"the-procedure","runs":[{"text":"The procedure"}]},{"t":"p","runs":[{"text":"List the removable units: each paragraph of the system prompt, each few-shot example, the repeated instruction near the question, the chunk header fields, the reranker, the query rewriter, and each numeric setting worth testing at a lower value."}]},{"t":"p","runs":[{"text":"Then run the set once per variant, changing exactly one thing."}]},{"t":"code","lang":"python","label":"Python","text":"variants = {\n    \"baseline\":        CONFIG,\n    \"no_tone_para\":    CONFIG.without(\"system.tone\"),\n    \"no_examples\":     CONFIG.with_(few_shot=[]),\n    \"no_reranker\":     CONFIG.with_(rerank=False),\n    \"k=5\":             CONFIG.with_(k=5),\n    \"no_chunk_dates\":  CONFIG.with_(header_fields=[\"path\", \"handle\"]),\n}"},{"t":"p","runs":[{"text":"For each, record accuracy by group, tokens per request, and latency. Three columns, one row per variant, and the row that matters is the one where accuracy did not move and tokens fell."}]},{"t":"h2","id":"what-you-will-find","runs":[{"text":"What you will find"}]},{"t":"p","runs":[{"text":"Two results are close to universal."}]},{"t":"p","runs":[{"text":"Something large does nothing.","b":true},{"text":" Typically a paragraph of 200 to 500 tokens whose removal moves no metric at all. It has been billed on every request since it was written. Teams are always slightly embarrassed by this and should not be: it is the expected outcome of a process with additions and no deletions, and finding it is the point."}]},{"t":"p","runs":[{"text":"Something small is load-bearing.","b":true},{"text":" A single sentence whose removal drops one group by fifteen points. Usually a constraint: the refusal condition, the instruction to cite, the rule about not inventing prices. Now you know, and, more usefully, it is now written down, which protects it from the future tidy-up where someone shortens the prompt for readability."}]},{"t":"p","runs":[{"text":"That second outcome is the underrated half. Ablation produces documentation nobody would otherwise write: a list of which parts of your context are actually doing the work, with a number beside each."}]},{"t":"h2","id":"ablate-more-than-the-prompt","runs":[{"text":"Ablate more than the prompt"}]},{"t":"p","runs":[{"text":"k.","b":true},{"text":" Almost always lower than it is set to. The k-curve is flat near its left edge, so k=5 frequently matches k=12 at less than half the tokens."}]},{"t":"p","runs":[{"text":"Few-shot examples.","b":true},{"text":" The most expensive instruction format there is. Test whether two work as well as five, and whether the instruction alone works as well as either. On capable models it increasingly does, and the examples stay because nobody re-tested them after the model changed."}]},{"t":"p","runs":[{"text":"Chunk header fields.","b":true},{"text":" Paid k times per request. A date field costing eight tokens with k=10 is 80 tokens a request; if versions never conflict in your corpus, it buys nothing."}]},{"t":"p","runs":[{"text":"The reranker.","b":true},{"text":" Real latency and real cost. Sometimes a large accuracy gain and sometimes, on a small clean corpus, none."}]},{"t":"p","runs":[{"text":"The repeated instruction.","b":true},{"text":" Test whether repeating it near the question beats stating it once at the top, on your model rather than on the model the advice was written for."}]},{"t":"h2","id":"the-trap","runs":[{"text":"The trap"}]},{"t":"p","runs":[{"text":"Ablation on a small set will hand you noise dressed as signal. Remove a paragraph, see accuracy rise by two points, and conclude the paragraph was harmful, when re-running the identical configuration would have moved it by two points anyway."}]},{"t":"p","runs":[{"text":"So: never act on a single small difference. Anything under a few points on a set of a couple of hundred cases should be treated as “no detectable effect”, and the correct response to "},{"text":"no detectable effect","i":true},{"text":" on an expensive block is still to remove it, because you are removing cost you can measure in exchange for a benefit you cannot."}]},{"t":"p","runs":[{"text":"The next lesson is how to tell the two apart properly."}]},{"t":"h2","id":"make-it-a-habit-not-a-project","runs":[{"text":"Make it a habit, not a project"}]},{"t":"p","runs":[{"text":"An ablation sweep of a dozen variants is one script and one evening of compute, and the results go stale as the model, corpus and prompt change."}]},{"t":"p","runs":[{"text":"Run it quarterly, and run it whenever you change model. The second one is the more valuable: a new model changes which blocks are load-bearing, often dramatically, and the examples or warnings that a previous generation needed are frequently the first things a newer one no longer requires. Prompts get longer over time and models get better at not needing them, and only a sweep will tell you that the two have crossed."}]}],"check":{"question":[{"text":"An ablation removes a 400-token paragraph and accuracy is unchanged within noise. What is the right conclusion?"}],"options":[{"text":[{"text":"Keep it, since an effect too small to measure may still be protecting against a rare failure the set omits."}],"correct":false,"tempting":[{"text":"A real risk, and the remedy is to add that case to the set rather than to pay indefinitely for an untested belief."}]},{"text":[{"text":"Re-run with a higher temperature to see whether the paragraph matters under more variable sampling."}],"correct":false,"tempting":[{"text":"Added variance makes any difference harder to detect, so this reduces the power of the test rather than sharpening it."}]},{"text":[{"text":"Remove it: the cost is measurable and the benefit is not."}],"correct":true,"tempting":null},{"text":[{"text":"The result is inconclusive until the paragraph is tested in combination with the other removable blocks."}],"correct":false,"tempting":[{"text":"Interaction effects exist, but waiting for a full combinatorial sweep is how a block nobody can justify stays in the prompt forever."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"offline-checks-on-the-assembler","title":"The tests that need no model at all","minutes":8},"next":{"slug":"is-the-difference-real","title":"Three points better, or three points of noise","minutes":10}}}