Take it out and see what happens

Prompts accumulate and nothing removes them

A prompt in production is a sedimentary record. Someone added a paragraph after a bad demo. Someone added three sentences after an incident. Someone added an example. Someone added a warning about a failure mode that was actually a retrieval bug, fixed since.

Nothing in the process removes any of it. Each addition was plausible, each was made under pressure, and none was ever tested alone. Two years later the system prompt is 2,400 tokens and nobody in the building can tell you what any individual paragraph does.

Ablation is the only instrument that answers the question. Remove one block. Re-run the golden set. Look at the difference.

The procedure

List the removable units: each paragraph of the system prompt, each few-shot example, the repeated instruction near the question, the chunk header fields, the reranker, the query rewriter, and each numeric setting worth testing at a lower value.

Then run the set once per variant, changing exactly one thing.

variants = {
    "baseline":        CONFIG,
    "no_tone_para":    CONFIG.without("system.tone"),
    "no_examples":     CONFIG.with_(few_shot=[]),
    "no_reranker":     CONFIG.with_(rerank=False),
    "k=5":             CONFIG.with_(k=5),
    "no_chunk_dates":  CONFIG.with_(header_fields=["path", "handle"]),
}

For each, record accuracy by group, tokens per request, and latency. Three columns, one row per variant, and the row that matters is the one where accuracy did not move and tokens fell.

What you will find

Two results are close to universal.

Something large does nothing. Typically a paragraph of 200 to 500 tokens whose removal moves no metric at all. It has been billed on every request since it was written. Teams are always slightly embarrassed by this and should not be: it is the expected outcome of a process with additions and no deletions, and finding it is the point.

Something small is load-bearing. A single sentence whose removal drops one group by fifteen points. Usually a constraint: the refusal condition, the instruction to cite, the rule about not inventing prices. Now you know, and, more usefully, it is now written down, which protects it from the future tidy-up where someone shortens the prompt for readability.

That second outcome is the underrated half. Ablation produces documentation nobody would otherwise write: a list of which parts of your context are actually doing the work, with a number beside each.

Ablate more than the prompt

k. Almost always lower than it is set to. The k-curve is flat near its left edge, so k=5 frequently matches k=12 at less than half the tokens.

Few-shot examples. The most expensive instruction format there is. Test whether two work as well as five, and whether the instruction alone works as well as either. On capable models it increasingly does, and the examples stay because nobody re-tested them after the model changed.

Chunk header fields. Paid k times per request. A date field costing eight tokens with k=10 is 80 tokens a request; if versions never conflict in your corpus, it buys nothing.

The reranker. Real latency and real cost. Sometimes a large accuracy gain and sometimes, on a small clean corpus, none.

The repeated instruction. Test whether repeating it near the question beats stating it once at the top, on your model rather than on the model the advice was written for.

The trap

Ablation on a small set will hand you noise dressed as signal. Remove a paragraph, see accuracy rise by two points, and conclude the paragraph was harmful, when re-running the identical configuration would have moved it by two points anyway.

So: never act on a single small difference. Anything under a few points on a set of a couple of hundred cases should be treated as “no detectable effect”, and the correct response to no detectable effect on an expensive block is still to remove it, because you are removing cost you can measure in exchange for a benefit you cannot.

The next lesson is how to tell the two apart properly.

Make it a habit, not a project

An ablation sweep of a dozen variants is one script and one evening of compute, and the results go stale as the model, corpus and prompt change.

Run it quarterly, and run it whenever you change model. The second one is the more valuable: a new model changes which blocks are load-bearing, often dramatically, and the examples or warnings that a previous generation needed are frequently the first things a newer one no longer requires. Prompts get longer over time and models get better at not needing them, and only a sweep will tell you that the two have crossed.