Writing the budget down

An allocation, not an aspiration

A context budget is a set of numbers that add up to less than the window, written in code, asserted at run time. Not a paragraph in a design document. The difference matters because the failure it prevents is silent: a request that fits on Tuesday and does not on Friday, when a customer pastes a longer email.

Here is the shape. Assume a 128,000-token window and a task that needs an 800-token answer.

WINDOW = 128_000
BUDGET = {
    "output_reserve": 800,     # subtracted first, always
    "safety_margin": 2_000,    # tokeniser drift, template, surprises
    "system":         900,     # measured, fixed
    "tools":        1_200,     # measured, fixed
    "exemplars":      600,     # two examples
    "history":      6_000,     # cap; compaction runs above this
    "retrieved":   12_000,     # cap; the packer fills up to it
    "user_turn":    2_000,     # cap; longer input is summarised, not dropped
}
assert sum(BUDGET.values()) <= WINDOW

Two things are already visible. The total is 25,500 against a 128,000-token window, which surprises people: most well-built systems use a small fraction of the space available, because more material is not free and usually not better. And every line is a cap that some component must respect, which means every component needs a truncation or compaction strategy of its own.

Reserve the output first

The output reserve is the first subtraction, not the last. The model cannot produce an answer it has no room for, and the failure when it runs out is not an error: it is a truncated response, mid-sentence, that looks to a user like a bug in your product.

Set it from measurement, not intuition. Take a hundred real responses, look at the distribution, and reserve at the 95th percentile rather than the mean. A summariser whose answers average 180 tokens still has a tail at 700 when it meets a long input, and the tail is the case where the truncation is most damaging.

A margin for things you cannot count

The safety margin covers what your arithmetic does not: chat template overhead, a tokeniser you cannot run locally, a vendor changing the serialisation of tool schemas in a point release, the fact that your estimate for the user’s turn was based on English.

Two percent of the window, or 2,000 tokens, whichever is larger, is a reasonable starting point. If you never hit it, lower it after a month of evidence. If you hit it weekly, something upstream is unbounded and the margin is masking it.

The cut order

When the assembled request exceeds the budget (and it will) something must go. Decide the order once, in code, rather than in an incident.

A defensible default, from first to cut to last:

1. Retrieved passages, lowest-ranked first. You already have them in a ranked list; dropping the tail is the least damaging cut available.
2. History, oldest first, or better, compacted rather than dropped.
3. Exemplars, from four to two. Measure the loss before making this permanent.
4. The user’s own input, summarised rather than truncated, and only with a visible note that you did so.
5. Tool definitions, by routing to a smaller tool set for this request.
6. Instructions. Last. Never automatically.

Most naive implementations do the opposite by accident: they build one long string and truncate the front, which removes the instructions and keeps the least relevant retrieved passage. If your code contains text[-N:] anywhere near a prompt, that is what it is doing.

Make the budget observable

The budget is only useful if you can see it being spent. Emit a structured line on every call:

log.info("context", extra={
    "tokens": {k: counted[k] for k in BUDGET},
    "total": total, "window": WINDOW,
    "cut": cuts,            # what was dropped, and how much
    "assembly_version": AV, # so you can diff later
})

With that line you can answer questions that are otherwise unanswerable after the fact: which component grew, on which day, for which customers. Without it, a cost increase is a mystery with six suspects.

The metric worth alerting on is not the average. It is the 99th percentile of total tokens, and the rate at which cuts occur. A system that trims retrieved passages on 0.5% of requests is healthy. One that trims on 30% is silently running a different product than the one you tested.

Why smaller budgets often win

The instinct with a large window is to fill it. Three forces push the other way.

Cost is linear in input tokens, and input is most of a typical request. Latency to first token scales with the amount of prefill, so a 40,000-token context is noticeably slower to start than a 6,000-token one on the same model. And quality does not improve monotonically with material: adding weakly relevant passages moves probability mass towards them, and a passage that is 60% on-topic can be worse than no passage at all because it looks like an answer.

The budget is the place where those three forces get resolved deliberately instead of by whatever the retriever happened to return.