Budgeting the window, line by line

Write down where it all goes

People discover their context budget when something breaks. It is better to compute it in advance, because every item on the list is a decision somebody made and can unmake.

A realistic assistant with a 128,000-token window:

system prompt and policy               1,800
tool / function definitions            2,400
few-shot examples                      1,200
retrieved documents (6 × 700)          4,200
conversation history (18 turns)        9,600
current user message                     240
------------------------------------------
input                                 19,440
reserved for the response              2,000
------------------------------------------
total                                 21,440   of 128,000

Two things become visible immediately. First, this application is nowhere near its limit, so “we need a bigger window” would be the wrong diagnosis for any problem it has. Second, history is nearly half the input and growing every turn, which is where the eventual pressure will come from.

The reservation is not optional

Almost every provider counts input and output against the same window. If your input is 127,000 tokens of a 128,000-token window, the model has room for 1,000 tokens of reply and will be cut off mid-sentence.

So reserve explicitly. Decide the maximum response length, subtract it, and treat the remainder as your real input budget. Truncation that happens silently in a client library is a bug you will chase for a day.

Four ways to trim history, in order of preference

Drop nothing yet. For most applications, conversations end long before the window does. Complexity added early is complexity for no reason.

Sliding window. Keep the system prompt, the first user message, and the last n turns. Simple, predictable, and it loses whatever was decided in the middle, which for a long support conversation is often the important part.

Summarise the old turns. Replace turns 1 to 12 with a model-written summary and keep the recent ones verbatim. Effective, and it introduces a compounding failure: summaries of summaries lose specifics, and the loss is invisible because the text still reads fluently. Keep the original transcript stored so you can recover.

Retrieve from history. Store past turns and pull back only the relevant ones. Best quality, most machinery, and it now inherits every retrieval failure mode in this module.

Tool definitions are the quiet cost

The line most teams have never examined is the tool definitions. Every function’s name, description, and full parameter schema is serialised into the prompt on every request, whether or not any tool is relevant. Forty tools with thorough descriptions can reach 8,000 tokens, sent on a request that asks what time the office closes.

Two fixes, both straightforward. Trim the schemas: a parameter description of three sentences rarely outperforms one clear sentence, and optional parameters nobody uses can go entirely. Then, if the count is large, select tools per request: a cheap classifier or an embedding lookup over tool descriptions can narrow forty to six before the prompt is built. Both changes reduce cost and improve selection accuracy at the same time, which is unusual enough to be worth taking.

Cost is not proportional to window size

A common misreading: a 1-million-token window does not mean using it is sensible. Every token you send is paid for and processed. Filling a large window costs the same as a large prompt anywhere else, plus the latency of prefill, plus the accuracy cost of the position effects in the next lesson.

The largest context windows are best understood as a capability for occasional heavy use (a whole codebase, a long deposition, a book) and not as licence to stop thinking about what goes in the prompt on ordinary requests.

Count before you guess

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
parts = {"system": SYSTEM, "tools": TOOLS_JSON, "examples": FEWSHOT}
for k, v in parts.items():
    print(f"{k:10} {len(enc.encode(v)):6,}")

Run this against your own application. The commonest discovery is that tool definitions or a long-untouched system prompt account for far more than anyone assumed, and that they are sent on every single request whether relevant or not.

The habit worth forming

Treat the context window as a shared resource with named claimants, review it the way you would review a bill, and require a justification for each line. Instructions accumulate: someone adds three sentences to the system prompt to fix an edge case, nobody removes them, and a year later the prompt is 4,000 tokens of overlapping and occasionally contradictory guidance that is degrading behaviour rather than improving it.