The biggest thing in the window is a tool result

The input you never sized

Every other block in the window has a bound you chose. The system prompt is as long as you wrote it. Retrieved passages are capped by k and by chunk size. Conversation history is capped by your compaction rule. The user’s message is capped by your input field.

Tool results have no such bound, because the thing on the other end of the tool is the internet, or a database, or a filesystem, none of which agreed to your budget.

Concrete sizes, so the scale is not abstract:

• A typical news article page, fetched as raw HTML: 200 to 400 KB, which is roughly 50,000 to 100,000 tokens. Most of it is script tags and inline CSS.
• SELECT * FROM orders WHERE customer_id = ? for a wholesale account: 4,000 rows, easily 200,000 tokens as JSON.
• A directory listing of a real repository: 30,000 paths.
• A stack trace from a deep framework: 400 tokens, of which about 20 are informative.

Any one of these overflows a window that was comfortable a moment earlier. The failure is not graceful. Depending on where the limit is hit, you get an API rejection, a truncation your framework performed silently, or a request that succeeds and costs forty times what you expected.

Convert before you truncate

The order of operations matters more than people expect.

If you truncate raw HTML to the first 4,000 tokens, you have kept the <head>, the analytics scripts and the navigation, and thrown away the article. The model receives a page of metadata and reports that the document does not contain the answer.

Convert first, then truncate. HTML to text or markdown with a readability extractor: trafilatura and readability-lxml are free, offline and good, and will reduce that 400 KB page to about 6 KB of actual prose. PDFs to text. HTML tables to one self-contained line per row. Only once the content is the content do you apply a limit.

The same principle applies to database results: select the columns the task needs rather than *, and aggregate in SQL rather than returning rows for the model to count. A COUNT(*) returning one number is not a smaller version of 4,000 rows, it is the answer.

Truncate in code, at the boundary

The truncation belongs in the tool wrapper, not in the prompt and not in a hope. Every tool your system can call gets a return-size cap enforced before the result is handed to the assembler.

MAX_TOOL_TOKENS = 3000

def cap(text: str, tool: str) -> str:
    toks = enc.encode(text)
    if len(toks) <= MAX_TOOL_TOKENS:
        return text
    head = enc.decode(toks[:MAX_TOOL_TOKENS - 600])
    tail = enc.decode(toks[-500:])
    omitted = len(toks) - MAX_TOOL_TOKENS + 100
    return (f"{head}\n\n[... {omitted} tokens omitted from the middle of this "
            f"{tool} result. Total length {len(toks)} tokens. Narrow the query "
            f"or call read_more with offset= to see the rest ...]\n\n{tail}")

Three things in that snippet earn their place.

Head and tail, not head alone. Documents put conclusions at the end. Logs put the error at the end. Keeping only the head is the most common implementation and the worst one.

The marker states the size. “Omitted” alone tells the model something is missing. “48,200 tokens omitted, total 51,200” tells it the material is vast and that a narrower query is the right next move rather than paging through.

The marker is actionable. It names the call that retrieves more. Without that, a model that needs the missing part has no legal move and will either give up or fabricate. With it, the next step is obvious.

Cap the total, not just each call

A per-call cap of 3,000 tokens does not protect you from twenty calls. An agent loop that reads thirty files at 3,000 tokens each has added 90,000 tokens to a window that also has to hold its instructions.

So keep a running total for the session and enforce a second, cumulative budget. When it is reached, the honest behaviour is to compact (stub the older observations, which is covered in the agent-loop lesson) rather than to refuse. Refusing mid-task is worse than forgetting an early file listing.

The judgement call

Truncation loses information, and sometimes the lost part was the answer. There is no setting that avoids this; you are choosing which failure you prefer.

A useful rule: make the cap generous enough that it fires rarely, and instrument it so you know when it fires. Log every truncation with the tool name and the original size. After a week you will have a short list of tools that truncate constantly, and each of those is not a truncation problem but a tool-design problem. A tool that always returns more than the window can hold should be returning a summary, a count, or a paginated slice, and the fix belongs on that side of the boundary rather than in the clipping.