What actually happens when it does not fit

Three different failures wearing one name

“The context overflowed” describes at least three distinct events, with different symptoms and different fixes. Confusing them wastes days.

The API rejects the request. You get a 400 with a message about exceeding the maximum. This is the friendly failure: loud, immediate, attributable. Handle it, but be glad when it happens.

Your own code truncates. Somewhere in the pipeline a slice keeps part of the material. This is the dangerous one, because the request succeeds, the response looks fine, and the only evidence is a passage that was silently not there. Most systems have at least one of these and do not know where.

A component upstream truncated. The document parser stopped at 50 pages. The tool returned a paginated result and nobody read past page one. The database query had a LIMIT 100 written in 2023. The window was never the constraint; something else quietly decided what you would see.

The third is the hardest to find because nothing in the model call is wrong.

Which end gets cut, and why it matters

When truncation is unavoidable, the end you cut decides which failure you get.

Cutting the front removes the instructions and the output contract. The model, given retrieved material and a question with no framing, produces something plausible in the wrong format. This is what naive text[-N:] does, and it is the worst available choice.

Cutting the back removes the question. Rarer, and it produces a summary of the material instead of an answer: a distinctive symptom worth recognising.

Cutting the middle (dropping the oldest history or the lowest-ranked passages) is what you want, and it only happens if you write it.

Sliding windows drop the wrong thing

The standard chat implementation keeps the last N messages. Two bugs follow, and both are common enough to check for by name.

First, the system message is a message. A naive messages[-20:] removes it once the conversation passes twenty turns, and the assistant’s behaviour changes character mid-conversation for no reason a user can see. Pin the system message outside the window.

Second, tool calls come in pairs. A tool-call message and its result must both be present or the API errors, or worse, the model sees a result with no request and hallucinates what was asked. Slice on turn boundaries, not on message count.

def window(messages, keep):
    head = [m for m in messages if m.role == "system"]
    tail = messages[-keep:]
    while tail and tail[0].role == "tool":   # never start on an orphan result
        tail = tail[1:]
    return head + tail

The tool result that ate the instructions

The most instructive production overflow looks like this. An agent calls a search tool. The tool returns 60,000 tokens of JSON because somebody’s query matched a large table. The framework appends it to the message list. The next call exceeds the window, so the framework’s automatic trimming removes the oldest messages, which include the system prompt and the user’s original request.

The agent then continues, working from a wall of JSON with no idea what it was asked, and produces something confident and unrelated. Nobody sees an error.

Three defences, in order of how much they help:

1. Cap tool output at the tool boundary, before it ever reaches the message list. A tool that can return unbounded data should return the first page plus a count and a handle to fetch more.
2. Assert on the assembled size in the assembler, and fail loudly rather than trimming silently.
3. Never automatically trim instructions. If the only way to fit is to remove the system prompt, the correct behaviour is to error and let a human see it.

Fail loudly, then degrade deliberately

A reasonable policy, in the order the assembler should apply it:

total = count(request)
if total <= budget:
    return request
request = drop_lowest_ranked_passages(request, budget)   # 1
if count(request) > budget:
    request = compact_history(request, budget)            # 2
if count(request) > budget:
    request = summarise_user_input(request, budget)       # 3, and say so
if count(request) > budget:
    raise ContextOverflow(breakdown=count_by_block(request))

The exception at the end is the important line. It carries a per-block breakdown, so the alert tells you which component grew rather than that something did. A system that never raises this exception is a system where you do not know whether the degradation path is being used.

Measure the near misses

Overflow is a threshold, so the metric that matters is not the failure count but the distance to it. Emit total_tokens / budget on every call and watch the 99th percentile.

A fleet running at a median of 30% and a 99th percentile of 96% is one unusual customer away from an incident, and the median tells you nothing about it. This is the same reasoning as tail latency: the average request is not the one that pages you at night.