{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"what-breaks-when-it-overflows","title":"What actually happens when it does not fit","url":"https://addaly.com/learn/context-engineering/what-breaks-when-it-overflows","position":10,"of":87,"minutes":8,"answer":"Overflow arrives as a rejected request, a silent truncation in your own code, or an upstream cut you never see, so cap unbounded sources at their boundary, never trim instructions automatically.","keyIdea":"Overflow arrives as a rejected request, a silent truncation in your own code, or an upstream cut you never see, so cap unbounded sources at their boundary, never trim instructions automatically, and alert on the 99th percentile of budget usage rather than on failures."},"body":{"format":"markdown","text":"## Three different failures wearing one name\n\n\"The context overflowed\" describes at least three distinct events, with different symptoms and different fixes. Confusing them wastes days.\n\n**The API rejects the request.** You get a 400 with a message about exceeding the maximum. This is the friendly failure: loud, immediate, attributable. Handle it, but be glad when it happens.\n\n**Your own code truncates.** Somewhere in the pipeline a slice keeps part of the material. This is the dangerous one, because the request succeeds, the response looks fine, and the only evidence is a passage that was silently not there. Most systems have at least one of these and do not know where.\n\n**A component upstream truncated.** The document parser stopped at 50 pages. The tool returned a paginated result and nobody read past page one. The database query had a `LIMIT 100` written in 2023. The window was never the constraint; something else quietly decided what you would see.\n\nThe third is the hardest to find because nothing in the model call is wrong.\n\n## Which end gets cut, and why it matters\n\nWhen truncation is unavoidable, the end you cut decides which failure you get.\n\nCutting the **front** removes the instructions and the output contract. The model, given retrieved material and a question with no framing, produces something plausible in the wrong format. This is what naive `text[-N:]` does, and it is the worst available choice.\n\nCutting the **back** removes the question. Rarer, and it produces a summary of the material instead of an answer: a distinctive symptom worth recognising.\n\nCutting the **middle** (dropping the oldest history or the lowest-ranked passages) is what you want, and it only happens if you write it.\n\n## Sliding windows drop the wrong thing\n\nThe standard chat implementation keeps the last N messages. Two bugs follow, and both are common enough to check for by name.\n\nFirst, the system message is a message. A naive `messages[-20:]` removes it once the conversation passes twenty turns, and the assistant's behaviour changes character mid-conversation for no reason a user can see. Pin the system message outside the window.\n\nSecond, tool calls come in pairs. A tool-call message and its result must both be present or the API errors, or worse, the model sees a result with no request and hallucinates what was asked. Slice on turn boundaries, not on message count.\n\n```python\ndef window(messages, keep):\n    head = [m for m in messages if m.role == \"system\"]\n    tail = messages[-keep:]\n    while tail and tail[0].role == \"tool\":   # never start on an orphan result\n        tail = tail[1:]\n    return head + tail\n```\n\n## The tool result that ate the instructions\n\nThe most instructive production overflow looks like this. An agent calls a search tool. The tool returns 60,000 tokens of JSON because somebody's query matched a large table. The framework appends it to the message list. The next call exceeds the window, so the framework's automatic trimming removes the oldest messages, which include the system prompt and the user's original request.\n\nThe agent then continues, working from a wall of JSON with no idea what it was asked, and produces something confident and unrelated. Nobody sees an error.\n\nThree defences, in order of how much they help:\n\n1. **Cap tool output at the tool boundary**, before it ever reaches the message list. A tool that can return unbounded data should return the first page plus a count and a handle to fetch more.\n2. **Assert on the assembled size** in the assembler, and fail loudly rather than trimming silently.\n3. **Never automatically trim instructions.** If the only way to fit is to remove the system prompt, the correct behaviour is to error and let a human see it.\n\n## Fail loudly, then degrade deliberately\n\nA reasonable policy, in the order the assembler should apply it:\n\n```python\ntotal = count(request)\nif total <= budget:\n    return request\nrequest = drop_lowest_ranked_passages(request, budget)   # 1\nif count(request) > budget:\n    request = compact_history(request, budget)            # 2\nif count(request) > budget:\n    request = summarise_user_input(request, budget)       # 3, and say so\nif count(request) > budget:\n    raise ContextOverflow(breakdown=count_by_block(request))\n```\n\nThe exception at the end is the important line. It carries a per-block breakdown, so the alert tells you which component grew rather than that something did. A system that never raises this exception is a system where you do not know whether the degradation path is being used.\n\n## Measure the near misses\n\nOverflow is a threshold, so the metric that matters is not the failure count but the distance to it. Emit `total_tokens / budget` on every call and watch the 99th percentile.\n\nA fleet running at a median of 30% and a 99th percentile of 96% is one unusual customer away from an incident, and the median tells you nothing about it. This is the same reasoning as tail latency: the average request is not the one that pages you at night."},"check":{"question":"An agent framework keeps the last 20 messages. After a tool returns a very large result, the agent starts answering a question nobody asked. What happened?","options":[{"text":"The tool result was itself truncated, so the agent worked from a partial observation.","correct":false,"tempting":"Plausible and would degrade quality, but a partial observation still leaves the original request in the window; it does not erase the task."},{"text":"The large result pushed the system prompt and the original request out of the 20-message window, leaving the agent with data and no task.","correct":true},{"text":"The model's effective context was exceeded, so it stopped attending to the earlier messages even though they were present.","correct":false,"tempting":"Reaches for gradual degradation to explain a total loss; effective-context effects weaken use of distant material rather than removing the task entirely."},{"text":"The tool result arrived with a role the model does not weight, so it was ignored and the model improvised.","correct":false,"tempting":"Inverts the symptom: the agent is clearly using the tool data, and the missing piece is the instruction rather than the observation."}]},"prev":{"slug":"context-assembly-as-a-function","title":"Making the assembler testable","url":"https://addaly.com/learn/context-engineering/context-assembly-as-a-function"},"next":{"slug":"retrieval-versus-long-context","title":"Retrieval versus long context: the real trade","url":"https://addaly.com/learn/context-engineering/retrieval-versus-long-context"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What actually happens when it does not fit', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/what-breaks-when-it-overflows (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What actually happens when it does not fit","courseTitle":"Context Engineering","module":{"index":1,"slug":"the-window-as-a-surface","title":"The window as an engineering surface"},"answer":null,"keyIdea":"Overflow arrives as a rejected request, a silent truncation in your own code, or an upstream cut you never see, so cap unbounded sources at their boundary, never trim instructions automatically, and alert on the 99th percentile of budget usage rather than on failures.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-different-failures-wearing-one-name","runs":[{"text":"Three different failures wearing one name"}]},{"t":"p","runs":[{"text":"“The context overflowed” describes at least three distinct events, with different symptoms and different fixes. Confusing them wastes days."}]},{"t":"p","runs":[{"text":"The API rejects the request.","b":true},{"text":" You get a 400 with a message about exceeding the maximum. This is the friendly failure: loud, immediate, attributable. Handle it, but be glad when it happens."}]},{"t":"p","runs":[{"text":"Your own code truncates.","b":true},{"text":" Somewhere in the pipeline a slice keeps part of the material. This is the dangerous one, because the request succeeds, the response looks fine, and the only evidence is a passage that was silently not there. Most systems have at least one of these and do not know where."}]},{"t":"p","runs":[{"text":"A component upstream truncated.","b":true},{"text":" The document parser stopped at 50 pages. The tool returned a paginated result and nobody read past page one. The database query had a "},{"text":"LIMIT 100","code":true},{"text":" written in 2023. The window was never the constraint; something else quietly decided what you would see."}]},{"t":"p","runs":[{"text":"The third is the hardest to find because nothing in the model call is wrong."}]},{"t":"h2","id":"which-end-gets-cut-and-why-it-matters","runs":[{"text":"Which end gets cut, and why it matters"}]},{"t":"p","runs":[{"text":"When truncation is unavoidable, the end you cut decides which failure you get."}]},{"t":"p","runs":[{"text":"Cutting the "},{"text":"front","b":true},{"text":" removes the instructions and the output contract. The model, given retrieved material and a question with no framing, produces something plausible in the wrong format. This is what naive "},{"text":"text[-N:]","code":true},{"text":" does, and it is the worst available choice."}]},{"t":"p","runs":[{"text":"Cutting the "},{"text":"back","b":true},{"text":" removes the question. Rarer, and it produces a summary of the material instead of an answer: a distinctive symptom worth recognising."}]},{"t":"p","runs":[{"text":"Cutting the "},{"text":"middle","b":true},{"text":" (dropping the oldest history or the lowest-ranked passages) is what you want, and it only happens if you write it."}]},{"t":"h2","id":"sliding-windows-drop-the-wrong-thing","runs":[{"text":"Sliding windows drop the wrong thing"}]},{"t":"p","runs":[{"text":"The standard chat implementation keeps the last N messages. Two bugs follow, and both are common enough to check for by name."}]},{"t":"p","runs":[{"text":"First, the system message is a message. A naive "},{"text":"messages[-20:]","code":true},{"text":" removes it once the conversation passes twenty turns, and the assistant’s behaviour changes character mid-conversation for no reason a user can see. Pin the system message outside the window."}]},{"t":"p","runs":[{"text":"Second, tool calls come in pairs. A tool-call message and its result must both be present or the API errors, or worse, the model sees a result with no request and hallucinates what was asked. Slice on turn boundaries, not on message count."}]},{"t":"code","lang":"python","label":"Python","text":"def window(messages, keep):\n    head = [m for m in messages if m.role == \"system\"]\n    tail = messages[-keep:]\n    while tail and tail[0].role == \"tool\":   # never start on an orphan result\n        tail = tail[1:]\n    return head + tail"},{"t":"h2","id":"the-tool-result-that-ate-the-instructions","runs":[{"text":"The tool result that ate the instructions"}]},{"t":"p","runs":[{"text":"The most instructive production overflow looks like this. An agent calls a search tool. The tool returns 60,000 tokens of JSON because somebody’s query matched a large table. The framework appends it to the message list. The next call exceeds the window, so the framework’s automatic trimming removes the oldest messages, which include the system prompt and the user’s original request."}]},{"t":"p","runs":[{"text":"The agent then continues, working from a wall of JSON with no idea what it was asked, and produces something confident and unrelated. Nobody sees an error."}]},{"t":"p","runs":[{"text":"Three defences, in order of how much they help:"}]},{"t":"ol","start":1,"items":[[{"text":"Cap tool output at the tool boundary","b":true},{"text":", before it ever reaches the message list. A tool that can return unbounded data should return the first page plus a count and a handle to fetch more."}],[{"text":"Assert on the assembled size","b":true},{"text":" in the assembler, and fail loudly rather than trimming silently."}],[{"text":"Never automatically trim instructions.","b":true},{"text":" If the only way to fit is to remove the system prompt, the correct behaviour is to error and let a human see it."}]]},{"t":"h2","id":"fail-loudly-then-degrade-deliberately","runs":[{"text":"Fail loudly, then degrade deliberately"}]},{"t":"p","runs":[{"text":"A reasonable policy, in the order the assembler should apply it:"}]},{"t":"code","lang":"python","label":"Python","text":"total = count(request)\nif total <= budget:\n    return request\nrequest = drop_lowest_ranked_passages(request, budget)   # 1\nif count(request) > budget:\n    request = compact_history(request, budget)            # 2\nif count(request) > budget:\n    request = summarise_user_input(request, budget)       # 3, and say so\nif count(request) > budget:\n    raise ContextOverflow(breakdown=count_by_block(request))"},{"t":"p","runs":[{"text":"The exception at the end is the important line. It carries a per-block breakdown, so the alert tells you which component grew rather than that something did. A system that never raises this exception is a system where you do not know whether the degradation path is being used."}]},{"t":"h2","id":"measure-the-near-misses","runs":[{"text":"Measure the near misses"}]},{"t":"p","runs":[{"text":"Overflow is a threshold, so the metric that matters is not the failure count but the distance to it. Emit "},{"text":"total_tokens / budget","code":true},{"text":" on every call and watch the 99th percentile."}]},{"t":"p","runs":[{"text":"A fleet running at a median of 30% and a 99th percentile of 96% is one unusual customer away from an incident, and the median tells you nothing about it. This is the same reasoning as tail latency: the average request is not the one that pages you at night."}]}],"check":{"question":[{"text":"An agent framework keeps the last 20 messages. After a tool returns a very large result, the agent starts answering a question nobody asked. What happened?"}],"options":[{"text":[{"text":"The tool result was itself truncated, so the agent worked from a partial observation."}],"correct":false,"tempting":[{"text":"Plausible and would degrade quality, but a partial observation still leaves the original request in the window; it does not erase the task."}]},{"text":[{"text":"The large result pushed the system prompt and the original request out of the 20-message window, leaving the agent with data and no task."}],"correct":true,"tempting":null},{"text":[{"text":"The model’s effective context was exceeded, so it stopped attending to the earlier messages even though they were present."}],"correct":false,"tempting":[{"text":"Reaches for gradual degradation to explain a total loss; effective-context effects weaken use of distant material rather than removing the task entirely."}]},{"text":[{"text":"The tool result arrived with a role the model does not weight, so it was ignored and the model improvised."}],"correct":false,"tempting":[{"text":"Inverts the symptom: the agent is clearly using the tool data, and the missing piece is the instruction rather than the observation."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"context-assembly-as-a-function","title":"Making the assembler testable","minutes":9},"next":{"slug":"retrieval-versus-long-context","title":"Retrieval versus long context: the real trade","minutes":9}}}