{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"what-fills-the-window","title":"Budgeting the window, line by line","url":"https://addaly.com/learn/how-llms-work/what-fills-the-window","position":31,"of":78,"minutes":8,"answer":"A context window is a budget shared by the system prompt, tool definitions, history, retrieved material and the response, and every one of those competes with the others.","keyIdea":"A context window is a budget shared by the system prompt, tool definitions, history, retrieved material and the response, and every one of those competes with the others."},"body":{"format":"markdown","text":"## Write down where it all goes\n\nPeople discover their context budget when something breaks. It is better to compute it in advance, because every item on the list is a decision somebody made and can unmake.\n\nA realistic assistant with a 128,000-token window:\n\n```text\nsystem prompt and policy               1,800\ntool / function definitions            2,400\nfew-shot examples                      1,200\nretrieved documents (6 × 700)          4,200\nconversation history (18 turns)        9,600\ncurrent user message                     240\n------------------------------------------\ninput                                 19,440\nreserved for the response              2,000\n------------------------------------------\ntotal                                 21,440   of 128,000\n```\n\nTwo things become visible immediately. First, this application is nowhere near its limit, so \"we need a bigger window\" would be the wrong diagnosis for any problem it has. Second, history is nearly half the input and growing every turn, which is where the eventual pressure will come from.\n\n> Figure: A 128,000-token window, and the 21,440 tokens actually claimed. Layers, top to bottom: System prompt and policy: 1,800 (Grows by three sentences each time somebody patches an edge case); Tool and function definitions: 2,400 (Serialised into every request, including the one asking when the office closes); Few-shot examples: 1,200 (Cheap to add, rarely revisited, occasionally contradicting the system prompt); Retrieved documents, six at 700: 4,200 (Six good passages beat forty mediocre ones, and not only on price); Conversation history, 18 turns: 9,600 (Nearly half the input already); The current user message: 240 (The only part a person actually wrote); Reserved for the response: 2,000 (Input and output share the window; unreserved, the reply is cut off mid-sentence). This application is nowhere near its limit, so a bigger window would fix nothing it has wrong. The pressure will come from history, which is the only line that grows every turn, and the line nobody has ever examined is the tool definitions.\n\n## The reservation is not optional\n\nAlmost every provider counts input and output against the same window. If your input is 127,000 tokens of a 128,000-token window, the model has room for 1,000 tokens of reply and will be cut off mid-sentence.\n\nSo reserve explicitly. Decide the maximum response length, subtract it, and treat the remainder as your real input budget. Truncation that happens silently in a client library is a bug you will chase for a day.\n\n## Four ways to trim history, in order of preference\n\n**Drop nothing yet.** For most applications, conversations end long before the window does. Complexity added early is complexity for no reason.\n\n**Sliding window.** Keep the system prompt, the first user message, and the last *n* turns. Simple, predictable, and it loses whatever was decided in the middle, which for a long support conversation is often the important part.\n\n**Summarise the old turns.** Replace turns 1 to 12 with a model-written summary and keep the recent ones verbatim. Effective, and it introduces a compounding failure: summaries of summaries lose specifics, and the loss is invisible because the text still reads fluently. Keep the original transcript stored so you can recover.\n\n**Retrieve from history.** Store past turns and pull back only the relevant ones. Best quality, most machinery, and it now inherits every retrieval failure mode in this module.\n\n## Tool definitions are the quiet cost\n\nThe line most teams have never examined is the tool definitions. Every function's name, description, and full parameter schema is serialised into the prompt on every request, whether or not any tool is relevant. Forty tools with thorough descriptions can reach 8,000 tokens, sent on a request that asks what time the office closes.\n\nTwo fixes, both straightforward. Trim the schemas: a parameter description of three sentences rarely outperforms one clear sentence, and optional parameters nobody uses can go entirely. Then, if the count is large, select tools per request: a cheap classifier or an embedding lookup over tool descriptions can narrow forty to six before the prompt is built. Both changes reduce cost and improve selection accuracy at the same time, which is unusual enough to be worth taking.\n\n## Cost is not proportional to window size\n\nA common misreading: a 1-million-token window does not mean using it is sensible. Every token you send is paid for and processed. Filling a large window costs the same as a large prompt anywhere else, plus the latency of prefill, plus the accuracy cost of the position effects in the next lesson.\n\nThe largest context windows are best understood as a capability for occasional heavy use (a whole codebase, a long deposition, a book) and not as licence to stop thinking about what goes in the prompt on ordinary requests.\n\n## Count before you guess\n\n```python\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nparts = {\"system\": SYSTEM, \"tools\": TOOLS_JSON, \"examples\": FEWSHOT}\nfor k, v in parts.items():\n    print(f\"{k:10} {len(enc.encode(v)):6,}\")\n```\n\nRun this against your own application. The commonest discovery is that tool definitions or a long-untouched system prompt account for far more than anyone assumed, and that they are sent on every single request whether relevant or not.\n\n## The habit worth forming\n\nTreat the context window as a shared resource with named claimants, review it the way you would review a bill, and require a justification for each line. Instructions accumulate: someone adds three sentences to the system prompt to fix an edge case, nobody removes them, and a year later the prompt is 4,000 tokens of overlapping and occasionally contradictory guidance that is degrading behaviour rather than improving it."},"check":{"question":"An assistant with a 128,000-token window starts returning replies that stop mid-sentence, and the input is around 126,000 tokens. What is happening?","options":[{"text":"Input and output share the same window, so with 126,000 tokens of input there is almost no room left for the response.","correct":true},{"text":"The provider truncates responses above a fixed length regardless of the window.","correct":false,"tempting":"Hard output caps do exist in some products, which makes a provider-side limit a reasonable first theory."},{"text":"The model has lost track of the instruction to complete its answers.","correct":false,"tempting":"Instruction-following really does degrade in very long prompts, so attributing it to attention failure fits a known effect."},{"text":"The response is being cut by a stop sequence appearing in the generated text.","correct":false,"tempting":"Stop sequences do cause abrupt endings, and this is exactly the symptom a misconfigured stop produces."}]},"prev":{"slug":"context-is-not-memory","title":"The context window is not memory","url":"https://addaly.com/learn/how-llms-work/context-is-not-memory"},"next":{"slug":"position-bias-and-effective-context","title":"Where in a long prompt things get missed","url":"https://addaly.com/learn/how-llms-work/position-bias-and-effective-context"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Budgeting the window, line by line', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/what-fills-the-window (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Budgeting the window, line by line","courseTitle":"How a Language Model Actually Works","module":{"index":4,"slug":"the-context-window","title":"What the model can see"},"answer":null,"keyIdea":"A context window is a budget shared by the system prompt, tool definitions, history, retrieved material and the response, and every one of those competes with the others.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"A 128,000-token window, and the 21,440 tokens actually claimed","layers":[{"label":"System prompt and policy: 1,800","detail":"Grows by three sentences each time somebody patches an edge case"},{"label":"Tool and function definitions: 2,400","detail":"Serialised into every request, including the one asking when the office closes"},{"label":"Few-shot examples: 1,200","detail":"Cheap to add, rarely revisited, occasionally contradicting the system prompt"},{"label":"Retrieved documents, six at 700: 4,200","detail":"Six good passages beat forty mediocre ones, and not only on price"},{"label":"Conversation history, 18 turns: 9,600","detail":"Nearly half the input already"},{"label":"The current user message: 240","detail":"The only part a person actually wrote"},{"label":"Reserved for the response: 2,000","detail":"Input and output share the window; unreserved, the reply is cut off mid-sentence"}],"caption":"This application is nowhere near its limit, so a bigger window would fix nothing it has wrong. The pressure will come from history, which is the only line that grows every turn, and the line nobody has ever examined is the tool definitions."},"alt":"Figure: A 128,000-token window, and the 21,440 tokens actually claimed. Layers, top to bottom: System prompt and policy: 1,800 (Grows by three sentences each time somebody patches an edge case); Tool and function definitions: 2,400 (Serialised into every request, including the one asking when the office closes); Few-shot examples: 1,200 (Cheap to add, rarely revisited, occasionally contradicting the system prompt); Retrieved documents, six at 700: 4,200 (Six good passages beat forty mediocre ones, and not only on price); Conversation history, 18 turns: 9,600 (Nearly half the input already); The current user message: 240 (The only part a person actually wrote); Reserved for the response: 2,000 (Input and output share the window; unreserved, the reply is cut off mid-sentence). This application is nowhere near its limit, so a bigger window would fix nothing it has wrong. The pressure will come from history, which is the only line that grows every turn, and the line nobody has ever examined is the tool definitions."}},"blocks":[{"t":"h2","id":"write-down-where-it-all-goes","runs":[{"text":"Write down where it all goes"}]},{"t":"p","runs":[{"text":"People discover their context budget when something breaks. It is better to compute it in advance, because every item on the list is a decision somebody made and can unmake."}]},{"t":"p","runs":[{"text":"A realistic assistant with a 128,000-token window:"}]},{"t":"code","lang":"text","label":"Text","text":"system prompt and policy               1,800\ntool / function definitions            2,400\nfew-shot examples                      1,200\nretrieved documents (6 × 700)          4,200\nconversation history (18 turns)        9,600\ncurrent user message                     240\n------------------------------------------\ninput                                 19,440\nreserved for the response              2,000\n------------------------------------------\ntotal                                 21,440   of 128,000"},{"t":"p","runs":[{"text":"Two things become visible immediately. First, this application is nowhere near its limit, so “we need a bigger window” would be the wrong diagnosis for any problem it has. Second, history is nearly half the input and growing every turn, which is where the eventual pressure will come from."}]},{"t":"h2","id":"the-reservation-is-not-optional","runs":[{"text":"The reservation is not optional"}]},{"t":"p","runs":[{"text":"Almost every provider counts input and output against the same window. If your input is 127,000 tokens of a 128,000-token window, the model has room for 1,000 tokens of reply and will be cut off mid-sentence."}]},{"t":"p","runs":[{"text":"So reserve explicitly. Decide the maximum response length, subtract it, and treat the remainder as your real input budget. Truncation that happens silently in a client library is a bug you will chase for a day."}]},{"t":"h2","id":"four-ways-to-trim-history-in-order-of-preference","runs":[{"text":"Four ways to trim history, in order of preference"}]},{"t":"p","runs":[{"text":"Drop nothing yet.","b":true},{"text":" For most applications, conversations end long before the window does. Complexity added early is complexity for no reason."}]},{"t":"p","runs":[{"text":"Sliding window.","b":true},{"text":" Keep the system prompt, the first user message, and the last "},{"text":"n","i":true},{"text":" turns. Simple, predictable, and it loses whatever was decided in the middle, which for a long support conversation is often the important part."}]},{"t":"p","runs":[{"text":"Summarise the old turns.","b":true},{"text":" Replace turns 1 to 12 with a model-written summary and keep the recent ones verbatim. Effective, and it introduces a compounding failure: summaries of summaries lose specifics, and the loss is invisible because the text still reads fluently. Keep the original transcript stored so you can recover."}]},{"t":"p","runs":[{"text":"Retrieve from history.","b":true},{"text":" Store past turns and pull back only the relevant ones. Best quality, most machinery, and it now inherits every retrieval failure mode in this module."}]},{"t":"h2","id":"tool-definitions-are-the-quiet-cost","runs":[{"text":"Tool definitions are the quiet cost"}]},{"t":"p","runs":[{"text":"The line most teams have never examined is the tool definitions. Every function’s name, description, and full parameter schema is serialised into the prompt on every request, whether or not any tool is relevant. Forty tools with thorough descriptions can reach 8,000 tokens, sent on a request that asks what time the office closes."}]},{"t":"p","runs":[{"text":"Two fixes, both straightforward. Trim the schemas: a parameter description of three sentences rarely outperforms one clear sentence, and optional parameters nobody uses can go entirely. Then, if the count is large, select tools per request: a cheap classifier or an embedding lookup over tool descriptions can narrow forty to six before the prompt is built. Both changes reduce cost and improve selection accuracy at the same time, which is unusual enough to be worth taking."}]},{"t":"h2","id":"cost-is-not-proportional-to-window-size","runs":[{"text":"Cost is not proportional to window size"}]},{"t":"p","runs":[{"text":"A common misreading: a 1-million-token window does not mean using it is sensible. Every token you send is paid for and processed. Filling a large window costs the same as a large prompt anywhere else, plus the latency of prefill, plus the accuracy cost of the position effects in the next lesson."}]},{"t":"p","runs":[{"text":"The largest context windows are best understood as a capability for occasional heavy use (a whole codebase, a long deposition, a book) and not as licence to stop thinking about what goes in the prompt on ordinary requests."}]},{"t":"h2","id":"count-before-you-guess","runs":[{"text":"Count before you guess"}]},{"t":"code","lang":"python","label":"Python","text":"import tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nparts = {\"system\": SYSTEM, \"tools\": TOOLS_JSON, \"examples\": FEWSHOT}\nfor k, v in parts.items():\n    print(f\"{k:10} {len(enc.encode(v)):6,}\")"},{"t":"p","runs":[{"text":"Run this against your own application. The commonest discovery is that tool definitions or a long-untouched system prompt account for far more than anyone assumed, and that they are sent on every single request whether relevant or not."}]},{"t":"h2","id":"the-habit-worth-forming","runs":[{"text":"The habit worth forming"}]},{"t":"p","runs":[{"text":"Treat the context window as a shared resource with named claimants, review it the way you would review a bill, and require a justification for each line. Instructions accumulate: someone adds three sentences to the system prompt to fix an edge case, nobody removes them, and a year later the prompt is 4,000 tokens of overlapping and occasionally contradictory guidance that is degrading behaviour rather than improving it."}]}],"check":{"question":[{"text":"An assistant with a 128,000-token window starts returning replies that stop mid-sentence, and the input is around 126,000 tokens. What is happening?"}],"options":[{"text":[{"text":"Input and output share the same window, so with 126,000 tokens of input there is almost no room left for the response."}],"correct":true,"tempting":null},{"text":[{"text":"The provider truncates responses above a fixed length regardless of the window."}],"correct":false,"tempting":[{"text":"Hard output caps do exist in some products, which makes a provider-side limit a reasonable first theory."}]},{"text":[{"text":"The model has lost track of the instruction to complete its answers."}],"correct":false,"tempting":[{"text":"Instruction-following really does degrade in very long prompts, so attributing it to attention failure fits a known effect."}]},{"text":[{"text":"The response is being cut by a stop sequence appearing in the generated text."}],"correct":false,"tempting":[{"text":"Stop sequences do cause abrupt endings, and this is exactly the symptom a misconfigured stop produces."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"context-is-not-memory","title":"The context window is not memory","minutes":7},"next":{"slug":"position-bias-and-effective-context","title":"Where in a long prompt things get missed","minutes":9}}}