{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"the-window-is-a-budget","title":"The window is a budget, not a container","url":"https://addaly.com/learn/context-engineering/the-window-is-a-budget","position":1,"of":87,"minutes":8,"answer":"Every token costs money, latency and attention, so treat the window as spend, not storage.","keyIdea":"Every token costs money, latency and attention, so treat the window as spend, not storage."},"body":{"format":"markdown","text":"## Fitting is not working\n\nYou have a 200,000-token window. Your document is 180,000 tokens. It fits.\n\nThat is not the same as it working. Three things get worse as you add tokens, and they get worse at different rates.\n\n**Money.** Input tokens are billed on every call, forever. At $3 per million input tokens, a 200k-token prompt costs $0.60 to send. Ten thousand calls a day is $6,000 a day, $180,000 a month. The same task trimmed to 6k tokens costs $0.018 a call: about $180 a day. Same model, often the same answers, roughly 33x the bill. For a team in Lagos or Lahore paying in dollars from local revenue, that difference is the whole business case.\n\n**Latency.** Before the model emits a single token, it reads yours. Prefill scales with input length. A 200k-token prompt typically means several seconds before the first character appears; an 8k prompt means a few hundred milliseconds. Users forgive a slow answer. They do not forgive a slow blank screen.\n\n**Attention.** This is the one people miss. A model does not read your context the way a database reads a row. It attends across all of it, and every irrelevant passage you add is a passage that can win the competition for relevance. Add the whole product manual and you have not just added the paragraph about refunds: you have added forty paragraphs that *look* like they are about refunds.\n\n## What the benchmarks hide\n\nNeedle-in-a-haystack tests (hide one odd sentence in 500k tokens, ask the model to find it) are close to saturated on frontier models. Teams read that and conclude long context is solved.\n\nIt is not the same task. The needle is semantically alien to its surroundings, and there is exactly one. Real work asks the model to combine four facts scattered across a long document, where three near-duplicates of each fact also appear and two of them are outdated. Accuracy on that shape of task degrades with length on every model measured, and it degrades well before the window fills.\n\nThe honest summary: long context is real and getting better, and it still costs you accuracy, not only money.\n\n## Spend it, do not fill it\n\nTreat the window like a monthly budget in a currency you actually earn. Give every component a line item, in tokens, and enforce it in code rather than hoping.\n\n> Figure: A 12,500-token ledger for one request. Bars (tokens): System prompt 1200; Tool schemas 1500; Few-shot examples 1000; Retrieved passages 6000; Conversation history 2000; The user's turn 800. The same allocation the code below enforces. Retrieved passages are nearly half of it, which is why the selection module comes before everything else. The point of writing it down is not the trimming: it is the warning that fires the day somebody uploads a PDF of scanned invoices and retrieval quietly starts returning forty thousand tokens.\n\n```python\nBUDGET = {\n    \"system\":       1_200,   # role, constraints, output contract\n    \"tools\":        1_500,   # tool schemas\n    \"examples\":     1_000,   # few-shot, static\n    \"retrieved\":    6_000,   # ~8-12 chunks after reranking\n    \"history\":      2_000,   # trimmed, oldest dropped first\n    \"user_turn\":      800,\n}\n# total 12,500 in, leaving room for output\n\ndef assemble(parts, budget):\n    out = {}\n    for name, limit in budget.items():\n        text = parts.get(name, \"\")\n        n = count_tokens(text)\n        if n > limit:\n            log.warning(\"over budget: %s used %d of %d\", name, n, limit)\n            text = trim(text, limit)          # per-part policy, not a blind cut\n        out[name] = text\n    return out\n```\n\nThe logging line matters more than the trimming. When retrieval quietly starts returning 40k tokens because someone uploaded a PDF of scanned invoices, you want a warning, not a surprise invoice.\n\n## The question to ask of every token\n\nFor each block you are about to include: *if I delete this, which specific answer gets worse?*\n\nIf you cannot name the answer, delete the block and measure. Most context bloat is defensive: someone added the full policy document because a single question about refunds went wrong once, and nobody ever took it out. Six months later that block is in every call, on every request, in the bill every month.\n\nThe cheapest optimisation available to you is usually deletion.\n\n## What this course does\n\nEverything that follows is a way of spending that budget better: where a token should sit so the model uses it, which tokens can be cached so they cost a tenth, when to fetch tokens instead of carrying them, how to cut documents without cutting meaning, and how to prove a change helped rather than believing it did."},"check":{"question":"A team replaces a retrieval step (5 chunks, about 3,000 tokens) with pasting their entire 90,000-token product manual into every call. The manual fits comfortably in the model's window. Accuracy on specific policy questions gets noticeably worse. What is the most likely explanation?","options":[{"text":"The manual needs converting to clean Markdown; the model is being confused by PDF formatting artefacts.","correct":false,"tempting":"Formatting does affect parsing, so it feels like the obvious first fix, but it would not explain a drop relative to chunks drawn from the same document."},{"text":"The 90,000 tokens contain many passages that resemble the answer, and the correct one now competes with dozens of plausible distractors.","correct":true},{"text":"The prompt exceeded the usable context and the provider silently truncated the middle of the manual.","correct":false,"tempting":"Silent truncation does happen with some client libraries, but the stated premise is that it fits, and providers error rather than trim on overflow."},{"text":"The model's knowledge cutoff predates the manual, so it falls back on outdated training data instead of reading the text.","correct":false,"tempting":"Cutoffs genuinely cause stale answers, but text placed in the context is read regardless of when it was written."}]},"prev":null,"next":{"slug":"what-context-engineering-is","title":"Nobody typed this","url":"https://addaly.com/learn/context-engineering/what-context-engineering-is"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The window is a budget, not a container', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/the-window-is-a-budget (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The window is a budget, not a container","courseTitle":"Context Engineering","module":{"index":1,"slug":"the-window-as-a-surface","title":"The window as an engineering surface"},"answer":null,"keyIdea":"Every token costs money, latency and attention, so treat the window as spend, not storage.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 1.1","figure":{"type":"bars","unit":"tokens","title":"A 12,500-token ledger for one request","series":[{"label":"System prompt","value":1200},{"label":"Tool schemas","value":1500},{"label":"Few-shot examples","value":1000},{"label":"Retrieved passages","value":6000,"highlight":true},{"label":"Conversation history","value":2000},{"label":"The user’s turn","value":800}],"caption":"The same allocation the code below enforces. Retrieved passages are nearly half of it, which is why the selection module comes before everything else. The point of writing it down is not the trimming: it is the warning that fires the day somebody uploads a PDF of scanned invoices and retrieval quietly starts returning forty thousand tokens."},"alt":"Figure: A 12,500-token ledger for one request. Bars (tokens): System prompt 1200; Tool schemas 1500; Few-shot examples 1000; Retrieved passages 6000; Conversation history 2000; The user’s turn 800. The same allocation the code below enforces. Retrieved passages are nearly half of it, which is why the selection module comes before everything else. The point of writing it down is not the trimming: it is the warning that fires the day somebody uploads a PDF of scanned invoices and retrieval quietly starts returning forty thousand tokens."}},"blocks":[{"t":"h2","id":"fitting-is-not-working","runs":[{"text":"Fitting is not working"}]},{"t":"p","runs":[{"text":"You have a 200,000-token window. Your document is 180,000 tokens. It fits."}]},{"t":"p","runs":[{"text":"That is not the same as it working. Three things get worse as you add tokens, and they get worse at different rates."}]},{"t":"p","runs":[{"text":"Money.","b":true},{"text":" Input tokens are billed on every call, forever. At $3 per million input tokens, a 200k-token prompt costs $0.60 to send. Ten thousand calls a day is $6,000 a day, $180,000 a month. The same task trimmed to 6k tokens costs $0.018 a call: about $180 a day. Same model, often the same answers, roughly 33x the bill. For a team in Lagos or Lahore paying in dollars from local revenue, that difference is the whole business case."}]},{"t":"p","runs":[{"text":"Latency.","b":true},{"text":" Before the model emits a single token, it reads yours. Prefill scales with input length. A 200k-token prompt typically means several seconds before the first character appears; an 8k prompt means a few hundred milliseconds. Users forgive a slow answer. They do not forgive a slow blank screen."}]},{"t":"p","runs":[{"text":"Attention.","b":true},{"text":" This is the one people miss. A model does not read your context the way a database reads a row. It attends across all of it, and every irrelevant passage you add is a passage that can win the competition for relevance. Add the whole product manual and you have not just added the paragraph about refunds: you have added forty paragraphs that "},{"text":"look","i":true},{"text":" like they are about refunds."}]},{"t":"h2","id":"what-the-benchmarks-hide","runs":[{"text":"What the benchmarks hide"}]},{"t":"p","runs":[{"text":"Needle-in-a-haystack tests (hide one odd sentence in 500k tokens, ask the model to find it) are close to saturated on frontier models. Teams read that and conclude long context is solved."}]},{"t":"p","runs":[{"text":"It is not the same task. The needle is semantically alien to its surroundings, and there is exactly one. Real work asks the model to combine four facts scattered across a long document, where three near-duplicates of each fact also appear and two of them are outdated. Accuracy on that shape of task degrades with length on every model measured, and it degrades well before the window fills."}]},{"t":"p","runs":[{"text":"The honest summary: long context is real and getting better, and it still costs you accuracy, not only money."}]},{"t":"h2","id":"spend-it-do-not-fill-it","runs":[{"text":"Spend it, do not fill it"}]},{"t":"p","runs":[{"text":"Treat the window like a monthly budget in a currency you actually earn. Give every component a line item, in tokens, and enforce it in code rather than hoping."}]},{"t":"code","lang":"python","label":"Python","text":"BUDGET = {\n    \"system\":       1_200,   # role, constraints, output contract\n    \"tools\":        1_500,   # tool schemas\n    \"examples\":     1_000,   # few-shot, static\n    \"retrieved\":    6_000,   # ~8-12 chunks after reranking\n    \"history\":      2_000,   # trimmed, oldest dropped first\n    \"user_turn\":      800,\n}\n# total 12,500 in, leaving room for output\n\ndef assemble(parts, budget):\n    out = {}\n    for name, limit in budget.items():\n        text = parts.get(name, \"\")\n        n = count_tokens(text)\n        if n > limit:\n            log.warning(\"over budget: %s used %d of %d\", name, n, limit)\n            text = trim(text, limit)          # per-part policy, not a blind cut\n        out[name] = text\n    return out"},{"t":"p","runs":[{"text":"The logging line matters more than the trimming. When retrieval quietly starts returning 40k tokens because someone uploaded a PDF of scanned invoices, you want a warning, not a surprise invoice."}]},{"t":"h2","id":"the-question-to-ask-of-every-token","runs":[{"text":"The question to ask of every token"}]},{"t":"p","runs":[{"text":"For each block you are about to include: "},{"text":"if I delete this, which specific answer gets worse?","i":true}]},{"t":"p","runs":[{"text":"If you cannot name the answer, delete the block and measure. Most context bloat is defensive: someone added the full policy document because a single question about refunds went wrong once, and nobody ever took it out. Six months later that block is in every call, on every request, in the bill every month."}]},{"t":"p","runs":[{"text":"The cheapest optimisation available to you is usually deletion."}]},{"t":"h2","id":"what-this-course-does","runs":[{"text":"What this course does"}]},{"t":"p","runs":[{"text":"Everything that follows is a way of spending that budget better: where a token should sit so the model uses it, which tokens can be cached so they cost a tenth, when to fetch tokens instead of carrying them, how to cut documents without cutting meaning, and how to prove a change helped rather than believing it did."}]}],"check":{"question":[{"text":"A team replaces a retrieval step (5 chunks, about 3,000 tokens) with pasting their entire 90,000-token product manual into every call. The manual fits comfortably in the model’s window. Accuracy on specific policy questions gets noticeably worse. What is the most likely explanation?"}],"options":[{"text":[{"text":"The manual needs converting to clean Markdown; the model is being confused by PDF formatting artefacts."}],"correct":false,"tempting":[{"text":"Formatting does affect parsing, so it feels like the obvious first fix, but it would not explain a drop relative to chunks drawn from the same document."}]},{"text":[{"text":"The 90,000 tokens contain many passages that resemble the answer, and the correct one now competes with dozens of plausible distractors."}],"correct":true,"tempting":null},{"text":[{"text":"The prompt exceeded the usable context and the provider silently truncated the middle of the manual."}],"correct":false,"tempting":[{"text":"Silent truncation does happen with some client libraries, but the stated premise is that it fits, and providers error rather than trim on overflow."}]},{"text":[{"text":"The model’s knowledge cutoff predates the manual, so it falls back on outdated training data instead of reading the text."}],"correct":false,"tempting":[{"text":"Cutoffs genuinely cause stale answers, but text placed in the context is read regardless of when it was written."}]}],"order":[0,1,3,2],"why":null},"prev":null,"next":{"slug":"what-context-engineering-is","title":"Nobody typed this","minutes":9}}}