The window is a budget, not a container

Fitting is not working

You have a 200,000-token window. Your document is 180,000 tokens. It fits.

That is not the same as it working. Three things get worse as you add tokens, and they get worse at different rates.

Money. Input tokens are billed on every call, forever. At $3 per million input tokens, a 200k-token prompt costs $0.60 to send. Ten thousand calls a day is $6,000 a day, $180,000 a month. The same task trimmed to 6k tokens costs $0.018 a call — about $180 a day. Same model, often the same answers, roughly 33x the bill. For a team in Lagos or Lahore paying in dollars from local revenue, that difference is the whole business case.

Latency. Before the model emits a single token, it reads yours. Prefill scales with input length. A 200k-token prompt typically means several seconds before the first character appears; an 8k prompt means a few hundred milliseconds. Users forgive a slow answer. They do not forgive a slow blank screen.

Attention. This is the one people miss. A model does not read your context the way a database reads a row. It attends across all of it, and every irrelevant passage you add is a passage that can win the competition for relevance. Add the whole product manual and you have not just added the paragraph about refunds — you have added forty paragraphs that *look* like they are about refunds.

What the benchmarks hide

Needle-in-a-haystack tests — hide one odd sentence in 500k tokens, ask the model to find it — are close to saturated on frontier models. Teams read that and conclude long context is solved.

It is not the same task. The needle is semantically alien to its surroundings, and there is exactly one. Real work asks the model to combine four facts scattered across a long document, where three near-duplicates of each fact also appear and two of them are outdated. Accuracy on that shape of task degrades with length on every model measured, and it degrades well before the window fills.

The honest summary: long context is real and getting better, and it still costs you accuracy, not only money.

Spend it, do not fill it

Treat the window like a monthly budget in a currency you actually earn. Give every component a line item, in tokens, and enforce it in code rather than hoping.

BUDGET = {
    "system":       1_200,   # role, constraints, output contract
    "tools":        1_500,   # tool schemas
    "examples":     1_000,   # few-shot, static
    "retrieved":    6_000,   # ~8-12 chunks after reranking
    "history":      2_000,   # trimmed, oldest dropped first
    "user_turn":      800,
}
# total 12,500 in, leaving room for output

def assemble(parts, budget):
    out = {}
    for name, limit in budget.items():
        text = parts.get(name, "")
        n = count_tokens(text)
        if n > limit:
            log.warning("over budget: %s used %d of %d", name, n, limit)
            text = trim(text, limit)          # per-part policy, not a blind cut
        out[name] = text
    return out

The logging line matters more than the trimming. When retrieval quietly starts returning 40k tokens because someone uploaded a PDF of scanned invoices, you want a warning, not a surprise invoice.

The question to ask of every token

For each block you are about to include: *if I delete this, which specific answer gets worse?*

If you cannot name the answer, delete the block and measure. Most context bloat is defensive — someone added the full policy document because a single question about refunds went wrong once, and nobody ever took it out. Six months later that block is in every call, on every request, in the bill every month.

The cheapest optimisation available to you is usually deletion.

What this course does

Everything that follows is a way of spending that budget better: where a token should sit so the model uses it, which tokens can be cached so they cost a tenth, when to fetch tokens instead of carrying them, how to cut documents without cutting meaning, and how to prove a change helped rather than believing it did.