Four kinds of context, four different problems

Sorting the window by what it is for

Everything you put in a context window belongs to one of four kinds. The kinds matter because each has a different half-life, a different trust level, a different cache behaviour, and a different thing that goes wrong with it. Mixing them up is the most common structural mistake in a system that has grown past its first version.

1. Instructions: what the system is

The role, the constraints, the refusal policy, the output contract, the tone. Written by you, changed on a deploy, identical for every user.

• Half-life: weeks.
• Trust: full. You wrote it.
• Cache: perfect, if you put it first and never interpolate anything variable into it.
• Failure mode: growth. Instructions accrete. Every incident adds a sentence, and nobody ever deletes one. A 300-token system prompt becomes 2,400 tokens of contradictory rules over eighteen months, and the contradictions are invisible because nobody reads it end to end.

2. Exemplars: what a good answer looks like

Few-shot examples, formatting demonstrations, a worked case.

• Half-life: months, but they rot silently when the task drifts.
• Trust: full, unless somebody pasted a real customer’s message into one, which happens more often than teams admit.
• Cache: perfect, if they sit with the instructions rather than being selected per request.
• Failure mode: they are expensive per unit of behaviour bought. Four examples at 250 tokens each is 1,000 tokens on every call forever. Sometimes one sentence of instruction does the same job.

3. Knowledge: what is true that the model does not know

Retrieved passages, an uploaded document, the row from your database, today’s price.

• Half-life: one request. It is chosen for this question.
• Trust: varies, and this is the important part. Your own product documentation is trusted. A PDF a stranger uploaded is not. Both arrive in the same field.
• Cache: poor by construction, since it changes per request. Anything you cache here you cache for a segment of users, not all.
• Failure mode: wrong selection, and it is invisible from the answer. The model produces a confident, well-formed reply based on the four passages you gave it and never mentions the fifth passage you did not.

4. State: what has happened so far

Conversation history, tool results, the agent’s scratchpad, the user’s profile, the current step of a workflow.

• Half-life: minutes, and growing.
• Trust: mixed and often mislabelled. A tool result is data from a system; the model’s own previous output is a guess it made; a user’s earlier message is user input. Teams routinely treat all three as equally reliable because they arrive in the same message list.
• Cache: good in the middle, because history is append-only, until you compact it, at which point the whole suffix invalidates.
• Failure mode: unbounded growth. A tool that returns 30,000 tokens of JSON on turn three is a design bug that surfaces as a cost bug on turn nine.

The fifth thing, which is not context

Output reserve. The window is shared between what you send and what comes back. On most APIs the limit you hit is the sum, and max_tokens is subtracted from the space available for input. Budget it first, not last: decide the answer needs 800 tokens, subtract it, and spend what remains.

This is the single most common cause of the error message that reads like a lie: a request that is comfortably under the context limit and still fails, because the requested completion length did not fit alongside it.

Why the sorting is useful

Once material is sorted, three decisions become mechanical instead of arguable.

Order. Stable before variable, because caching matches from the first token forward. Instructions and exemplars are stable; knowledge and state are not. That single rule decides most of your prefix layout, and it is covered properly in its own lesson.

Trust. Instructions and exemplars are yours. Knowledge and state largely are not. The boundary between them is where your defences go, and it is a boundary you can draw on a diagram only if the four kinds are separated in the code that builds the request.

What to cut when it does not fit. State compacts. Knowledge is re-ranked and truncated. Exemplars can be reduced from four to two and measured. Instructions are the last thing you touch, because they are the only part guaranteeing that the output is usable at all. Most naive truncation does exactly the opposite: it drops from the front, which is where the instructions live.

The audit worth doing today

Take one real production request. Print each of the four kinds with a token count beside it. Teams doing this for the first time usually find one of three things: tool definitions costing more than the instructions, a history that nobody has ever capped, or retrieved passages making up two thirds of the window while the retrieval quality has never been measured on its own.

All three are fixable this week. None of them is visible from the model’s output.