Long context in practice

The advertised number is a capacity, not a competence

A model advertised at 128k will accept 128,000 tokens without error. Whether it can use them is a separate question with a much less flattering answer.

The standard demonstration (hide one sentence in a long document and ask for it) is the easiest possible version of the test. Models pass it well past the point where they fail at anything harder. Treat a passing result on that test as evidence of nothing much.

What actually degrades

Three distinct failures, and they arrive at different lengths.

Position effects. Material at the start and end of a long context is used more reliably than material in the middle. Put the important passage in the middle of forty pages and it is measurably less likely to be found.

Multi-fact tasks. Retrieving one fact is far easier than retrieving three from different places, or comparing two, or noticing that a later statement contradicts an earlier one. Models in the 4B-to-32B band typically start failing these somewhere between 8k and 32k, well short of the advertised figure.

Distractor sensitivity. Text that resembles the answer but is not the answer pulls the model towards it. A contract with five similar clauses is much harder than a contract with one, at the same length.

A model does not announce any of this. It produces a fluent answer built from whatever it managed to attend to, which is exactly the failure that is hardest to notice.

Measure your own working limit

Twenty minutes, and the result is worth more than any published number because it uses your documents and your questions.

1. Take a real document at your real length.
2. Insert three facts you know the answer to, at the start, the middle and the end.
3. Ask for all three in one question. Then ask a question requiring two of them together.
4. Repeat at 4k, 8k, 16k, 32k by truncating or padding.
5. Note where the answers start to go wrong.

That length is your working limit for this model on this kind of document. It will be shorter than the advertised context and probably shorter than you hoped.

The cost side

Long context is expensive in three ways simultaneously.

Memory. The KV cache grows linearly. At 128k on a modern 8B, the cache is several times the size of the weights.

Prefill time. Processing 100,000 tokens takes real seconds even on good hardware, and a minute or more on a CPU. This lands entirely before the first output token.

Attention cost per token. Each generated token attends over the whole cache, so generation slows as the context fills. A conversation that started at 50 tokens per second can be at 25 by the time it is deep into a long document.

Retrieval beats stuffing, almost always

The instinct is to paste everything and let the model sort it out. Five well-chosen passages beat a hundred pages on quality, cost and speed at the same time, and the quality part surprises people, because it seems as though more information should help.

It does not, for a mechanical reason: attention is a limited resource spread across the context, and irrelevant material dilutes it. Adding forty pages of unrelated text around the relevant paragraph makes the relevant paragraph harder to use, not easier.

So the pattern from the retrieval lesson is not a workaround for small context windows. It is the better design even when everything fits.

Practical management

Cap conversation history. Chat interfaces accumulate. By turn twenty you may be sending 20,000 tokens for a one-line question, paying for it in latency on every turn.

Summarise rather than truncate. Dropping the oldest turns loses established facts. Replacing them with a short summary keeps them, and costs one extra call. Note that this breaks prefix caching for that conversation, so do it on a schedule rather than every turn.

Put the question last. Both because instructions at the end are followed best, and because it keeps the document prefix stable and cacheable.

Keep the document ordering stable across queries, for the same caching reason.

Check the token count you are sending. The usage.prompt_tokens field on every response, against your configured window and against your measured working limit. Two numbers, and they explain most unexpected answers on long inputs.