The advertised window and the usable one

Two numbers, one of which is marketing

A model advertises a context window of 200,000 or a million tokens. That number is a hard limit enforced by the API. It is not a promise about quality, and the gap between the limit and the length at which the model still works well is one of the most consequential unstated facts in the field.

The pattern, repeated across families and generations: accuracy on tasks that require finding and combining information holds steady for a while and then falls, often long before the advertised limit. How much earlier depends on the model and enormously on the task.

Why the easy test is misleading

The test everybody runs is needle-in-a-haystack: hide a distinctive sentence in a long document and ask for it back. Modern models score close to perfect on this at very long lengths, and vendors publish the resulting green grids.

The test is easy for a mechanical reason. The needle is lexically distinctive: it shares rare tokens with the question, so attention has an easy target. Nothing else in the haystack competes.

Change one thing and the picture changes. Paraphrase the needle so it shares no distinctive words with the question, and scores fall sharply at long lengths on models that scored perfectly on the literal version. Require two needles to be combined, and they fall further. Add distractors that are topically similar but wrong, and they fall further still.

The harder public suites (the ones that vary needle count, paraphrase, and add distractors) consistently show usable performance somewhere well below the advertised number for tasks with any reasoning in them. The honest summary is a shape rather than a constant: retrieval of a distinctive string degrades late, retrieval of a paraphrased fact degrades earlier, and combining several facts scattered through the window degrades earliest of all.

The mechanisms, so the shape is not a surprise

Three things are going on, and knowing them tells you when to expect trouble.

Attention dilution. Softmax attention distributes a fixed total of one across all positions. With 200 candidate positions, a relevant one can hold a large share. With 200,000, the same absolute score is a much smaller share of the total, and weakly relevant material in aggregate can outweigh it.

Training length distribution. Long-context ability is largely trained in, and long training documents are scarce and expensive. Much of the very long capability comes from a comparatively short extension phase, so the model has seen far fewer examples of reasoning across 300,000 tokens than across 3,000.

Position encoding extrapolation. Techniques that extend a model beyond its trained length (interpolating or scaling rotary embeddings, for instance) work, and they degrade gracefully rather than cliff-edge. Graceful degradation is still degradation.

What this means for your budget

Do not plan a system around the advertised number. Plan it around a measured one.

The measurement is not hard and you should run it on your own material, because the answer depends on your task far more than on any published average:

1. Take 50 real questions whose answers you know are in one specific passage.
2. Build contexts at several lengths (4k, 16k, 64k, 128k) where the true passage is present and the rest is filled with your own real but irrelevant material, not lorem ipsum. Distractors from the same domain are the point.
3. Place the true passage at a random position each time, so you are measuring length and not position.
4. Score exact-match or a simple containment check on the answer.
5. Plot accuracy against length.

You now have your model’s effective context for your task, expressed as a curve rather than a marketing number. Budget below the knee. The exercise costs a few dollars of inference and settles arguments that otherwise run for months.

Cost and latency do not degrade: they grow

Quality falls off gradually. The bill does not. Input tokens are charged linearly, and prefill time is roughly linear in input length too, with attention cost growing faster than linearly in the underlying computation even where serving optimisations hide much of it.

So the trade at 200,000 tokens is: you pay perhaps twenty times the price and several times the time-to-first-token of a 10,000-token request, for a context in which the model is measurably worse at combining facts. That is a bad trade to make by default, and a reasonable one to make deliberately when the alternative is missing the relevant material entirely.

Where long context wins

None of this argues against long windows. It argues against filling them without measurement. Long context earns its cost when:

• The corpus is small enough to fit entirely and retrieval would risk missing the one relevant line: a single contract, one codebase module, one patient record.
• The task needs global structure that chunking destroys: “is this document internally consistent”, “summarise the argument”, “find every place the tone changes”.
• You are prototyping and want to establish an upper bound on quality before spending a week on a retrieval pipeline. This is an excellent use of a big window and the most underrated one.

The common thread: use the whole window when the whole thing is relevant, not when you could not decide what was.