What a 128k context window really means

Advertised, supported, effective

Three different numbers travel under the same name.

Advertised is what the announcement says: 128k, 256k, a million. It comes from max_position_embeddings and from whatever context extension the lab applied.

Supported is what your serving stack will actually run, which is set by your memory. The previous lesson’s arithmetic decides this, and for most people it is between 8k and 32k regardless of what the model claims.

Effective is the length at which the model still does the task well. It is usually far below the advertised number, and it is the only one that matters.

Why the gap exists

Position information in current models comes from rotary embeddings, which rotate the query and key vectors by an angle proportional to position. The rotation frequencies are set by rope_theta. A model trained with rope_theta at 10,000 over 4k-token documents has never seen the rotations that correspond to position 90,000, and its behaviour there is extrapolation.

Labs extend context in one of three ways. Linear interpolation squeezes positions into the trained range, at some cost to fine positional resolution. NTK-aware scaling and YaRN adjust the frequencies unevenly so that high-frequency components, which carry local ordering, are disturbed least. All of them are usually followed by a short continued-pretraining run on long documents, and the amount of that training decides how real the extension is.

So a model can honestly advertise 128k and be trained on very few long documents. Nothing breaks at 100k; it just gets vaguer.

Needle tests pass, and prove less than they look

The standard demonstration hides a sentence in a long document and asks the model to find it. Modern models pass this at very long lengths, and the resulting green charts are what the announcements show.

Retrieving one distinctive sentence is close to the easiest possible long-context task. Harder evaluations (multiple needles, tracking a variable through a long trace, aggregating facts spread across the document, or answering when the answer is absent) show much shorter effective lengths. Published work along these lines has repeatedly found models whose effective length on demanding tasks is a quarter or less of what they advertise.

There is also a well-documented positional effect: material at the start and end of a long context is used more reliably than material in the middle. Put the instruction and the most important document at the edges, not buried at 60%.

The costs nobody quotes

Prefill is quadratic in attention. Processing a 100,000-token prompt is not twelve times a 8,000-token prompt; the attention part grows with the square. Time to first token on a very long prompt can be tens of seconds on hardware where short prompts feel instant.

The cache is linear and large. 128k of context on the 8B model from the last lesson is about 17 GB of KV cache, which is more than the weights.

Rented long context is expensive. You pay per input token, every request. A 50,000-token prompt sent a thousand times a day is 50 million input tokens daily, and at typical prices that is a real monthly bill for something a retrieval step would have reduced to 3,000 tokens.

What to do instead, mostly

For nearly every application, retrieving the right 3,000 tokens beats sending 100,000. It is cheaper, faster, more accurate, and it gives you a citation trail. The exceptions are genuine: a single document that must be reasoned over as a whole, a long code file, a legal contract where any clause may matter. Those are real, and they are rarer than long-context marketing implies.

When you do need it, three habits help. Put instructions at the end as well as the beginning. Number or label the sections, so the model can refer to them. Ask for the location of the evidence along with the answer, which both improves accuracy and gives you something to check.

Measure your own effective length

Take five real documents at your longest realistic size. Ask three questions of each whose answers sit at different depths: near the start, in the middle, near the end. Score the answers. Then halve the document length and repeat.

If accuracy at half the length is much better, your effective length is below where you are operating, and no amount of prompt work will fix that. This takes an hour and is the only version of this number that describes your material.