Context settings that truncate in silence

Three different numbers called context

Confusion here is the source of a whole class of bug where the model appears to have amnesia.

1. What the model was trained for. A property of the weights, in config.json as max_position_embeddings.
2. What the server allocated. A runtime setting: -c in llama.cpp, num_ctx in Ollama, --max-model-len in vLLM. It costs KV cache memory, so it defaults low.
3. What the client sent. The application decides what history to include, and many silently drop the oldest messages.

All three can differ. The smallest one wins, and only the third produces a visible error.

The default that catches everyone

Ollama allocates a modest window by default regardless of what the model supports. Send more than that and the oldest tokens are dropped with no error and no warning. The model appears to forget the beginning of the document you just pasted.

# per session
/set parameter num_ctx 16384

# for the whole server
OLLAMA_CONTEXT_LENGTH=16384 ollama serve

llama-server’s -c behaves differently and traps people in another way: with --parallel, the value is the total budget shared across slots. --parallel 4 -c 32768 gives each of four users 8,192 tokens. Teams meet this as truncation that only happens when several people are working at once.

vLLM is the honest one. --max-model-len is per request and exceeding it returns an error rather than dropping text, which is much better behaviour and occasionally annoying.

Extending beyond the trained length

Models encode position with rotary embeddings, which have a base frequency. Increase the wavelength and positions spread out further, so the model can address a longer sequence than it saw in training.

• Linear scaling stretches all positions uniformly. Simple, and it degrades short-range precision because nearby tokens become less distinguishable.
• NTK-aware scaling stretches low frequencies more than high ones, preserving local resolution.
• YaRN refines that further with a temperature adjustment, and is what most models shipping extended context now use.

# llama.cpp: run a 32k model at 64k
llama-server -m model.gguf -c 65536 --rope-scaling yarn --rope-scale 2

Two honest caveats. Applying scaling when you do not need it makes short-context performance worse, so do not enable it by default. And extension is not free capability: a model stretched from 32k to 128k can attend across 128k without necessarily being able to use what it finds there.

Advertised context and usable context

A model advertised at 128k rarely performs at 128k. The single-fact retrieval test (hide a sentence in a long document and ask for it) is the easy version, and most models pass it well beyond the point where they fail at anything harder. Ask for two facts from different places, or a comparison across the document, or a fact that contradicts something stated earlier, and performance falls off much sooner. For models in the 4B-to-32B band, the point where multi-fact tasks start failing is often somewhere between 8k and 32k.

Test it yourself, because the number is task-specific: take a real document at your real length, place three facts you know, and ask for all three. That takes ten minutes and tells you your working limit rather than the marketing one.

The practical settings

Set the window to what you need plus headroom, not to the maximum. Every unused token of window is memory you could have spent on a better quant.

Watch what the client sends. Chat interfaces accumulate history, and by the twentieth turn you may be sending 20,000 tokens for a one-line question. Most have a setting to cap or summarise history; find it.

Prefer retrieval over stuffing. Five relevant passages beat a hundred pages on quality, on speed, and on memory. This is true even when the whole document fits.

Watch for the silent drop. If a long input seems to be half-understood, log the token count you are actually sending and compare it with the allocated window. Most servers report prompt tokens in the response usage field; that number against your configured limit answers the question immediately.