{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"context-settings-that-truncate-in-silence","title":"Context settings that truncate in silence","url":"https://addaly.com/learn/running-models-yourself/context-settings-that-truncate-in-silence","position":33,"of":82,"minutes":9,"answer":"Trained length, allocated window and what the client sends are three different numbers, and the smallest wins silently: check the prompt token count against the configured window before blaming the model's memory.","keyIdea":"Trained length, allocated window and what the client sends are three different numbers, and the smallest wins silently: check the prompt token count against the configured window before blaming the model's memory."},"body":{"format":"markdown","text":"## Three different numbers called context\n\nConfusion here is the source of a whole class of bug where the model appears to have amnesia.\n\n1. **What the model was trained for.** A property of the weights, in `config.json` as `max_position_embeddings`.\n2. **What the server allocated.** A runtime setting: `-c` in llama.cpp, `num_ctx` in Ollama, `--max-model-len` in vLLM. It costs KV cache memory, so it defaults low.\n3. **What the client sent.** The application decides what history to include, and many silently drop the oldest messages.\n\nAll three can differ. The smallest one wins, and only the third produces a visible error.\n\n## The default that catches everyone\n\nOllama allocates a modest window by default regardless of what the model supports. Send more than that and the oldest tokens are dropped with no error and no warning. The model appears to forget the beginning of the document you just pasted.\n\n> Figure: Three numbers called context, and the one that wins. Bars (tokens): What the weights were trained for 40960; What the server allocated by default 4096; What the client sent 12000; What the model actually saw 4096. The smallest number wins, and only the client's own limit is ever reported as an error. Everything past the allocated window is dropped in silence, which a reader experiences as a model that has forgotten the beginning of the document they just pasted.\n\n```bash\n# per session\n/set parameter num_ctx 16384\n\n# for the whole server\nOLLAMA_CONTEXT_LENGTH=16384 ollama serve\n```\n\nllama-server's `-c` behaves differently and traps people in another way: with `--parallel`, the value is the *total* budget shared across slots. `--parallel 4 -c 32768` gives each of four users 8,192 tokens. Teams meet this as truncation that only happens when several people are working at once.\n\nvLLM is the honest one. `--max-model-len` is per request and exceeding it returns an error rather than dropping text, which is much better behaviour and occasionally annoying.\n\n## Extending beyond the trained length\n\nModels encode position with rotary embeddings, which have a base frequency. Increase the wavelength and positions spread out further, so the model can address a longer sequence than it saw in training.\n\n- **Linear scaling** stretches all positions uniformly. Simple, and it degrades short-range precision because nearby tokens become less distinguishable.\n- **NTK-aware scaling** stretches low frequencies more than high ones, preserving local resolution.\n- **YaRN** refines that further with a temperature adjustment, and is what most models shipping extended context now use.\n\n```bash\n# llama.cpp: run a 32k model at 64k\nllama-server -m model.gguf -c 65536 --rope-scaling yarn --rope-scale 2\n```\n\nTwo honest caveats. Applying scaling when you do not need it makes short-context performance worse, so do not enable it by default. And extension is not free capability: a model stretched from 32k to 128k can attend across 128k without necessarily being able to use what it finds there.\n\n## Advertised context and usable context\n\nA model advertised at 128k rarely performs at 128k. The single-fact retrieval test (hide a sentence in a long document and ask for it) is the easy version, and most models pass it well beyond the point where they fail at anything harder. Ask for two facts from different places, or a comparison across the document, or a fact that contradicts something stated earlier, and performance falls off much sooner. For models in the 4B-to-32B band, the point where multi-fact tasks start failing is often somewhere between 8k and 32k.\n\nTest it yourself, because the number is task-specific: take a real document at your real length, place three facts you know, and ask for all three. That takes ten minutes and tells you your working limit rather than the marketing one.\n\n## The practical settings\n\n**Set the window to what you need plus headroom, not to the maximum.** Every unused token of window is memory you could have spent on a better quant.\n\n**Watch what the client sends.** Chat interfaces accumulate history, and by the twentieth turn you may be sending 20,000 tokens for a one-line question. Most have a setting to cap or summarise history; find it.\n\n**Prefer retrieval over stuffing.** Five relevant passages beat a hundred pages on quality, on speed, and on memory. This is true even when the whole document fits.\n\n**Watch for the silent drop.** If a long input seems to be half-understood, log the token count you are actually sending and compare it with the allocated window. Most servers report prompt tokens in the response `usage` field; that number against your configured limit answers the question immediately."},"check":{"question":"A team runs llama-server with `--parallel 4 -c 32768`. One person pasting a 20,000-token document gets good summaries when working alone, but truncated ones during busy periods. What is happening?","options":[{"text":"Continuous batching evicts the oldest tokens from a request's cache when memory pressure rises, so long prompts lose their beginning under load.","correct":false,"tempting":"Describes an eviction policy that engines do not apply to an in-flight request's own prompt; a full cache blocks admission rather than trimming context."},{"text":"The context value is the total budget shared across the four slots, so each concurrent request gets about 8,192 tokens.","correct":true},{"text":"Prefix caching is sharing cache blocks between users, and a collision is overwriting part of the long prompt.","correct":false,"tempting":"Invents a correctness bug in a mechanism that keys on exact token prefixes and cannot mix one user's tokens into another's sequence."},{"text":"Under load the server falls back to a shorter rotary scaling factor to save memory, reducing the usable window.","correct":false,"tempting":"Attributes a dynamic behaviour to a static setting; rope scaling is fixed at load time and does not vary with concurrency."}]},"prev":{"slug":"chat-templates","title":"Chat templates: the most common silent failure","url":"https://addaly.com/learn/running-models-yourself/chat-templates"},"next":{"slug":"the-openai-compatible-api","title":"The API every engine speaks","url":"https://addaly.com/learn/running-models-yourself/the-openai-compatible-api"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Context settings that truncate in silence', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/context-settings-that-truncate-in-silence (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Context settings that truncate in silence","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"Trained length, allocated window and what the client sends are three different numbers, and the smallest wins silently: check the prompt token count against the configured window before blaming the model’s memory.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens","title":"Three numbers called context, and the one that wins","series":[{"label":"What the weights were trained for","value":40960},{"label":"What the server allocated by default","value":4096},{"label":"What the client sent","value":12000},{"label":"What the model actually saw","value":4096,"highlight":true}],"caption":"The smallest number wins, and only the client’s own limit is ever reported as an error. Everything past the allocated window is dropped in silence, which a reader experiences as a model that has forgotten the beginning of the document they just pasted."},"alt":"Figure: Three numbers called context, and the one that wins. Bars (tokens): What the weights were trained for 40960; What the server allocated by default 4096; What the client sent 12000; What the model actually saw 4096. The smallest number wins, and only the client’s own limit is ever reported as an error. Everything past the allocated window is dropped in silence, which a reader experiences as a model that has forgotten the beginning of the document they just pasted."}},"blocks":[{"t":"h2","id":"three-different-numbers-called-context","runs":[{"text":"Three different numbers called context"}]},{"t":"p","runs":[{"text":"Confusion here is the source of a whole class of bug where the model appears to have amnesia."}]},{"t":"ol","start":1,"items":[[{"text":"What the model was trained for.","b":true},{"text":" A property of the weights, in "},{"text":"config.json","code":true},{"text":" as "},{"text":"max_position_embeddings","code":true},{"text":"."}],[{"text":"What the server allocated.","b":true},{"text":" A runtime setting: "},{"text":"-c","code":true},{"text":" in llama.cpp, "},{"text":"num_ctx","code":true},{"text":" in Ollama, "},{"text":"--max-model-len","code":true},{"text":" in vLLM. It costs KV cache memory, so it defaults low."}],[{"text":"What the client sent.","b":true},{"text":" The application decides what history to include, and many silently drop the oldest messages."}]]},{"t":"p","runs":[{"text":"All three can differ. The smallest one wins, and only the third produces a visible error."}]},{"t":"h2","id":"the-default-that-catches-everyone","runs":[{"text":"The default that catches everyone"}]},{"t":"p","runs":[{"text":"Ollama allocates a modest window by default regardless of what the model supports. Send more than that and the oldest tokens are dropped with no error and no warning. The model appears to forget the beginning of the document you just pasted."}]},{"t":"code","lang":"bash","label":"Bash","text":"# per session\n/set parameter num_ctx 16384\n\n# for the whole server\nOLLAMA_CONTEXT_LENGTH=16384 ollama serve"},{"t":"p","runs":[{"text":"llama-server’s "},{"text":"-c","code":true},{"text":" behaves differently and traps people in another way: with "},{"text":"--parallel","code":true},{"text":", the value is the "},{"text":"total","i":true},{"text":" budget shared across slots. "},{"text":"--parallel 4 -c 32768","code":true},{"text":" gives each of four users 8,192 tokens. Teams meet this as truncation that only happens when several people are working at once."}]},{"t":"p","runs":[{"text":"vLLM is the honest one. "},{"text":"--max-model-len","code":true},{"text":" is per request and exceeding it returns an error rather than dropping text, which is much better behaviour and occasionally annoying."}]},{"t":"h2","id":"extending-beyond-the-trained-length","runs":[{"text":"Extending beyond the trained length"}]},{"t":"p","runs":[{"text":"Models encode position with rotary embeddings, which have a base frequency. Increase the wavelength and positions spread out further, so the model can address a longer sequence than it saw in training."}]},{"t":"ul","items":[[{"text":"Linear scaling","b":true},{"text":" stretches all positions uniformly. Simple, and it degrades short-range precision because nearby tokens become less distinguishable."}],[{"text":"NTK-aware scaling","b":true},{"text":" stretches low frequencies more than high ones, preserving local resolution."}],[{"text":"YaRN","b":true},{"text":" refines that further with a temperature adjustment, and is what most models shipping extended context now use."}]]},{"t":"code","lang":"bash","label":"Bash","text":"# llama.cpp: run a 32k model at 64k\nllama-server -m model.gguf -c 65536 --rope-scaling yarn --rope-scale 2"},{"t":"p","runs":[{"text":"Two honest caveats. Applying scaling when you do not need it makes short-context performance worse, so do not enable it by default. And extension is not free capability: a model stretched from 32k to 128k can attend across 128k without necessarily being able to use what it finds there."}]},{"t":"h2","id":"advertised-context-and-usable-context","runs":[{"text":"Advertised context and usable context"}]},{"t":"p","runs":[{"text":"A model advertised at 128k rarely performs at 128k. The single-fact retrieval test (hide a sentence in a long document and ask for it) is the easy version, and most models pass it well beyond the point where they fail at anything harder. Ask for two facts from different places, or a comparison across the document, or a fact that contradicts something stated earlier, and performance falls off much sooner. For models in the 4B-to-32B band, the point where multi-fact tasks start failing is often somewhere between 8k and 32k."}]},{"t":"p","runs":[{"text":"Test it yourself, because the number is task-specific: take a real document at your real length, place three facts you know, and ask for all three. That takes ten minutes and tells you your working limit rather than the marketing one."}]},{"t":"h2","id":"the-practical-settings","runs":[{"text":"The practical settings"}]},{"t":"p","runs":[{"text":"Set the window to what you need plus headroom, not to the maximum.","b":true},{"text":" Every unused token of window is memory you could have spent on a better quant."}]},{"t":"p","runs":[{"text":"Watch what the client sends.","b":true},{"text":" Chat interfaces accumulate history, and by the twentieth turn you may be sending 20,000 tokens for a one-line question. Most have a setting to cap or summarise history; find it."}]},{"t":"p","runs":[{"text":"Prefer retrieval over stuffing.","b":true},{"text":" Five relevant passages beat a hundred pages on quality, on speed, and on memory. This is true even when the whole document fits."}]},{"t":"p","runs":[{"text":"Watch for the silent drop.","b":true},{"text":" If a long input seems to be half-understood, log the token count you are actually sending and compare it with the allocated window. Most servers report prompt tokens in the response "},{"text":"usage","code":true},{"text":" field; that number against your configured limit answers the question immediately."}]}],"check":{"question":[{"text":"A team runs llama-server with "},{"text":"--parallel 4 -c 32768","code":true},{"text":". One person pasting a 20,000-token document gets good summaries when working alone, but truncated ones during busy periods. What is happening?"}],"options":[{"text":[{"text":"Continuous batching evicts the oldest tokens from a request’s cache when memory pressure rises, so long prompts lose their beginning under load."}],"correct":false,"tempting":[{"text":"Describes an eviction policy that engines do not apply to an in-flight request’s own prompt; a full cache blocks admission rather than trimming context."}]},{"text":[{"text":"The context value is the total budget shared across the four slots, so each concurrent request gets about 8,192 tokens."}],"correct":true,"tempting":null},{"text":[{"text":"Prefix caching is sharing cache blocks between users, and a collision is overwriting part of the long prompt."}],"correct":false,"tempting":[{"text":"Invents a correctness bug in a mechanism that keys on exact token prefixes and cannot mix one user’s tokens into another’s sequence."}]},{"text":[{"text":"Under load the server falls back to a shorter rotary scaling factor to save memory, reducing the usable window."}],"correct":false,"tempting":[{"text":"Attributes a dynamic behaviour to a static setting; rope scaling is fixed at load time and does not vary with concurrency."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"chat-templates","title":"Chat templates: the most common silent failure","minutes":10},"next":{"slug":"the-openai-compatible-api","title":"The API every engine speaks","minutes":8}}}