When it does not fit: the ordered remedies

Fix the right term

Out of memory is not one problem. Work out which term is too large before changing anything, because the remedies for weights and for cache are different and applying the wrong one costs quality for nothing.

The sum, once more: weights + KV cache + overhead. Compute all three. If the cache is a quarter of the total, halving it saves an eighth; if it is two thirds, halving it saves a third. That arithmetic decides the order of everything below.

The ladder, cheapest quality cost first

1. Reduce the context window. Free. Zero quality cost if you do not need the length. Most people set 32k out of optimism and send 3,000 tokens. Set it to what you actually use plus headroom. In Ollama that is num_ctx; in llama.cpp -c; in vLLM --max-model-len.

2. Quantise the KV cache to 8-bit. Halves the cache term for a difference most tasks cannot detect. Requires flash attention in llama.cpp. This is the best value change available and it is routinely skipped.

3. Close the other things using the memory. A browser with hardware acceleration, a desktop compositor, another model still resident. nvidia-smi will name them. On Ollama, OLLAMA_KEEP_ALIVE=0 unloads a model as soon as it goes idle rather than holding it for five minutes.

4. Drop one quantisation tier. Q5_K_M to Q4_K_M is a real but small cost and buys about 15% of the weight size. Q6 to Q5 is smaller still. Stop before Q3 unless nothing else works.

5. Use a smaller model in the same family. Often better than crushing a bigger one. A 4B at Q5 frequently beats an 8B at Q2 on anything requiring precision, and it is twice as fast.

6. Use a task-specialised model. A code-specific 7B in place of a general 14B; an embedding model in place of an LLM for retrieval; a small classifier in place of a general model for labelling. This is a redesign rather than a setting, and it is often the largest win available.

7. Consider a mixture-of-experts model. If you are short of GPU memory and rich in system RAM, this changes the problem rather than trading against it.

8. Partial offload. Real, and expensive in the way the harmonic mean makes it. Accept it when the CPU share is small, or when the model is MoE and you can offload expert tensors specifically.

9. Quantise the cache to 4-bit. Deliberately below offload, because it damages exactly the long-context recall you are usually extending context to get.

10. Buy memory. Honest, last, and sometimes correct. System RAM is cheap and helps MoE and CPU inference; VRAM is expensive and is the only thing that helps a dense model on a GPU.

Fitting into named sizes

Some concrete landing points, all at Q4_K_M with 8k context and roughly 1 GB of overhead:

• 4 GB VRAM. A 3B fits with room. A 4B is tight. An 8B does not fit and partial offload is painful.
• 8 GB. An 8B at 8k fits with about a gigabyte spare. At 32k it does not, unless the cache is quantised.
• 12 GB. A 14B fits at 8k. An 8B fits with long context and a quantised cache.
• 16 GB. A 14B with generous context, or a 22B tightly.
• 24 GB. A 32B at 8k, with little room to spare. A 14B with very long context comfortably.
• 2 x 24 GB or 64 GB unified. A 70B at Q4 with moderate context.

Recompute these for your model rather than trusting them; a model with a fat KV cache moves every line.

The failure modes to recognise

CUDA out of memory during a long conversation, not at startup. The cache grew. Cap the context and reserve for it up front rather than discovering the limit at token 6,000.

Fits, then fails when a second user arrives. Each concurrent sequence needs its own cache. Either cap concurrency or use an engine with paged cache blocks.

Fits on Linux, fails on Windows with the same card. The desktop compositor holds several hundred megabytes of VRAM. Real, and it is why the same configuration behaves differently across operating systems.

Works, then stops working after an update. Engines change default context lengths and default cache types between versions. Pin your flags explicitly rather than relying on defaults, and the problem disappears permanently.