Does it fit: the arithmetic

Stop looking up tables of “models that fit in 8GB”. Learn the two sums and you can answer for any model, any quant, any context length.

Total memory = weights + KV cache + overhead.

Weights

bytes = parameters x bytes_per_weight

Bytes per weight comes from the format. Useful figures, in bits per weight (divide by 8):

The 4-bit formats are above 4 bits because each block of weights carries a scale. Lesson 3 explains why.

So Llama 3.1 8B at Q4_K_M: 8.03e9 x 4.8 / 8 = 4.8 GB. The published file is 4.9 GB. Close enough to plan with.

A 70B at Q4_K_M: 70e9 x 0.6 = 42 GB. Two 24 GB cards, or a 64 GB Mac, or it does not happen.

KV cache

Every token you have already processed leaves behind a key and a value vector in every layer. This is what “context costs memory” means.

bytes_per_token = 2 x n_layers x n_kv_heads x head_dim x bytes_per_element

The leading 2 is K and V. All four numbers are in the model’s config.json on Hugging Face: look for num_hidden_layers, num_key_value_heads, and hidden_size / num_attention_heads for head dim.

Llama 3.1 8B: 32 layers, 8 KV heads, head dim 128, fp16 cache.

2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token

At 8,192 tokens: 1.0 GiB. At 32,768: 4.0 GiB. At 128k: 16 GiB: four times the weights.

Now contrast Llama 2 13B, which predates grouped-query attention and has 40 KV heads across 40 layers:

2 x 40 x 40 x 128 x 2 = 819,200 bytes = 800 KiB per token

That is 3.1 GiB at only 4k context. Grouped-query attention, which shares one KV head across several query heads, is the reason long context became affordable. Two models of similar size can differ sixfold here.

Overhead

CUDA context, compute buffers, activations, the framework itself. Budget 0.6 to 1.5 GB, or add 15%. It is not nothing and it is what turns “it should just fit” into an out-of-memory error.

A worked decision

An 8 GB card. Llama 3.1 8B Q4_K_M, 8k context:

• weights 4.9 GB
• KV 1.0 GB
• overhead 0.8 GB
• total 6.7 GB: fits, with headroom.

Same model at 32k context: KV becomes 4.0 GB, total 9.7 GB. Does not fit. Your options, in order of what they cost you:

1. Quantise the KV cache to 8-bit. Halves it to 2.0 GB, total 7.7 GB. Fits. Quality cost is small at q8_0; q4_0 cache is noticeably worse for long recall.
2. Drop to 16k context. Total 7.7 GB too, with no quality cost at all, if you do not need 32k.
3. Drop weights to Q3_K_M (3.9 GB). Saves 1 GB and costs more quality than either of the above. Last resort.

In llama.cpp those first two are:

llama-server -m model.gguf -c 32768 --flash-attn \
  --cache-type-k q8_0 --cache-type-v q8_0

Partial offload

If it does not fit, llama.cpp will run some layers on the GPU and the rest on the CPU (-ngl 24 for 24 layers). This works, and it is slow roughly in proportion to how much stayed on the CPU. Ten percent of layers on CPU is not a ten percent slowdown; it is often a fifty percent one, because every token waits for those layers.

Do the sum before you download 40 GB.