Will it fit: the memory budget of a running model

Four line items

“Will it fit?” has an answer you can compute before downloading anything. Four things occupy memory when a model runs, and you can estimate each.

Weights. Parameters times bytes per parameter. Two bytes in bf16, one in int8, and about 0.55 to 0.6 at four bits once the scales that the next lesson explains are included. An 8-billion-parameter model is 16, 8 or roughly 5 gigabytes.

The key-value cache. Module two gave the per-token formula (two, times layers, times key-value heads, times head dimension, times bytes) and the result of 131 kilobytes per token for that 8-billion model in bf16. Multiply by the context length, and again by the number of sequences being served at once.

Activations and workspace. The intermediate vectors of the forward pass and the scratch memory attention needs. For inference these are modest (hundreds of megabytes to a few gigabytes depending on batch size) because, unlike training, nothing has to be kept for a backward pass. Modern attention kernels keep this from growing with the square of the sequence length.

Framework overhead and fragmentation. The GPU runtime takes a few hundred megabytes just to exist. Allocators fragment. Serving engines reserve a fraction of the card up front (vLLM’s default is 90 per cent) and refuse to start if the model plus a minimum cache does not fit inside it.

The rule of thumb that falls out: gigabytes of weights ≈ parameters in billions × bytes per parameter, plus ten per cent, and then everything left over is your cache budget.

Worked example: the 24-gigabyte card

An 8-billion model in bf16 on a consumer card with 24 gigabytes.

• Weights: 16.0 GB.
• Overhead: about 0.5 GB.
• Left for cache: 7.5 GB, which is 7.5 billion ÷ 131,000 ≈ 57,000 tokens of cache in total.

One user with a 32,000-token conversation fits. Two do. A fourth arrives with the same context and the total wanted is 128,000 tokens of cache: an out-of-memory error, from the line item that grows per user while the weights stay shared.

Now the same model at four bits: about 5 GB of weights, 18.5 GB free, around 140,000 tokens of cache. Quantise the cache itself to 8-bit (nearly free in quality) and it is 280,000. The card that served two users now serves eight.

Worked example: the 70-billion model

In bf16, 140 gigabytes of weights. That does not fit on one 80-gigabyte card and does not comfortably fit on two, once cache is added. This is why 70-billion-class models are usually served either quantised or split across several cards, which is a later lesson.

At four bits: about 40 gigabytes. One 80-gigabyte card holds it with 35 gigabytes to spare. The 70-billion architecture has 80 layers, eight key-value heads and a head dimension of 128, so a token costs 2 × 80 × 1,024 × 2 = 327 kilobytes in bf16, and 35 gigabytes is about 107,000 tokens of cache. A single card, a four-bit 70-billion model, and a handful of long conversations at once: that is a real, common deployment.

A laptop with 64 gigabytes of unified memory also holds the four-bit 70-billion model. The previous lesson says how fast: 400 gigabytes a second over 40 gigabytes is a bound of ten tokens a second. It runs. It is not quick.

The mixture-of-experts trap

Module two separated a sparse model’s total parameters from its active parameters. Memory uses the total: every expert must be resident, because the next token may route to any of them. A model advertised as “17 billion active, 109 billion total” needs memory for 109 billion (55 gigabytes at four bits) while its decode speed follows the 17 billion that are read per token. Both numbers matter, and they answer different questions.

When it does not fit, in order

1. Quantise the weights. Halves or quarters the largest line item. The next lesson says what it costs.
2. Quantise the cache. 8-bit is close to free. 4-bit costs measurable long-context recall.
3. Cap the context. Serving engines take a maximum sequence length; setting it to what your use actually needs frees cache for more users.
4. Offload layers to system RAM. llama.cpp and its front-ends let you place some layers on the GPU and the rest on the CPU. Speed falls steeply, because the offloaded layers are read at CPU-memory bandwidth or across a bus a tenth as fast as GPU memory. As a rough guide, offloading a tenth of the layers costs a third or more of the speed.
5. Split across GPUs. The lesson on parallelism.
6. Use a smaller model. Often the right answer, and the distillation lesson explains why small models became viable.

Training is a different budget

For contrast: module five’s sixteen bytes per parameter means the 8-billion model needs 128 gigabytes of optimiser state and gradients before any activations. QLoRA brings fine-tuning it down to under ten. The gap between that and the five gigabytes of inference is why running models is a laptop task and training them is not.

Measure after you load

On a GPU, nvidia-smi shows what is taken once the model is up; a serving engine’s startup log reports how many cache blocks it allocated. On a Mac, Activity Monitor’s memory pressure tells you when you have crossed into swap, which is when tokens per second collapses. The number to watch is free memory after the weights load. That, divided by the per-token cache cost, is the context and concurrency you can actually afford, and it is a better guide than anything on a model card.