The memory arithmetic you can do in your head

One multiplication decides most of it

weights in bytes = parameters × bytes per parameter

The bytes per parameter come from the dtype:

• float32: 4 bytes. The default if you say nothing, and almost never what you want.
• float16 / bfloat16: 2 bytes. The normal choice on a GPU.
• int8: 1 byte.
• int4: about 0.5 bytes, plus a small overhead for scales.

So a 7-billion-parameter model is 28 GB at float32, 14 GB at bfloat16, 7 GB at int8 and roughly 4 GB at int4. A 1.5B model is 3 GB at bfloat16 and fits comfortably in a free Colab session with room to work. A 70B model is 140 GB at bfloat16, which is two 80 GB cards, and about 40 GB at int4, which is one.

You can now look at any repository name and know, before downloading anything, whether it is worth downloading.

float16 and bfloat16 are not interchangeable

Both take two bytes. They spend those bytes differently.

• float16 gives more bits to precision and has a narrow exponent range, with a maximum around 65,504.
• bfloat16 keeps float32’s exponent range and sacrifices precision.

For neural networks the exponent range matters more, because activations occasionally spike. A value above 65,504 becomes infinity in float16, infinity times zero becomes NaN, and the NaN spreads through the network within one layer. This is the mechanism behind the T5 and older-checkpoint failures from the debugging lesson.

Practical rule: use bfloat16 if the hardware supports it: Ampere-generation GPUs and newer, which includes the A100 and the free-tier T4’s successors but not the T4 itself. On a T4, float16 works for most modern models and fails on a few. On CPU, use float32; half precision there is usually emulated and slower.

model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16)
print(next(model.parameters()).dtype)

Check the second line. Loading without specifying gives you float32 on some paths regardless of what config.json says the weights were saved as, and doubling your memory by accident is a routine mistake.

Weights are not the whole bill

Three other things occupy memory during inference:

1. Activations: the intermediate tensors of the forward pass. Modest for a single short sequence, and they scale with batch size and sequence length.
2. The KV cache: for generation, the stored keys and values for every token so far. This is the one that grows, and the formula is worth carrying:

KV bytes ≈ 2 × layers × kv_heads × head_dim × sequence_length × batch × bytes_per_element

For an 8B model with 32 layers, 8 key-value heads and a head dimension of 128 at bfloat16, one 4,096-token sequence costs roughly 0.5 GB. Eight concurrent conversations at that length cost about 4 GB: comparable to the weights of a quantized model.

1. Framework overhead: CUDA context, allocator fragmentation, the library itself. Budget one to two gigabytes and stop being surprised.

So the honest working estimate for inference is:

weights + KV cache + 1 to 2 GB

And for training, roughly four times the weights before you count the data, because you also hold gradients and two optimiser states per parameter under Adam. That is why a model you can run happily is a model you cannot fine-tune fully on the same card, and why the adapter methods in the training module exist.

Grouped-query attention changed the numbers

Older models gave every attention head its own keys and values. Most models since 2023 share them across groups, which cuts the KV cache by a factor of four to eight. It is why a modern 8B model handles a long context on hardware where a 2022 7B model would not. The relevant fields are num_attention_heads and num_key_value_heads in config.json; when the second is much smaller than the first, the cache is cheap.

Reading your own usage

import torch
torch.cuda.reset_peak_memory_stats()
_ = model.generate(**enc, max_new_tokens=256)
print(f"peak: {torch.cuda.max_memory_allocated()/1e9:.2f} GB")

max_memory_allocated is the number that matters, not the current allocation: out-of-memory errors happen at the peak, which may occur for a few milliseconds inside one layer. On CPU, psutil.Process().memory_info().rss gives the equivalent.

Do this now

Pick three models you might use: a small one, a mid one, and one that is clearly too large. For each, write down the bf16 weight size, the int4 size, and the KV cache for 4,096 tokens. Compare against the memory of the machine you actually have. Most people discover the constraint is the cache, not the weights.