{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"dtype-and-the-memory-bill","title":"The memory arithmetic you can do in your head","url":"https://addaly.com/learn/hugging-face/dtype-and-the-memory-bill","position":20,"of":75,"minutes":9,"answer":"Weights cost parameters times bytes-per-dtype.","keyIdea":"Weights cost parameters times bytes-per-dtype, but the bill that decides whether generation fits is weights plus a KV cache that grows with sequence length and batch. And bfloat16 is preferred over float16 not for size but because it keeps float32's exponent range and so does not overflow to NaN."},"body":{"format":"markdown","text":"## One multiplication decides most of it\n\n> **weights in bytes = parameters × bytes per parameter**\n\nThe bytes per parameter come from the dtype:\n\n- `float32`: 4 bytes. The default if you say nothing, and almost never what you want.\n- `float16` / `bfloat16`: 2 bytes. The normal choice on a GPU.\n- `int8`: 1 byte.\n- `int4`: about 0.5 bytes, plus a small overhead for scales.\n\nSo a 7-billion-parameter model is 28 GB at float32, 14 GB at bfloat16, 7 GB at int8 and roughly 4 GB at int4. A 1.5B model is 3 GB at bfloat16 and fits comfortably in a free Colab session with room to work. A 70B model is 140 GB at bfloat16, which is two 80 GB cards, and about 40 GB at int4, which is one.\n\nYou can now look at any repository name and know, before downloading anything, whether it is worth downloading.\n\n## float16 and bfloat16 are not interchangeable\n\nBoth take two bytes. They spend those bytes differently.\n\n> Figure: A seven-billion-parameter model, by dtype. Bars (GB): float32, four bytes a parameter 28; bfloat16, two bytes a parameter 14; int8, one byte a parameter 7; 4-bit nf4, about half a byte 4; KV cache, one 4,096-token sequence 0.5. The first four bars are the weights alone. The fifth is what a single 4,096-token conversation adds on top of whichever of them you chose: eight concurrent conversations of that length cost about four gigabytes, which is the whole of the quantized model again. Most people discover the constraint is the cache, not the weights.\n\n- **float16** gives more bits to precision and has a narrow exponent range, with a maximum around 65,504.\n- **bfloat16** keeps float32's exponent range and sacrifices precision.\n\nFor neural networks the exponent range matters more, because activations occasionally spike. A value above 65,504 becomes infinity in float16, infinity times zero becomes NaN, and the NaN spreads through the network within one layer. This is the mechanism behind the T5 and older-checkpoint failures from the debugging lesson.\n\nPractical rule: **use bfloat16 if the hardware supports it**: Ampere-generation GPUs and newer, which includes the A100 and the free-tier T4's successors but not the T4 itself. On a T4, float16 works for most modern models and fails on a few. On CPU, use float32; half precision there is usually emulated and slower.\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16)\nprint(next(model.parameters()).dtype)\n```\n\nCheck the second line. Loading without specifying gives you float32 on some paths regardless of what `config.json` says the weights were saved as, and doubling your memory by accident is a routine mistake.\n\n## Weights are not the whole bill\n\nThree other things occupy memory during inference:\n\n1. **Activations**: the intermediate tensors of the forward pass. Modest for a single short sequence, and they scale with batch size and sequence length.\n2. **The KV cache**: for generation, the stored keys and values for every token so far. This is the one that grows, and the formula is worth carrying:\n\n   > KV bytes ≈ 2 × layers × kv_heads × head_dim × sequence_length × batch × bytes_per_element\n\n   For an 8B model with 32 layers, 8 key-value heads and a head dimension of 128 at bfloat16, one 4,096-token sequence costs roughly 0.5 GB. Eight concurrent conversations at that length cost about 4 GB: comparable to the weights of a quantized model.\n3. **Framework overhead**: CUDA context, allocator fragmentation, the library itself. Budget one to two gigabytes and stop being surprised.\n\nSo the honest working estimate for inference is:\n\n> **weights + KV cache + 1 to 2 GB**\n\nAnd for training, roughly **four times the weights** before you count the data, because you also hold gradients and two optimiser states per parameter under Adam. That is why a model you can run happily is a model you cannot fine-tune fully on the same card, and why the adapter methods in the training module exist.\n\n## Grouped-query attention changed the numbers\n\nOlder models gave every attention head its own keys and values. Most models since 2023 share them across groups, which cuts the KV cache by a factor of four to eight. It is why a modern 8B model handles a long context on hardware where a 2022 7B model would not. The relevant fields are `num_attention_heads` and `num_key_value_heads` in `config.json`; when the second is much smaller than the first, the cache is cheap.\n\n## Reading your own usage\n\n```python\nimport torch\ntorch.cuda.reset_peak_memory_stats()\n_ = model.generate(**enc, max_new_tokens=256)\nprint(f\"peak: {torch.cuda.max_memory_allocated()/1e9:.2f} GB\")\n```\n\n`max_memory_allocated` is the number that matters, not the current allocation: out-of-memory errors happen at the peak, which may occur for a few milliseconds inside one layer. On CPU, `psutil.Process().memory_info().rss` gives the equivalent.\n\n## Do this now\n\nPick three models you might use: a small one, a mid one, and one that is clearly too large. For each, write down the bf16 weight size, the int4 size, and the KV cache for 4,096 tokens. Compare against the memory of the machine you actually have. Most people discover the constraint is the cache, not the weights."},"check":{"question":"A 7B model in bfloat16 loads fine on a 16 GB GPU and serves single short prompts, then runs out of memory once several users hold long conversations. What grew?","options":[{"text":"The KV cache, which stores keys and values for every token in every active sequence and scales with length and concurrency.","correct":true},{"text":"The weights, because transformers promotes cached layers to float32 during multi-request serving to preserve numerical stability.","correct":false,"tempting":"Invents a promotion step; weight dtype is fixed at load and does not change with traffic."},{"text":"Activation memory, which grows with the number of tokens already generated in the same way the cache does.","correct":false,"tempting":"Names a real consumer, but activations are transient per forward pass and do not accumulate across a conversation."},{"text":"Allocator fragmentation, which is the usual cause when a model loads successfully and later fails under identical shapes.","correct":false,"tempting":"Fragmentation is real but secondary here, and the shapes are not identical: conversations are getting longer."}]},"prev":{"slug":"running-it-locally","title":"Your disk fills up before your patience does","url":"https://addaly.com/learn/hugging-face/running-it-locally"},"next":{"slug":"device-map-and-offloading","title":"Spreading a model across whatever you have","url":"https://addaly.com/learn/hugging-face/device-map-and-offloading"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The memory arithmetic you can do in your head', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/dtype-and-the-memory-bill (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The memory arithmetic you can do in your head","courseTitle":"Hugging Face, End to End","module":{"index":3,"slug":"running-it-on-what-you-have","title":"Running it on the hardware you actually have"},"answer":null,"keyIdea":"Weights cost parameters times bytes-per-dtype, but the bill that decides whether generation fits is weights plus a KV cache that grows with sequence length and batch. And bfloat16 is preferred over float16 not for size but because it keeps float32’s exponent range and so does not overflow to NaN.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB","title":"A seven-billion-parameter model, by dtype","series":[{"label":"float32, four bytes a parameter","value":28},{"label":"bfloat16, two bytes a parameter","value":14},{"label":"int8, one byte a parameter","value":7},{"label":"4-bit nf4, about half a byte","value":4,"highlight":true},{"label":"KV cache, one 4,096-token sequence","value":0.5}],"caption":"The first four bars are the weights alone. The fifth is what a single 4,096-token conversation adds on top of whichever of them you chose: eight concurrent conversations of that length cost about four gigabytes, which is the whole of the quantized model again. Most people discover the constraint is the cache, not the weights."},"alt":"Figure: A seven-billion-parameter model, by dtype. Bars (GB): float32, four bytes a parameter 28; bfloat16, two bytes a parameter 14; int8, one byte a parameter 7; 4-bit nf4, about half a byte 4; KV cache, one 4,096-token sequence 0.5. The first four bars are the weights alone. The fifth is what a single 4,096-token conversation adds on top of whichever of them you chose: eight concurrent conversations of that length cost about four gigabytes, which is the whole of the quantized model again. Most people discover the constraint is the cache, not the weights."}},"blocks":[{"t":"h2","id":"one-multiplication-decides-most-of-it","runs":[{"text":"One multiplication decides most of it"}]},{"t":"quote","runs":[{"text":"weights in bytes = parameters × bytes per parameter","b":true}]},{"t":"p","runs":[{"text":"The bytes per parameter come from the dtype:"}]},{"t":"ul","items":[[{"text":"float32","code":true},{"text":": 4 bytes. The default if you say nothing, and almost never what you want."}],[{"text":"float16","code":true},{"text":" / "},{"text":"bfloat16","code":true},{"text":": 2 bytes. The normal choice on a GPU."}],[{"text":"int8","code":true},{"text":": 1 byte."}],[{"text":"int4","code":true},{"text":": about 0.5 bytes, plus a small overhead for scales."}]]},{"t":"p","runs":[{"text":"So a 7-billion-parameter model is 28 GB at float32, 14 GB at bfloat16, 7 GB at int8 and roughly 4 GB at int4. A 1.5B model is 3 GB at bfloat16 and fits comfortably in a free Colab session with room to work. A 70B model is 140 GB at bfloat16, which is two 80 GB cards, and about 40 GB at int4, which is one."}]},{"t":"p","runs":[{"text":"You can now look at any repository name and know, before downloading anything, whether it is worth downloading."}]},{"t":"h2","id":"float16-and-bfloat16-are-not-interchangeable","runs":[{"text":"float16 and bfloat16 are not interchangeable"}]},{"t":"p","runs":[{"text":"Both take two bytes. They spend those bytes differently."}]},{"t":"ul","items":[[{"text":"float16","b":true},{"text":" gives more bits to precision and has a narrow exponent range, with a maximum around 65,504."}],[{"text":"bfloat16","b":true},{"text":" keeps float32’s exponent range and sacrifices precision."}]]},{"t":"p","runs":[{"text":"For neural networks the exponent range matters more, because activations occasionally spike. A value above 65,504 becomes infinity in float16, infinity times zero becomes NaN, and the NaN spreads through the network within one layer. This is the mechanism behind the T5 and older-checkpoint failures from the debugging lesson."}]},{"t":"p","runs":[{"text":"Practical rule: "},{"text":"use bfloat16 if the hardware supports it","b":true},{"text":": Ampere-generation GPUs and newer, which includes the A100 and the free-tier T4’s successors but not the T4 itself. On a T4, float16 works for most modern models and fails on a few. On CPU, use float32; half precision there is usually emulated and slower."}]},{"t":"code","lang":"python","label":"Python","text":"model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16)\nprint(next(model.parameters()).dtype)"},{"t":"p","runs":[{"text":"Check the second line. Loading without specifying gives you float32 on some paths regardless of what "},{"text":"config.json","code":true},{"text":" says the weights were saved as, and doubling your memory by accident is a routine mistake."}]},{"t":"h2","id":"weights-are-not-the-whole-bill","runs":[{"text":"Weights are not the whole bill"}]},{"t":"p","runs":[{"text":"Three other things occupy memory during inference:"}]},{"t":"ol","start":1,"items":[[{"text":"Activations","b":true},{"text":": the intermediate tensors of the forward pass. Modest for a single short sequence, and they scale with batch size and sequence length."}],[{"text":"The KV cache","b":true},{"text":": for generation, the stored keys and values for every token so far. This is the one that grows, and the formula is worth carrying:"}]]},{"t":"quote","runs":[{"text":"KV bytes ≈ 2 × layers × kv_heads × head_dim × sequence_length × batch × bytes_per_element"}]},{"t":"p","runs":[{"text":"For an 8B model with 32 layers, 8 key-value heads and a head dimension of 128 at bfloat16, one 4,096-token sequence costs roughly 0.5 GB. Eight concurrent conversations at that length cost about 4 GB: comparable to the weights of a quantized model."}]},{"t":"ol","start":3,"items":[[{"text":"Framework overhead","b":true},{"text":": CUDA context, allocator fragmentation, the library itself. Budget one to two gigabytes and stop being surprised."}]]},{"t":"p","runs":[{"text":"So the honest working estimate for inference is:"}]},{"t":"quote","runs":[{"text":"weights + KV cache + 1 to 2 GB","b":true}]},{"t":"p","runs":[{"text":"And for training, roughly "},{"text":"four times the weights","b":true},{"text":" before you count the data, because you also hold gradients and two optimiser states per parameter under Adam. That is why a model you can run happily is a model you cannot fine-tune fully on the same card, and why the adapter methods in the training module exist."}]},{"t":"h2","id":"grouped-query-attention-changed-the-numbers","runs":[{"text":"Grouped-query attention changed the numbers"}]},{"t":"p","runs":[{"text":"Older models gave every attention head its own keys and values. Most models since 2023 share them across groups, which cuts the KV cache by a factor of four to eight. It is why a modern 8B model handles a long context on hardware where a 2022 7B model would not. The relevant fields are "},{"text":"num_attention_heads","code":true},{"text":" and "},{"text":"num_key_value_heads","code":true},{"text":" in "},{"text":"config.json","code":true},{"text":"; when the second is much smaller than the first, the cache is cheap."}]},{"t":"h2","id":"reading-your-own-usage","runs":[{"text":"Reading your own usage"}]},{"t":"code","lang":"python","label":"Python","text":"import torch\ntorch.cuda.reset_peak_memory_stats()\n_ = model.generate(**enc, max_new_tokens=256)\nprint(f\"peak: {torch.cuda.max_memory_allocated()/1e9:.2f} GB\")"},{"t":"p","runs":[{"text":"max_memory_allocated","code":true},{"text":" is the number that matters, not the current allocation: out-of-memory errors happen at the peak, which may occur for a few milliseconds inside one layer. On CPU, "},{"text":"psutil.Process().memory_info().rss","code":true},{"text":" gives the equivalent."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Pick three models you might use: a small one, a mid one, and one that is clearly too large. For each, write down the bf16 weight size, the int4 size, and the KV cache for 4,096 tokens. Compare against the memory of the machine you actually have. Most people discover the constraint is the cache, not the weights."}]}],"check":{"question":[{"text":"A 7B model in bfloat16 loads fine on a 16 GB GPU and serves single short prompts, then runs out of memory once several users hold long conversations. What grew?"}],"options":[{"text":[{"text":"The KV cache, which stores keys and values for every token in every active sequence and scales with length and concurrency."}],"correct":true,"tempting":null},{"text":[{"text":"The weights, because transformers promotes cached layers to float32 during multi-request serving to preserve numerical stability."}],"correct":false,"tempting":[{"text":"Invents a promotion step; weight dtype is fixed at load and does not change with traffic."}]},{"text":[{"text":"Activation memory, which grows with the number of tokens already generated in the same way the cache does."}],"correct":false,"tempting":[{"text":"Names a real consumer, but activations are transient per forward pass and do not accumulate across a conversation."}]},{"text":[{"text":"Allocator fragmentation, which is the usual cause when a model loads successfully and later fails under identical shapes."}],"correct":false,"tempting":[{"text":"Fragmentation is real but secondary here, and the shapes are not identical: conversations are getting longer."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"running-it-locally","title":"Your disk fills up before your patience does","minutes":11},"next":{"slug":"device-map-and-offloading","title":"Spreading a model across whatever you have","minutes":8}}}