{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"will-it-fit","title":"Will it fit: the memory budget of a running model","url":"https://addaly.com/learn/how-llms-work/will-it-fit","position":60,"of":78,"minutes":9,"answer":"A running model's memory is weights plus a key-value cache that grows with every token of every concurrent user.","keyIdea":"A running model's memory is weights plus a key-value cache that grows with every token of every concurrent user, so the context and concurrency you can afford is free memory after the weights load divided by the per-token cache cost."},"body":{"format":"markdown","text":"## Four line items\n\n\"Will it fit?\" has an answer you can compute before downloading anything. Four things occupy memory when a model runs, and you can estimate each.\n\n**Weights.** Parameters times bytes per parameter. Two bytes in bf16, one in int8, and about 0.55 to 0.6 at four bits once the scales that the next lesson explains are included. An 8-billion-parameter model is 16, 8 or roughly 5 gigabytes.\n\n**The key-value cache.** Module two gave the per-token formula (two, times layers, times key-value heads, times head dimension, times bytes) and the result of 131 kilobytes per token for that 8-billion model in bf16. Multiply by the context length, and again by the number of sequences being served at once.\n\n**Activations and workspace.** The intermediate vectors of the forward pass and the scratch memory attention needs. For inference these are modest (hundreds of megabytes to a few gigabytes depending on batch size) because, unlike training, nothing has to be kept for a backward pass. Modern attention kernels keep this from growing with the square of the sequence length.\n\n**Framework overhead and fragmentation.** The GPU runtime takes a few hundred megabytes just to exist. Allocators fragment. Serving engines reserve a fraction of the card up front (vLLM's default is 90 per cent) and refuse to start if the model plus a minimum cache does not fit inside it.\n\nThe rule of thumb that falls out: **gigabytes of weights ≈ parameters in billions × bytes per parameter, plus ten per cent**, and then everything left over is your cache budget.\n\n## Worked example: the 24-gigabyte card\n\nAn 8-billion model in bf16 on a consumer card with 24 gigabytes.\n\n- Weights: 16.0 GB.\n- Overhead: about 0.5 GB.\n- Left for cache: 7.5 GB, which is 7.5 billion ÷ 131,000 ≈ **57,000 tokens** of cache in total.\n\nOne user with a 32,000-token conversation fits. Two do. A fourth arrives with the same context and the total wanted is 128,000 tokens of cache: an out-of-memory error, from the line item that grows per user while the weights stay shared.\n\nNow the same model at four bits: about 5 GB of weights, 18.5 GB free, around **140,000 tokens** of cache. Quantise the cache itself to 8-bit (nearly free in quality) and it is 280,000. The card that served two users now serves eight.\n\n> Figure: The same 24-gigabyte card, twice. Layers, top to bottom: bf16 weights: 16.0 GB (Loaded once and shared by every user on the card); Framework overhead: 0.5 GB (The runtime takes a few hundred megabytes simply to exist); Left for cache: 7.5 GB, about 57,000 tokens (At 131 KB a token: three 32,000-token conversations, and the fourth fails); Four-bit weights instead: 5.0 GB (Same model, about 0.6 bytes a parameter once the group scales are counted); Left for cache: 18.5 GB, about 140,000 tokens (Quantise the cache to 8-bit as well and it is 280,000: the card now serves eight). The weights are a fixed cost and the cache is the variable one, so the concurrency a card can afford is free memory after the weights load, divided by the per-token cache cost. That division is a better guide than anything printed on a model card.\n\n## Worked example: the 70-billion model\n\nIn bf16, 140 gigabytes of weights. That does not fit on one 80-gigabyte card and does not comfortably fit on two, once cache is added. This is why 70-billion-class models are usually served either quantised or split across several cards, which is a later lesson.\n\nAt four bits: about 40 gigabytes. One 80-gigabyte card holds it with 35 gigabytes to spare. The 70-billion architecture has 80 layers, eight key-value heads and a head dimension of 128, so a token costs 2 × 80 × 1,024 × 2 = 327 kilobytes in bf16, and 35 gigabytes is about 107,000 tokens of cache. A single card, a four-bit 70-billion model, and a handful of long conversations at once: that is a real, common deployment.\n\nA laptop with 64 gigabytes of unified memory also holds the four-bit 70-billion model. The previous lesson says how fast: 400 gigabytes a second over 40 gigabytes is a bound of ten tokens a second. It runs. It is not quick.\n\n## The mixture-of-experts trap\n\nModule two separated a sparse model's total parameters from its active parameters. Memory uses the **total**: every expert must be resident, because the next token may route to any of them. A model advertised as \"17 billion active, 109 billion total\" needs memory for 109 billion (55 gigabytes at four bits) while its decode speed follows the 17 billion that are read per token. Both numbers matter, and they answer different questions.\n\n## When it does not fit, in order\n\n1. **Quantise the weights.** Halves or quarters the largest line item. The next lesson says what it costs.\n2. **Quantise the cache.** 8-bit is close to free. 4-bit costs measurable long-context recall.\n3. **Cap the context.** Serving engines take a maximum sequence length; setting it to what your use actually needs frees cache for more users.\n4. **Offload layers to system RAM.** `llama.cpp` and its front-ends let you place some layers on the GPU and the rest on the CPU. Speed falls steeply, because the offloaded layers are read at CPU-memory bandwidth or across a bus a tenth as fast as GPU memory. As a rough guide, offloading a tenth of the layers costs a third or more of the speed.\n5. **Split across GPUs.** The lesson on parallelism.\n6. **Use a smaller model.** Often the right answer, and the distillation lesson explains why small models became viable.\n\n## Training is a different budget\n\nFor contrast: module five's sixteen bytes per parameter means the 8-billion model needs 128 gigabytes of optimiser state and gradients before any activations. QLoRA brings fine-tuning it down to under ten. The gap between that and the five gigabytes of inference is why running models is a laptop task and training them is not.\n\n## Measure after you load\n\nOn a GPU, `nvidia-smi` shows what is taken once the model is up; a serving engine's startup log reports how many cache blocks it allocated. On a Mac, Activity Monitor's memory pressure tells you when you have crossed into swap, which is when tokens per second collapses. The number to watch is free memory *after* the weights load. That, divided by the per-token cache cost, is the context and concurrency you can actually afford, and it is a better guide than anything on a model card."},"check":{"question":"A 24-gigabyte GPU runs an 8-billion-parameter model in bf16 comfortably for one user at 8,000 tokens of context, but throws an out-of-memory error when a fourth concurrent user with the same context connects. Which line item overflowed?","options":[{"text":"The key-value cache, which grows with every concurrent sequence while the weights stay shared.","correct":true},{"text":"The weights, because the serving engine loads a separate copy of the model for each connected user.","correct":false,"tempting":"A natural assumption from how ordinary applications scale, but weights are loaded once and shared; only the per-sequence state multiplies."},{"text":"The attention activations, which grow with the square of the context length and so quadruple when the total context across users doubles.","correct":false,"tempting":"Attention was once quadratic in memory, which makes this sound informed, but modern kernels removed that, and the total context here grew linearly with users in any case."},{"text":"The framework's fragmentation overhead, which roughly doubles each time a new connection is opened and is only reclaimed when the server restarts.","correct":false,"tempting":"Fragmentation is real and does waste memory, but it is a fixed slice of the card rather than something that scales with connections; the per-user growth is the cache."}]},"prev":{"slug":"the-bandwidth-wall","title":"The bandwidth wall: why decode speed is a division","url":"https://addaly.com/learn/how-llms-work/the-bandwidth-wall"},"next":{"slug":"quantisation-mechanics","title":"Quantisation: what the numbers lose","url":"https://addaly.com/learn/how-llms-work/quantisation-mechanics"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Will it fit: the memory budget of a running model', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/will-it-fit (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Will it fit: the memory budget of a running model","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"A running model’s memory is weights plus a key-value cache that grows with every token of every concurrent user, so the context and concurrency you can afford is free memory after the weights load divided by the per-token cache cost.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"The same 24-gigabyte card, twice","layers":[{"label":"bf16 weights: 16.0 GB","detail":"Loaded once and shared by every user on the card"},{"label":"Framework overhead: 0.5 GB","detail":"The runtime takes a few hundred megabytes simply to exist"},{"label":"Left for cache: 7.5 GB, about 57,000 tokens","detail":"At 131 KB a token: three 32,000-token conversations, and the fourth fails"},{"label":"Four-bit weights instead: 5.0 GB","detail":"Same model, about 0.6 bytes a parameter once the group scales are counted"},{"label":"Left for cache: 18.5 GB, about 140,000 tokens","detail":"Quantise the cache to 8-bit as well and it is 280,000: the card now serves eight"}],"caption":"The weights are a fixed cost and the cache is the variable one, so the concurrency a card can afford is free memory after the weights load, divided by the per-token cache cost. That division is a better guide than anything printed on a model card."},"alt":"Figure: The same 24-gigabyte card, twice. Layers, top to bottom: bf16 weights: 16.0 GB (Loaded once and shared by every user on the card); Framework overhead: 0.5 GB (The runtime takes a few hundred megabytes simply to exist); Left for cache: 7.5 GB, about 57,000 tokens (At 131 KB a token: three 32,000-token conversations, and the fourth fails); Four-bit weights instead: 5.0 GB (Same model, about 0.6 bytes a parameter once the group scales are counted); Left for cache: 18.5 GB, about 140,000 tokens (Quantise the cache to 8-bit as well and it is 280,000: the card now serves eight). The weights are a fixed cost and the cache is the variable one, so the concurrency a card can afford is free memory after the weights load, divided by the per-token cache cost. That division is a better guide than anything printed on a model card."}},"blocks":[{"t":"h2","id":"four-line-items","runs":[{"text":"Four line items"}]},{"t":"p","runs":[{"text":"“Will it fit?” has an answer you can compute before downloading anything. Four things occupy memory when a model runs, and you can estimate each."}]},{"t":"p","runs":[{"text":"Weights.","b":true},{"text":" Parameters times bytes per parameter. Two bytes in bf16, one in int8, and about 0.55 to 0.6 at four bits once the scales that the next lesson explains are included. An 8-billion-parameter model is 16, 8 or roughly 5 gigabytes."}]},{"t":"p","runs":[{"text":"The key-value cache.","b":true},{"text":" Module two gave the per-token formula (two, times layers, times key-value heads, times head dimension, times bytes) and the result of 131 kilobytes per token for that 8-billion model in bf16. Multiply by the context length, and again by the number of sequences being served at once."}]},{"t":"p","runs":[{"text":"Activations and workspace.","b":true},{"text":" The intermediate vectors of the forward pass and the scratch memory attention needs. For inference these are modest (hundreds of megabytes to a few gigabytes depending on batch size) because, unlike training, nothing has to be kept for a backward pass. Modern attention kernels keep this from growing with the square of the sequence length."}]},{"t":"p","runs":[{"text":"Framework overhead and fragmentation.","b":true},{"text":" The GPU runtime takes a few hundred megabytes just to exist. Allocators fragment. Serving engines reserve a fraction of the card up front (vLLM’s default is 90 per cent) and refuse to start if the model plus a minimum cache does not fit inside it."}]},{"t":"p","runs":[{"text":"The rule of thumb that falls out: "},{"text":"gigabytes of weights ≈ parameters in billions × bytes per parameter, plus ten per cent","b":true},{"text":", and then everything left over is your cache budget."}]},{"t":"h2","id":"worked-example-the-24-gigabyte-card","runs":[{"text":"Worked example: the 24-gigabyte card"}]},{"t":"p","runs":[{"text":"An 8-billion model in bf16 on a consumer card with 24 gigabytes."}]},{"t":"ul","items":[[{"text":"Weights: 16.0 GB."}],[{"text":"Overhead: about 0.5 GB."}],[{"text":"Left for cache: 7.5 GB, which is 7.5 billion ÷ 131,000 ≈ "},{"text":"57,000 tokens","b":true},{"text":" of cache in total."}]]},{"t":"p","runs":[{"text":"One user with a 32,000-token conversation fits. Two do. A fourth arrives with the same context and the total wanted is 128,000 tokens of cache: an out-of-memory error, from the line item that grows per user while the weights stay shared."}]},{"t":"p","runs":[{"text":"Now the same model at four bits: about 5 GB of weights, 18.5 GB free, around "},{"text":"140,000 tokens","b":true},{"text":" of cache. Quantise the cache itself to 8-bit (nearly free in quality) and it is 280,000. The card that served two users now serves eight."}]},{"t":"h2","id":"worked-example-the-70-billion-model","runs":[{"text":"Worked example: the 70-billion model"}]},{"t":"p","runs":[{"text":"In bf16, 140 gigabytes of weights. That does not fit on one 80-gigabyte card and does not comfortably fit on two, once cache is added. This is why 70-billion-class models are usually served either quantised or split across several cards, which is a later lesson."}]},{"t":"p","runs":[{"text":"At four bits: about 40 gigabytes. One 80-gigabyte card holds it with 35 gigabytes to spare. The 70-billion architecture has 80 layers, eight key-value heads and a head dimension of 128, so a token costs 2 × 80 × 1,024 × 2 = 327 kilobytes in bf16, and 35 gigabytes is about 107,000 tokens of cache. A single card, a four-bit 70-billion model, and a handful of long conversations at once: that is a real, common deployment."}]},{"t":"p","runs":[{"text":"A laptop with 64 gigabytes of unified memory also holds the four-bit 70-billion model. The previous lesson says how fast: 400 gigabytes a second over 40 gigabytes is a bound of ten tokens a second. It runs. It is not quick."}]},{"t":"h2","id":"the-mixture-of-experts-trap","runs":[{"text":"The mixture-of-experts trap"}]},{"t":"p","runs":[{"text":"Module two separated a sparse model’s total parameters from its active parameters. Memory uses the "},{"text":"total","b":true},{"text":": every expert must be resident, because the next token may route to any of them. A model advertised as “17 billion active, 109 billion total” needs memory for 109 billion (55 gigabytes at four bits) while its decode speed follows the 17 billion that are read per token. Both numbers matter, and they answer different questions."}]},{"t":"h2","id":"when-it-does-not-fit-in-order","runs":[{"text":"When it does not fit, in order"}]},{"t":"ol","start":1,"items":[[{"text":"Quantise the weights.","b":true},{"text":" Halves or quarters the largest line item. The next lesson says what it costs."}],[{"text":"Quantise the cache.","b":true},{"text":" 8-bit is close to free. 4-bit costs measurable long-context recall."}],[{"text":"Cap the context.","b":true},{"text":" Serving engines take a maximum sequence length; setting it to what your use actually needs frees cache for more users."}],[{"text":"Offload layers to system RAM.","b":true},{"text":" "},{"text":"llama.cpp","code":true},{"text":" and its front-ends let you place some layers on the GPU and the rest on the CPU. Speed falls steeply, because the offloaded layers are read at CPU-memory bandwidth or across a bus a tenth as fast as GPU memory. As a rough guide, offloading a tenth of the layers costs a third or more of the speed."}],[{"text":"Split across GPUs.","b":true},{"text":" The lesson on parallelism."}],[{"text":"Use a smaller model.","b":true},{"text":" Often the right answer, and the distillation lesson explains why small models became viable."}]]},{"t":"h2","id":"training-is-a-different-budget","runs":[{"text":"Training is a different budget"}]},{"t":"p","runs":[{"text":"For contrast: module five’s sixteen bytes per parameter means the 8-billion model needs 128 gigabytes of optimiser state and gradients before any activations. QLoRA brings fine-tuning it down to under ten. The gap between that and the five gigabytes of inference is why running models is a laptop task and training them is not."}]},{"t":"h2","id":"measure-after-you-load","runs":[{"text":"Measure after you load"}]},{"t":"p","runs":[{"text":"On a GPU, "},{"text":"nvidia-smi","code":true},{"text":" shows what is taken once the model is up; a serving engine’s startup log reports how many cache blocks it allocated. On a Mac, Activity Monitor’s memory pressure tells you when you have crossed into swap, which is when tokens per second collapses. The number to watch is free memory "},{"text":"after","i":true},{"text":" the weights load. That, divided by the per-token cache cost, is the context and concurrency you can actually afford, and it is a better guide than anything on a model card."}]}],"check":{"question":[{"text":"A 24-gigabyte GPU runs an 8-billion-parameter model in bf16 comfortably for one user at 8,000 tokens of context, but throws an out-of-memory error when a fourth concurrent user with the same context connects. Which line item overflowed?"}],"options":[{"text":[{"text":"The key-value cache, which grows with every concurrent sequence while the weights stay shared."}],"correct":true,"tempting":null},{"text":[{"text":"The weights, because the serving engine loads a separate copy of the model for each connected user."}],"correct":false,"tempting":[{"text":"A natural assumption from how ordinary applications scale, but weights are loaded once and shared; only the per-sequence state multiplies."}]},{"text":[{"text":"The attention activations, which grow with the square of the context length and so quadruple when the total context across users doubles."}],"correct":false,"tempting":[{"text":"Attention was once quadratic in memory, which makes this sound informed, but modern kernels removed that, and the total context here grew linearly with users in any case."}]},{"text":[{"text":"The framework’s fragmentation overhead, which roughly doubles each time a new connection is opened and is only reclaimed when the server restarts."}],"correct":false,"tempting":[{"text":"Fragmentation is real and does waste memory, but it is a fixed slice of the card rather than something that scales with connections; the per-user growth is the cache."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"the-bandwidth-wall","title":"The bandwidth wall: why decode speed is a division","minutes":9},"next":{"slug":"quantisation-mechanics","title":"Quantisation: what the numbers lose","minutes":10}}}