{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"where-the-memory-goes","title":"Where the memory actually goes","url":"https://addaly.com/learn/fine-tuning/where-the-memory-goes","position":16,"of":84,"minutes":10,"answer":"Memory is weights, gradients, optimiser state, activations and logits, and at long context the quadratic attention matrix and the vocabulary-sized logits tensor are usually the two surprises.","keyIdea":"Memory is weights, gradients, optimiser state, activations and logits, and at long context the quadratic attention matrix and the vocabulary-sized logits tensor are usually the two surprises, both removable with flash attention and a fused cross-entropy before you touch batch size."},"body":{"format":"markdown","text":"## Five consumers, in order of surprise\n\nWhen a run reports `CUDA out of memory`, people reach for a smaller batch size. Sometimes that is right. Often the batch was not the problem, and knowing which of five consumers is eating the card saves an afternoon.\n\n**1. Weights.** Parameters times bytes per parameter. A 7B model is 14 GB in bf16, 28 GB in fp32, about 3.9 GB in 4-bit NF4. Fixed for the whole run, easy to compute, rarely the surprise.\n\n**2. Gradients.** Same size as the *trainable* weights, in the gradient dtype. Full fine-tuning a 7B in bf16: another 14 GB. LoRA with 20M trainable parameters: 80 MB. This single line is why parameter-efficient methods exist.\n\n**3. Optimiser state.** 8 bytes per trainable parameter for AdamW. 56 GB for a full 7B; 160 MB for the LoRA.\n\n**4. Activations.** The intermediate tensors kept from the forward pass so the backward pass can use them. This is the one people underestimate, and it scales with things they change casually:\n\n```\nactivations ≈ batch × seq_len × hidden × layers × bytes × c\n```\n\nwhere `c` is a small constant: roughly 10 to 20 depending on architecture and what is fused. For a 7B model (32 layers, hidden 4,096) at batch 4 and sequence 2,048 in bf16, that is on the order of 10 to 20 GB. It grows linearly in both batch and sequence length, so doubling your context doubles it.\n\n**5. Attention, if it is not flash attention.** A naive attention implementation materialises the full score matrix: `batch × heads × seq² × bytes`. At sequence 8,192 with 32 heads in bf16 that is about 4.3 GB **per layer**, transiently. This is the quadratic term everybody has heard of and few have costed. FlashAttention computes attention in tiles without ever writing the matrix, turning that term from quadratic to linear in memory. Switch it on (`attn_implementation=\"flash_attention_2\"`, or the PyTorch SDPA backend) before you touch anything else at long context.\n\n## Gradient checkpointing, and what it actually trades\n\nGradient checkpointing throws away most activations after the forward pass and recomputes them during the backward pass from a small number of saved boundaries. Memory for activations drops from roughly linear in depth to roughly the square root of it; compute rises by about 30%.\n\nIt is nearly always worth it on a memory-bound run, because the memory you free usually buys a larger batch, and a larger batch recovers much of the lost throughput.\n\nOne trap: with LoRA and gradient checkpointing together, you may need `model.enable_input_require_grads()`. Without it the checkpointed segments see no input requiring grad, the graph is not built, and you get a run where the loss never changes and nothing obviously errors. This has cost a great many people an evening.\n\n## The logits, again\n\nFrom the first lesson of this module: `batch × seq × vocab` can exceed the weights. At batch 4, sequence 2,048 and a 128k vocabulary that is 1 billion values: 2 GB in bf16, 4 GB in fp32, and typically both exist at once during the loss computation.\n\nFused cross-entropy kernels avoid materialising it. Liger Kernel and cut cross-entropy are the common options, both free and both a one-line change in TRL. On large-vocabulary models this is frequently the single largest saving available.\n\n## The diagnostic sequence\n\nWhen you are out of memory, work down this list rather than guessing:\n\n1. Print `torch.cuda.max_memory_allocated()` and compare against the card. If allocated is far below capacity, you have **fragmentation**, not a shortage: set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` and try again.\n2. Is flash attention on? At sequence length over 4,000 this is usually the answer.\n3. Is gradient checkpointing on?\n4. Is the cross-entropy fused?\n5. Only now reduce batch size, raising gradient accumulation to keep the effective batch constant.\n6. Then reduce sequence length, and measure how many of your examples get truncated by the new limit, because a max length that cuts 30% of your completions is a data problem, not a memory fix.\n\n## The estimate before you rent\n\nDo the arithmetic on paper first. For QLoRA on a 7B at sequence 2,048, batch 2: 3.9 GB weights, negligible gradients and optimiser state, perhaps 5 to 8 GB of activations with checkpointing off, 1 GB of logits. Around 11 GB, which fits a 16 GB T4 with room and comfortably fits a 24 GB card.\n\nIf the arithmetic says 70 GB and you were planning to use a 24 GB card, you learned that for free.\n\n> Figure: An 11 GB QLoRA run, by consumer. Bars (GB): Activations kept for the backward pass 6; Frozen base weights in 4-bit NF4 3.9; Logits, 128k vocabulary 1; Adapter gradients and optimiser state 0.2. A 7B base at sequence 2,048, batch 2. Lowering the batch size touches one row of this chart, and it is often not the largest one."},"check":{"question":"A QLoRA run on a 24 GB card fails with out-of-memory at sequence length 8,192 but succeeds at 2,048. The 4-bit base weights are under 4 GB and the adapter is tiny. Which term grew fastest, and why?","options":[{"text":"The optimiser state, which is allocated per position in the sequence during long-context training.","correct":false,"tempting":"Invents a dependency on sequence length; optimiser state is per parameter and is identical at both lengths."},{"text":"The attention score matrix, which is quadratic in sequence length without flash attention.","correct":true},{"text":"The 4-bit base weights, which must be dequantised to bf16 for long sequences and so quadruple in size.","correct":false,"tempting":"Dequantisation happens per matrix multiplication regardless of sequence length, and is transient rather than scaling with context."},{"text":"The gradients, which are proportional to the number of tokens processed and so grew four-fold.","correct":false,"tempting":"Gradients are the same shape as the trainable parameters and do not depend on how many tokens produced them."}]},"prev":{"slug":"precision-and-mixed-precision","title":"Precision, and why bf16 ended a class of bugs","url":"https://addaly.com/learn/fine-tuning/precision-and-mixed-precision"},"next":{"slug":"epochs-and-the-loss-curve","title":"Epochs, overfitting and checkpoint selection","url":"https://addaly.com/learn/fine-tuning/epochs-and-the-loss-curve"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the memory actually goes', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/where-the-memory-goes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the memory actually goes","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"Memory is weights, gradients, optimiser state, activations and logits, and at long context the quadratic attention matrix and the vocabulary-sized logits tensor are usually the two surprises, both removable with flash attention and a fused cross-entropy before you touch batch size.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB","title":"An 11 GB QLoRA run, by consumer","series":[{"label":"Activations kept for the backward pass","value":6,"highlight":true},{"label":"Frozen base weights in 4-bit NF4","value":3.9},{"label":"Logits, 128k vocabulary","value":1},{"label":"Adapter gradients and optimiser state","value":0.2}],"caption":"A 7B base at sequence 2,048, batch 2. Lowering the batch size touches one row of this chart, and it is often not the largest one."},"alt":"Figure: An 11 GB QLoRA run, by consumer. Bars (GB): Activations kept for the backward pass 6; Frozen base weights in 4-bit NF4 3.9; Logits, 128k vocabulary 1; Adapter gradients and optimiser state 0.2. A 7B base at sequence 2,048, batch 2. Lowering the batch size touches one row of this chart, and it is often not the largest one."}},"blocks":[{"t":"h2","id":"five-consumers-in-order-of-surprise","runs":[{"text":"Five consumers, in order of surprise"}]},{"t":"p","runs":[{"text":"When a run reports "},{"text":"CUDA out of memory","code":true},{"text":", people reach for a smaller batch size. Sometimes that is right. Often the batch was not the problem, and knowing which of five consumers is eating the card saves an afternoon."}]},{"t":"p","runs":[{"text":"1. Weights.","b":true},{"text":" Parameters times bytes per parameter. A 7B model is 14 GB in bf16, 28 GB in fp32, about 3.9 GB in 4-bit NF4. Fixed for the whole run, easy to compute, rarely the surprise."}]},{"t":"p","runs":[{"text":"2. Gradients.","b":true},{"text":" Same size as the "},{"text":"trainable","i":true},{"text":" weights, in the gradient dtype. Full fine-tuning a 7B in bf16: another 14 GB. LoRA with 20M trainable parameters: 80 MB. This single line is why parameter-efficient methods exist."}]},{"t":"p","runs":[{"text":"3. Optimiser state.","b":true},{"text":" 8 bytes per trainable parameter for AdamW. 56 GB for a full 7B; 160 MB for the LoRA."}]},{"t":"p","runs":[{"text":"4. Activations.","b":true},{"text":" The intermediate tensors kept from the forward pass so the backward pass can use them. This is the one people underestimate, and it scales with things they change casually:"}]},{"t":"code","lang":"","label":"Text","text":"activations ≈ batch × seq_len × hidden × layers × bytes × c"},{"t":"p","runs":[{"text":"where "},{"text":"c","code":true},{"text":" is a small constant: roughly 10 to 20 depending on architecture and what is fused. For a 7B model (32 layers, hidden 4,096) at batch 4 and sequence 2,048 in bf16, that is on the order of 10 to 20 GB. It grows linearly in both batch and sequence length, so doubling your context doubles it."}]},{"t":"p","runs":[{"text":"5. Attention, if it is not flash attention.","b":true},{"text":" A naive attention implementation materialises the full score matrix: "},{"text":"batch × heads × seq² × bytes","code":true},{"text":". At sequence 8,192 with 32 heads in bf16 that is about 4.3 GB "},{"text":"per layer","b":true},{"text":", transiently. This is the quadratic term everybody has heard of and few have costed. FlashAttention computes attention in tiles without ever writing the matrix, turning that term from quadratic to linear in memory. Switch it on ("},{"text":"attn_implementation=\"flash_attention_2\"","code":true},{"text":", or the PyTorch SDPA backend) before you touch anything else at long context."}]},{"t":"h2","id":"gradient-checkpointing-and-what-it-actually-trades","runs":[{"text":"Gradient checkpointing, and what it actually trades"}]},{"t":"p","runs":[{"text":"Gradient checkpointing throws away most activations after the forward pass and recomputes them during the backward pass from a small number of saved boundaries. Memory for activations drops from roughly linear in depth to roughly the square root of it; compute rises by about 30%."}]},{"t":"p","runs":[{"text":"It is nearly always worth it on a memory-bound run, because the memory you free usually buys a larger batch, and a larger batch recovers much of the lost throughput."}]},{"t":"p","runs":[{"text":"One trap: with LoRA and gradient checkpointing together, you may need "},{"text":"model.enable_input_require_grads()","code":true},{"text":". Without it the checkpointed segments see no input requiring grad, the graph is not built, and you get a run where the loss never changes and nothing obviously errors. This has cost a great many people an evening."}]},{"t":"h2","id":"the-logits-again","runs":[{"text":"The logits, again"}]},{"t":"p","runs":[{"text":"From the first lesson of this module: "},{"text":"batch × seq × vocab","code":true},{"text":" can exceed the weights. At batch 4, sequence 2,048 and a 128k vocabulary that is 1 billion values: 2 GB in bf16, 4 GB in fp32, and typically both exist at once during the loss computation."}]},{"t":"p","runs":[{"text":"Fused cross-entropy kernels avoid materialising it. Liger Kernel and cut cross-entropy are the common options, both free and both a one-line change in TRL. On large-vocabulary models this is frequently the single largest saving available."}]},{"t":"h2","id":"the-diagnostic-sequence","runs":[{"text":"The diagnostic sequence"}]},{"t":"p","runs":[{"text":"When you are out of memory, work down this list rather than guessing:"}]},{"t":"ol","start":1,"items":[[{"text":"Print "},{"text":"torch.cuda.max_memory_allocated()","code":true},{"text":" and compare against the card. If allocated is far below capacity, you have "},{"text":"fragmentation","b":true},{"text":", not a shortage: set "},{"text":"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True","code":true},{"text":" and try again."}],[{"text":"Is flash attention on? At sequence length over 4,000 this is usually the answer."}],[{"text":"Is gradient checkpointing on?"}],[{"text":"Is the cross-entropy fused?"}],[{"text":"Only now reduce batch size, raising gradient accumulation to keep the effective batch constant."}],[{"text":"Then reduce sequence length, and measure how many of your examples get truncated by the new limit, because a max length that cuts 30% of your completions is a data problem, not a memory fix."}]]},{"t":"h2","id":"the-estimate-before-you-rent","runs":[{"text":"The estimate before you rent"}]},{"t":"p","runs":[{"text":"Do the arithmetic on paper first. For QLoRA on a 7B at sequence 2,048, batch 2: 3.9 GB weights, negligible gradients and optimiser state, perhaps 5 to 8 GB of activations with checkpointing off, 1 GB of logits. Around 11 GB, which fits a 16 GB T4 with room and comfortably fits a 24 GB card."}]},{"t":"p","runs":[{"text":"If the arithmetic says 70 GB and you were planning to use a 24 GB card, you learned that for free."}]}],"check":{"question":[{"text":"A QLoRA run on a 24 GB card fails with out-of-memory at sequence length 8,192 but succeeds at 2,048. The 4-bit base weights are under 4 GB and the adapter is tiny. Which term grew fastest, and why?"}],"options":[{"text":[{"text":"The optimiser state, which is allocated per position in the sequence during long-context training."}],"correct":false,"tempting":[{"text":"Invents a dependency on sequence length; optimiser state is per parameter and is identical at both lengths."}]},{"text":[{"text":"The attention score matrix, which is quadratic in sequence length without flash attention."}],"correct":true,"tempting":null},{"text":[{"text":"The 4-bit base weights, which must be dequantised to bf16 for long sequences and so quadruple in size."}],"correct":false,"tempting":[{"text":"Dequantisation happens per matrix multiplication regardless of sequence length, and is transient rather than scaling with context."}]},{"text":[{"text":"The gradients, which are proportional to the number of tokens processed and so grew four-fold."}],"correct":false,"tempting":[{"text":"Gradients are the same shape as the trainable parameters and do not depend on how many tokens produced them."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"precision-and-mixed-precision","title":"Precision, and why bf16 ended a class of bugs","minutes":9},"next":{"slug":"epochs-and-the-loss-curve","title":"Epochs, overfitting and checkpoint selection","minutes":9}}}