{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"memory-arithmetic","title":"Full, LoRA, QLoRA, and the Memory Arithmetic","url":"https://addaly.com/learn/fine-tuning/memory-arithmetic","position":19,"of":84,"minutes":10,"answer":"Full fine-tuning costs about 16 bytes per parameter; LoRA charges that only on the 1% you train.","keyIdea":"Full fine-tuning costs about 16 bytes per parameter; LoRA charges that only on the 1% you train."},"body":{"format":"markdown","text":"## Count the bytes\n\nEvery training setup holds four things in memory: weights, gradients, optimizer state, and activations. Only the last depends on batch size, which is why \"just lower the batch size\" so often fails to save you.\n\nFor a model with P parameters, standard mixed-precision training with AdamW costs roughly:\n\n| Component | Bytes per parameter |\n|---|---|\n| Weights (bf16) | 2 |\n| Gradients (bf16) | 2 |\n| fp32 master weights | 4 |\n| Adam first moment | 4 |\n| Adam second moment | 4 |\n| **Total** | **16** |\n\nA 7B model: 7e9 x 16 = **112 GB**, before a single activation. That does not fit on one 80 GB card. Full fine-tuning of a 7B is a multi-GPU job with FSDP or DeepSpeed, or an 8-bit-optimizer-plus-offload arrangement that is slow and fiddly. This is why almost nobody does it, and why the rest of this lesson exists.\n\n## LoRA: freeze the base, train a small correction\n\nLoRA freezes W and learns a low-rank update, W + BA. For a weight of shape d_in x d_out, B and A hold r(d_in + d_out) parameters. A 4096 x 4096 projection at r=16 is 16 x 8192 = 131,072 parameters against 16.7M: about 0.8%. Across a 7B model with adapters on every linear layer at r=16, you train roughly 20-40M parameters.\n\nThe 16-byte tax now applies only to those. The frozen base still sits in memory at 2 bytes per parameter.\n\n- 7B base in bf16: **14 GB**\n- adapters, their gradients and optimizer state: **under 1 GB**\n- activations with gradient checkpointing, sequence 1024, small batch: **2-5 GB**\n\nCall it 18-20 GB. That fits a 24 GB card (RTX 3090, 4090, A10G). It does not fit a 16 GB T4.\n\n## QLoRA: quantize the frozen base\n\nThe base is frozen, so it does not need to be precise: it only has to produce good forward activations. QLoRA stores it in 4-bit NF4, about 0.5 bytes per parameter plus quantization constants, and dequantizes each block on the fly during the forward pass. Adapters stay in 16-bit.\n\n- 7B base in NF4: **about 4 GB**\n- everything else as before: **3-6 GB**\n\nSeven to ten gigabytes total. That fits a free Colab T4, a Kaggle P100, or a 12 GB consumer card. The cost is speed: expect 20-40% lower throughput than 16-bit LoRA because of the dequantization work.\n\n```python\nfrom transformers import BitsAndBytesConfig\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=\"bfloat16\",  # use float16 on a T4; Turing has no bf16\n    bnb_4bit_use_double_quant=True,     # quantizes the quantization constants too\n)\n```\n\n## Scaling to your model\n\nRough rule: QLoRA needs about `0.6 x P` bytes for the base, plus 3-6 GB of working memory.\n\n- 3B → about 2 GB base → comfortable on 8 GB\n- 7-8B → about 4.5 GB → fits 16 GB\n- 13B → about 8 GB → tight on 16 GB, fine on 24 GB\n- 70B → about 40 GB → needs one 48 GB or 80 GB card\n\n## If you have 8 GB of RAM and no GPU\n\nThe honest thing: you cannot usefully fine-tune a 7B model on a CPU. A step that takes 0.4 seconds on a rented GPU takes minutes, and a run that takes twenty minutes takes a fortnight.\n\nWhat you can do, and it is not a consolation prize:\n\n- **Fine-tune a small model locally.** A 135M-600M parameter model trains on CPU in hours on a few hundred examples. Every part of the pipeline (chat templates, loss masking, evaluation) is identical to the 7B version, and those are the parts where mistakes actually happen.\n- **Use free GPU hours.** Kaggle gives roughly 30 GPU-hours a week on a T4 or P100, with sessions that survive better than Colab's free tier. Colab free works but disconnects; checkpoint every few hundred steps and push somewhere persistent.\n- **Rent.** A 24 GB card runs $0.30-0.80 an hour on the spot-style marketplaces. Most LoRA runs in this course cost less than a coffee. Prices move constantly; check on the day.\n\nThe memory table is the thing to remember. Batch size touches one row of it.\n\n> Figure: A 7B model in memory, three ways. Bars (GB): Full fine-tuning with AdamW 112; LoRA r=16, bf16 base 19; QLoRA, NF4 base 8. Sixteen bytes per parameter against roughly half a byte. The free T4's sixteen gigabytes is the line that decides what a learner with no budget can train."},"check":{"question":"You have one free Colab T4: 16 GB, no bf16 support. You want to fine-tune a 7B model. Which plan is workable, and for the right reason?","options":[{"text":"Full fine-tuning, with batch size 1 and gradient checkpointing to keep memory down.","correct":false,"tempting":"Batch size and checkpointing really do control activation memory, but optimizer state is about 12 bytes per parameter regardless of batch: roughly 84 GB for a 7B model."},{"text":"Sixteen-bit LoRA, since only the adapters carry gradients and optimizer state.","correct":false,"tempting":"The trainable side is indeed tiny, but the frozen base still occupies 14 GB in 16-bit, leaving almost nothing for activations, and a T4 has no bf16 at all."},{"text":"QLoRA: the frozen base in 4-bit is about 4 GB, leaving room for adapters and checkpointed activations, at maybe 20-40% lower throughput.","correct":true},{"text":"Any of them, using gradient accumulation to spread the memory across steps.","correct":false,"tempting":"Accumulation does simulate a large batch on small hardware, but it only splits activation memory; weights, gradients and optimizer state stay resident throughout."}]},"prev":{"slug":"regularisation-that-helps","title":"Regularisation that helps, and the kind that does not","url":"https://addaly.com/learn/fine-tuning/regularisation-that-helps"},"next":{"slug":"what-lora-is-doing","title":"What LoRA is actually doing","url":"https://addaly.com/learn/fine-tuning/what-lora-is-doing"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Full, LoRA, QLoRA, and the Memory Arithmetic', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/memory-arithmetic (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Full, LoRA, QLoRA, and the Memory Arithmetic","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"Full fine-tuning costs about 16 bytes per parameter; LoRA charges that only on the 1% you train.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 3.1","figure":{"type":"bars","unit":"GB","title":"A 7B model in memory, three ways","series":[{"label":"Full fine-tuning with AdamW","value":112},{"label":"LoRA r=16, bf16 base","value":19},{"label":"QLoRA, NF4 base","value":8,"highlight":true}],"caption":"Sixteen bytes per parameter against roughly half a byte. The free T4’s sixteen gigabytes is the line that decides what a learner with no budget can train."},"alt":"Figure: A 7B model in memory, three ways. Bars (GB): Full fine-tuning with AdamW 112; LoRA r=16, bf16 base 19; QLoRA, NF4 base 8. Sixteen bytes per parameter against roughly half a byte. The free T4’s sixteen gigabytes is the line that decides what a learner with no budget can train."}},"blocks":[{"t":"h2","id":"count-the-bytes","runs":[{"text":"Count the bytes"}]},{"t":"p","runs":[{"text":"Every training setup holds four things in memory: weights, gradients, optimizer state, and activations. Only the last depends on batch size, which is why “just lower the batch size” so often fails to save you."}]},{"t":"p","runs":[{"text":"For a model with P parameters, standard mixed-precision training with AdamW costs roughly:"}]},{"t":"table","head":[[{"text":"Component"}],[{"text":"Bytes per parameter"}]],"align":["left","left"],"rows":[[[{"text":"Weights (bf16)"}],[{"text":"2"}]],[[{"text":"Gradients (bf16)"}],[{"text":"2"}]],[[{"text":"fp32 master weights"}],[{"text":"4"}]],[[{"text":"Adam first moment"}],[{"text":"4"}]],[[{"text":"Adam second moment"}],[{"text":"4"}]],[[{"text":"Total","b":true}],[{"text":"16","b":true}]]]},{"t":"p","runs":[{"text":"A 7B model: 7e9 x 16 = "},{"text":"112 GB","b":true},{"text":", before a single activation. That does not fit on one 80 GB card. Full fine-tuning of a 7B is a multi-GPU job with FSDP or DeepSpeed, or an 8-bit-optimizer-plus-offload arrangement that is slow and fiddly. This is why almost nobody does it, and why the rest of this lesson exists."}]},{"t":"h2","id":"lora-freeze-the-base-train-a-small-correction","runs":[{"text":"LoRA: freeze the base, train a small correction"}]},{"t":"p","runs":[{"text":"LoRA freezes W and learns a low-rank update, W + BA. For a weight of shape d_in x d_out, B and A hold r(d_in + d_out) parameters. A 4096 x 4096 projection at r=16 is 16 x 8192 = 131,072 parameters against 16.7M: about 0.8%. Across a 7B model with adapters on every linear layer at r=16, you train roughly 20-40M parameters."}]},{"t":"p","runs":[{"text":"The 16-byte tax now applies only to those. The frozen base still sits in memory at 2 bytes per parameter."}]},{"t":"ul","items":[[{"text":"7B base in bf16: "},{"text":"14 GB","b":true}],[{"text":"adapters, their gradients and optimizer state: "},{"text":"under 1 GB","b":true}],[{"text":"activations with gradient checkpointing, sequence 1024, small batch: "},{"text":"2-5 GB","b":true}]]},{"t":"p","runs":[{"text":"Call it 18-20 GB. That fits a 24 GB card (RTX 3090, 4090, A10G). It does not fit a 16 GB T4."}]},{"t":"h2","id":"qlora-quantize-the-frozen-base","runs":[{"text":"QLoRA: quantize the frozen base"}]},{"t":"p","runs":[{"text":"The base is frozen, so it does not need to be precise: it only has to produce good forward activations. QLoRA stores it in 4-bit NF4, about 0.5 bytes per parameter plus quantization constants, and dequantizes each block on the fly during the forward pass. Adapters stay in 16-bit."}]},{"t":"ul","items":[[{"text":"7B base in NF4: "},{"text":"about 4 GB","b":true}],[{"text":"everything else as before: "},{"text":"3-6 GB","b":true}]]},{"t":"p","runs":[{"text":"Seven to ten gigabytes total. That fits a free Colab T4, a Kaggle P100, or a 12 GB consumer card. The cost is speed: expect 20-40% lower throughput than 16-bit LoRA because of the dequantization work."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import BitsAndBytesConfig\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=\"bfloat16\",  # use float16 on a T4; Turing has no bf16\n    bnb_4bit_use_double_quant=True,     # quantizes the quantization constants too\n)"},{"t":"h2","id":"scaling-to-your-model","runs":[{"text":"Scaling to your model"}]},{"t":"p","runs":[{"text":"Rough rule: QLoRA needs about "},{"text":"0.6 x P","code":true},{"text":" bytes for the base, plus 3-6 GB of working memory."}]},{"t":"ul","items":[[{"text":"3B → about 2 GB base → comfortable on 8 GB"}],[{"text":"7-8B → about 4.5 GB → fits 16 GB"}],[{"text":"13B → about 8 GB → tight on 16 GB, fine on 24 GB"}],[{"text":"70B → about 40 GB → needs one 48 GB or 80 GB card"}]]},{"t":"h2","id":"if-you-have-8-gb-of-ram-and-no-gpu","runs":[{"text":"If you have 8 GB of RAM and no GPU"}]},{"t":"p","runs":[{"text":"The honest thing: you cannot usefully fine-tune a 7B model on a CPU. A step that takes 0.4 seconds on a rented GPU takes minutes, and a run that takes twenty minutes takes a fortnight."}]},{"t":"p","runs":[{"text":"What you can do, and it is not a consolation prize:"}]},{"t":"ul","items":[[{"text":"Fine-tune a small model locally.","b":true},{"text":" A 135M-600M parameter model trains on CPU in hours on a few hundred examples. Every part of the pipeline (chat templates, loss masking, evaluation) is identical to the 7B version, and those are the parts where mistakes actually happen."}],[{"text":"Use free GPU hours.","b":true},{"text":" Kaggle gives roughly 30 GPU-hours a week on a T4 or P100, with sessions that survive better than Colab’s free tier. Colab free works but disconnects; checkpoint every few hundred steps and push somewhere persistent."}],[{"text":"Rent.","b":true},{"text":" A 24 GB card runs $0.30-0.80 an hour on the spot-style marketplaces. Most LoRA runs in this course cost less than a coffee. Prices move constantly; check on the day."}]]},{"t":"p","runs":[{"text":"The memory table is the thing to remember. Batch size touches one row of it."}]}],"check":{"question":[{"text":"You have one free Colab T4: 16 GB, no bf16 support. You want to fine-tune a 7B model. Which plan is workable, and for the right reason?"}],"options":[{"text":[{"text":"Full fine-tuning, with batch size 1 and gradient checkpointing to keep memory down."}],"correct":false,"tempting":[{"text":"Batch size and checkpointing really do control activation memory, but optimizer state is about 12 bytes per parameter regardless of batch: roughly 84 GB for a 7B model."}]},{"text":[{"text":"Sixteen-bit LoRA, since only the adapters carry gradients and optimizer state."}],"correct":false,"tempting":[{"text":"The trainable side is indeed tiny, but the frozen base still occupies 14 GB in 16-bit, leaving almost nothing for activations, and a T4 has no bf16 at all."}]},{"text":[{"text":"QLoRA: the frozen base in 4-bit is about 4 GB, leaving room for adapters and checkpointed activations, at maybe 20-40% lower throughput."}],"correct":true,"tempting":null},{"text":[{"text":"Any of them, using gradient accumulation to spread the memory across steps."}],"correct":false,"tempting":[{"text":"Accumulation does simulate a large batch on small hardware, but it only splits activation memory; weights, gradients and optimizer state stay resident throughout."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"regularisation-that-helps","title":"Regularisation that helps, and the kind that does not","minutes":8},"next":{"slug":"what-lora-is-doing","title":"What LoRA is actually doing","minutes":9}}}