{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"qlora-and-memory","title":"Fitting a fine-tune in 16 GB","url":"https://addaly.com/learn/open-models/qlora-and-memory","position":53,"of":71,"minutes":9,"answer":"Training memory is weights plus gradients plus optimiser state plus activations, and QLoRA collapses the first three by freezing a 4-bit base and training a small adapter, leaving activations, controlled by sequence length.","keyIdea":"Training memory is weights plus gradients plus optimiser state plus activations, and QLoRA collapses the first three by freezing a 4-bit base and training a small adapter, leaving activations, controlled by sequence length, batch size and gradient checkpointing, as the dial you actually turn."},"body":{"format":"markdown","text":"## Where the memory goes in training\n\nInference needs weights plus a cache. Training needs considerably more, and knowing the four components tells you exactly which dial to turn when it will not fit.\n\nFor a full fine-tune of a 7B model in mixed precision:\n\n- **Weights**, 2 bytes each: 14 GB\n- **Gradients**, one per weight, 2 bytes: 14 GB\n- **Optimiser state**, Adam keeps two moments in 32-bit plus a master copy: roughly 12 bytes per parameter, 84 GB\n- **Activations**, everything saved for the backward pass: depends on batch size and sequence length, often tens of GB\n\nThat is 110 GB or more, which means several datacentre GPUs. It is why almost nobody full-tunes a 7B model.\n\n> Figure: Full fine-tune of a 7B: where the memory goes. Bars (GB): Optimiser state, about 12 bytes per parameter 84; Activations, at batch 8 and 2,048 tokens 20; Weights at 2 bytes 14; Gradients at 2 bytes 14. About 110 GB, which is why almost nobody full-tunes a 7B. QLoRA freezes a 4-bit base at 3.5 GB and trains an adapter, taking the first, third and fourth bars to under five in total, leaving activations, the one bar you control, with sequence length, batch size and gradient checkpointing.\n\n## What QLoRA removes\n\nQLoRA combines three ideas and the result fits on a free notebook GPU.\n\n**The base is quantised to 4-bit and frozen.** Weights drop from 14 GB to about 3.5 GB. Because the base is never updated, quantisation error does not accumulate: it is a fixed distortion the adapter learns against.\n\n**Only the adapter trains.** With 40 million trainable parameters instead of 7 billion, gradients and optimiser state fall from 98 GB to under a gigabyte.\n\n**Paged optimiser state** moves optimiser memory to CPU RAM when a spike would otherwise overflow, which turns an occasional crash into an occasional slowdown.\n\nThe format is NF4, a 4-bit representation shaped for normally distributed values, plus double quantisation, which quantises the block scales themselves for a further small saving.\n\nNew total for a 7B: about 3.5 GB of base, under 1 GB of adapter and optimiser, plus activations. On a 16 GB card that leaves 11 GB for activations, which is the number you now control.\n\n## Activations are the dial you actually turn\n\nActivation memory scales with **batch size × sequence length**, roughly linearly in both. When a run fails with an out-of-memory error, this is nearly always what to change.\n\n**Gradient checkpointing** is the biggest single lever. Instead of storing every intermediate activation for the backward pass, it stores a few and recomputes the rest. Memory falls a lot; compute rises by roughly 30%. Turn it on by default.\n\n**Gradient accumulation** gives you a large effective batch with a small real one. Batch size 1 with 16 accumulation steps has the same optimisation behaviour as batch size 16 and a sixteenth of the activation memory.\n\n**Sequence length** is the one people forget. Training at 4,096 tokens when your examples are 600 tokens long wastes memory in proportion. Set `max_seq_length` to your actual data, and use example packing to fill sequences efficiently.\n\n```python\nfrom transformers import BitsAndBytesConfig, TrainingArguments\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True, bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=\"bfloat16\", bnb_4bit_use_double_quant=True,\n)\nargs = TrainingArguments(\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,      # effective batch 16\n    gradient_checkpointing=True,\n    optim=\"paged_adamw_8bit\",\n    bf16=True, learning_rate=2e-4, num_train_epochs=2,\n)\n```\n\n## What fits where\n\n**16 GB (a free notebook T4 or a consumer card):** QLoRA on a 7B or 8B at sequence length 1,024 to 2,048. A 13B is tight and works at short sequence lengths.\n\n**24 GB (a 3090 or 4090):** QLoRA on a 13B comfortably, a 34B at short sequences, an 8B at long sequence lengths.\n\n**48 GB:** QLoRA on a 70B at modest sequence length. This is where a single rented card becomes interesting.\n\n**80 GB:** 70B QLoRA with room, or full fine-tuning of a 7B.\n\n## The cost in quality\n\nQLoRA against full fine-tuning loses very little on typical instruction-tuning tasks: the published comparisons find the gap small enough to be inside evaluation noise for most work. The base being 4-bit does cost something on tasks that were already fragile at that quantisation, which by the earlier lesson means long context and lower-resource languages.\n\nIf you have the memory, `load_in_4bit=False` with a 16-bit frozen base and LoRA adapters is slightly better and still cheap. Use QLoRA because the memory is not there, not out of habit.\n\n> The practical order when a run will not fit: gradient checkpointing on, sequence length down to your real data, batch size 1 with accumulation, then 4-bit base. Reducing rank is the last thing to try, because it is the only one that reduces what the run can learn."},"check":{"question":"A QLoRA run on a 16 GB GPU fails with an out-of-memory error partway through the first epoch, having loaded successfully. Which change addresses the actual cause?","options":[{"text":"Reduce the LoRA rank from 32 to 8, cutting trainable parameters by four.","correct":false,"tempting":"Saves well under a gigabyte of gradient and optimiser state while removing capacity, so it is the least effective change available."},{"text":"Switch from NF4 to 8-bit quantisation of the base, which is more stable.","correct":false,"tempting":"Moves in the wrong direction: an 8-bit base doubles the frozen weight memory."},{"text":"Enable gradient checkpointing and cut the sequence length to match the data, since activation memory grows with batch size and sequence length.","correct":true},{"text":"Increase gradient accumulation steps, since more steps spread the memory over time.","correct":false,"tempting":"Accumulation helps only if the per-device batch size falls with it; raising the step count alone changes nothing about peak memory."}]},"prev":{"slug":"lora-mechanics","title":"What a LoRA actually is","url":"https://addaly.com/learn/open-models/lora-mechanics"},"next":{"slug":"the-fine-tuning-dataset","title":"Five hundred examples, done properly","url":"https://addaly.com/learn/open-models/the-fine-tuning-dataset"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fitting a fine-tune in 16 GB', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/qlora-and-memory (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fitting a fine-tune in 16 GB","courseTitle":"The Open Model Ecosystem","module":{"index":6,"slug":"adapting-a-model","title":"Adapting a model to your work"},"answer":null,"keyIdea":"Training memory is weights plus gradients plus optimiser state plus activations, and QLoRA collapses the first three by freezing a 4-bit base and training a small adapter, leaving activations, controlled by sequence length, batch size and gradient checkpointing, as the dial you actually turn.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB","title":"Full fine-tune of a 7B: where the memory goes","series":[{"label":"Optimiser state, about 12 bytes per parameter","value":84,"highlight":true},{"label":"Activations, at batch 8 and 2,048 tokens","value":20},{"label":"Weights at 2 bytes","value":14},{"label":"Gradients at 2 bytes","value":14}],"caption":"About 110 GB, which is why almost nobody full-tunes a 7B. QLoRA freezes a 4-bit base at 3.5 GB and trains an adapter, taking the first, third and fourth bars to under five in total, leaving activations, the one bar you control, with sequence length, batch size and gradient checkpointing."},"alt":"Figure: Full fine-tune of a 7B: where the memory goes. Bars (GB): Optimiser state, about 12 bytes per parameter 84; Activations, at batch 8 and 2,048 tokens 20; Weights at 2 bytes 14; Gradients at 2 bytes 14. About 110 GB, which is why almost nobody full-tunes a 7B. QLoRA freezes a 4-bit base at 3.5 GB and trains an adapter, taking the first, third and fourth bars to under five in total, leaving activations, the one bar you control, with sequence length, batch size and gradient checkpointing."}},"blocks":[{"t":"h2","id":"where-the-memory-goes-in-training","runs":[{"text":"Where the memory goes in training"}]},{"t":"p","runs":[{"text":"Inference needs weights plus a cache. Training needs considerably more, and knowing the four components tells you exactly which dial to turn when it will not fit."}]},{"t":"p","runs":[{"text":"For a full fine-tune of a 7B model in mixed precision:"}]},{"t":"ul","items":[[{"text":"Weights","b":true},{"text":", 2 bytes each: 14 GB"}],[{"text":"Gradients","b":true},{"text":", one per weight, 2 bytes: 14 GB"}],[{"text":"Optimiser state","b":true},{"text":", Adam keeps two moments in 32-bit plus a master copy: roughly 12 bytes per parameter, 84 GB"}],[{"text":"Activations","b":true},{"text":", everything saved for the backward pass: depends on batch size and sequence length, often tens of GB"}]]},{"t":"p","runs":[{"text":"That is 110 GB or more, which means several datacentre GPUs. It is why almost nobody full-tunes a 7B model."}]},{"t":"h2","id":"what-qlora-removes","runs":[{"text":"What QLoRA removes"}]},{"t":"p","runs":[{"text":"QLoRA combines three ideas and the result fits on a free notebook GPU."}]},{"t":"p","runs":[{"text":"The base is quantised to 4-bit and frozen.","b":true},{"text":" Weights drop from 14 GB to about 3.5 GB. Because the base is never updated, quantisation error does not accumulate: it is a fixed distortion the adapter learns against."}]},{"t":"p","runs":[{"text":"Only the adapter trains.","b":true},{"text":" With 40 million trainable parameters instead of 7 billion, gradients and optimiser state fall from 98 GB to under a gigabyte."}]},{"t":"p","runs":[{"text":"Paged optimiser state","b":true},{"text":" moves optimiser memory to CPU RAM when a spike would otherwise overflow, which turns an occasional crash into an occasional slowdown."}]},{"t":"p","runs":[{"text":"The format is NF4, a 4-bit representation shaped for normally distributed values, plus double quantisation, which quantises the block scales themselves for a further small saving."}]},{"t":"p","runs":[{"text":"New total for a 7B: about 3.5 GB of base, under 1 GB of adapter and optimiser, plus activations. On a 16 GB card that leaves 11 GB for activations, which is the number you now control."}]},{"t":"h2","id":"activations-are-the-dial-you-actually-turn","runs":[{"text":"Activations are the dial you actually turn"}]},{"t":"p","runs":[{"text":"Activation memory scales with "},{"text":"batch size × sequence length","b":true},{"text":", roughly linearly in both. When a run fails with an out-of-memory error, this is nearly always what to change."}]},{"t":"p","runs":[{"text":"Gradient checkpointing","b":true},{"text":" is the biggest single lever. Instead of storing every intermediate activation for the backward pass, it stores a few and recomputes the rest. Memory falls a lot; compute rises by roughly 30%. Turn it on by default."}]},{"t":"p","runs":[{"text":"Gradient accumulation","b":true},{"text":" gives you a large effective batch with a small real one. Batch size 1 with 16 accumulation steps has the same optimisation behaviour as batch size 16 and a sixteenth of the activation memory."}]},{"t":"p","runs":[{"text":"Sequence length","b":true},{"text":" is the one people forget. Training at 4,096 tokens when your examples are 600 tokens long wastes memory in proportion. Set "},{"text":"max_seq_length","code":true},{"text":" to your actual data, and use example packing to fill sequences efficiently."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import BitsAndBytesConfig, TrainingArguments\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True, bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=\"bfloat16\", bnb_4bit_use_double_quant=True,\n)\nargs = TrainingArguments(\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,      # effective batch 16\n    gradient_checkpointing=True,\n    optim=\"paged_adamw_8bit\",\n    bf16=True, learning_rate=2e-4, num_train_epochs=2,\n)"},{"t":"h2","id":"what-fits-where","runs":[{"text":"What fits where"}]},{"t":"p","runs":[{"text":"16 GB (a free notebook T4 or a consumer card):","b":true},{"text":" QLoRA on a 7B or 8B at sequence length 1,024 to 2,048. A 13B is tight and works at short sequence lengths."}]},{"t":"p","runs":[{"text":"24 GB (a 3090 or 4090):","b":true},{"text":" QLoRA on a 13B comfortably, a 34B at short sequences, an 8B at long sequence lengths."}]},{"t":"p","runs":[{"text":"48 GB:","b":true},{"text":" QLoRA on a 70B at modest sequence length. This is where a single rented card becomes interesting."}]},{"t":"p","runs":[{"text":"80 GB:","b":true},{"text":" 70B QLoRA with room, or full fine-tuning of a 7B."}]},{"t":"h2","id":"the-cost-in-quality","runs":[{"text":"The cost in quality"}]},{"t":"p","runs":[{"text":"QLoRA against full fine-tuning loses very little on typical instruction-tuning tasks: the published comparisons find the gap small enough to be inside evaluation noise for most work. The base being 4-bit does cost something on tasks that were already fragile at that quantisation, which by the earlier lesson means long context and lower-resource languages."}]},{"t":"p","runs":[{"text":"If you have the memory, "},{"text":"load_in_4bit=False","code":true},{"text":" with a 16-bit frozen base and LoRA adapters is slightly better and still cheap. Use QLoRA because the memory is not there, not out of habit."}]},{"t":"quote","runs":[{"text":"The practical order when a run will not fit: gradient checkpointing on, sequence length down to your real data, batch size 1 with accumulation, then 4-bit base. Reducing rank is the last thing to try, because it is the only one that reduces what the run can learn."}]}],"check":{"question":[{"text":"A QLoRA run on a 16 GB GPU fails with an out-of-memory error partway through the first epoch, having loaded successfully. Which change addresses the actual cause?"}],"options":[{"text":[{"text":"Reduce the LoRA rank from 32 to 8, cutting trainable parameters by four."}],"correct":false,"tempting":[{"text":"Saves well under a gigabyte of gradient and optimiser state while removing capacity, so it is the least effective change available."}]},{"text":[{"text":"Switch from NF4 to 8-bit quantisation of the base, which is more stable."}],"correct":false,"tempting":[{"text":"Moves in the wrong direction: an 8-bit base doubles the frozen weight memory."}]},{"text":[{"text":"Enable gradient checkpointing and cut the sequence length to match the data, since activation memory grows with batch size and sequence length."}],"correct":true,"tempting":null},{"text":[{"text":"Increase gradient accumulation steps, since more steps spread the memory over time."}],"correct":false,"tempting":[{"text":"Accumulation helps only if the per-device batch size falls with it; raising the step count alone changes nothing about peak memory."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"lora-mechanics","title":"What a LoRA actually is","minutes":9},"next":{"slug":"the-fine-tuning-dataset","title":"Five hundred examples, done properly","minutes":9}}}