{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"qlora-on-one-card","title":"QLoRA: training a 7B on the card you have","url":"https://addaly.com/learn/running-models-yourself/qlora-on-one-card","position":76,"of":82,"minutes":10,"answer":"LoRA removes gradients and optimiser state for the frozen base and QLoRA loads that base in 4-bit NF4, bringing a 7B run onto a 12GB card. Sequence length is the memory knob that bites first.","keyIdea":"LoRA removes gradients and optimiser state for the frozen base and QLoRA loads that base in 4-bit NF4, bringing a 7B run onto a 12GB card. Sequence length is the memory knob that bites first, and an adapter trained against a 4-bit base must either be served on that base or re-evaluated after merging into 16-bit weights."},"body":{"format":"markdown","text":"## Where training memory actually goes\n\nInference needs weights plus a KV cache. Training needs four things, and three of them are invisible until you run out:\n\n- **Weights.** 7B at 16-bit is about 14GB.\n- **Gradients.** One per trainable parameter, same precision. Another 14GB for a full fine-tune.\n- **Optimiser state.** Adam keeps two moments per parameter, commonly in 32-bit: about 56GB for 7B.\n- **Activations.** Everything the backward pass needs, which scales with batch size and sequence length and is the term people forget.\n\nAdd them up and you already have 84GB before activations. Standard mixed-precision training also keeps a 32-bit master copy of the weights, so weights, gradients and optimiser state come to 16 to 18 bytes per parameter: roughly 112 to 126GB for a 7B model before activations. LoRA removes the second and third for the frozen base, because frozen parameters have no gradients and no optimiser state. You are left with 14GB of weights, a few megabytes of adapter gradients and states, and the activations.\n\nQLoRA removes most of the remainder by loading the base in 4-bit. The 14GB becomes roughly 4GB. Training happens in a higher precision, dequantising each weight block as it is needed for the forward and backward pass: the quantised copy is what lives in memory, not what the arithmetic uses.\n\n> Figure: Where the memory goes when you train a 7B model. Bars (GB of GPU memory): Full fine-tune: weights, gradients, optimiser 90; LoRA on a 16-bit base 20; QLoRA on a 4-bit NF4 base 10. A frozen parameter needs no gradient and no optimiser state, which is where seventy of those gigabytes go. Loading the base in 4-bit NF4 removes most of what is left. Sequence length is what bites next: halving it from 4,096 to 2,048 frees more memory than any other single change.\n\nThe practical result is that a 7B model becomes trainable on a 12GB card, and an 8B on a 16GB one, at moderate sequence lengths.\n\n## A configuration that fits\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",           # 4-bit NormalFloat, not plain int4\n    bnb_4bit_compute_dtype=torch.bfloat16,\n    bnb_4bit_use_double_quant=True,      # quantise the quantisation constants too\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"Qwen/Qwen2.5-7B-Instruct\", quantization_config=bnb, device_map=\"auto\")\nmodel.gradient_checkpointing_enable()\nmodel.config.use_cache = False           # incompatible with checkpointing\n```\n\nThree of those settings deserve explanation.\n\n**NF4** is a 4-bit format whose levels are spaced to match the roughly normal distribution of neural network weights, rather than spaced evenly. It costs the same memory as int4 and loses less. There is no reason to choose otherwise for this.\n\n**Double quantisation** quantises the per-block scaling constants as well as the weights, saving roughly a further 0.4 bits per parameter. On a 7B model that is a few hundred megabytes, which is often the difference between fitting and not.\n\n**Gradient checkpointing** discards intermediate activations during the forward pass and recomputes them during the backward pass. It cuts activation memory substantially and costs roughly 20 to 30% in speed. On a card that would otherwise not fit, that trade is not close.\n\n## Sequence length is the knob that bites\n\nActivation memory grows with sequence length, and attention's contribution grows faster than linearly. Halving `max_seq_length` from 4096 to 2048 typically frees more memory than any other single change, and it is the first thing to try when a run dies in the first few steps.\n\nThe cost is that the model never sees examples longer than that during training, which matters if your real inputs are long. The usual compromise: train at the length that covers the great majority of your examples, and accept that behaviour on the tail is untrained rather than learned.\n\nIf you must keep the length, reduce `per_device_train_batch_size` to 1 and recover the effective batch size with `gradient_accumulation_steps`. Sixteen accumulation steps at batch size 1 gives the same gradient as batch size 16, at the same memory as batch size 1 and roughly the same total time.\n\n## Use a paged optimiser\n\nMemory use during training is spiky, and a single spike kills a run that had been fine for two hours. `optim=\"paged_adamw_8bit\"` keeps the optimiser state in 8-bit and allows it to page to host memory under pressure. It is slightly slower and it converts a class of hard failures into a class of slowdowns.\n\n## The honest catch, which arrives at merge time\n\nQLoRA trains an adapter against a base that was quantised to 4-bit. If you then merge that adapter into the original 16-bit weights, the base the adapter was fitted to is not the base it is now attached to. The discrepancy is usually small and it is not always small.\n\nThere are two defensible paths. Serve the adapter on the same 4-bit base it was trained against, which is exactly consistent and is what a multi-adapter server does anyway. Or merge into the 16-bit base and then **re-run your evaluation**, treating the merged model as a new artefact rather than as the thing you tested. What is not defensible is merging and assuming the numbers carry over.\n\n## The free path\n\nThis entire lesson runs on a free hosted notebook tier. A 7B QLoRA at 2048 tokens on a few thousand examples fits within a free session's memory and time budget, with the usual caveats: mount your own storage so the adapter survives, checkpoint often, and expect the session to end without warning. For learning this technique, no hardware purchase is required at any point."},"check":{"question":"A QLoRA run on a 12GB card fails with an out-of-memory error a few steps in, after loading successfully. Which change addresses the actual cause most directly?","options":[{"text":"Reduce the LoRA rank from 32 to 8 to shrink the trainable parameters.","correct":false,"tempting":"Adapter parameters are a few megabytes at any usable rank, so this frees almost nothing compared with what activations consume."},{"text":"Disable gradient checkpointing, since recomputation increases peak memory during the backward pass.","correct":false,"tempting":"Reverses what checkpointing does: it trades speed for lower activation memory, so turning it off makes the failure arrive sooner."},{"text":"Switch from NF4 to plain int4 quantisation to save memory on the base weights.","correct":false,"tempting":"Both formats use four bits, so this changes quality rather than footprint, and the base was already loaded successfully."},{"text":"Reduce max sequence length, and recover the effective batch size with gradient accumulation.","correct":true}]},"prev":{"slug":"what-a-lora-actually-is","title":"What a LoRA actually is","url":"https://addaly.com/learn/running-models-yourself/what-a-lora-actually-is"},"next":{"slug":"building-the-dataset","title":"The dataset is the project","url":"https://addaly.com/learn/running-models-yourself/building-the-dataset"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'QLoRA: training a 7B on the card you have', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/qlora-on-one-card (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"QLoRA: training a 7B on the card you have","courseTitle":"Running Models Yourself","module":{"index":9,"slug":"making-it-yours","title":"Making it yours: adapters and training"},"answer":null,"keyIdea":"LoRA removes gradients and optimiser state for the frozen base and QLoRA loads that base in 4-bit NF4, bringing a 7B run onto a 12GB card. Sequence length is the memory knob that bites first, and an adapter trained against a 4-bit base must either be served on that base or re-evaluated after merging into 16-bit weights.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB of GPU memory","title":"Where the memory goes when you train a 7B model","series":[{"label":"Full fine-tune: weights, gradients, optimiser","value":90},{"label":"LoRA on a 16-bit base","value":20},{"label":"QLoRA on a 4-bit NF4 base","value":10,"highlight":true}],"caption":"A frozen parameter needs no gradient and no optimiser state, which is where seventy of those gigabytes go. Loading the base in 4-bit NF4 removes most of what is left. Sequence length is what bites next: halving it from 4,096 to 2,048 frees more memory than any other single change."},"alt":"Figure: Where the memory goes when you train a 7B model. Bars (GB of GPU memory): Full fine-tune: weights, gradients, optimiser 90; LoRA on a 16-bit base 20; QLoRA on a 4-bit NF4 base 10. A frozen parameter needs no gradient and no optimiser state, which is where seventy of those gigabytes go. Loading the base in 4-bit NF4 removes most of what is left. Sequence length is what bites next: halving it from 4,096 to 2,048 frees more memory than any other single change."}},"blocks":[{"t":"h2","id":"where-training-memory-actually-goes","runs":[{"text":"Where training memory actually goes"}]},{"t":"p","runs":[{"text":"Inference needs weights plus a KV cache. Training needs four things, and three of them are invisible until you run out:"}]},{"t":"ul","items":[[{"text":"Weights.","b":true},{"text":" 7B at 16-bit is about 14GB."}],[{"text":"Gradients.","b":true},{"text":" One per trainable parameter, same precision. Another 14GB for a full fine-tune."}],[{"text":"Optimiser state.","b":true},{"text":" Adam keeps two moments per parameter, commonly in 32-bit: about 56GB for 7B."}],[{"text":"Activations.","b":true},{"text":" Everything the backward pass needs, which scales with batch size and sequence length and is the term people forget."}]]},{"t":"p","runs":[{"text":"Add them up and you already have 84GB before activations. Standard mixed-precision training also keeps a 32-bit master copy of the weights, so weights, gradients and optimiser state come to 16 to 18 bytes per parameter: roughly 112 to 126GB for a 7B model before activations. LoRA removes the second and third for the frozen base, because frozen parameters have no gradients and no optimiser state. You are left with 14GB of weights, a few megabytes of adapter gradients and states, and the activations."}]},{"t":"p","runs":[{"text":"QLoRA removes most of the remainder by loading the base in 4-bit. The 14GB becomes roughly 4GB. Training happens in a higher precision, dequantising each weight block as it is needed for the forward and backward pass: the quantised copy is what lives in memory, not what the arithmetic uses."}]},{"t":"p","runs":[{"text":"The practical result is that a 7B model becomes trainable on a 12GB card, and an 8B on a 16GB one, at moderate sequence lengths."}]},{"t":"h2","id":"a-configuration-that-fits","runs":[{"text":"A configuration that fits"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",           # 4-bit NormalFloat, not plain int4\n    bnb_4bit_compute_dtype=torch.bfloat16,\n    bnb_4bit_use_double_quant=True,      # quantise the quantisation constants too\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"Qwen/Qwen2.5-7B-Instruct\", quantization_config=bnb, device_map=\"auto\")\nmodel.gradient_checkpointing_enable()\nmodel.config.use_cache = False           # incompatible with checkpointing"},{"t":"p","runs":[{"text":"Three of those settings deserve explanation."}]},{"t":"p","runs":[{"text":"NF4","b":true},{"text":" is a 4-bit format whose levels are spaced to match the roughly normal distribution of neural network weights, rather than spaced evenly. It costs the same memory as int4 and loses less. There is no reason to choose otherwise for this."}]},{"t":"p","runs":[{"text":"Double quantisation","b":true},{"text":" quantises the per-block scaling constants as well as the weights, saving roughly a further 0.4 bits per parameter. On a 7B model that is a few hundred megabytes, which is often the difference between fitting and not."}]},{"t":"p","runs":[{"text":"Gradient checkpointing","b":true},{"text":" discards intermediate activations during the forward pass and recomputes them during the backward pass. It cuts activation memory substantially and costs roughly 20 to 30% in speed. On a card that would otherwise not fit, that trade is not close."}]},{"t":"h2","id":"sequence-length-is-the-knob-that-bites","runs":[{"text":"Sequence length is the knob that bites"}]},{"t":"p","runs":[{"text":"Activation memory grows with sequence length, and attention’s contribution grows faster than linearly. Halving "},{"text":"max_seq_length","code":true},{"text":" from 4096 to 2048 typically frees more memory than any other single change, and it is the first thing to try when a run dies in the first few steps."}]},{"t":"p","runs":[{"text":"The cost is that the model never sees examples longer than that during training, which matters if your real inputs are long. The usual compromise: train at the length that covers the great majority of your examples, and accept that behaviour on the tail is untrained rather than learned."}]},{"t":"p","runs":[{"text":"If you must keep the length, reduce "},{"text":"per_device_train_batch_size","code":true},{"text":" to 1 and recover the effective batch size with "},{"text":"gradient_accumulation_steps","code":true},{"text":". Sixteen accumulation steps at batch size 1 gives the same gradient as batch size 16, at the same memory as batch size 1 and roughly the same total time."}]},{"t":"h2","id":"use-a-paged-optimiser","runs":[{"text":"Use a paged optimiser"}]},{"t":"p","runs":[{"text":"Memory use during training is spiky, and a single spike kills a run that had been fine for two hours. "},{"text":"optim=\"paged_adamw_8bit\"","code":true},{"text":" keeps the optimiser state in 8-bit and allows it to page to host memory under pressure. It is slightly slower and it converts a class of hard failures into a class of slowdowns."}]},{"t":"h2","id":"the-honest-catch-which-arrives-at-merge-time","runs":[{"text":"The honest catch, which arrives at merge time"}]},{"t":"p","runs":[{"text":"QLoRA trains an adapter against a base that was quantised to 4-bit. If you then merge that adapter into the original 16-bit weights, the base the adapter was fitted to is not the base it is now attached to. The discrepancy is usually small and it is not always small."}]},{"t":"p","runs":[{"text":"There are two defensible paths. Serve the adapter on the same 4-bit base it was trained against, which is exactly consistent and is what a multi-adapter server does anyway. Or merge into the 16-bit base and then "},{"text":"re-run your evaluation","b":true},{"text":", treating the merged model as a new artefact rather than as the thing you tested. What is not defensible is merging and assuming the numbers carry over."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"This entire lesson runs on a free hosted notebook tier. A 7B QLoRA at 2048 tokens on a few thousand examples fits within a free session’s memory and time budget, with the usual caveats: mount your own storage so the adapter survives, checkpoint often, and expect the session to end without warning. For learning this technique, no hardware purchase is required at any point."}]}],"check":{"question":[{"text":"A QLoRA run on a 12GB card fails with an out-of-memory error a few steps in, after loading successfully. Which change addresses the actual cause most directly?"}],"options":[{"text":[{"text":"Reduce the LoRA rank from 32 to 8 to shrink the trainable parameters."}],"correct":false,"tempting":[{"text":"Adapter parameters are a few megabytes at any usable rank, so this frees almost nothing compared with what activations consume."}]},{"text":[{"text":"Disable gradient checkpointing, since recomputation increases peak memory during the backward pass."}],"correct":false,"tempting":[{"text":"Reverses what checkpointing does: it trades speed for lower activation memory, so turning it off makes the failure arrive sooner."}]},{"text":[{"text":"Switch from NF4 to plain int4 quantisation to save memory on the base weights."}],"correct":false,"tempting":[{"text":"Both formats use four bits, so this changes quality rather than footprint, and the base was already loaded successfully."}]},{"text":[{"text":"Reduce max sequence length, and recover the effective batch size with gradient accumulation."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"what-a-lora-actually-is","title":"What a LoRA actually is","minutes":9},"next":{"slug":"building-the-dataset","title":"The dataset is the project","minutes":9}}}