{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"qlora-and-nf4","title":"QLoRA: quantising the part you are not training","url":"https://addaly.com/learn/fine-tuning/qlora-and-nf4","position":22,"of":84,"minutes":10,"answer":"QLoRA quantises only the frozen base to NF4 with block-wise scales and double-quantised constants, buying a 4x memory reduction for roughly 30% throughput and an adapter that was fitted against a slightly wrong model.","keyIdea":"QLoRA quantises only the frozen base to NF4 with block-wise scales and double-quantised constants, buying a 4x memory reduction for roughly 30% throughput and an adapter that was fitted against a slightly wrong model, which is why merging it back is the awkward part."},"body":{"format":"markdown","text":"## The idea in one line\n\nThe frozen base model receives no gradients, so it does not need to be stored precisely. Quantise it to 4 bits, keep the small trainable adapter in higher precision, and a 70B model becomes trainable on a single 48 GB card.\n\nThe numbers for an 8B model: bf16 weights are 16 GB, NF4 weights are about **4.1 GB**. For a 70B: 140 GB becomes about 36 GB.\n\n## NF4, and why not plain int4\n\nUniform 4-bit quantisation carves the range into 16 evenly spaced buckets. Neural network weights are not uniformly distributed (they are roughly zero-centred and bell-shaped) so even spacing wastes most of its buckets on values that barely occur and crowds the region where the weights actually live.\n\n**NormalFloat 4** places its 16 levels so that, for a normally distributed tensor, each bucket receives an equal share of the values. It is information-theoretically matched to the distribution the weights actually have, and it measurably beats uniform int4 at the same bit width.\n\nTwo further pieces make it work:\n\n**Block-wise scaling.** Quantisation is applied per block of 64 weights, each with its own scaling constant, rather than per tensor. One outlier weight then distorts 64 values instead of 16 million.\n\n**Double quantisation.** Those per-block constants are themselves numbers you have to store: one fp32 per 64 weights is 0.5 bits per parameter of overhead. Quantising *them* to 8 bits, in blocks of 256, brings the overhead down to about 0.127 bits per parameter. On a 65B model that saving is around 3 GB, which is the difference between fitting and not.\n\n## What it costs\n\nThree honest costs, none of them usually decisive but all of them real:\n\n1. **Speed.** Every matrix multiplication dequantises its weights to bf16 on the fly. QLoRA typically runs 25 to 40% slower per step than the same LoRA on unquantised weights. You are trading time for the ability to run at all.\n2. **Quantisation error in the forward pass.** The gradients computed for your adapter are gradients through a slightly wrong model. The QLoRA paper's central claim is that the final quality matches 16-bit LoRA on the tasks measured, and that has held up broadly, but \"broadly\" is the honest word. On tasks that are sensitive to small numeric differences, and at very small model sizes where quantisation error is proportionally larger, the gap is not always zero.\n3. **Merging is awkward.** Your adapter was fitted against the *quantised* base. Merge it into the full-precision base and you get a model that is not quite what you trained; merge it into the quantised base and you must requantise, which introduces its own error. The practical answers are to keep the adapter separate at serving time, or to use a method designed for this (QA-LoRA and similar) rather than merging naively and hoping.\n\n## The configuration\n\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",          # not \"fp4\"\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_compute_dtype=torch.bfloat16,  # float16 on a T4\n)\n```\n\n> Figure: What QLoRA is made of. Layers, top to bottom: NF4, not plain int4 (Sixteen levels placed so each receives an equal share of a normally distributed tensor, rather than evenly spaced across a range the weights do not occupy.); Block-wise scaling (One scaling constant per 64 weights, so a single outlier distorts 64 values instead of sixteen million.); Double quantisation (Those constants quantised too, in blocks of 256: overhead falls from about 0.5 bits per parameter to about 0.127.); A higher-precision adapter on top (The frozen base is 4-bit; only the small trained part is kept and updated in 16-bit. Three precisions in one model, and that is normal.). Each idea is what makes the next affordable. On a 65B model, double quantisation alone saves about three gigabytes: the difference between fitting and not.\n\nFour lines, and three of them have a wrong value that works. `fp4` instead of `nf4` trains without complaint and is slightly worse. Leaving double quantisation off costs memory you may need. Setting the compute dtype to fp32 quietly removes most of the speed benefit.\n\nPair it with `optim=\"paged_adamw_8bit\"`. Paged optimisers use CUDA unified memory so that a memory spike on one unusually long batch is absorbed by CPU RAM instead of ending the run.\n\n## When *not* to use it\n\nIf the model fits in bf16 on the card you have, use plain LoRA. QLoRA is a memory technique, and paying 30% throughput for memory you did not need is a straight loss. A 3B model on a 24 GB card has no business being quantised for training.\n\nThe decision is arithmetic, not taste: parameters × 2 bytes, plus activations, plus logits. If that is under your card's capacity with headroom, skip the quantisation.\n\n## The free path\n\nThis is the technique that put fine-tuning on free hardware. A 7B or 8B model in NF4 at sequence length 1,024, batch 1 with gradient accumulation and checkpointing on, fits inside a free Colab or Kaggle T4's 16 GB. It is slow (expect several hours for a few thousand steps) and Colab will disconnect you, so save checkpoints to Drive every couple of hundred steps. But it costs nothing, and the resulting adapter is the same kind of artefact a rented A100 would have produced."},"check":{"question":"Why does double quantisation exist, given that the weights are already at 4 bits?","options":[{"text":"It applies a second quantisation pass to the weights themselves, reducing them from 4 bits to an effective 2 bits.","correct":false,"tempting":"Misreads the target: the second pass acts on the scaling constants, and 2-bit weights would destroy quality rather than preserve it."},{"text":"It quantises the adapter matrices as well as the base, so the whole model is uniformly 4-bit during training.","correct":false,"tempting":"Would break training entirely, since the adapter is the only thing receiving gradients and must be updatable at a useful precision."},{"text":"It quantises the per-block scaling constants, which otherwise add about 0.5 bits per parameter.","correct":true},{"text":"It quantises the activations in addition to the weights, halving the memory used by the forward pass.","correct":false,"tempting":"Names a different technique; QLoRA's activations stay in the bf16 compute dtype and are not what double quantisation touches."}]},"prev":{"slug":"rank-alpha-and-targets","title":"Choosing rank, alpha and which modules to touch","url":"https://addaly.com/learn/fine-tuning/rank-alpha-and-targets"},"next":{"slug":"dora-rslora-and-variants","title":"DoRA, rsLoRA and the variant question","url":"https://addaly.com/learn/fine-tuning/dora-rslora-and-variants"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'QLoRA: quantising the part you are not training', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/qlora-and-nf4 (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"QLoRA: quantising the part you are not training","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"QLoRA quantises only the frozen base to NF4 with block-wise scales and double-quantised constants, buying a 4x memory reduction for roughly 30% throughput and an adapter that was fitted against a slightly wrong model, which is why merging it back is the awkward part.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"What QLoRA is made of","layers":[{"label":"NF4, not plain int4","detail":"Sixteen levels placed so each receives an equal share of a normally distributed tensor, rather than evenly spaced across a range the weights do not occupy."},{"label":"Block-wise scaling","detail":"One scaling constant per 64 weights, so a single outlier distorts 64 values instead of sixteen million."},{"label":"Double quantisation","detail":"Those constants quantised too, in blocks of 256: overhead falls from about 0.5 bits per parameter to about 0.127."},{"label":"A higher-precision adapter on top","detail":"The frozen base is 4-bit; only the small trained part is kept and updated in 16-bit. Three precisions in one model, and that is normal."}],"caption":"Each idea is what makes the next affordable. On a 65B model, double quantisation alone saves about three gigabytes: the difference between fitting and not."},"alt":"Figure: What QLoRA is made of. Layers, top to bottom: NF4, not plain int4 (Sixteen levels placed so each receives an equal share of a normally distributed tensor, rather than evenly spaced across a range the weights do not occupy.); Block-wise scaling (One scaling constant per 64 weights, so a single outlier distorts 64 values instead of sixteen million.); Double quantisation (Those constants quantised too, in blocks of 256: overhead falls from about 0.5 bits per parameter to about 0.127.); A higher-precision adapter on top (The frozen base is 4-bit; only the small trained part is kept and updated in 16-bit. Three precisions in one model, and that is normal.). Each idea is what makes the next affordable. On a 65B model, double quantisation alone saves about three gigabytes: the difference between fitting and not."}},"blocks":[{"t":"h2","id":"the-idea-in-one-line","runs":[{"text":"The idea in one line"}]},{"t":"p","runs":[{"text":"The frozen base model receives no gradients, so it does not need to be stored precisely. Quantise it to 4 bits, keep the small trainable adapter in higher precision, and a 70B model becomes trainable on a single 48 GB card."}]},{"t":"p","runs":[{"text":"The numbers for an 8B model: bf16 weights are 16 GB, NF4 weights are about "},{"text":"4.1 GB","b":true},{"text":". For a 70B: 140 GB becomes about 36 GB."}]},{"t":"h2","id":"nf4-and-why-not-plain-int4","runs":[{"text":"NF4, and why not plain int4"}]},{"t":"p","runs":[{"text":"Uniform 4-bit quantisation carves the range into 16 evenly spaced buckets. Neural network weights are not uniformly distributed (they are roughly zero-centred and bell-shaped) so even spacing wastes most of its buckets on values that barely occur and crowds the region where the weights actually live."}]},{"t":"p","runs":[{"text":"NormalFloat 4","b":true},{"text":" places its 16 levels so that, for a normally distributed tensor, each bucket receives an equal share of the values. It is information-theoretically matched to the distribution the weights actually have, and it measurably beats uniform int4 at the same bit width."}]},{"t":"p","runs":[{"text":"Two further pieces make it work:"}]},{"t":"p","runs":[{"text":"Block-wise scaling.","b":true},{"text":" Quantisation is applied per block of 64 weights, each with its own scaling constant, rather than per tensor. One outlier weight then distorts 64 values instead of 16 million."}]},{"t":"p","runs":[{"text":"Double quantisation.","b":true},{"text":" Those per-block constants are themselves numbers you have to store: one fp32 per 64 weights is 0.5 bits per parameter of overhead. Quantising "},{"text":"them","i":true},{"text":" to 8 bits, in blocks of 256, brings the overhead down to about 0.127 bits per parameter. On a 65B model that saving is around 3 GB, which is the difference between fitting and not."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"Three honest costs, none of them usually decisive but all of them real:"}]},{"t":"ol","start":1,"items":[[{"text":"Speed.","b":true},{"text":" Every matrix multiplication dequantises its weights to bf16 on the fly. QLoRA typically runs 25 to 40% slower per step than the same LoRA on unquantised weights. You are trading time for the ability to run at all."}],[{"text":"Quantisation error in the forward pass.","b":true},{"text":" The gradients computed for your adapter are gradients through a slightly wrong model. The QLoRA paper’s central claim is that the final quality matches 16-bit LoRA on the tasks measured, and that has held up broadly, but “broadly” is the honest word. On tasks that are sensitive to small numeric differences, and at very small model sizes where quantisation error is proportionally larger, the gap is not always zero."}],[{"text":"Merging is awkward.","b":true},{"text":" Your adapter was fitted against the "},{"text":"quantised","i":true},{"text":" base. Merge it into the full-precision base and you get a model that is not quite what you trained; merge it into the quantised base and you must requantise, which introduces its own error. The practical answers are to keep the adapter separate at serving time, or to use a method designed for this (QA-LoRA and similar) rather than merging naively and hoping."}]]},{"t":"h2","id":"the-configuration","runs":[{"text":"The configuration"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import BitsAndBytesConfig\nimport torch\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",          # not \"fp4\"\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_compute_dtype=torch.bfloat16,  # float16 on a T4\n)"},{"t":"p","runs":[{"text":"Four lines, and three of them have a wrong value that works. "},{"text":"fp4","code":true},{"text":" instead of "},{"text":"nf4","code":true},{"text":" trains without complaint and is slightly worse. Leaving double quantisation off costs memory you may need. Setting the compute dtype to fp32 quietly removes most of the speed benefit."}]},{"t":"p","runs":[{"text":"Pair it with "},{"text":"optim=\"paged_adamw_8bit\"","code":true},{"text":". Paged optimisers use CUDA unified memory so that a memory spike on one unusually long batch is absorbed by CPU RAM instead of ending the run."}]},{"t":"h2","id":"when-not-to-use-it","runs":[{"text":"When "},{"text":"not","i":true},{"text":" to use it"}]},{"t":"p","runs":[{"text":"If the model fits in bf16 on the card you have, use plain LoRA. QLoRA is a memory technique, and paying 30% throughput for memory you did not need is a straight loss. A 3B model on a 24 GB card has no business being quantised for training."}]},{"t":"p","runs":[{"text":"The decision is arithmetic, not taste: parameters × 2 bytes, plus activations, plus logits. If that is under your card’s capacity with headroom, skip the quantisation."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"This is the technique that put fine-tuning on free hardware. A 7B or 8B model in NF4 at sequence length 1,024, batch 1 with gradient accumulation and checkpointing on, fits inside a free Colab or Kaggle T4’s 16 GB. It is slow (expect several hours for a few thousand steps) and Colab will disconnect you, so save checkpoints to Drive every couple of hundred steps. But it costs nothing, and the resulting adapter is the same kind of artefact a rented A100 would have produced."}]}],"check":{"question":[{"text":"Why does double quantisation exist, given that the weights are already at 4 bits?"}],"options":[{"text":[{"text":"It applies a second quantisation pass to the weights themselves, reducing them from 4 bits to an effective 2 bits."}],"correct":false,"tempting":[{"text":"Misreads the target: the second pass acts on the scaling constants, and 2-bit weights would destroy quality rather than preserve it."}]},{"text":[{"text":"It quantises the adapter matrices as well as the base, so the whole model is uniformly 4-bit during training."}],"correct":false,"tempting":[{"text":"Would break training entirely, since the adapter is the only thing receiving gradients and must be updatable at a useful precision."}]},{"text":[{"text":"It quantises the per-block scaling constants, which otherwise add about 0.5 bits per parameter."}],"correct":true,"tempting":null},{"text":[{"text":"It quantises the activations in addition to the weights, halving the memory used by the forward pass."}],"correct":false,"tempting":[{"text":"Names a different technique; QLoRA’s activations stay in the bf16 compute dtype and are not what double quantisation touches."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"rank-alpha-and-targets","title":"Choosing rank, alpha and which modules to touch","minutes":9},"next":{"slug":"dora-rslora-and-variants","title":"DoRA, rsLoRA and the variant question","minutes":9}}}