{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"qlora-on-one-card","title":"QLoRA, and fine-tuning a 7B model on a free GPU","url":"https://addaly.com/learn/hugging-face/qlora-on-one-card","position":61,"of":75,"minutes":8,"answer":"QLoRA keeps a frozen base in 4-bit and trains precise adapters over it, dequantizing per matrix multiply, so a 7B model fits a free GPU at the cost of speed rather than memory.","keyIdea":"QLoRA keeps a frozen base in 4-bit and trains precise adapters over it, dequantizing per matrix multiply, so a 7B model fits a free GPU at the cost of speed rather than memory, and `prepare_model_for_kbit_training` is the line whose absence makes training quietly fail instead of raising."},"body":{"format":"markdown","text":"## Combining two ideas\n\nLoRA removed the optimiser memory. The weights themselves are still there: 14 GB for a 7B model at bfloat16, which does not leave room on a 16 GB card for activations and a KV cache.\n\nQLoRA loads the **base model in 4-bit** and trains LoRA adapters on top of it in higher precision. The frozen base is quantized because nothing updates it; the trainable part stays precise because precision is what it needs. A 7B model then trains in roughly 6 to 8 GB, which fits a free Colab T4 with room to spare.\n\n```python\nimport torch\nfrom transformers import BitsAndBytesConfig, AutoModelForCausalLM\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16,\n    bnb_4bit_use_double_quant=True,\n)\n\nbase = AutoModelForCausalLM.from_pretrained(\n    \"Qwen/Qwen2.5-7B-Instruct\", quantization_config=bnb, device_map=\"auto\"\n)\nbase = prepare_model_for_kbit_training(base, use_gradient_checkpointing=True)\nmodel = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, task_type=\"CAUSAL_LM\",\n                                        target_modules=\"all-linear\"))\n```\n\n## What each setting is doing\n\n**`nf4`** is a 4-bit data type designed for weights that are roughly normally distributed, which neural network weights are. It preserves more information than plain 4-bit integers at the same size.\n\n**`bnb_4bit_compute_dtype=torch.bfloat16`** means weights are *stored* in 4 bits and *computed* in bfloat16: dequantized on the fly for each matrix multiply. This is why QLoRA is slower than bfloat16 training per step even though it uses far less memory. You are trading time for space, deliberately.\n\n**`use_double_quant`** quantizes the quantization constants themselves, saving a further few hundred megabytes on a 7B model.\n\n**`prepare_model_for_kbit_training`** casts layer norms to float32, enables gradient checkpointing and makes the input embeddings require gradients. Skip it and training is unstable or silently fails to learn: one of those lines whose absence produces a confusing result rather than an error.\n\n## What it costs in quality\n\nThe published QLoRA results, and most community replications, find results close to 16-bit LoRA on adaptation tasks. \"Close\" is doing work in that sentence: on tasks where the quantization damage falls (arithmetic, exact structured output, rarely seen languages) the gap is larger, for the mechanism in module three.\n\nThe practical stance: use QLoRA when the alternative is not training at all, which for most people on free hardware it is. If you have a card that fits the model at bfloat16, use bfloat16: it is faster and marginally better.\n\n## Merging a QLoRA adapter, and the subtlety\n\nYou trained against 4-bit weights. Merging the adapter into the **full-precision** base is standard practice and is not exactly the arithmetic you trained: the adapter learned to correct a quantized base, and it is being applied to an unquantized one. In practice this works and is what almost everybody does, and it is worth evaluating the merged model rather than assuming the numbers carry over.\n\nThe cleaner path for serving is to keep the adapter separate and load it over whichever base precision you serve with. It also lets you swap adapters, which merging forfeits.\n\n## A realistic budget\n\nOn a free Colab T4, a 7B model with QLoRA, 2,000 examples of about 512 tokens, three epochs, batch size 1 with accumulation to 16: roughly two to four hours. On Kaggle's longer sessions, comfortable. Rented on a 24 GB card at community marketplace rates, a few dollars.\n\nThe binding constraint is usually the session length rather than the money, which brings back the module-three discipline: `save_steps`, `push_to_hub`, `resume_from_checkpoint`.\n\n## The honest alternative\n\nIf you want the result rather than the understanding, AutoTrain on the Hub and libraries like Unsloth and Axolotl wrap all of this behind configuration. Unsloth in particular reports substantial speed and memory improvements through custom kernels, and it is free and open source for single-GPU use. Using them is not cheating. Understanding what they are doing is what this lesson is for, because when a run fails you will need it.\n\n## Do this now\n\nLoad a 7B model with the QLoRA configuration above on a free GPU and print `torch.cuda.max_memory_allocated()` after one training step. Compare that against the 56 GB full fine-tuning would have needed. That single comparison is the reason this technique changed who can do this work."},"check":{"question":"Why is QLoRA slower per training step than 16-bit LoRA despite using far less memory?","options":[{"text":"Gradient checkpointing is mandatory with 4-bit bases, and recomputing activations dominates the step time.","correct":false,"tempting":"Checkpointing does cost time and is commonly enabled, but it is a separate choice rather than what quantization itself imposes."},{"text":"Quantized weights must be dequantized to the compute dtype for every matrix multiply.","correct":true},{"text":"4-bit arithmetic is emulated in software on most GPUs, so each operation takes several native instructions.","correct":false,"tempting":"The arithmetic is not performed in 4 bits at all; the weights are unpacked before computation."},{"text":"Smaller memory footprints reduce the achievable batch size, so more steps are needed for the same data.","correct":false,"tempting":"Inverts the effect: less memory allows larger batches, and the question is about per-step cost anyway."}]},"prev":{"slug":"lora-and-peft","title":"LoRA: training 0.1% of the parameters","url":"https://addaly.com/learn/hugging-face/lora-and-peft"},"next":{"slug":"trl-and-instruction-tuning","title":"TRL, and training on conversations","url":"https://addaly.com/learn/hugging-face/trl-and-instruction-tuning"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'QLoRA, and fine-tuning a 7B model on a free GPU', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/qlora-on-one-card (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"QLoRA, and fine-tuning a 7B model on a free GPU","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"QLoRA keeps a frozen base in 4-bit and trains precise adapters over it, dequantizing per matrix multiply, so a 7B model fits a free GPU at the cost of speed rather than memory, and `prepare_model_for_kbit_training` is the line whose absence makes training quietly fail instead of raising.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"combining-two-ideas","runs":[{"text":"Combining two ideas"}]},{"t":"p","runs":[{"text":"LoRA removed the optimiser memory. The weights themselves are still there: 14 GB for a 7B model at bfloat16, which does not leave room on a 16 GB card for activations and a KV cache."}]},{"t":"p","runs":[{"text":"QLoRA loads the "},{"text":"base model in 4-bit","b":true},{"text":" and trains LoRA adapters on top of it in higher precision. The frozen base is quantized because nothing updates it; the trainable part stays precise because precision is what it needs. A 7B model then trains in roughly 6 to 8 GB, which fits a free Colab T4 with room to spare."}]},{"t":"code","lang":"python","label":"Python","text":"import torch\nfrom transformers import BitsAndBytesConfig, AutoModelForCausalLM\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16,\n    bnb_4bit_use_double_quant=True,\n)\n\nbase = AutoModelForCausalLM.from_pretrained(\n    \"Qwen/Qwen2.5-7B-Instruct\", quantization_config=bnb, device_map=\"auto\"\n)\nbase = prepare_model_for_kbit_training(base, use_gradient_checkpointing=True)\nmodel = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, task_type=\"CAUSAL_LM\",\n                                        target_modules=\"all-linear\"))"},{"t":"h2","id":"what-each-setting-is-doing","runs":[{"text":"What each setting is doing"}]},{"t":"p","runs":[{"text":"nf4","b":true,"code":true},{"text":" is a 4-bit data type designed for weights that are roughly normally distributed, which neural network weights are. It preserves more information than plain 4-bit integers at the same size."}]},{"t":"p","runs":[{"text":"bnb_4bit_compute_dtype=torch.bfloat16","b":true,"code":true},{"text":" means weights are "},{"text":"stored","i":true},{"text":" in 4 bits and "},{"text":"computed","i":true},{"text":" in bfloat16: dequantized on the fly for each matrix multiply. This is why QLoRA is slower than bfloat16 training per step even though it uses far less memory. You are trading time for space, deliberately."}]},{"t":"p","runs":[{"text":"use_double_quant","b":true,"code":true},{"text":" quantizes the quantization constants themselves, saving a further few hundred megabytes on a 7B model."}]},{"t":"p","runs":[{"text":"prepare_model_for_kbit_training","b":true,"code":true},{"text":" casts layer norms to float32, enables gradient checkpointing and makes the input embeddings require gradients. Skip it and training is unstable or silently fails to learn: one of those lines whose absence produces a confusing result rather than an error."}]},{"t":"h2","id":"what-it-costs-in-quality","runs":[{"text":"What it costs in quality"}]},{"t":"p","runs":[{"text":"The published QLoRA results, and most community replications, find results close to 16-bit LoRA on adaptation tasks. “Close” is doing work in that sentence: on tasks where the quantization damage falls (arithmetic, exact structured output, rarely seen languages) the gap is larger, for the mechanism in module three."}]},{"t":"p","runs":[{"text":"The practical stance: use QLoRA when the alternative is not training at all, which for most people on free hardware it is. If you have a card that fits the model at bfloat16, use bfloat16: it is faster and marginally better."}]},{"t":"h2","id":"merging-a-qlora-adapter-and-the-subtlety","runs":[{"text":"Merging a QLoRA adapter, and the subtlety"}]},{"t":"p","runs":[{"text":"You trained against 4-bit weights. Merging the adapter into the "},{"text":"full-precision","b":true},{"text":" base is standard practice and is not exactly the arithmetic you trained: the adapter learned to correct a quantized base, and it is being applied to an unquantized one. In practice this works and is what almost everybody does, and it is worth evaluating the merged model rather than assuming the numbers carry over."}]},{"t":"p","runs":[{"text":"The cleaner path for serving is to keep the adapter separate and load it over whichever base precision you serve with. It also lets you swap adapters, which merging forfeits."}]},{"t":"h2","id":"a-realistic-budget","runs":[{"text":"A realistic budget"}]},{"t":"p","runs":[{"text":"On a free Colab T4, a 7B model with QLoRA, 2,000 examples of about 512 tokens, three epochs, batch size 1 with accumulation to 16: roughly two to four hours. On Kaggle’s longer sessions, comfortable. Rented on a 24 GB card at community marketplace rates, a few dollars."}]},{"t":"p","runs":[{"text":"The binding constraint is usually the session length rather than the money, which brings back the module-three discipline: "},{"text":"save_steps","code":true},{"text":", "},{"text":"push_to_hub","code":true},{"text":", "},{"text":"resume_from_checkpoint","code":true},{"text":"."}]},{"t":"h2","id":"the-honest-alternative","runs":[{"text":"The honest alternative"}]},{"t":"p","runs":[{"text":"If you want the result rather than the understanding, AutoTrain on the Hub and libraries like Unsloth and Axolotl wrap all of this behind configuration. Unsloth in particular reports substantial speed and memory improvements through custom kernels, and it is free and open source for single-GPU use. Using them is not cheating. Understanding what they are doing is what this lesson is for, because when a run fails you will need it."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load a 7B model with the QLoRA configuration above on a free GPU and print "},{"text":"torch.cuda.max_memory_allocated()","code":true},{"text":" after one training step. Compare that against the 56 GB full fine-tuning would have needed. That single comparison is the reason this technique changed who can do this work."}]}],"check":{"question":[{"text":"Why is QLoRA slower per training step than 16-bit LoRA despite using far less memory?"}],"options":[{"text":[{"text":"Gradient checkpointing is mandatory with 4-bit bases, and recomputing activations dominates the step time."}],"correct":false,"tempting":[{"text":"Checkpointing does cost time and is commonly enabled, but it is a separate choice rather than what quantization itself imposes."}]},{"text":[{"text":"Quantized weights must be dequantized to the compute dtype for every matrix multiply."}],"correct":true,"tempting":null},{"text":[{"text":"4-bit arithmetic is emulated in software on most GPUs, so each operation takes several native instructions."}],"correct":false,"tempting":[{"text":"The arithmetic is not performed in 4 bits at all; the weights are unpacked before computation."}]},{"text":[{"text":"Smaller memory footprints reduce the achievable batch size, so more steps are needed for the same data."}],"correct":false,"tempting":[{"text":"Inverts the effect: less memory allows larger batches, and the question is about per-step cost anyway."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"lora-and-peft","title":"LoRA: training 0.1% of the parameters","minutes":9},"next":{"slug":"trl-and-instruction-tuning","title":"TRL, and training on conversations","minutes":9}}}