{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-run","title":"Running a LoRA, and What It Costs","url":"https://addaly.com/learn/fine-tuning/the-run","position":49,"of":84,"minutes":10,"answer":"GPU time is the cheap part: a 7B LoRA run costs cents, while the dataset and the evaluation cost days.","keyIdea":"GPU time is the cheap part: a 7B LoRA run costs cents, while the dataset and the evaluation cost days."},"body":{"format":"markdown","text":"## A configuration that works\n\nStart here, and change one thing at a time.\n\n```python\nfrom peft import LoraConfig\npeft_config = LoraConfig(\n    r=16,\n    lora_alpha=32,              # convention: twice the rank\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n)\n```\n\nAdapting every linear layer rather than attention alone is the finding from the QLoRA work, and it is close to free: the trainable count is still under 1%.\n\n```python\nfrom trl import SFTConfig\nargs = SFTConfig(\n    output_dir=\"out\",\n    num_train_epochs=2,\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,   # effective batch of 16\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    warmup_ratio=0.03,\n    max_length=1024,\n    gradient_checkpointing=True,\n    bf16=True,                        # fp16=True on a T4\n    logging_steps=10,\n    save_steps=100,\n    optim=\"paged_adamw_8bit\",\n)\n```\n\nPin your library versions. This part of the ecosystem changes fast enough that a six-month-old notebook usually will not run.\n\n## The four settings that matter\n\n- **Learning rate.** LoRA wants 1e-4 to 2e-4, roughly ten to a hundred times higher than full fine-tuning, because you are moving very few parameters. Too low and nothing changes; too high and the model degrades in ways that look like a data problem. If outputs turn erratic, halve it before touching anything else.\n- **Epochs.** One to three. On a small dataset, three passes is already the memorisation zone. Save a checkpoint each epoch and evaluate all of them: the best is often not the last.\n- **Rank.** 8 or 16 for style and format. 32 to 64 if you are teaching a new task. Higher rank means more capacity and more forgetting; it is not a quality dial.\n- **Sequence length.** Set it from your data, not from habit. Look at the token-length distribution and cover the 95th percentile. Memory grows with it, and every token past your longest real example is money spent on padding.\n\n## Where to run it\n\n**Free.** Kaggle gives roughly 30 GPU-hours a week on a T4 or P100, with sessions that survive better than Colab's free tier. Colab free gives a T4 that can disconnect at any moment, so keep `save_steps` low and push checkpoints somewhere persistent. On either, QLoRA on a 7B is fine; 16-bit LoRA on a 7B is not.\n\n**Rented.** On the spot-style marketplaces, an RTX 4090 (24 GB) runs roughly $0.30-0.80 an hour, an A100 80 GB roughly $1.20-2.00, an H100 $2-3. These prices move constantly and vary by provider and region, so check on the day rather than trusting any number, including these.\n\n**Hosted fine-tuning APIs** charge per training token, typically a few dollars for a small dataset. They handle the infrastructure and hand you back an endpoint. The trade is that you cannot inspect the run, and you usually cannot export the weights, which puts you back where you started on cost and portability.\n\n## What a run actually costs\n\nDo the arithmetic before you rent anything. Two thousand examples averaging 600 tokens is 1.2M tokens per epoch, 2.4M for two epochs. A 7B QLoRA on a 24 GB card processes very roughly 1,500-3,000 tokens per second depending on sequence length and settings.\n\n2.4M divided by 2,000 is about 1,200 seconds, so twenty minutes. At $0.50 an hour that is about **17 cents**.\n\nThat number is the point of this lesson. The GPU has never been the expensive part of fine-tuning. The dataset took three days and the evaluation will take another two. Anyone selling you a fine-tuning story that is mostly about hardware is selling hardware.\n\n## Watch these while it runs\n\n- **Loss falls, then flattens.** That is healthy. A cliff to near zero means memorisation or a masking bug.\n- **Loss rising, or NaN.** Learning rate too high, or fp16 overflow on a T4. Lower the rate first.\n- **Grad-norm spikes** every few hundred steps usually mean one pathological long example. Find it.\n- **A flat line from step one** means the adapter is attached to nothing. Print `model.print_trainable_parameters()`; if it reports zero, your `target_modules` names do not match this architecture.\n\nThen evaluate before you celebrate, which is the next lesson.\n\n> Figure: Reading a run while it is still running. Healthy: loss falls, then flattens; gradient norm settles into a band; the three sample generations still read like your data. Not healthy: a cliff to near zero in fifty steps: the answer is visible in the input, or the labels were not shifted; rising loss or NaN: the learning rate, or fp16 overflow on a T4; a flat line from step one: the adapter is attached to nothing. Print the trainable parameter count before you launch. On a misconfigured LoRA it is sometimes zero, and a run that trains nothing produces a perfectly flat, perfectly innocent-looking loss curve."},"check":{"question":"800 examples, rank 64, learning rate 2e-4, ten epochs. Training loss reaches 0.05, and on new inputs the model reproduces training answers almost word for word. What is the right read?","options":[{"text":"It memorised. Drop to one or two epochs, lower the learning rate, lower the rank, and judge checkpoints on the held-out set rather than the loss.","correct":true},{"text":"It worked: a loss of 0.05 means the model learned the task well.","correct":false,"tempting":"The trainer is correctly minimising exactly what you asked it to, but what it measures is reproduction of your data, and perfect reproduction is the failure itself."},{"text":"The dataset is too small; collect 8,000 more examples and rerun.","correct":false,"tempting":"More data does dilute memorisation, but these settings are several multiples off, and you would spend weeks fixing what three numbers fix today."},{"text":"Rank 64 is not enough capacity to generalise; raise it to 128.","correct":false,"tempting":"More capacity generalising better is the lesson people carry over from pretraining, but here extra capacity buys more memorisation and more forgetting."}]},"prev":{"slug":"templates-and-masking","title":"Chat Templates and Loss Masking","url":"https://addaly.com/learn/fine-tuning/templates-and-masking"},"next":{"slug":"the-tokenizer-is-the-model","title":"The tokenizer is part of the model","url":"https://addaly.com/learn/fine-tuning/the-tokenizer-is-the-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Running a LoRA, and What It Costs', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-run (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Running a LoRA, and What It Costs","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"GPU time is the cheap part: a 7B LoRA run costs cents, while the dataset and the evaluation cost days.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Healthy","items":["loss falls, then flattens","gradient norm settles into a band","the three sample generations still read like your data"]},"type":"compare","right":{"title":"Not healthy","items":["a cliff to near zero in fifty steps: the answer is visible in the input, or the labels were not shifted","rising loss or NaN: the learning rate, or fp16 overflow on a T4","a flat line from step one: the adapter is attached to nothing"]},"title":"Reading a run while it is still running","caption":"Print the trainable parameter count before you launch. On a misconfigured LoRA it is sometimes zero, and a run that trains nothing produces a perfectly flat, perfectly innocent-looking loss curve."},"alt":"Figure: Reading a run while it is still running. Healthy: loss falls, then flattens; gradient norm settles into a band; the three sample generations still read like your data. Not healthy: a cliff to near zero in fifty steps: the answer is visible in the input, or the labels were not shifted; rising loss or NaN: the learning rate, or fp16 overflow on a T4; a flat line from step one: the adapter is attached to nothing. Print the trainable parameter count before you launch. On a misconfigured LoRA it is sometimes zero, and a run that trains nothing produces a perfectly flat, perfectly innocent-looking loss curve."}},"blocks":[{"t":"h2","id":"a-configuration-that-works","runs":[{"text":"A configuration that works"}]},{"t":"p","runs":[{"text":"Start here, and change one thing at a time."}]},{"t":"code","lang":"python","label":"Python","text":"from peft import LoraConfig\npeft_config = LoraConfig(\n    r=16,\n    lora_alpha=32,              # convention: twice the rank\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n)"},{"t":"p","runs":[{"text":"Adapting every linear layer rather than attention alone is the finding from the QLoRA work, and it is close to free: the trainable count is still under 1%."}]},{"t":"code","lang":"python","label":"Python","text":"from trl import SFTConfig\nargs = SFTConfig(\n    output_dir=\"out\",\n    num_train_epochs=2,\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,   # effective batch of 16\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    warmup_ratio=0.03,\n    max_length=1024,\n    gradient_checkpointing=True,\n    bf16=True,                        # fp16=True on a T4\n    logging_steps=10,\n    save_steps=100,\n    optim=\"paged_adamw_8bit\",\n)"},{"t":"p","runs":[{"text":"Pin your library versions. This part of the ecosystem changes fast enough that a six-month-old notebook usually will not run."}]},{"t":"h2","id":"the-four-settings-that-matter","runs":[{"text":"The four settings that matter"}]},{"t":"ul","items":[[{"text":"Learning rate.","b":true},{"text":" LoRA wants 1e-4 to 2e-4, roughly ten to a hundred times higher than full fine-tuning, because you are moving very few parameters. Too low and nothing changes; too high and the model degrades in ways that look like a data problem. If outputs turn erratic, halve it before touching anything else."}],[{"text":"Epochs.","b":true},{"text":" One to three. On a small dataset, three passes is already the memorisation zone. Save a checkpoint each epoch and evaluate all of them: the best is often not the last."}],[{"text":"Rank.","b":true},{"text":" 8 or 16 for style and format. 32 to 64 if you are teaching a new task. Higher rank means more capacity and more forgetting; it is not a quality dial."}],[{"text":"Sequence length.","b":true},{"text":" Set it from your data, not from habit. Look at the token-length distribution and cover the 95th percentile. Memory grows with it, and every token past your longest real example is money spent on padding."}]]},{"t":"h2","id":"where-to-run-it","runs":[{"text":"Where to run it"}]},{"t":"p","runs":[{"text":"Free.","b":true},{"text":" Kaggle gives roughly 30 GPU-hours a week on a T4 or P100, with sessions that survive better than Colab’s free tier. Colab free gives a T4 that can disconnect at any moment, so keep "},{"text":"save_steps","code":true},{"text":" low and push checkpoints somewhere persistent. On either, QLoRA on a 7B is fine; 16-bit LoRA on a 7B is not."}]},{"t":"p","runs":[{"text":"Rented.","b":true},{"text":" On the spot-style marketplaces, an RTX 4090 (24 GB) runs roughly $0.30-0.80 an hour, an A100 80 GB roughly $1.20-2.00, an H100 $2-3. These prices move constantly and vary by provider and region, so check on the day rather than trusting any number, including these."}]},{"t":"p","runs":[{"text":"Hosted fine-tuning APIs","b":true},{"text":" charge per training token, typically a few dollars for a small dataset. They handle the infrastructure and hand you back an endpoint. The trade is that you cannot inspect the run, and you usually cannot export the weights, which puts you back where you started on cost and portability."}]},{"t":"h2","id":"what-a-run-actually-costs","runs":[{"text":"What a run actually costs"}]},{"t":"p","runs":[{"text":"Do the arithmetic before you rent anything. Two thousand examples averaging 600 tokens is 1.2M tokens per epoch, 2.4M for two epochs. A 7B QLoRA on a 24 GB card processes very roughly 1,500-3,000 tokens per second depending on sequence length and settings."}]},{"t":"p","runs":[{"text":"2.4M divided by 2,000 is about 1,200 seconds, so twenty minutes. At $0.50 an hour that is about "},{"text":"17 cents","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"That number is the point of this lesson. The GPU has never been the expensive part of fine-tuning. The dataset took three days and the evaluation will take another two. Anyone selling you a fine-tuning story that is mostly about hardware is selling hardware."}]},{"t":"h2","id":"watch-these-while-it-runs","runs":[{"text":"Watch these while it runs"}]},{"t":"ul","items":[[{"text":"Loss falls, then flattens.","b":true},{"text":" That is healthy. A cliff to near zero means memorisation or a masking bug."}],[{"text":"Loss rising, or NaN.","b":true},{"text":" Learning rate too high, or fp16 overflow on a T4. Lower the rate first."}],[{"text":"Grad-norm spikes","b":true},{"text":" every few hundred steps usually mean one pathological long example. Find it."}],[{"text":"A flat line from step one","b":true},{"text":" means the adapter is attached to nothing. Print "},{"text":"model.print_trainable_parameters()","code":true},{"text":"; if it reports zero, your "},{"text":"target_modules","code":true},{"text":" names do not match this architecture."}]]},{"t":"p","runs":[{"text":"Then evaluate before you celebrate, which is the next lesson."}]}],"check":{"question":[{"text":"800 examples, rank 64, learning rate 2e-4, ten epochs. Training loss reaches 0.05, and on new inputs the model reproduces training answers almost word for word. What is the right read?"}],"options":[{"text":[{"text":"It memorised. Drop to one or two epochs, lower the learning rate, lower the rank, and judge checkpoints on the held-out set rather than the loss."}],"correct":true,"tempting":null},{"text":[{"text":"It worked: a loss of 0.05 means the model learned the task well."}],"correct":false,"tempting":[{"text":"The trainer is correctly minimising exactly what you asked it to, but what it measures is reproduction of your data, and perfect reproduction is the failure itself."}]},{"text":[{"text":"The dataset is too small; collect 8,000 more examples and rerun."}],"correct":false,"tempting":[{"text":"More data does dilute memorisation, but these settings are several multiples off, and you would spend weeks fixing what three numbers fix today."}]},{"text":[{"text":"Rank 64 is not enough capacity to generalise; raise it to 128."}],"correct":false,"tempting":[{"text":"More capacity generalising better is the lesson people carry over from pretraining, but here extra capacity buys more memorisation and more forgetting."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"templates-and-masking","title":"Chat Templates and Loss Masking","minutes":9},"next":{"slug":"the-tokenizer-is-the-model","title":"The tokenizer is part of the model","minutes":9}}}