Running a LoRA, and What It Costs

A configuration that works

Start here, and change one thing at a time.

from peft import LoraConfig
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,              # convention: twice the rank
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

Adapting every linear layer rather than attention alone is the finding from the QLoRA work, and it is close to free: the trainable count is still under 1%.

from trl import SFTConfig
args = SFTConfig(
    output_dir="out",
    num_train_epochs=2,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,   # effective batch of 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    max_length=1024,
    gradient_checkpointing=True,
    bf16=True,                        # fp16=True on a T4
    logging_steps=10,
    save_steps=100,
    optim="paged_adamw_8bit",
)

Pin your library versions. This part of the ecosystem changes fast enough that a six-month-old notebook usually will not run.

The four settings that matter

• Learning rate. LoRA wants 1e-4 to 2e-4, roughly ten to a hundred times higher than full fine-tuning, because you are moving very few parameters. Too low and nothing changes; too high and the model degrades in ways that look like a data problem. If outputs turn erratic, halve it before touching anything else.
• Epochs. One to three. On a small dataset, three passes is already the memorisation zone. Save a checkpoint each epoch and evaluate all of them: the best is often not the last.
• Rank. 8 or 16 for style and format. 32 to 64 if you are teaching a new task. Higher rank means more capacity and more forgetting; it is not a quality dial.
• Sequence length. Set it from your data, not from habit. Look at the token-length distribution and cover the 95th percentile. Memory grows with it, and every token past your longest real example is money spent on padding.

Where to run it

Free. Kaggle gives roughly 30 GPU-hours a week on a T4 or P100, with sessions that survive better than Colab’s free tier. Colab free gives a T4 that can disconnect at any moment, so keep save_steps low and push checkpoints somewhere persistent. On either, QLoRA on a 7B is fine; 16-bit LoRA on a 7B is not.

Rented. On the spot-style marketplaces, an RTX 4090 (24 GB) runs roughly $0.30-0.80 an hour, an A100 80 GB roughly $1.20-2.00, an H100 $2-3. These prices move constantly and vary by provider and region, so check on the day rather than trusting any number, including these.

Hosted fine-tuning APIs charge per training token, typically a few dollars for a small dataset. They handle the infrastructure and hand you back an endpoint. The trade is that you cannot inspect the run, and you usually cannot export the weights, which puts you back where you started on cost and portability.

What a run actually costs

Do the arithmetic before you rent anything. Two thousand examples averaging 600 tokens is 1.2M tokens per epoch, 2.4M for two epochs. A 7B QLoRA on a 24 GB card processes very roughly 1,500-3,000 tokens per second depending on sequence length and settings.

2.4M divided by 2,000 is about 1,200 seconds, so twenty minutes. At $0.50 an hour that is about 17 cents.

That number is the point of this lesson. The GPU has never been the expensive part of fine-tuning. The dataset took three days and the evaluation will take another two. Anyone selling you a fine-tuning story that is mostly about hardware is selling hardware.

Watch these while it runs

• Loss falls, then flattens. That is healthy. A cliff to near zero means memorisation or a masking bug.
• Loss rising, or NaN. Learning rate too high, or fp16 overflow on a T4. Lower the rate first.
• Grad-norm spikes every few hundred steps usually mean one pathological long example. Find it.
• A flat line from step one means the adapter is attached to nothing. Print model.print_trainable_parameters(); if it reports zero, your target_modules names do not match this architecture.

Then evaluate before you celebrate, which is the next lesson.