Running the training run

The tools, all free

Four options cover essentially everything, and none of them costs money.

Unsloth is the fastest single-GPU path and the lowest-friction. It patches the attention and MLP kernels for the common architectures and typically trains roughly twice as fast with noticeably less memory than a stock setup. Apache-2.0 licensed. Its limitation is coverage: it supports a list of architectures, and a model outside that list needs something else.

TRL with PEFT is the Hugging Face path. Most flexible, best documented, works with any architecture that loads, slower than Unsloth. This is what to learn if you want to understand what is happening.

Axolotl is configuration-driven: you write a YAML file instead of a script. Excellent for reproducibility and for running many variants, and it hides mechanics you may want to see the first time.

llama.cpp includes a fine-tuning path that runs on CPU and on modest hardware. It is slow and limited, and for someone with no GPU at all and no access to a free notebook tier, it is a real option rather than a theoretical one.

A run that works

from trl import SFTTrainer, SFTConfig

args = SFTConfig(
    output_dir="out",
    num_train_epochs=2,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,      # effective batch size 16
    learning_rate=2e-4,                  # LoRA rates are ~10x full fine-tuning
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    max_seq_length=2048,
    logging_steps=10,
    eval_strategy="steps", eval_steps=50,
    save_steps=50, save_total_limit=3,
    optim="paged_adamw_8bit",
    bf16=True,
    seed=3407,
)
trainer = SFTTrainer(model=model, args=args, peft_config=cfg,
                     train_dataset=train_ds, eval_dataset=eval_ds)
trainer.train()

The five settings that matter, in order

Learning rate. LoRA tolerates and needs a much higher rate than full fine-tuning, because only a small low-rank update is being learned. Between 1e-4 and 2e-4 is the working range for most adapters. Below 5e-5 the adapter often barely moves; above 5e-4 runs become unstable and the loss curve shows it.

Epochs. Two is a good default; three is often the most a few hundred examples will take before memorising them. If you find yourself wanting five, the honest reading is that the dataset is too small rather than that the model needs longer.

Effective batch size. Batch size times accumulation steps. Sixteen to thirty-two is a reasonable range. Very small effective batches make the loss curve noisy and the run harder to read.

Warmup. A few per cent of total steps. Cheap insurance against an early instability.

Sequence length. Set by your data and your memory, as the previous lesson described. Longer than your data needs is wasted memory.

Notice what is not on this list. Rank, dropout, and the scheduler shape all matter less than any of the above, and people spend disproportionate time on them.

Print three examples before you launch

The highest-value thirty seconds in this whole process:

for i in range(3):
    ids = train_ds[i]["input_ids"]
    labels = train_ds[i]["labels"]
    print(tok.decode(ids))
    print("TRAINED ON:", tok.decode([t for t in labels if t != -100]))
    print("---")

That second line is the one that catches real bugs. It shows exactly which tokens contribute to the loss. If it prints the user’s message, your masking is wrong. If it prints nothing, your masking is inverted and the run will do nothing for two hours. If it prints the assistant’s reply, you are ready.

Most first runs fail on data formatting, not on hyperparameters, and this check finds nearly all of it.

What the run should look like

On a consumer card, a 7B QLoRA over about 1,000 examples at 2,048 tokens for two epochs is typically well under an hour. If your estimate is wildly different from that shape (eight hours, or ninety seconds) something is wrong with the data size, the sequence length, or whether the GPU is being used at all. Check nvidia-smi during the first minute: utilisation should be high and steady.

Checkpoint every 50 steps with save_total_limit so a crash costs minutes rather than the run. Set the seed and record it, along with the base model revision, the dataset hash and the full configuration. A fine-tune you cannot reproduce is a result you cannot defend, and “which version of the data was that” is a question you will be asked.

The run is the easy part. Everything that decides whether it worked happened before it started, or happens in the evaluation afterwards.