The six arguments worth understanding

TrainingArguments has over a hundred fields

Six of them decide almost everything. The rest have sensible defaults and you can leave them alone until a specific problem says otherwise.

1. Learning rate

The most important number in the file, and the one wrong by an order of magnitude most often.

• Full fine-tuning of a pretrained model: 2e-5 to 5e-5. Higher and you destroy what pretraining learned: the loss falls fast and the model gets worse at everything it used to do.
• LoRA and adapters: 1e-4 to 3e-4, ten times higher, because you are training a small number of fresh parameters rather than nudging trained ones.
• A newly initialised head only: 1e-3 is reasonable.

If the loss explodes to NaN, the learning rate is too high. If the loss barely moves after a full epoch, it is too low. Those two symptoms are unambiguous and they are the fastest diagnostic you have.

2. Batch size, and the way to fake a big one

per_device_train_batch_size=4,
gradient_accumulation_steps=8,      # effective batch size = 32

Gradient accumulation runs four examples, keeps the gradients, runs four more, and updates after eight rounds. Mathematically close to a batch of 32, with the memory of 4. This is how a 16 GB free GPU trains with batch sizes that would otherwise need 80 GB, and it is the single most useful trick in the file.

It costs wall-clock time, not quality. Effective batch size is what matters for stability, and it is the product of the two numbers times the number of devices.

3. Epochs, and when to stop

Two to three epochs for most fine-tuning. More than that on a small dataset and the model memorises: training loss keeps falling, validation loss turns upward. That divergence is overfitting, and it is visible in the logged numbers long before it is visible in the output.

eval_strategy="steps", eval_steps=100,
load_best_model_at_end=True,
metric_for_best_model="eval_loss", greater_is_better=False,

With those set, you keep the best checkpoint rather than the last one, which is not the same thing and is frequently better.

4. Precision

bf16=True,      # Ampere-generation GPUs and newer
fp16=True,      # older cards such as the free-tier T4

Roughly twice as fast and half the memory. bf16 where supported, for the exponent-range reason from module three. Never both.

5. Memory arguments, in the order to reach for them

When you hit out-of-memory, work down this list:

1. per_device_train_batch_size down, gradient_accumulation_steps up. Free, no quality cost.
2. gradient_checkpointing=True. Recomputes activations in the backward pass instead of storing them. Cuts activation memory substantially and costs roughly 20 to 30% speed. Almost always worth it.
3. optim="adamw_bnb_8bit". Adam keeps two state tensors per parameter; 8-bit quantizes them, saving about three quarters of that. For a 7B model that is several gigabytes.
4. Shorter max_length. Attention memory grows with sequence length; halving it helps a lot.
5. LoRA instead of full fine-tuning, which is the next lesson and dwarfs all of the above.

6. Warmup and schedule

warmup_ratio=0.03,
lr_scheduler_type="cosine",

Warmup ramps the learning rate from zero over the first few per cent of steps. Without it, the first large update to a pretrained model can do damage that later training never recovers from. Cosine decay then brings the rate down smoothly toward the end. These two are close to free and close to always correct.

Length-grouped batching, for free speed

group_by_length=True,

Batches items of similar length together, so padding is minimal. On variable-length text this frequently cuts training time by a third or more, for one argument and no quality effect. It is the same insight as module two’s sorting trick.

A working starting point

TrainingArguments(
    output_dir="out",
    learning_rate=2e-4,                  # LoRA
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    num_train_epochs=3,
    bf16=True,
    gradient_checkpointing=True,
    group_by_length=True,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    eval_strategy="steps", eval_steps=100,
    save_steps=100, save_total_limit=2,
    load_best_model_at_end=True,
    report_to="tensorboard",
    seed=42,
)

Change the learning rate for full fine-tuning and leave the rest until something goes wrong.

Do this now

Run the same tiny training twice with learning rates ten times apart and plot both loss curves. Too-high and too-low have distinct shapes, and recognising them on sight is worth more than any table of recommended values.