{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"training-arguments-that-matter","title":"The six arguments worth understanding","url":"https://addaly.com/learn/hugging-face/training-arguments-that-matter","position":59,"of":75,"minutes":9,"answer":"Learning rate sets the scale of everything and differs tenfold between full fine-tuning and LoRA, while gradient accumulation buys a large effective batch on small hardware and gradient_checkpointing.","keyIdea":"Learning rate sets the scale of everything and differs tenfold between full fine-tuning and LoRA, while gradient accumulation buys a large effective batch on small hardware and `gradient_checkpointing`, 8-bit optimisers and shorter sequences are the ordered responses to running out of memory."},"body":{"format":"markdown","text":"## `TrainingArguments` has over a hundred fields\n\nSix of them decide almost everything. The rest have sensible defaults and you can leave them alone until a specific problem says otherwise.\n\n## 1. Learning rate\n\nThe most important number in the file, and the one wrong by an order of magnitude most often.\n\n- **Full fine-tuning of a pretrained model**: `2e-5` to `5e-5`. Higher and you destroy what pretraining learned: the loss falls fast and the model gets worse at everything it used to do.\n- **LoRA and adapters**: `1e-4` to `3e-4`, ten times higher, because you are training a small number of fresh parameters rather than nudging trained ones.\n- **A newly initialised head only**: `1e-3` is reasonable.\n\nIf the loss explodes to NaN, the learning rate is too high. If the loss barely moves after a full epoch, it is too low. Those two symptoms are unambiguous and they are the fastest diagnostic you have.\n\n## 2. Batch size, and the way to fake a big one\n\n```python\nper_device_train_batch_size=4,\ngradient_accumulation_steps=8,      # effective batch size = 32\n```\n\nGradient accumulation runs four examples, keeps the gradients, runs four more, and updates after eight rounds. Mathematically close to a batch of 32, with the memory of 4. This is how a 16 GB free GPU trains with batch sizes that would otherwise need 80 GB, and it is the single most useful trick in the file.\n\nIt costs wall-clock time, not quality. Effective batch size is what matters for stability, and it is the product of the two numbers times the number of devices.\n\n## 3. Epochs, and when to stop\n\nTwo to three epochs for most fine-tuning. More than that on a small dataset and the model memorises: training loss keeps falling, validation loss turns upward. That divergence is overfitting, and it is visible in the logged numbers long before it is visible in the output.\n\n```python\neval_strategy=\"steps\", eval_steps=100,\nload_best_model_at_end=True,\nmetric_for_best_model=\"eval_loss\", greater_is_better=False,\n```\n\nWith those set, you keep the best checkpoint rather than the last one, which is not the same thing and is frequently better.\n\n## 4. Precision\n\n```python\nbf16=True,      # Ampere-generation GPUs and newer\nfp16=True,      # older cards such as the free-tier T4\n```\n\nRoughly twice as fast and half the memory. `bf16` where supported, for the exponent-range reason from module three. Never both.\n\n## 5. Memory arguments, in the order to reach for them\n\nWhen you hit out-of-memory, work down this list:\n\n1. `per_device_train_batch_size` down, `gradient_accumulation_steps` up. Free, no quality cost.\n2. `gradient_checkpointing=True`. Recomputes activations in the backward pass instead of storing them. Cuts activation memory substantially and costs roughly 20 to 30% speed. Almost always worth it.\n3. `optim=\"adamw_bnb_8bit\"`. Adam keeps two state tensors per parameter; 8-bit quantizes them, saving about three quarters of that. For a 7B model that is several gigabytes.\n4. Shorter `max_length`. Attention memory grows with sequence length; halving it helps a lot.\n5. LoRA instead of full fine-tuning, which is the next lesson and dwarfs all of the above.\n\n## 6. Warmup and schedule\n\n```python\nwarmup_ratio=0.03,\nlr_scheduler_type=\"cosine\",\n```\n\nWarmup ramps the learning rate from zero over the first few per cent of steps. Without it, the first large update to a pretrained model can do damage that later training never recovers from. Cosine decay then brings the rate down smoothly toward the end. These two are close to free and close to always correct.\n\n## Length-grouped batching, for free speed\n\n```python\ngroup_by_length=True,\n```\n\nBatches items of similar length together, so padding is minimal. On variable-length text this frequently cuts training time by a third or more, for one argument and no quality effect. It is the same insight as module two's sorting trick.\n\n## A working starting point\n\n```python\nTrainingArguments(\n    output_dir=\"out\",\n    learning_rate=2e-4,                  # LoRA\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=8,\n    num_train_epochs=3,\n    bf16=True,\n    gradient_checkpointing=True,\n    group_by_length=True,\n    warmup_ratio=0.03,\n    lr_scheduler_type=\"cosine\",\n    logging_steps=10,\n    eval_strategy=\"steps\", eval_steps=100,\n    save_steps=100, save_total_limit=2,\n    load_best_model_at_end=True,\n    report_to=\"tensorboard\",\n    seed=42,\n)\n```\n\nChange the learning rate for full fine-tuning and leave the rest until something goes wrong.\n\n## Do this now\n\nRun the same tiny training twice with learning rates ten times apart and plot both loss curves. Too-high and too-low have distinct shapes, and recognising them on sight is worth more than any table of recommended values."},"check":{"question":"Training a 7B model with LoRA runs out of memory at batch size 4. Which change reduces memory without changing the effective batch size or the optimisation?","options":[{"text":"Halving the learning rate, which reduces the magnitude of the stored gradients.","correct":false,"tempting":"Learning rate scales updates at application time; gradient tensors are the same size regardless."},{"text":"Dropping to batch size 2 and doubling gradient accumulation steps.","correct":true},{"text":"Switching from bf16 to fp16, which uses less memory per activation on most hardware.","correct":false,"tempting":"Both are two bytes per value, so the memory is identical and only the numerical range differs."},{"text":"Reducing the number of epochs, so fewer checkpoints are held in memory during the run.","correct":false,"tempting":"Checkpoints are written to disk and epochs do not affect peak memory during a step."}]},"prev":{"slug":"the-trainer-api","title":"Trainer, and what it saves you writing","url":"https://addaly.com/learn/hugging-face/the-trainer-api"},"next":{"slug":"lora-and-peft","title":"LoRA: training 0.1% of the parameters","url":"https://addaly.com/learn/hugging-face/lora-and-peft"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The six arguments worth understanding', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/training-arguments-that-matter (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The six arguments worth understanding","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"Learning rate sets the scale of everything and differs tenfold between full fine-tuning and LoRA, while gradient accumulation buys a large effective batch on small hardware and `gradient_checkpointing`, 8-bit optimisers and shorter sequences are the ordered responses to running out of memory.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"trainingarguments-has-over-a-hundred-fields","runs":[{"text":"TrainingArguments","code":true},{"text":" has over a hundred fields"}]},{"t":"p","runs":[{"text":"Six of them decide almost everything. The rest have sensible defaults and you can leave them alone until a specific problem says otherwise."}]},{"t":"h2","id":"1-learning-rate","runs":[{"text":"1. Learning rate"}]},{"t":"p","runs":[{"text":"The most important number in the file, and the one wrong by an order of magnitude most often."}]},{"t":"ul","items":[[{"text":"Full fine-tuning of a pretrained model","b":true},{"text":": "},{"text":"2e-5","code":true},{"text":" to "},{"text":"5e-5","code":true},{"text":". Higher and you destroy what pretraining learned: the loss falls fast and the model gets worse at everything it used to do."}],[{"text":"LoRA and adapters","b":true},{"text":": "},{"text":"1e-4","code":true},{"text":" to "},{"text":"3e-4","code":true},{"text":", ten times higher, because you are training a small number of fresh parameters rather than nudging trained ones."}],[{"text":"A newly initialised head only","b":true},{"text":": "},{"text":"1e-3","code":true},{"text":" is reasonable."}]]},{"t":"p","runs":[{"text":"If the loss explodes to NaN, the learning rate is too high. If the loss barely moves after a full epoch, it is too low. Those two symptoms are unambiguous and they are the fastest diagnostic you have."}]},{"t":"h2","id":"2-batch-size-and-the-way-to-fake-a-big-one","runs":[{"text":"2. Batch size, and the way to fake a big one"}]},{"t":"code","lang":"python","label":"Python","text":"per_device_train_batch_size=4,\ngradient_accumulation_steps=8,      # effective batch size = 32"},{"t":"p","runs":[{"text":"Gradient accumulation runs four examples, keeps the gradients, runs four more, and updates after eight rounds. Mathematically close to a batch of 32, with the memory of 4. This is how a 16 GB free GPU trains with batch sizes that would otherwise need 80 GB, and it is the single most useful trick in the file."}]},{"t":"p","runs":[{"text":"It costs wall-clock time, not quality. Effective batch size is what matters for stability, and it is the product of the two numbers times the number of devices."}]},{"t":"h2","id":"3-epochs-and-when-to-stop","runs":[{"text":"3. Epochs, and when to stop"}]},{"t":"p","runs":[{"text":"Two to three epochs for most fine-tuning. More than that on a small dataset and the model memorises: training loss keeps falling, validation loss turns upward. That divergence is overfitting, and it is visible in the logged numbers long before it is visible in the output."}]},{"t":"code","lang":"python","label":"Python","text":"eval_strategy=\"steps\", eval_steps=100,\nload_best_model_at_end=True,\nmetric_for_best_model=\"eval_loss\", greater_is_better=False,"},{"t":"p","runs":[{"text":"With those set, you keep the best checkpoint rather than the last one, which is not the same thing and is frequently better."}]},{"t":"h2","id":"4-precision","runs":[{"text":"4. Precision"}]},{"t":"code","lang":"python","label":"Python","text":"bf16=True,      # Ampere-generation GPUs and newer\nfp16=True,      # older cards such as the free-tier T4"},{"t":"p","runs":[{"text":"Roughly twice as fast and half the memory. "},{"text":"bf16","code":true},{"text":" where supported, for the exponent-range reason from module three. Never both."}]},{"t":"h2","id":"5-memory-arguments-in-the-order-to-reach-for-them","runs":[{"text":"5. Memory arguments, in the order to reach for them"}]},{"t":"p","runs":[{"text":"When you hit out-of-memory, work down this list:"}]},{"t":"ol","start":1,"items":[[{"text":"per_device_train_batch_size","code":true},{"text":" down, "},{"text":"gradient_accumulation_steps","code":true},{"text":" up. Free, no quality cost."}],[{"text":"gradient_checkpointing=True","code":true},{"text":". Recomputes activations in the backward pass instead of storing them. Cuts activation memory substantially and costs roughly 20 to 30% speed. Almost always worth it."}],[{"text":"optim=\"adamw_bnb_8bit\"","code":true},{"text":". Adam keeps two state tensors per parameter; 8-bit quantizes them, saving about three quarters of that. For a 7B model that is several gigabytes."}],[{"text":"Shorter "},{"text":"max_length","code":true},{"text":". Attention memory grows with sequence length; halving it helps a lot."}],[{"text":"LoRA instead of full fine-tuning, which is the next lesson and dwarfs all of the above."}]]},{"t":"h2","id":"6-warmup-and-schedule","runs":[{"text":"6. Warmup and schedule"}]},{"t":"code","lang":"python","label":"Python","text":"warmup_ratio=0.03,\nlr_scheduler_type=\"cosine\","},{"t":"p","runs":[{"text":"Warmup ramps the learning rate from zero over the first few per cent of steps. Without it, the first large update to a pretrained model can do damage that later training never recovers from. Cosine decay then brings the rate down smoothly toward the end. These two are close to free and close to always correct."}]},{"t":"h2","id":"length-grouped-batching-for-free-speed","runs":[{"text":"Length-grouped batching, for free speed"}]},{"t":"code","lang":"python","label":"Python","text":"group_by_length=True,"},{"t":"p","runs":[{"text":"Batches items of similar length together, so padding is minimal. On variable-length text this frequently cuts training time by a third or more, for one argument and no quality effect. It is the same insight as module two’s sorting trick."}]},{"t":"h2","id":"a-working-starting-point","runs":[{"text":"A working starting point"}]},{"t":"code","lang":"python","label":"Python","text":"TrainingArguments(\n    output_dir=\"out\",\n    learning_rate=2e-4,                  # LoRA\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=8,\n    num_train_epochs=3,\n    bf16=True,\n    gradient_checkpointing=True,\n    group_by_length=True,\n    warmup_ratio=0.03,\n    lr_scheduler_type=\"cosine\",\n    logging_steps=10,\n    eval_strategy=\"steps\", eval_steps=100,\n    save_steps=100, save_total_limit=2,\n    load_best_model_at_end=True,\n    report_to=\"tensorboard\",\n    seed=42,\n)"},{"t":"p","runs":[{"text":"Change the learning rate for full fine-tuning and leave the rest until something goes wrong."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Run the same tiny training twice with learning rates ten times apart and plot both loss curves. Too-high and too-low have distinct shapes, and recognising them on sight is worth more than any table of recommended values."}]}],"check":{"question":[{"text":"Training a 7B model with LoRA runs out of memory at batch size 4. Which change reduces memory without changing the effective batch size or the optimisation?"}],"options":[{"text":[{"text":"Halving the learning rate, which reduces the magnitude of the stored gradients."}],"correct":false,"tempting":[{"text":"Learning rate scales updates at application time; gradient tensors are the same size regardless."}]},{"text":[{"text":"Dropping to batch size 2 and doubling gradient accumulation steps."}],"correct":true,"tempting":null},{"text":[{"text":"Switching from bf16 to fp16, which uses less memory per activation on most hardware."}],"correct":false,"tempting":[{"text":"Both are two bytes per value, so the memory is identical and only the numerical range differs."}]},{"text":[{"text":"Reducing the number of epochs, so fewer checkpoints are held in memory during the run."}],"correct":false,"tempting":[{"text":"Checkpoints are written to disk and epochs do not affect peak memory during a step."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"the-trainer-api","title":"Trainer, and what it saves you writing","minutes":9},"next":{"slug":"lora-and-peft","title":"LoRA: training 0.1% of the parameters","minutes":9}}}