{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"a-config-line-by-line","title":"A working config, line by line","url":"https://addaly.com/learn/fine-tuning/a-config-line-by-line","position":52,"of":84,"minutes":10,"answer":"A working config is a set of decisions, not defaults: the instruct variant as base, all-linear targets, 2e-4 for LoRA, non-reentrant checkpointing so gradients reach the adapter.","keyIdea":"A working config is a set of decisions, not defaults: the instruct variant as base, all-linear targets, 2e-4 for LoRA, non-reentrant checkpointing so gradients reach the adapter, and length-grouped batching instead of packing until packing is verified."},"body":{"format":"markdown","text":"## The whole thing, then the reasons\n\nThis is a complete supervised fine-tuning setup for an 8B model on a 24 GB card, using TRL, PEFT and bitsandbytes, all free, all open source.\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\nfrom peft import LoraConfig\nfrom trl import SFTConfig, SFTTrainer\nimport torch\n\nMODEL = \"meta-llama/Llama-3.1-8B-Instruct\"\n\nbnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\",\n                         bnb_4bit_use_double_quant=True,\n                         bnb_4bit_compute_dtype=torch.bfloat16)\n\ntok = AutoTokenizer.from_pretrained(MODEL)\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL, quantization_config=bnb, torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\", device_map={\"\": 0})\n\npeft_cfg = LoraConfig(\n    r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n    target_modules=\"all-linear\", task_type=\"CAUSAL_LM\")\n\nargs = SFTConfig(\n    output_dir=\"out/triage-v3\",\n    num_train_epochs=2,\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=8,\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    warmup_ratio=0.03,\n    max_grad_norm=1.0,\n    bf16=True,\n    optim=\"paged_adamw_8bit\",\n    gradient_checkpointing=True,\n    gradient_checkpointing_kwargs={\"use_reentrant\": False},\n    max_length=2048,\n    packing=False,\n    group_by_length=True,\n    assistant_only_loss=True,\n    logging_steps=10,\n    eval_strategy=\"steps\", eval_steps=100,\n    save_steps=200, save_total_limit=3,\n    seed=0,\n    report_to=\"tensorboard\")\n\ntrainer = SFTTrainer(model=model, args=args, peft_config=peft_cfg,\n                     train_dataset=train, eval_dataset=val,\n                     processing_class=tok)\ntrainer.train()\n```\n\n## The lines that are decisions\n\n**`Llama-3.1-8B-Instruct`, not `-8B`.** Start from the instruct variant when your task is instruction-shaped. It already follows instructions, has a chat template, and needs far less data to specialise. Start from the base only for continued pretraining or when you want no inherited behaviour at all.\n\n**`attn_implementation=\"flash_attention_2\"`.** Halves memory on the attention path and speeds it up. Requires Ampere or newer; on a T4 omit this and PyTorch's SDPA backend will be used instead.\n\n**`device_map={\"\": 0}`.** Everything on GPU 0. Using `\"auto\"` on a single-GPU machine can silently offload layers to CPU when memory is tight, turning a slow run into a very slow one for reasons that are not logged prominently.\n\n**`target_modules=\"all-linear\"`.** Architecture-agnostic, so this config survives a change of model family.\n\n**`num_train_epochs=2`.** Two passes. Check the resulting step count: with 6,000 examples at an effective batch of 16 that is 750 steps, comfortably in range.\n\n**`learning_rate=2e-4`.** Standard for LoRA. For full fine-tuning this line becomes `2e-5`, and forgetting that is the single most expensive typo in this file.\n\n**`gradient_checkpointing_kwargs={\"use_reentrant\": False}`.** The non-reentrant implementation is the one that composes correctly with PEFT. With the old default you can get a run where gradients never reach the adapter and the loss is perfectly flat.\n\n**`packing=False` with `group_by_length=True`.** The conservative choice: most of the padding saving, none of the cross-document attention risk. Turn packing on once you have verified that your version separates documents correctly.\n\n**`assistant_only_loss=True`.** Loss on the assistant turns only. This requires the model's chat template to mark generation spans; if it does not, use the completion-only collator instead and pass the exact response template string.\n\n**`optim=\"paged_adamw_8bit\"`.** Paged, so a spike on one long batch is absorbed by CPU memory rather than ending the run.\n\n**`seed=0`.** Recorded, so the run is as reproducible as the hardware allows.\n\n## What changes for a full fine-tune\n\nThe same file, five lines different:\n\n```python\n# drop the BitsAndBytesConfig entirely, and peft_config=None\nlearning_rate = 2e-5          # not 2e-4\nweight_decay  = 0.01          # not 0\noptim         = \"adamw_bnb_8bit\"\nnum_train_epochs = 1          # full fine-tuning forgets faster\n```\n\nAnd the memory arithmetic from module three applies, so this configuration belongs on a sharded multi-GPU setup for anything above about 1.5B parameters.\n\n## The evaluation callback\n\nOne addition worth making from the first run rather than the fifth. Alongside `eval_loss`, generate on three fixed prompts at every evaluation step and log the text, and run the canary and refusal sets if they are cheap enough:\n\n```python\nclass Watch(TrainerCallback):\n    def on_evaluate(self, args, state, control, model=None, **kw):\n        for p in FIXED_PROMPTS:\n            print(state.global_step, generate(model, p)[:200])\n```\n\nEvery checkpoint then arrives with evidence attached, which is what makes checkpoint selection on a task metric possible rather than aspirational.\n\n## The four lines to change first\n\nIf something is wrong, in this order: `learning_rate`, `max_length` (check how many examples it truncates), `num_train_epochs`, and whether the masking line is actually taking effect.\n\n## Doing this without writing Python\n\nAxolotl and Llama-Factory express the same configuration as a YAML file and handle the wiring:\n\n```yaml\nbase_model: meta-llama/Llama-3.1-8B-Instruct\nload_in_4bit: true\nadapter: qlora\nlora_r: 16\nlora_alpha: 32\nlora_target_linear: true\nsequence_len: 2048\nmicro_batch_size: 2\ngradient_accumulation_steps: 8\nnum_epochs: 2\nlearning_rate: 0.0002\nlr_scheduler: cosine\nwarmup_ratio: 0.03\nbf16: auto\ngradient_checkpointing: true\ndatasets:\n  - path: ./train.jsonl\n    type: chat_template\n```\n\nBoth are free. For a first project a config file is better than a script, because a file diffs cleanly, versions naturally, and is much harder to make subtly inconsistent between two runs."},"check":{"question":"A run using QLoRA with gradient checkpointing produces a perfectly flat loss curve from step one, with a non-zero trainable-parameter count and a correct learning rate. Which config line is the likeliest culprit?","options":[{"text":"packing=False, which leaves padding in every batch and dilutes the gradient signal to nothing.","correct":false,"tempting":"Padding is masked out of the loss, so it dilutes throughput rather than gradient direction, and would never flatten the curve entirely."},{"text":"Reentrant gradient checkpointing, which can stop gradients reaching the adapter.","correct":true},{"text":"group_by_length=True, which sorts examples so early batches contain only the shortest sequences.","correct":false,"tempting":"Changes batch composition and therefore step-to-step noise, but does not stop the loss moving across a whole run."},{"text":"optim=\"paged_adamw_8bit\", whose quantised state cannot represent the small updates a LoRA adapter produces.","correct":false,"tempting":"Attributes a total failure to learn to optimiser quantisation, which is block-scaled and represents small updates without difficulty."}]},"prev":{"slug":"adding-your-own-tokens","title":"Adding tokens, and initialising them properly","url":"https://addaly.com/learn/fine-tuning/adding-your-own-tokens"},"next":{"slug":"debugging-a-run","title":"Debugging a run","url":"https://addaly.com/learn/fine-tuning/debugging-a-run"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A working config, line by line', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/a-config-line-by-line (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A working config, line by line","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"A working config is a set of decisions, not defaults: the instruct variant as base, all-linear targets, 2e-4 for LoRA, non-reentrant checkpointing so gradients reach the adapter, and length-grouped batching instead of packing until packing is verified.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-whole-thing-then-the-reasons","runs":[{"text":"The whole thing, then the reasons"}]},{"t":"p","runs":[{"text":"This is a complete supervised fine-tuning setup for an 8B model on a 24 GB card, using TRL, PEFT and bitsandbytes, all free, all open source."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\nfrom peft import LoraConfig\nfrom trl import SFTConfig, SFTTrainer\nimport torch\n\nMODEL = \"meta-llama/Llama-3.1-8B-Instruct\"\n\nbnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\",\n                         bnb_4bit_use_double_quant=True,\n                         bnb_4bit_compute_dtype=torch.bfloat16)\n\ntok = AutoTokenizer.from_pretrained(MODEL)\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL, quantization_config=bnb, torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\", device_map={\"\": 0})\n\npeft_cfg = LoraConfig(\n    r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n    target_modules=\"all-linear\", task_type=\"CAUSAL_LM\")\n\nargs = SFTConfig(\n    output_dir=\"out/triage-v3\",\n    num_train_epochs=2,\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=8,\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    warmup_ratio=0.03,\n    max_grad_norm=1.0,\n    bf16=True,\n    optim=\"paged_adamw_8bit\",\n    gradient_checkpointing=True,\n    gradient_checkpointing_kwargs={\"use_reentrant\": False},\n    max_length=2048,\n    packing=False,\n    group_by_length=True,\n    assistant_only_loss=True,\n    logging_steps=10,\n    eval_strategy=\"steps\", eval_steps=100,\n    save_steps=200, save_total_limit=3,\n    seed=0,\n    report_to=\"tensorboard\")\n\ntrainer = SFTTrainer(model=model, args=args, peft_config=peft_cfg,\n                     train_dataset=train, eval_dataset=val,\n                     processing_class=tok)\ntrainer.train()"},{"t":"h2","id":"the-lines-that-are-decisions","runs":[{"text":"The lines that are decisions"}]},{"t":"p","runs":[{"text":"Llama-3.1-8B-Instruct","b":true,"code":true},{"text":", not ","b":true},{"text":"-8B","b":true,"code":true},{"text":".","b":true},{"text":" Start from the instruct variant when your task is instruction-shaped. It already follows instructions, has a chat template, and needs far less data to specialise. Start from the base only for continued pretraining or when you want no inherited behaviour at all."}]},{"t":"p","runs":[{"text":"attn_implementation=\"flash_attention_2\"","b":true,"code":true},{"text":".","b":true},{"text":" Halves memory on the attention path and speeds it up. Requires Ampere or newer; on a T4 omit this and PyTorch’s SDPA backend will be used instead."}]},{"t":"p","runs":[{"text":"device_map={\"\": 0}","b":true,"code":true},{"text":".","b":true},{"text":" Everything on GPU 0. Using "},{"text":"\"auto\"","code":true},{"text":" on a single-GPU machine can silently offload layers to CPU when memory is tight, turning a slow run into a very slow one for reasons that are not logged prominently."}]},{"t":"p","runs":[{"text":"target_modules=\"all-linear\"","b":true,"code":true},{"text":".","b":true},{"text":" Architecture-agnostic, so this config survives a change of model family."}]},{"t":"p","runs":[{"text":"num_train_epochs=2","b":true,"code":true},{"text":".","b":true},{"text":" Two passes. Check the resulting step count: with 6,000 examples at an effective batch of 16 that is 750 steps, comfortably in range."}]},{"t":"p","runs":[{"text":"learning_rate=2e-4","b":true,"code":true},{"text":".","b":true},{"text":" Standard for LoRA. For full fine-tuning this line becomes "},{"text":"2e-5","code":true},{"text":", and forgetting that is the single most expensive typo in this file."}]},{"t":"p","runs":[{"text":"gradient_checkpointing_kwargs={\"use_reentrant\": False}","b":true,"code":true},{"text":".","b":true},{"text":" The non-reentrant implementation is the one that composes correctly with PEFT. With the old default you can get a run where gradients never reach the adapter and the loss is perfectly flat."}]},{"t":"p","runs":[{"text":"packing=False","b":true,"code":true},{"text":" with ","b":true},{"text":"group_by_length=True","b":true,"code":true},{"text":".","b":true},{"text":" The conservative choice: most of the padding saving, none of the cross-document attention risk. Turn packing on once you have verified that your version separates documents correctly."}]},{"t":"p","runs":[{"text":"assistant_only_loss=True","b":true,"code":true},{"text":".","b":true},{"text":" Loss on the assistant turns only. This requires the model’s chat template to mark generation spans; if it does not, use the completion-only collator instead and pass the exact response template string."}]},{"t":"p","runs":[{"text":"optim=\"paged_adamw_8bit\"","b":true,"code":true},{"text":".","b":true},{"text":" Paged, so a spike on one long batch is absorbed by CPU memory rather than ending the run."}]},{"t":"p","runs":[{"text":"seed=0","b":true,"code":true},{"text":".","b":true},{"text":" Recorded, so the run is as reproducible as the hardware allows."}]},{"t":"h2","id":"what-changes-for-a-full-fine-tune","runs":[{"text":"What changes for a full fine-tune"}]},{"t":"p","runs":[{"text":"The same file, five lines different:"}]},{"t":"code","lang":"python","label":"Python","text":"# drop the BitsAndBytesConfig entirely, and peft_config=None\nlearning_rate = 2e-5          # not 2e-4\nweight_decay  = 0.01          # not 0\noptim         = \"adamw_bnb_8bit\"\nnum_train_epochs = 1          # full fine-tuning forgets faster"},{"t":"p","runs":[{"text":"And the memory arithmetic from module three applies, so this configuration belongs on a sharded multi-GPU setup for anything above about 1.5B parameters."}]},{"t":"h2","id":"the-evaluation-callback","runs":[{"text":"The evaluation callback"}]},{"t":"p","runs":[{"text":"One addition worth making from the first run rather than the fifth. Alongside "},{"text":"eval_loss","code":true},{"text":", generate on three fixed prompts at every evaluation step and log the text, and run the canary and refusal sets if they are cheap enough:"}]},{"t":"code","lang":"python","label":"Python","text":"class Watch(TrainerCallback):\n    def on_evaluate(self, args, state, control, model=None, **kw):\n        for p in FIXED_PROMPTS:\n            print(state.global_step, generate(model, p)[:200])"},{"t":"p","runs":[{"text":"Every checkpoint then arrives with evidence attached, which is what makes checkpoint selection on a task metric possible rather than aspirational."}]},{"t":"h2","id":"the-four-lines-to-change-first","runs":[{"text":"The four lines to change first"}]},{"t":"p","runs":[{"text":"If something is wrong, in this order: "},{"text":"learning_rate","code":true},{"text":", "},{"text":"max_length","code":true},{"text":" (check how many examples it truncates), "},{"text":"num_train_epochs","code":true},{"text":", and whether the masking line is actually taking effect."}]},{"t":"h2","id":"doing-this-without-writing-python","runs":[{"text":"Doing this without writing Python"}]},{"t":"p","runs":[{"text":"Axolotl and Llama-Factory express the same configuration as a YAML file and handle the wiring:"}]},{"t":"code","lang":"yaml","label":"YAML","text":"base_model: meta-llama/Llama-3.1-8B-Instruct\nload_in_4bit: true\nadapter: qlora\nlora_r: 16\nlora_alpha: 32\nlora_target_linear: true\nsequence_len: 2048\nmicro_batch_size: 2\ngradient_accumulation_steps: 8\nnum_epochs: 2\nlearning_rate: 0.0002\nlr_scheduler: cosine\nwarmup_ratio: 0.03\nbf16: auto\ngradient_checkpointing: true\ndatasets:\n  - path: ./train.jsonl\n    type: chat_template"},{"t":"p","runs":[{"text":"Both are free. For a first project a config file is better than a script, because a file diffs cleanly, versions naturally, and is much harder to make subtly inconsistent between two runs."}]}],"check":{"question":[{"text":"A run using QLoRA with gradient checkpointing produces a perfectly flat loss curve from step one, with a non-zero trainable-parameter count and a correct learning rate. Which config line is the likeliest culprit?"}],"options":[{"text":[{"text":"packing=False, which leaves padding in every batch and dilutes the gradient signal to nothing."}],"correct":false,"tempting":[{"text":"Padding is masked out of the loss, so it dilutes throughput rather than gradient direction, and would never flatten the curve entirely."}]},{"text":[{"text":"Reentrant gradient checkpointing, which can stop gradients reaching the adapter."}],"correct":true,"tempting":null},{"text":[{"text":"group_by_length=True, which sorts examples so early batches contain only the shortest sequences."}],"correct":false,"tempting":[{"text":"Changes batch composition and therefore step-to-step noise, but does not stop the loss moving across a whole run."}]},{"text":[{"text":"optim=”paged_adamw_8bit”, whose quantised state cannot represent the small updates a LoRA adapter produces."}],"correct":false,"tempting":[{"text":"Attributes a total failure to learn to optimiser quantisation, which is block-scaled and represents small updates without difficulty."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"adding-your-own-tokens","title":"Adding tokens, and initialising them properly","minutes":8},"next":{"slug":"debugging-a-run","title":"Debugging a run","minutes":10}}}