{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"trl-and-instruction-tuning","title":"TRL, and training on conversations","url":"https://addaly.com/learn/hugging-face/trl-and-instruction-tuning","position":62,"of":75,"minutes":9,"answer":"SFTTrainer renders chat templates, masks the prompt, packs and truncates conversations for you, so the remaining decisions are about data: a few hundred diverse.","keyIdea":"`SFTTrainer` renders chat templates, masks the prompt, packs and truncates conversations for you, so the remaining decisions are about data: a few hundred diverse, well-written examples in the messages format beat tens of thousands of scraped ones, and the register of your responses becomes the model's register."},"body":{"format":"markdown","text":"## Why a separate library\n\n`Trainer` trains on tensors. Instruction data is conversations, which have to be templated, masked so loss falls only on the assistant's turns, packed sensibly and truncated without cutting a turn in half. Module five did that by hand once, which is the right way to understand it and the wrong way to do it every time.\n\nTRL's `SFTTrainer` does all of it from a dataset of messages.\n\n```bash\npip install trl\n```\n\n```python\nfrom trl import SFTTrainer, SFTConfig\n\n# each row: {\"messages\": [{\"role\": \"user\", ...}, {\"role\": \"assistant\", ...}]}\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=ds[\"train\"],\n    eval_dataset=ds[\"validation\"],\n    peft_config=lora_cfg,\n    args=SFTConfig(\n        output_dir=\"out\",\n        max_length=1024,\n        packing=True,\n        learning_rate=2e-4,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=4,\n        bf16=True,\n    ),\n)\ntrainer.train()\n```\n\n`SFTConfig` extends `TrainingArguments`, so everything from the previous lesson still applies. `peft_config` wires LoRA in without a separate `get_peft_model` call.\n\n## The data format\n\nThe conversational format is a `messages` list per row, and it is the one to use because the tokenizer's own chat template renders it:\n\n```json\n{\"messages\": [\n  {\"role\": \"system\", \"content\": \"You answer in Hindi.\"},\n  {\"role\": \"user\", \"content\": \"पार्सल कब आएगा?\"},\n  {\"role\": \"assistant\", \"content\": \"आपका पार्सल मंगलवार तक पहुँच जाएगा।\"}\n]}\n```\n\nStoring messages rather than rendered prompts is the same rule as module two, and here it pays twice: the same dataset trains any model family, because rendering happens at training time with that model's template.\n\n## What makes instruction data good\n\nThis matters more than any hyperparameter, and it is where people underinvest.\n\n- **Quality over quantity.** A thousand carefully written examples beat fifty thousand scraped ones. Published work on instruction tuning has repeatedly found small curated sets competitive with much larger noisy ones.\n- **Diversity of instruction, not of topic.** Twenty ways of asking for the same thing teaches robustness; twenty topics with one phrasing each does not.\n- **Responses in the register you want.** The model copies length, formality and structure. If your examples are three paragraphs, expect three paragraphs forever.\n- **Include refusals and edge cases.** If every example has a confident answer, the model learns that confident answers are always available.\n- **Check for leakage** between your training conversations and your evaluation ones. Module five's check applies unchanged.\n\n## Packing, and its caveat\n\n`packing=True` concatenates short conversations into full-length blocks so no compute is spent on padding. On a dataset of short exchanges this is a large speed-up.\n\nThe caveat: packed examples share a sequence, so without attention masking between them one conversation can attend to the previous one. TRL handles this correctly for supported models; a hand-rolled implementation usually does not, which is why doing it yourself is a good exercise and a bad production choice.\n\n## Preference tuning, briefly\n\nAfter supervised tuning, preference methods adjust a model toward chosen responses and away from rejected ones. `DPOTrainer` implements direct preference optimisation, which needs no separate reward model:\n\n```python\n# each row: {\"prompt\": ..., \"chosen\": ..., \"rejected\": ...}\nfrom trl import DPOTrainer, DPOConfig\n```\n\nThis is what turns a model that *can* answer into one that answers the way people prefer. It needs preference pairs, which are more expensive to collect than instructions, and it is a second stage rather than a replacement for the first. TRL also implements GRPO and related methods for training against a programmatic reward: useful when correctness is checkable, such as code that must compile or arithmetic that must be right.\n\nFor most people reading this, supervised tuning is enough and preference tuning is a later problem. Knowing the sequence (pretrain, instruction tune, preference tune) explains why the models you download behave as they do.\n\n## Before you launch a long run\n\n```python\nb = next(iter(trainer.get_train_dataloader()))\nprint(tok.decode(b[\"input_ids\"][0]))\nprint(tok.decode([t for t in b[\"labels\"][0] if t != -100]))\n```\n\nRead both. The first is what the model sees; the second is what it is graded on. If the second contains the user's words, your masking is wrong and three hours are about to be wasted.\n\n## Do this now\n\nBuild fifty instruction examples by hand for a task you care about, in the messages format, and run one epoch. Fifty is too few to work and enough to prove the pipeline. Then spend the time you saved on making two hundred good examples rather than two thousand mediocre ones."},"check":{"question":"A team fine-tunes on 40,000 scraped instruction pairs and gets worse results than a colleague using 800 hand-written ones. What best explains it?","options":[{"text":"The larger dataset needed a lower learning rate, and the run diverged before converging properly.","correct":false,"tempting":"A tuning issue would show in the loss curve rather than producing a consistently weaker model."},{"text":"40,000 examples require more epochs to see each example enough times, so the run was undertrained.","correct":false,"tempting":"Inverts the relationship; more data means more updates per epoch, not fewer."},{"text":"The model copies the register and quality of its examples, so noisy scraped responses teach noisy responses.","correct":true},{"text":"Packing was enabled on the larger set, so unrelated conversations attended to each other and corrupted training.","correct":false,"tempting":"A real hazard in hand-rolled packing, but TRL masks between packed examples for supported models."}]},"prev":{"slug":"qlora-on-one-card","title":"QLoRA, and fine-tuning a 7B model on a free GPU","url":"https://addaly.com/learn/hugging-face/qlora-on-one-card"},"next":{"slug":"tracking-and-checkpointing","title":"Watching a run you cannot sit next to","url":"https://addaly.com/learn/hugging-face/tracking-and-checkpointing"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'TRL, and training on conversations', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/trl-and-instruction-tuning (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"TRL, and training on conversations","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"`SFTTrainer` renders chat templates, masks the prompt, packs and truncates conversations for you, so the remaining decisions are about data: a few hundred diverse, well-written examples in the messages format beat tens of thousands of scraped ones, and the register of your responses becomes the model’s register.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-a-separate-library","runs":[{"text":"Why a separate library"}]},{"t":"p","runs":[{"text":"Trainer","code":true},{"text":" trains on tensors. Instruction data is conversations, which have to be templated, masked so loss falls only on the assistant’s turns, packed sensibly and truncated without cutting a turn in half. Module five did that by hand once, which is the right way to understand it and the wrong way to do it every time."}]},{"t":"p","runs":[{"text":"TRL’s "},{"text":"SFTTrainer","code":true},{"text":" does all of it from a dataset of messages."}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install trl"},{"t":"code","lang":"python","label":"Python","text":"from trl import SFTTrainer, SFTConfig\n\n# each row: {\"messages\": [{\"role\": \"user\", ...}, {\"role\": \"assistant\", ...}]}\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=ds[\"train\"],\n    eval_dataset=ds[\"validation\"],\n    peft_config=lora_cfg,\n    args=SFTConfig(\n        output_dir=\"out\",\n        max_length=1024,\n        packing=True,\n        learning_rate=2e-4,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=4,\n        bf16=True,\n    ),\n)\ntrainer.train()"},{"t":"p","runs":[{"text":"SFTConfig","code":true},{"text":" extends "},{"text":"TrainingArguments","code":true},{"text":", so everything from the previous lesson still applies. "},{"text":"peft_config","code":true},{"text":" wires LoRA in without a separate "},{"text":"get_peft_model","code":true},{"text":" call."}]},{"t":"h2","id":"the-data-format","runs":[{"text":"The data format"}]},{"t":"p","runs":[{"text":"The conversational format is a "},{"text":"messages","code":true},{"text":" list per row, and it is the one to use because the tokenizer’s own chat template renders it:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"messages\": [\n  {\"role\": \"system\", \"content\": \"You answer in Hindi.\"},\n  {\"role\": \"user\", \"content\": \"पार्सल कब आएगा?\"},\n  {\"role\": \"assistant\", \"content\": \"आपका पार्सल मंगलवार तक पहुँच जाएगा।\"}\n]}"},{"t":"p","runs":[{"text":"Storing messages rather than rendered prompts is the same rule as module two, and here it pays twice: the same dataset trains any model family, because rendering happens at training time with that model’s template."}]},{"t":"h2","id":"what-makes-instruction-data-good","runs":[{"text":"What makes instruction data good"}]},{"t":"p","runs":[{"text":"This matters more than any hyperparameter, and it is where people underinvest."}]},{"t":"ul","items":[[{"text":"Quality over quantity.","b":true},{"text":" A thousand carefully written examples beat fifty thousand scraped ones. Published work on instruction tuning has repeatedly found small curated sets competitive with much larger noisy ones."}],[{"text":"Diversity of instruction, not of topic.","b":true},{"text":" Twenty ways of asking for the same thing teaches robustness; twenty topics with one phrasing each does not."}],[{"text":"Responses in the register you want.","b":true},{"text":" The model copies length, formality and structure. If your examples are three paragraphs, expect three paragraphs forever."}],[{"text":"Include refusals and edge cases.","b":true},{"text":" If every example has a confident answer, the model learns that confident answers are always available."}],[{"text":"Check for leakage","b":true},{"text":" between your training conversations and your evaluation ones. Module five’s check applies unchanged."}]]},{"t":"h2","id":"packing-and-its-caveat","runs":[{"text":"Packing, and its caveat"}]},{"t":"p","runs":[{"text":"packing=True","code":true},{"text":" concatenates short conversations into full-length blocks so no compute is spent on padding. On a dataset of short exchanges this is a large speed-up."}]},{"t":"p","runs":[{"text":"The caveat: packed examples share a sequence, so without attention masking between them one conversation can attend to the previous one. TRL handles this correctly for supported models; a hand-rolled implementation usually does not, which is why doing it yourself is a good exercise and a bad production choice."}]},{"t":"h2","id":"preference-tuning-briefly","runs":[{"text":"Preference tuning, briefly"}]},{"t":"p","runs":[{"text":"After supervised tuning, preference methods adjust a model toward chosen responses and away from rejected ones. "},{"text":"DPOTrainer","code":true},{"text":" implements direct preference optimisation, which needs no separate reward model:"}]},{"t":"code","lang":"python","label":"Python","text":"# each row: {\"prompt\": ..., \"chosen\": ..., \"rejected\": ...}\nfrom trl import DPOTrainer, DPOConfig"},{"t":"p","runs":[{"text":"This is what turns a model that "},{"text":"can","i":true},{"text":" answer into one that answers the way people prefer. It needs preference pairs, which are more expensive to collect than instructions, and it is a second stage rather than a replacement for the first. TRL also implements GRPO and related methods for training against a programmatic reward: useful when correctness is checkable, such as code that must compile or arithmetic that must be right."}]},{"t":"p","runs":[{"text":"For most people reading this, supervised tuning is enough and preference tuning is a later problem. Knowing the sequence (pretrain, instruction tune, preference tune) explains why the models you download behave as they do."}]},{"t":"h2","id":"before-you-launch-a-long-run","runs":[{"text":"Before you launch a long run"}]},{"t":"code","lang":"python","label":"Python","text":"b = next(iter(trainer.get_train_dataloader()))\nprint(tok.decode(b[\"input_ids\"][0]))\nprint(tok.decode([t for t in b[\"labels\"][0] if t != -100]))"},{"t":"p","runs":[{"text":"Read both. The first is what the model sees; the second is what it is graded on. If the second contains the user’s words, your masking is wrong and three hours are about to be wasted."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Build fifty instruction examples by hand for a task you care about, in the messages format, and run one epoch. Fifty is too few to work and enough to prove the pipeline. Then spend the time you saved on making two hundred good examples rather than two thousand mediocre ones."}]}],"check":{"question":[{"text":"A team fine-tunes on 40,000 scraped instruction pairs and gets worse results than a colleague using 800 hand-written ones. What best explains it?"}],"options":[{"text":[{"text":"The larger dataset needed a lower learning rate, and the run diverged before converging properly."}],"correct":false,"tempting":[{"text":"A tuning issue would show in the loss curve rather than producing a consistently weaker model."}]},{"text":[{"text":"40,000 examples require more epochs to see each example enough times, so the run was undertrained."}],"correct":false,"tempting":[{"text":"Inverts the relationship; more data means more updates per epoch, not fewer."}]},{"text":[{"text":"The model copies the register and quality of its examples, so noisy scraped responses teach noisy responses."}],"correct":true,"tempting":null},{"text":[{"text":"Packing was enabled on the larger set, so unrelated conversations attended to each other and corrupted training."}],"correct":false,"tempting":[{"text":"A real hazard in hand-rolled packing, but TRL masks between packed examples for supported models."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"qlora-on-one-card","title":"QLoRA, and fine-tuning a 7B model on a free GPU","minutes":8},"next":{"slug":"tracking-and-checkpointing","title":"Watching a run you cannot sit next to","minutes":8}}}