TRL, and training on conversations

Why a separate library

Trainer trains on tensors. Instruction data is conversations, which have to be templated, masked so loss falls only on the assistant’s turns, packed sensibly and truncated without cutting a turn in half. Module five did that by hand once, which is the right way to understand it and the wrong way to do it every time.

TRL’s SFTTrainer does all of it from a dataset of messages.

pip install trl

from trl import SFTTrainer, SFTConfig

# each row: {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}
trainer = SFTTrainer(
    model=model,
    train_dataset=ds["train"],
    eval_dataset=ds["validation"],
    peft_config=lora_cfg,
    args=SFTConfig(
        output_dir="out",
        max_length=1024,
        packing=True,
        learning_rate=2e-4,
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        bf16=True,
    ),
)
trainer.train()

SFTConfig extends TrainingArguments, so everything from the previous lesson still applies. peft_config wires LoRA in without a separate get_peft_model call.

The data format

The conversational format is a messages list per row, and it is the one to use because the tokenizer’s own chat template renders it:

{"messages": [
  {"role": "system", "content": "You answer in Hindi."},
  {"role": "user", "content": "पार्सल कब आएगा?"},
  {"role": "assistant", "content": "आपका पार्सल मंगलवार तक पहुँच जाएगा।"}
]}

Storing messages rather than rendered prompts is the same rule as module two, and here it pays twice: the same dataset trains any model family, because rendering happens at training time with that model’s template.

What makes instruction data good

This matters more than any hyperparameter, and it is where people underinvest.

• Quality over quantity. A thousand carefully written examples beat fifty thousand scraped ones. Published work on instruction tuning has repeatedly found small curated sets competitive with much larger noisy ones.
• Diversity of instruction, not of topic. Twenty ways of asking for the same thing teaches robustness; twenty topics with one phrasing each does not.
• Responses in the register you want. The model copies length, formality and structure. If your examples are three paragraphs, expect three paragraphs forever.
• Include refusals and edge cases. If every example has a confident answer, the model learns that confident answers are always available.
• Check for leakage between your training conversations and your evaluation ones. Module five’s check applies unchanged.

Packing, and its caveat

packing=True concatenates short conversations into full-length blocks so no compute is spent on padding. On a dataset of short exchanges this is a large speed-up.

The caveat: packed examples share a sequence, so without attention masking between them one conversation can attend to the previous one. TRL handles this correctly for supported models; a hand-rolled implementation usually does not, which is why doing it yourself is a good exercise and a bad production choice.

Preference tuning, briefly

After supervised tuning, preference methods adjust a model toward chosen responses and away from rejected ones. DPOTrainer implements direct preference optimisation, which needs no separate reward model:

# each row: {"prompt": ..., "chosen": ..., "rejected": ...}
from trl import DPOTrainer, DPOConfig

This is what turns a model that can answer into one that answers the way people prefer. It needs preference pairs, which are more expensive to collect than instructions, and it is a second stage rather than a replacement for the first. TRL also implements GRPO and related methods for training against a programmatic reward: useful when correctness is checkable, such as code that must compile or arithmetic that must be right.

For most people reading this, supervised tuning is enough and preference tuning is a later problem. Knowing the sequence (pretrain, instruction tune, preference tune) explains why the models you download behave as they do.

Before you launch a long run

b = next(iter(trainer.get_train_dataloader()))
print(tok.decode(b["input_ids"][0]))
print(tok.decode([t for t in b["labels"][0] if t != -100]))

Read both. The first is what the model sees; the second is what it is graded on. If the second contains the user’s words, your masking is wrong and three hours are about to be wasted.

Do this now

Build fifty instruction examples by hand for a task you care about, in the messages format, and run one epoch. Fifty is too few to work and enough to prove the pipeline. Then spend the time you saved on making two hundred good examples rather than two thousand mediocre ones.