The dataset is the project

Five hundred good examples beat fifty thousand scraped ones

Every decision in a fine-tuning project is downstream of the data, and the amount of data people imagine they need is usually an order of magnitude more than they do. For a behaviour, a format, or a classification boundary, several hundred carefully chosen examples routinely outperform tens of thousands of noisy ones, because the noisy set teaches the noise too, and the model has no way to tell which parts you meant.

Spend the effort here. A week on the dataset and an afternoon on hyperparameters is the correct ratio, and it is almost always inverted by people doing this for the first time.

Use the template you will serve with

The most common silent failure in fine-tuning is a training format that does not match the inference format. If you train on plain text like Q: ... A: ... and then serve through a chat endpoint that wraps messages in the model’s own special tokens, the model at inference time is seeing a structure it never trained on. Behaviour is degraded, nothing errors, and the cause is invisible.

Store examples as messages and render them with the model’s own template:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(MODEL)
row = {"messages": [
  {"role": "system", "content": "You are a triage assistant for a clinic."},
  {"role": "user", "content": "Appointment moved to Friday, please confirm."},
  {"role": "assistant", "content": "{\"intent\":\"reschedule\",\"urgency\":\"low\"}"},
]}
print(tok.apply_chat_template(row["messages"], tokenize=False))

Print that rendered string and read it before you train anything. Ten seconds of reading catches a missing system turn, a doubled beginning-of-sequence token, or a template that puts the assistant’s reply somewhere you did not expect.

Mask the prompt, train on the response

By default, a language model trains on every token in the sequence, including the user’s message. That teaches it to generate user turns, which is not what you want, and it dilutes the gradient with tokens whose content you are not trying to change.

Instruction tuning should compute loss on the assistant’s tokens only. Most frameworks expose this directly; in TRL it is a completion-only collator, and Unsloth ships a train_on_responses_only helper. It is one line and it reliably improves the result.

The exception is worth knowing: if you are teaching a completion behaviour rather than a chat behaviour, training on the whole sequence is correct. The rule is that loss should be computed on exactly the tokens the model will be asked to produce.

What to put in, beyond the happy path

A dataset of successes teaches a model to always succeed, including when it should not.

• Include refusals and uncertainty. If you never show it an input it should decline or ask about, it will confidently handle those too. A tenth of the set being “I do not have enough information to do this” changes the model’s behaviour on ambiguous inputs markedly.
• Include the awkward shapes. Empty fields, the wrong language, a message that is three words long, a message that is four pages. Whatever your real traffic contains.
• Watch the length distribution. Output length is learned. If every training response is two sentences, you will get two sentences forever, including when the user asks for a detailed explanation. Vary it deliberately to match what you want.
• Balance the classes you care about. A rare label at 2% of the data will stay rare in the model’s behaviour. Oversample it, or accept that it will be under-predicted.

Split before you look

Hold out 10 to 20% before you start, stratified by the input shapes and the labels that matter, and do not look at it. Every decision you make after seeing the held-out set leaks into it, and the number it gives you afterwards is no longer a measurement.

Keep a third, smaller set that is not from the same source at all (collected a month later, or from a different team) because a train-and-test split from one collection shares its quirks, and a model that looks excellent on both can still fail on real traffic for reasons the collection never contained.

Provenance, recorded as you go

Write down where every example came from, at the time, in the file. It takes seconds then and is impossible to reconstruct later.

This matters practically and legally. Data scraped from a site carries that site’s terms. Data from a public dataset carries its licence, which is frequently more restrictive than people assume. Output generated by a hosted model carries that provider’s terms of use, and several providers restrict using their output to develop competing models. Whether a narrow task adapter counts as a competing model is contested and largely untested. Personal data in a training set raises its own obligations, complicated by the fact that removing an example from a dataset does not remove what a model learned from it.

This course cannot tell you your legal position and is not legal advice. What it can tell you is that a provenance column costs nothing to keep and is the only thing that lets anyone answer these questions later.