Five hundred examples, done properly

The dataset is the model

Everything about a fine-tuned model that you did not inherit from the base came from your examples. Every inconsistency in your labels becomes an inconsistency in the model. Every systematic error becomes a systematic behaviour. This is the part of the work that decides the outcome, and it is the part people rush.

How many, and of what

300 to 1,000 examples is enough for most behavioural tuning. Below 200 the run is unstable and few-shot prompting is usually as good; above a few thousand the returns flatten unless the task is broad.

Quality dominates quantity at this scale. A careful 400 beats a careless 4,000, because the careless set teaches the model to be careless in the same places.

The format

One conversation per line, as JSONL:

{"messages":[{"role":"system","content":"Classify the ticket."},
             {"role":"user","content":"Card declined twice at checkout."},
             {"role":"assistant","content":"{\"label\":\"billing\",\"urgency\":\"high\"}"}]}

The trainer applies the model’s chat template and, correctly configured, computes loss only on the assistant turn. Check this: training on the loss of the user’s tokens teaches the model to generate user messages, and it is a real and quiet defect in hand-rolled training scripts.

Six properties that matter, in order

1. Diversity of inputs. More important than volume. Four hundred examples covering the full range of what arrives beats four thousand variations on the same three shapes. Sample from real traffic across time, source and user type, not from whatever was easy to collect.

2. Consistency of output format. Byte-identical. Same key order, same date format, same capitalisation, same phrasing of the standard sentences. The model copies your format exactly, including the inconsistency.

3. The edge cases you want handled. Empty input, off-topic input, ambiguous input, the case that should be escalated, the thing that should be refused. If none of your 500 examples shows an abstention, the model will never abstain.

4. Real inputs, not invented ones. Invented examples are cleaner, better punctuated and more polite than what people send. A model trained on them handles real messiness worse, and the gap only appears in production.

5. Label consistency you have measured. Take 30 items, have a second person label them independently, and count agreement. If two careful people agree 80% of the time, the ceiling on your model is around 80%, and no amount of tuning gets past your own disagreement. This measurement takes an hour and reframes the whole project.

6. Splits, held from the start. Training, development and a held-out set you touch once. Deduplicate across them: near-duplicates leaking from training into held-out is the most common reason a fine-tune reports a wonderful number and disappoints in production.

Where the data comes from

Production logs, corrected by a person. The best source by a distance, because the inputs are real by construction. Take the model’s current output, have someone fix it, keep the fixed version. Correction is several times faster than writing from scratch.

A larger model, with the terms checked. Generate candidate outputs with a stronger model, then have a human accept, edit or reject each. Two cautions: the terms of service of closed commercial APIs generally prohibit using their outputs to train competing models, and an open-weights teacher passes its own licence conditions to the result. And the student inherits the teacher’s style and its errors, so review rather than accept wholesale.

Templating and augmentation. Useful for coverage of rare categories and dangerous as the bulk of a set, because templated variety is not real variety and the model learns the template.

The hour that saves the project

Before training, read 50 of your examples end to end as if you were the model. Ask of each: is the output actually derivable from the input? A surprising number of hand-built datasets contain examples where the correct answer depended on information the model was never given: a system the annotator could see, a convention they knew, context from the previous ticket.

Every such example teaches the model to guess. Remove them or add the missing context, and the model gets better at the whole task, not just those items.