The Dataset Decides Everything

Build the test set before the training set

Do this first, before you write a single training example.

Take 100-300 real inputs from real traffic. Not invented ones — invented inputs are always cleaner, shorter and more polite than what people actually send. Write or approve the ideal output for each. Freeze the file. Never train on it. Split it in half: a dev half you look at while iterating, and a test half you open exactly twice, once at the start to record the base model's score and once at the end.

Teams that skip this cannot answer the only question that matters at the end, which is whether the work helped. They ship on vibes, and vibes favour whatever they just spent a week building.

Then write twenty by hand

Yourself. Not generated, not delegated. You will discover that the task you thought was well defined has four cases you had never decided, and that two of your "obviously correct" outputs contradict each other. Better to find that now than inside a 4,000-row dataset where the contradiction is trained in.

Those twenty become the style guide for everything that follows.

The model learns the whole distribution

Including the parts you were not thinking about.

• If 30% of your answers open with "Certainly," the tuned model opens with "Certainly."
• If your answers average 400 words because you wrote them carefully, it will write 400 words in reply to a yes-or-no question.
• If every input is a well-formed question, it will handle a typo-ridden fragment badly.
• If no example says "I don't know," it will never say it.

So vary the inputs hard — length, register, typos, ambiguity, off-topic, hostile, and the other languages you serve — and keep the outputs consistent in the ways you care about. Include the hard cases, the refusals, the ones where the right answer is a clarifying question. A dataset of easy cases produces a model that is confident on hard ones.

Deduplicate

Near-duplicates inflate your evaluation and waste training. Exact dedup on normalised text takes a minute:

import json, hashlib
seen, out = set(), []
for line in open("train.jsonl"):
    row = json.loads(line)
    text = row["messages"][0]["content"]
    key = hashlib.sha1(" ".join(text.split()).lower().encode()).hexdigest()
    if key not in seen:
        seen.add(key)
        out.append(row)
print(len(out))

Then check the same way for overlap between train and test. If a test input also appears in training, your score is fiction.

Synthetic data, honestly

Generating training data by prompting a stronger model is normal, effective, and widely done. Two conditions. Check the provider's terms, because several forbid using outputs to train competing models. And understand what you are buying: the stronger model's style, its mistakes and its blind spots, transferred faithfully. Have a human read and fix a sample. A 500-example set you have actually read beats a 5,000-example set nobody has.

Never draw the test set from the same generator. It would measure agreement with the generator, not quality.

How many examples

Honest ranges, assuming clean data and LoRA on a competent instruct base:

• 50-200 — tone, register, format, output schema. This works better than people expect, and it is where most teams should stop.
• 500-2,000 — a narrow task with real accuracy gains: routing, extraction, structured summarisation in your house form.
• 5,000-50,000 — a genuinely new behaviour, or wide domain coverage. Expect weeks of data work.

LIMA is the paper everyone cites for "a thousand examples is enough." Read what it measured: human preference for helpfulness and style, starting from a strong 65B base. It is real evidence about style. It is not evidence that a thousand examples teaches a task that requires being right.

Find your own number with an ablation

Do not guess. Train on 25%, 50% and 100% of your data with everything else fixed, and score each on the dev set.

• Still climbing at 100%? More of the same data will help.
• Flat between 50% and 100%? More of the same will not. You need *different* data — harder cases, different input styles — or the bottleneck is somewhere else entirely.

Three small runs cost a few dollars and replace an argument that would otherwise last a month.

One aside worth the whole lesson

If your task is classification over a fixed label set and you have 5,000 or more labelled examples, fine-tune a small encoder model — the BERT family, 100-400M parameters — before you fine-tune an LLM. It is often more accurate, it serves on a CPU for a fraction of the cost, and it trains in twenty minutes. The unfashionable answer is frequently the right one.