Where preference data comes from

Four sources, and their biases

1. Product feedback. Thumbs up and thumbs down on your existing model’s answers. Free, continuous, and already flowing if you built the button. The catch is that it is binary and unpaired: you have judgements on single responses, not comparisons between two. Most preference algorithms want pairs. KTO, covered in the next-but-one lesson, is the one designed for exactly this data, and that is the reason to know it exists.

The bias: people rate when annoyed far more than when satisfied. A raw thumbs dataset is heavily weighted towards failures, and towards the kinds of failure that are noticeable rather than the kinds that are serious.

2. Pairwise human comparison. Generate two responses to the same input, show both, ask which is better. This is the clean source, and it is expensive: 30 to 90 seconds per comparison for a careful judgement, so 3,000 pairs is 40 to 70 hours.

Three biases to control. Position bias: annotators favour whichever is shown first, so randomise order and record the randomisation. Length bias: longer answers are preferred at rates that exceed their actual quality, which is the single most documented effect in this area. Fluency bias: a confidently wrong answer beats a hedged correct one unless the annotator is asked specifically about correctness.

3. AI feedback. Ask a model which of two responses is better. Cheap, fast, consistent, and it makes tens of thousands of pairs affordable. It carries the same length bias as humans and adds self-preference: a model asked to judge tends to prefer text produced by itself or by a model like it.

The workable version is layered: have the model judge everything, have a person judge 200 of the same pairs, and compute the agreement rate. If model and human agree 85% of the time, the model’s labels are useful with a stated caveat. If they agree 61% of the time, you have a random number generator with good manners.

4. Constructed pairs. The cheapest source and the most under-used. Where you already know what bad looks like, build the pair rather than collecting it.

pair = {
  "prompt": ticket,
  "chosen":  good_answer,                     # your SFT gold answer
  "rejected": model_output_before_sft,        # the failure you are fixing
}

Or generate deliberately degraded versions: the same answer with the apology restored, with the JSON keys reordered, with an invented order number inserted. Each targets one specific behaviour, which is exactly what preference learning is good at.

The on-policy requirement

This is the piece most often got wrong. The rejected response should be something your current model actually produces. A pair whose rejected side came from a different model, or from a much older checkpoint, is teaching your model to avoid behaviour it was not going to exhibit anyway, and worse, the gradient that pushes down an already-unlikely response can push down neighbouring plausible text with it.

The correct loop:

1. Take prompts from your traffic.
2. Generate several responses from the model you are about to train, at a temperature that gives variety.
3. Rank them, by a person, a judge or a verifier.
4. Pair the best against the worst.
5. Train. Then regenerate, because the model has moved.

That last line is why preference tuning is iterative in a way SFT is not. After one round the model’s failures are different failures, and a preference set collected against the previous checkpoint is partly stale.

How many pairs

Fewer than people expect, if they are on-policy and targeted.

• 500 to 1,000 for one specific behaviour (a format habit, a tone, a length preference) with constructed or tightly-filtered pairs.
• 3,000 to 10,000 for general helpfulness or a broad style shift.
• Beyond that, returns diminish quickly relative to the labelling cost, and your effort is better spent on the quality of the pairs than on their number.

The pair quality test

Before training, read fifty pairs and ask of each: can I state in one clause why the chosen one is better?

If you cannot (if they are two acceptable answers and somebody picked one) that pair is noise. It contributes a gradient with no consistent direction, and enough of them will teach the model something arbitrary with the same confidence as everything else. Drop the pairs where the margin is thin. A preference dataset of 800 clear comparisons beats one of 5,000 where half are coin flips, and the second is much more expensive.