{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"where-preference-data-comes-from","title":"Where preference data comes from","url":"https://addaly.com/learn/fine-tuning/where-preference-data-comes-from","position":40,"of":84,"minutes":9,"answer":"Preference pairs must be on-policy (the rejected side generated by the model you are about to train) or you are teaching it to avoid behaviour it never had.","keyIdea":"Preference pairs must be on-policy (the rejected side generated by the model you are about to train) or you are teaching it to avoid behaviour it never had; and pairs where you cannot state in one clause why the chosen answer is better are noise that trains something arbitrary."},"body":{"format":"markdown","text":"## Four sources, and their biases\n\n**1. Product feedback.** Thumbs up and thumbs down on your existing model's answers. Free, continuous, and already flowing if you built the button. The catch is that it is *binary and unpaired*: you have judgements on single responses, not comparisons between two. Most preference algorithms want pairs. KTO, covered in the next-but-one lesson, is the one designed for exactly this data, and that is the reason to know it exists.\n\nThe bias: people rate when annoyed far more than when satisfied. A raw thumbs dataset is heavily weighted towards failures, and towards the kinds of failure that are noticeable rather than the kinds that are serious.\n\n**2. Pairwise human comparison.** Generate two responses to the same input, show both, ask which is better. This is the clean source, and it is expensive: 30 to 90 seconds per comparison for a careful judgement, so 3,000 pairs is 40 to 70 hours.\n\nThree biases to control. **Position bias**: annotators favour whichever is shown first, so randomise order and record the randomisation. **Length bias**: longer answers are preferred at rates that exceed their actual quality, which is the single most documented effect in this area. **Fluency bias**: a confidently wrong answer beats a hedged correct one unless the annotator is asked specifically about correctness.\n\n**3. AI feedback.** Ask a model which of two responses is better. Cheap, fast, consistent, and it makes tens of thousands of pairs affordable. It carries the same length bias as humans and adds self-preference: a model asked to judge tends to prefer text produced by itself or by a model like it.\n\nThe workable version is layered: have the model judge everything, have a person judge 200 of the same pairs, and compute the agreement rate. If model and human agree 85% of the time, the model's labels are useful with a stated caveat. If they agree 61% of the time, you have a random number generator with good manners.\n\n**4. Constructed pairs.** The cheapest source and the most under-used. Where you already know what bad looks like, build the pair rather than collecting it.\n\n```python\npair = {\n  \"prompt\": ticket,\n  \"chosen\":  good_answer,                     # your SFT gold answer\n  \"rejected\": model_output_before_sft,        # the failure you are fixing\n}\n```\n\nOr generate deliberately degraded versions: the same answer with the apology restored, with the JSON keys reordered, with an invented order number inserted. Each targets one specific behaviour, which is exactly what preference learning is good at.\n\n## The on-policy requirement\n\nThis is the piece most often got wrong. The rejected response should be something **your current model actually produces**. A pair whose rejected side came from a different model, or from a much older checkpoint, is teaching your model to avoid behaviour it was not going to exhibit anyway, and worse, the gradient that pushes down an already-unlikely response can push down neighbouring plausible text with it.\n\nThe correct loop:\n\n1. Take prompts from your traffic.\n2. Generate several responses **from the model you are about to train**, at a temperature that gives variety.\n3. Rank them, by a person, a judge or a verifier.\n4. Pair the best against the worst.\n5. Train. Then regenerate, because the model has moved.\n\nThat last line is why preference tuning is iterative in a way SFT is not. After one round the model's failures are different failures, and a preference set collected against the previous checkpoint is partly stale.\n\n## How many pairs\n\nFewer than people expect, if they are on-policy and targeted.\n\n- **500 to 1,000** for one specific behaviour (a format habit, a tone, a length preference) with constructed or tightly-filtered pairs.\n- **3,000 to 10,000** for general helpfulness or a broad style shift.\n- **Beyond that**, returns diminish quickly relative to the labelling cost, and your effort is better spent on the quality of the pairs than on their number.\n\n## The pair quality test\n\nBefore training, read fifty pairs and ask of each: *can I state in one clause why the chosen one is better?*\n\nIf you cannot (if they are two acceptable answers and somebody picked one) that pair is noise. It contributes a gradient with no consistent direction, and enough of them will teach the model something arbitrary with the same confidence as everything else. Drop the pairs where the margin is thin. A preference dataset of 800 clear comparisons beats one of 5,000 where half are coin flips, and the second is much more expensive."},"check":{"question":"A team builds 5,000 preference pairs where the rejected responses were generated by a much weaker model. Training runs cleanly but the tuned model shows little improvement on the behaviours they targeted. What went wrong?","options":[{"text":"Five thousand pairs is too few to shift a model's behaviour, and the gap only closes above ten thousand.","correct":false,"tempting":"Reaches for volume; on-policy pairs at a fraction of that count routinely produce the targeted change."},{"text":"Weaker models produce shorter responses, so the length bias in preference learning pushed the model towards brevity rather than the targeted behaviour.","correct":false,"tempting":"Names a real bias, but it would have produced a visible change in length rather than little improvement of any kind."},{"text":"Mixing outputs from two models makes the reference model's log-probabilities inconsistent, which destabilises the loss.","correct":false,"tempting":"Invokes a reference-model inconsistency; the reference scores whatever text it is given and does not care which model produced it."},{"text":"The rejected responses were already unlikely under their model, so the gradient had little to act on.","correct":true}]},"prev":{"slug":"sft-is-imitation","title":"What supervised fine-tuning cannot express","url":"https://addaly.com/learn/fine-tuning/sft-is-imitation"},"next":{"slug":"dpo-from-first-principles","title":"DPO, from first principles","url":"https://addaly.com/learn/fine-tuning/dpo-from-first-principles"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where preference data comes from', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/where-preference-data-comes-from (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where preference data comes from","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"Preference pairs must be on-policy (the rejected side generated by the model you are about to train) or you are teaching it to avoid behaviour it never had; and pairs where you cannot state in one clause why the chosen answer is better are noise that trains something arbitrary.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-sources-and-their-biases","runs":[{"text":"Four sources, and their biases"}]},{"t":"p","runs":[{"text":"1. Product feedback.","b":true},{"text":" Thumbs up and thumbs down on your existing model’s answers. Free, continuous, and already flowing if you built the button. The catch is that it is "},{"text":"binary and unpaired","i":true},{"text":": you have judgements on single responses, not comparisons between two. Most preference algorithms want pairs. KTO, covered in the next-but-one lesson, is the one designed for exactly this data, and that is the reason to know it exists."}]},{"t":"p","runs":[{"text":"The bias: people rate when annoyed far more than when satisfied. A raw thumbs dataset is heavily weighted towards failures, and towards the kinds of failure that are noticeable rather than the kinds that are serious."}]},{"t":"p","runs":[{"text":"2. Pairwise human comparison.","b":true},{"text":" Generate two responses to the same input, show both, ask which is better. This is the clean source, and it is expensive: 30 to 90 seconds per comparison for a careful judgement, so 3,000 pairs is 40 to 70 hours."}]},{"t":"p","runs":[{"text":"Three biases to control. "},{"text":"Position bias","b":true},{"text":": annotators favour whichever is shown first, so randomise order and record the randomisation. "},{"text":"Length bias","b":true},{"text":": longer answers are preferred at rates that exceed their actual quality, which is the single most documented effect in this area. "},{"text":"Fluency bias","b":true},{"text":": a confidently wrong answer beats a hedged correct one unless the annotator is asked specifically about correctness."}]},{"t":"p","runs":[{"text":"3. AI feedback.","b":true},{"text":" Ask a model which of two responses is better. Cheap, fast, consistent, and it makes tens of thousands of pairs affordable. It carries the same length bias as humans and adds self-preference: a model asked to judge tends to prefer text produced by itself or by a model like it."}]},{"t":"p","runs":[{"text":"The workable version is layered: have the model judge everything, have a person judge 200 of the same pairs, and compute the agreement rate. If model and human agree 85% of the time, the model’s labels are useful with a stated caveat. If they agree 61% of the time, you have a random number generator with good manners."}]},{"t":"p","runs":[{"text":"4. Constructed pairs.","b":true},{"text":" The cheapest source and the most under-used. Where you already know what bad looks like, build the pair rather than collecting it."}]},{"t":"code","lang":"python","label":"Python","text":"pair = {\n  \"prompt\": ticket,\n  \"chosen\":  good_answer,                     # your SFT gold answer\n  \"rejected\": model_output_before_sft,        # the failure you are fixing\n}"},{"t":"p","runs":[{"text":"Or generate deliberately degraded versions: the same answer with the apology restored, with the JSON keys reordered, with an invented order number inserted. Each targets one specific behaviour, which is exactly what preference learning is good at."}]},{"t":"h2","id":"the-on-policy-requirement","runs":[{"text":"The on-policy requirement"}]},{"t":"p","runs":[{"text":"This is the piece most often got wrong. The rejected response should be something "},{"text":"your current model actually produces","b":true},{"text":". A pair whose rejected side came from a different model, or from a much older checkpoint, is teaching your model to avoid behaviour it was not going to exhibit anyway, and worse, the gradient that pushes down an already-unlikely response can push down neighbouring plausible text with it."}]},{"t":"p","runs":[{"text":"The correct loop:"}]},{"t":"ol","start":1,"items":[[{"text":"Take prompts from your traffic."}],[{"text":"Generate several responses "},{"text":"from the model you are about to train","b":true},{"text":", at a temperature that gives variety."}],[{"text":"Rank them, by a person, a judge or a verifier."}],[{"text":"Pair the best against the worst."}],[{"text":"Train. Then regenerate, because the model has moved."}]]},{"t":"p","runs":[{"text":"That last line is why preference tuning is iterative in a way SFT is not. After one round the model’s failures are different failures, and a preference set collected against the previous checkpoint is partly stale."}]},{"t":"h2","id":"how-many-pairs","runs":[{"text":"How many pairs"}]},{"t":"p","runs":[{"text":"Fewer than people expect, if they are on-policy and targeted."}]},{"t":"ul","items":[[{"text":"500 to 1,000","b":true},{"text":" for one specific behaviour (a format habit, a tone, a length preference) with constructed or tightly-filtered pairs."}],[{"text":"3,000 to 10,000","b":true},{"text":" for general helpfulness or a broad style shift."}],[{"text":"Beyond that","b":true},{"text":", returns diminish quickly relative to the labelling cost, and your effort is better spent on the quality of the pairs than on their number."}]]},{"t":"h2","id":"the-pair-quality-test","runs":[{"text":"The pair quality test"}]},{"t":"p","runs":[{"text":"Before training, read fifty pairs and ask of each: "},{"text":"can I state in one clause why the chosen one is better?","i":true}]},{"t":"p","runs":[{"text":"If you cannot (if they are two acceptable answers and somebody picked one) that pair is noise. It contributes a gradient with no consistent direction, and enough of them will teach the model something arbitrary with the same confidence as everything else. Drop the pairs where the margin is thin. A preference dataset of 800 clear comparisons beats one of 5,000 where half are coin flips, and the second is much more expensive."}]}],"check":{"question":[{"text":"A team builds 5,000 preference pairs where the rejected responses were generated by a much weaker model. Training runs cleanly but the tuned model shows little improvement on the behaviours they targeted. What went wrong?"}],"options":[{"text":[{"text":"Five thousand pairs is too few to shift a model’s behaviour, and the gap only closes above ten thousand."}],"correct":false,"tempting":[{"text":"Reaches for volume; on-policy pairs at a fraction of that count routinely produce the targeted change."}]},{"text":[{"text":"Weaker models produce shorter responses, so the length bias in preference learning pushed the model towards brevity rather than the targeted behaviour."}],"correct":false,"tempting":[{"text":"Names a real bias, but it would have produced a visible change in length rather than little improvement of any kind."}]},{"text":[{"text":"Mixing outputs from two models makes the reference model’s log-probabilities inconsistent, which destabilises the loss."}],"correct":false,"tempting":[{"text":"Invokes a reference-model inconsistency; the reference scores whatever text it is given and does not care which model produced it."}]},{"text":[{"text":"The rejected responses were already unlikely under their model, so the gradient had little to act on."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"sft-is-imitation","title":"What supervised fine-tuning cannot express","minutes":9},"next":{"slug":"dpo-from-first-principles","title":"DPO, from first principles","minutes":10}}}