{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"preference-tuning","title":"Teaching it which answer is better","url":"https://addaly.com/learn/open-models/preference-tuning","position":56,"of":71,"minutes":9,"answer":"Preference tuning learns from pairs or binary judgements about which output is better, fixing verbosity, tone and refusal calibration that supervised tuning cannot.","keyIdea":"Preference tuning learns from pairs or binary judgements about which output is better, fixing verbosity, tone and refusal calibration that supervised tuning cannot, at an order of magnitude lower learning rate and with a strong tendency to make models longer."},"body":{"format":"markdown","text":"## The gap supervised tuning leaves\n\nSupervised fine-tuning shows the model one good answer per input. It never sees a bad one, so it learns what a good answer looks like without learning what to avoid.\n\nThat leaves a specific set of problems unsolved: the model is too verbose, or too terse; it hedges when it should commit; it adds a summary paragraph nobody wanted; it refuses things it should handle and handles things it should refuse; it picks the second-best of two acceptable formats. These are preferences between outputs, and you often cannot write down the rule: you only know it when you see two answers side by side.\n\nPreference tuning learns from exactly that: pairs of answers with a judgement about which is better.\n\n## The three methods you will meet\n\n**DPO: direct preference optimisation.** Takes triples of prompt, chosen answer and rejected answer, and adjusts the model to raise the relative likelihood of the chosen one. It needs a frozen reference copy of the starting model, and a parameter `beta` controls how far the tuned model is allowed to drift from it. Lower beta means more change and more risk.\n\n**ORPO.** Combines supervised and preference learning in a single stage with no reference model, so it is cheaper in memory and simpler to run. Useful when you are starting from a base model and would otherwise do two runs.\n\n**KTO.** Needs only a binary judgement (this output was good, that one was bad) rather than matched pairs. This matters enormously in practice, because binary feedback is what real products collect. A thumbs-up button produces KTO data; producing DPO pairs requires someone to compare two answers to the same input deliberately.\n\n```python\nfrom trl import DPOTrainer, DPOConfig\ntrainer = DPOTrainer(\n    model=sft_model, ref_model=None,          # None uses the adapter-disabled base\n    args=DPOConfig(beta=0.1, learning_rate=5e-6, num_train_epochs=1,\n                   per_device_train_batch_size=1, gradient_accumulation_steps=16),\n    train_dataset=pairs,                       # prompt / chosen / rejected\n    tokenizer=tok, peft_config=lora_cfg,\n)\n```\n\nNote the learning rate: preference tuning wants roughly an order of magnitude lower than supervised tuning, around `5e-6` to `5e-7`. Using an SFT learning rate here is the standard way to destroy a model in twenty minutes.\n\n## Where the pairs come from\n\n**From your own model.** Sample two completions at temperature 0.8 for each of 500 real prompts, and have a person pick the better one. Fast (a judgement takes seconds) and the pairs are on-distribution, which matters: preference data drawn from a different model teaches your model less.\n\n**From corrections.** The original output is the rejected answer; the human-edited version is the chosen one. This is free if you are already correcting output, and it is the highest-quality source available.\n\n**From a judge model**, with a rubric, spot-checked by a person. Cheap and biased in ways the judge is biased, so audit twenty judgements before trusting the rest.\n\nFive hundred to two thousand pairs is the usual range. As with supervised data, consistency matters more than volume; pairs judged by three people with different tastes teach the average of three tastes, which is nobody's.\n\n## The order\n\nSupervised tuning first, preference tuning second. Preference tuning adjusts a model that already does the task; it is a poor way to teach the task itself. Running DPO on a base model that has never seen your format mostly produces confusion.\n\n## The damage to watch for\n\nPreference tuning is the stage most likely to make a model worse in ways your task metric will not show.\n\n- **Verbosity drift.** Human raters, and judge models even more, prefer longer answers. Uncontrolled preference tuning reliably makes models more verbose. If your data was rated by a model, check length distributions before and after.\n- **Capability loss.** Too many epochs or too low a beta pulls the model far from the reference and general ability degrades. One epoch is usually correct.\n- **Reward hacking on format.** The model learns the surface features that correlated with being chosen (headings, bullet points, a confident opening) rather than the substance.\n\nThe defence is the same in every case: a regression set covering general capability, other languages and refusal behaviour, run before and after. The next lesson is about building one, and preference tuning is the stage that most needs it."},"check":{"question":"After DPO on 1,500 pairs judged by a strong model, task accuracy is unchanged but responses are 60% longer and users complain. What most likely happened?","options":[{"text":"Beta was set too high, so the model drifted too far from the reference.","correct":false,"tempting":"Inverts the parameter: a higher beta constrains drift, and a lower one permits it."},{"text":"The pairs contained too few examples of short answers, so the model never learned brevity was acceptable.","correct":false,"tempting":"Plausible on the surface, but the systematic length increase points to the judge's preference rather than a coverage gap."},{"text":"Judge models systematically prefer longer answers, so the preference signal encoded length as a proxy for quality.","correct":true},{"text":"The learning rate was too low, so only superficial features such as length were learned.","correct":false,"tempting":"A low rate produces smaller changes overall rather than selectively teaching surface features."}]},"prev":{"slug":"running-your-first-fine-tune","title":"A real run, on free hardware","url":"https://addaly.com/learn/open-models/running-your-first-fine-tune"},"next":{"slug":"distilling-a-bigger-model","title":"Distilling a larger model into a smaller one","url":"https://addaly.com/learn/open-models/distilling-a-bigger-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Teaching it which answer is better', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/preference-tuning (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Teaching it which answer is better","courseTitle":"The Open Model Ecosystem","module":{"index":6,"slug":"adapting-a-model","title":"Adapting a model to your work"},"answer":null,"keyIdea":"Preference tuning learns from pairs or binary judgements about which output is better, fixing verbosity, tone and refusal calibration that supervised tuning cannot, at an order of magnitude lower learning rate and with a strong tendency to make models longer.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-gap-supervised-tuning-leaves","runs":[{"text":"The gap supervised tuning leaves"}]},{"t":"p","runs":[{"text":"Supervised fine-tuning shows the model one good answer per input. It never sees a bad one, so it learns what a good answer looks like without learning what to avoid."}]},{"t":"p","runs":[{"text":"That leaves a specific set of problems unsolved: the model is too verbose, or too terse; it hedges when it should commit; it adds a summary paragraph nobody wanted; it refuses things it should handle and handles things it should refuse; it picks the second-best of two acceptable formats. These are preferences between outputs, and you often cannot write down the rule: you only know it when you see two answers side by side."}]},{"t":"p","runs":[{"text":"Preference tuning learns from exactly that: pairs of answers with a judgement about which is better."}]},{"t":"h2","id":"the-three-methods-you-will-meet","runs":[{"text":"The three methods you will meet"}]},{"t":"p","runs":[{"text":"DPO: direct preference optimisation.","b":true},{"text":" Takes triples of prompt, chosen answer and rejected answer, and adjusts the model to raise the relative likelihood of the chosen one. It needs a frozen reference copy of the starting model, and a parameter "},{"text":"beta","code":true},{"text":" controls how far the tuned model is allowed to drift from it. Lower beta means more change and more risk."}]},{"t":"p","runs":[{"text":"ORPO.","b":true},{"text":" Combines supervised and preference learning in a single stage with no reference model, so it is cheaper in memory and simpler to run. Useful when you are starting from a base model and would otherwise do two runs."}]},{"t":"p","runs":[{"text":"KTO.","b":true},{"text":" Needs only a binary judgement (this output was good, that one was bad) rather than matched pairs. This matters enormously in practice, because binary feedback is what real products collect. A thumbs-up button produces KTO data; producing DPO pairs requires someone to compare two answers to the same input deliberately."}]},{"t":"code","lang":"python","label":"Python","text":"from trl import DPOTrainer, DPOConfig\ntrainer = DPOTrainer(\n    model=sft_model, ref_model=None,          # None uses the adapter-disabled base\n    args=DPOConfig(beta=0.1, learning_rate=5e-6, num_train_epochs=1,\n                   per_device_train_batch_size=1, gradient_accumulation_steps=16),\n    train_dataset=pairs,                       # prompt / chosen / rejected\n    tokenizer=tok, peft_config=lora_cfg,\n)"},{"t":"p","runs":[{"text":"Note the learning rate: preference tuning wants roughly an order of magnitude lower than supervised tuning, around "},{"text":"5e-6","code":true},{"text":" to "},{"text":"5e-7","code":true},{"text":". Using an SFT learning rate here is the standard way to destroy a model in twenty minutes."}]},{"t":"h2","id":"where-the-pairs-come-from","runs":[{"text":"Where the pairs come from"}]},{"t":"p","runs":[{"text":"From your own model.","b":true},{"text":" Sample two completions at temperature 0.8 for each of 500 real prompts, and have a person pick the better one. Fast (a judgement takes seconds) and the pairs are on-distribution, which matters: preference data drawn from a different model teaches your model less."}]},{"t":"p","runs":[{"text":"From corrections.","b":true},{"text":" The original output is the rejected answer; the human-edited version is the chosen one. This is free if you are already correcting output, and it is the highest-quality source available."}]},{"t":"p","runs":[{"text":"From a judge model","b":true},{"text":", with a rubric, spot-checked by a person. Cheap and biased in ways the judge is biased, so audit twenty judgements before trusting the rest."}]},{"t":"p","runs":[{"text":"Five hundred to two thousand pairs is the usual range. As with supervised data, consistency matters more than volume; pairs judged by three people with different tastes teach the average of three tastes, which is nobody’s."}]},{"t":"h2","id":"the-order","runs":[{"text":"The order"}]},{"t":"p","runs":[{"text":"Supervised tuning first, preference tuning second. Preference tuning adjusts a model that already does the task; it is a poor way to teach the task itself. Running DPO on a base model that has never seen your format mostly produces confusion."}]},{"t":"h2","id":"the-damage-to-watch-for","runs":[{"text":"The damage to watch for"}]},{"t":"p","runs":[{"text":"Preference tuning is the stage most likely to make a model worse in ways your task metric will not show."}]},{"t":"ul","items":[[{"text":"Verbosity drift.","b":true},{"text":" Human raters, and judge models even more, prefer longer answers. Uncontrolled preference tuning reliably makes models more verbose. If your data was rated by a model, check length distributions before and after."}],[{"text":"Capability loss.","b":true},{"text":" Too many epochs or too low a beta pulls the model far from the reference and general ability degrades. One epoch is usually correct."}],[{"text":"Reward hacking on format.","b":true},{"text":" The model learns the surface features that correlated with being chosen (headings, bullet points, a confident opening) rather than the substance."}]]},{"t":"p","runs":[{"text":"The defence is the same in every case: a regression set covering general capability, other languages and refusal behaviour, run before and after. The next lesson is about building one, and preference tuning is the stage that most needs it."}]}],"check":{"question":[{"text":"After DPO on 1,500 pairs judged by a strong model, task accuracy is unchanged but responses are 60% longer and users complain. What most likely happened?"}],"options":[{"text":[{"text":"Beta was set too high, so the model drifted too far from the reference."}],"correct":false,"tempting":[{"text":"Inverts the parameter: a higher beta constrains drift, and a lower one permits it."}]},{"text":[{"text":"The pairs contained too few examples of short answers, so the model never learned brevity was acceptable."}],"correct":false,"tempting":[{"text":"Plausible on the surface, but the systematic length increase points to the judge’s preference rather than a coverage gap."}]},{"text":[{"text":"Judge models systematically prefer longer answers, so the preference signal encoded length as a proxy for quality."}],"correct":true,"tempting":null},{"text":[{"text":"The learning rate was too low, so only superficial features such as length were learned."}],"correct":false,"tempting":[{"text":"A low rate produces smaller changes overall rather than selectively teaching surface features."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"running-your-first-fine-tune","title":"A real run, on free hardware","minutes":10},"next":{"slug":"distilling-a-bigger-model","title":"Distilling a larger model into a smaller one","minutes":9}}}