{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"dpo-from-first-principles","title":"DPO, from first principles","url":"https://addaly.com/learn/fine-tuning/dpo-from-first-principles","position":41,"of":84,"minutes":10,"answer":"DPO replaces the reward model with an implicit reward expressed as the log-probability ratio against a frozen reference, needs no second model copy when you use LoRA.","keyIdea":"DPO replaces the reward model with an implicit reward expressed as the log-probability ratio against a frozen reference, needs no second model copy when you use LoRA, and fails in a characteristic way (both chosen and rejected probabilities falling) which is why rewards/chosen is the statistic to watch."},"body":{"format":"markdown","text":"## The problem DPO removed\n\nThe classical approach to learning from preferences trains a separate reward model on the pairs, then uses reinforcement learning to maximise that reward while staying close to the original model. Four models in memory, an unstable optimisation, and a reward model that can be gamed.\n\nDirect Preference Optimisation's contribution is an algebraic observation: for that KL-constrained objective, the optimal policy has a closed form in terms of the reward, and that relationship can be inverted. The reward is therefore expressible in terms of the policy itself, which means you can optimise directly on preference pairs and never build a reward model at all.\n\nThe loss that falls out is a binary classification loss:\n\n```\nL = -log σ( β · [ log π(y_w|x)/π_ref(y_w|x)  −  log π(y_l|x)/π_ref(y_l|x) ] )\n```\n\nRead it in pieces:\n\n- `π` is the model being trained, `π_ref` the frozen starting point.\n- `log π(y|x)/π_ref(y|x)` is how much more likely your model has made a response than the reference did. This ratio *is* the implicit reward.\n- The bracket is the reward of the chosen response minus the reward of the rejected one: the margin.\n- `σ` and the negative log turn it into a loss that falls as the margin grows.\n- `β` controls how far the model may move from the reference.\n\n## β, the one parameter\n\nβ is the KL penalty strength, and it is the only hyperparameter with a big effect.\n\n- **β = 0.1** is the standard default and a sensible start.\n- **Lower (0.01 to 0.05)**: the model is freer to move. Bigger behaviour change, more risk of drifting away from the coherent model you started with.\n- **Higher (0.3 to 0.5)**: strongly anchored to the reference. Safer, smaller effect.\n\nIf your DPO run produces a model that has become strange (repetitive, oddly formatted, degraded on things you did not touch) raise β before you change anything else.\n\n## The reference model, and a LoRA trick\n\nDPO needs the reference model's log-probabilities for both responses. Naively that means a second copy of the model in memory, doubling your weight footprint.\n\nWith LoRA you get it free. The reference model *is* the base model, and the base is right there under the adapter, so you compute reference log-probabilities by temporarily disabling the adapter:\n\n```python\nwith model.disable_adapter():\n    ref_logps = compute_logps(model, batch)\n```\n\nTRL's `DPOTrainer` does this automatically when it is given a PEFT model and no explicit `ref_model`. This is one of the strongest practical arguments for doing preference tuning with adapters: the memory cost of the reference model disappears entirely.\n\n## The failure everyone meets\n\nWatch the two reward statistics TRL logs: `rewards/chosen` and `rewards/rejected`. What you expect is chosen rising and rejected falling.\n\nWhat you frequently see is **both falling**, with the margin widening only because rejected falls faster.\n\nThis is a well-documented property of the objective, not a bug in your setup. The loss only cares about the difference, so reducing the probability of the chosen response is acceptable to it as long as the rejected one is reduced more. Taken far enough, the model assigns low probability to good answers and produces something neither chosen nor rejected: often shorter, blander, or degenerate.\n\nThree mitigations:\n\n1. **Watch the statistic and stop early.** If `rewards/chosen` has been falling for a few hundred steps, you have gone too far. This is the single most useful thing to monitor in a DPO run.\n2. **Add an SFT term.** A small negative-log-likelihood loss on the chosen response, mixed in with a coefficient around 0.1 (`rpo_alpha` in TRL), anchors the chosen probability. This is standard practice now.\n3. **Raise β.**\n\n## Length, again\n\nDPO reliably increases response length. Part of that is real (longer answers often are better) and part is an artefact of how the objective interacts with sequence probability, plus the length bias already sitting in whoever labelled your pairs.\n\nSo measure it. Log the mean token count of generated responses before and after. If the model got 40% longer and your win rate went up 6 points, you should find out how much of the 6 points survives a length-controlled comparison, because a judge that prefers longer answers will show you a gain that your users will not feel.\n\n## The whole configuration\n\n```python\nfrom trl import DPOConfig, DPOTrainer\n\ncfg = DPOConfig(beta=0.1, rpo_alpha=0.1,\n                learning_rate=5e-6,        # much lower than SFT\n                num_train_epochs=1,\n                max_length=1024, max_prompt_length=512)\n```\n\nNote the learning rate: **5e-6 to 5e-7**, one to two orders of magnitude below an SFT LoRA run. DPO moves a model that is already good, and moving it hard is how you break it. One epoch is usually right; a second often makes things worse.\n\n> Figure: The two reward statistics, and what they usually do. What you expect: rewards/chosen rises; rewards/rejected falls; the margin widens; good answers stay likely. What you frequently see: rewards/chosen falls as well; rewards/rejected falls faster; the margin widens all the same; the model drifts towards something neither chosen nor rejected: shorter, blander, sometimes degenerate. The loss only cares about the difference, so reducing the chosen response is acceptable to it as long as the rejected one falls faster. Watch rewards over chosen and stop when it has been falling for a few hundred steps."},"check":{"question":"Midway through a DPO run, rewards/chosen has fallen steadily while rewards/rejected has fallen faster, so the margin keeps widening and the loss keeps dropping. What does this mean?","options":[{"text":"The objective rewards only the difference, so the model makes good answers less likely too and drifts.","correct":true},{"text":"Training is healthy; only the margin matters to the objective, so the absolute levels are not informative.","correct":false,"tempting":"States the objective's indifference correctly and then draws the wrong practical conclusion, since the model's output quality depends on those absolute levels."},{"text":"The reference model is misconfigured, since correct reference log-probabilities would keep the chosen reward positive.","correct":false,"tempting":"Blames configuration for a property of the objective that appears with a correctly frozen reference model."},{"text":"Beta is too high, over-constraining the model to the reference and preventing chosen probabilities from rising.","correct":false,"tempting":"Gets the direction of beta backwards; a high beta restrains movement away from the reference, which is what would prevent this drift."}]},"prev":{"slug":"where-preference-data-comes-from","title":"Where preference data comes from","url":"https://addaly.com/learn/fine-tuning/where-preference-data-comes-from"},"next":{"slug":"the-dpo-family","title":"KTO, ORPO, SimPO and choosing among them","url":"https://addaly.com/learn/fine-tuning/the-dpo-family"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'DPO, from first principles', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/dpo-from-first-principles (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"DPO, from first principles","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"DPO replaces the reward model with an implicit reward expressed as the log-probability ratio against a frozen reference, needs no second model copy when you use LoRA, and fails in a characteristic way (both chosen and rejected probabilities falling) which is why rewards/chosen is the statistic to watch.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"What you expect","items":["rewards/chosen rises","rewards/rejected falls","the margin widens","good answers stay likely"]},"type":"compare","right":{"title":"What you frequently see","items":["rewards/chosen falls as well","rewards/rejected falls faster","the margin widens all the same","the model drifts towards something neither chosen nor rejected: shorter, blander, sometimes degenerate"]},"title":"The two reward statistics, and what they usually do","caption":"The loss only cares about the difference, so reducing the chosen response is acceptable to it as long as the rejected one falls faster. Watch rewards over chosen and stop when it has been falling for a few hundred steps."},"alt":"Figure: The two reward statistics, and what they usually do. What you expect: rewards/chosen rises; rewards/rejected falls; the margin widens; good answers stay likely. What you frequently see: rewards/chosen falls as well; rewards/rejected falls faster; the margin widens all the same; the model drifts towards something neither chosen nor rejected: shorter, blander, sometimes degenerate. The loss only cares about the difference, so reducing the chosen response is acceptable to it as long as the rejected one falls faster. Watch rewards over chosen and stop when it has been falling for a few hundred steps."}},"blocks":[{"t":"h2","id":"the-problem-dpo-removed","runs":[{"text":"The problem DPO removed"}]},{"t":"p","runs":[{"text":"The classical approach to learning from preferences trains a separate reward model on the pairs, then uses reinforcement learning to maximise that reward while staying close to the original model. Four models in memory, an unstable optimisation, and a reward model that can be gamed."}]},{"t":"p","runs":[{"text":"Direct Preference Optimisation’s contribution is an algebraic observation: for that KL-constrained objective, the optimal policy has a closed form in terms of the reward, and that relationship can be inverted. The reward is therefore expressible in terms of the policy itself, which means you can optimise directly on preference pairs and never build a reward model at all."}]},{"t":"p","runs":[{"text":"The loss that falls out is a binary classification loss:"}]},{"t":"code","lang":"","label":"Text","text":"L = -log σ( β · [ log π(y_w|x)/π_ref(y_w|x)  −  log π(y_l|x)/π_ref(y_l|x) ] )"},{"t":"p","runs":[{"text":"Read it in pieces:"}]},{"t":"ul","items":[[{"text":"π","code":true},{"text":" is the model being trained, "},{"text":"π_ref","code":true},{"text":" the frozen starting point."}],[{"text":"log π(y|x)/π_ref(y|x)","code":true},{"text":" is how much more likely your model has made a response than the reference did. This ratio "},{"text":"is","i":true},{"text":" the implicit reward."}],[{"text":"The bracket is the reward of the chosen response minus the reward of the rejected one: the margin."}],[{"text":"σ","code":true},{"text":" and the negative log turn it into a loss that falls as the margin grows."}],[{"text":"β","code":true},{"text":" controls how far the model may move from the reference."}]]},{"t":"h2","id":"the-one-parameter","runs":[{"text":"β, the one parameter"}]},{"t":"p","runs":[{"text":"β is the KL penalty strength, and it is the only hyperparameter with a big effect."}]},{"t":"ul","items":[[{"text":"β = 0.1","b":true},{"text":" is the standard default and a sensible start."}],[{"text":"Lower (0.01 to 0.05)","b":true},{"text":": the model is freer to move. Bigger behaviour change, more risk of drifting away from the coherent model you started with."}],[{"text":"Higher (0.3 to 0.5)","b":true},{"text":": strongly anchored to the reference. Safer, smaller effect."}]]},{"t":"p","runs":[{"text":"If your DPO run produces a model that has become strange (repetitive, oddly formatted, degraded on things you did not touch) raise β before you change anything else."}]},{"t":"h2","id":"the-reference-model-and-a-lora-trick","runs":[{"text":"The reference model, and a LoRA trick"}]},{"t":"p","runs":[{"text":"DPO needs the reference model’s log-probabilities for both responses. Naively that means a second copy of the model in memory, doubling your weight footprint."}]},{"t":"p","runs":[{"text":"With LoRA you get it free. The reference model "},{"text":"is","i":true},{"text":" the base model, and the base is right there under the adapter, so you compute reference log-probabilities by temporarily disabling the adapter:"}]},{"t":"code","lang":"python","label":"Python","text":"with model.disable_adapter():\n    ref_logps = compute_logps(model, batch)"},{"t":"p","runs":[{"text":"TRL’s "},{"text":"DPOTrainer","code":true},{"text":" does this automatically when it is given a PEFT model and no explicit "},{"text":"ref_model","code":true},{"text":". This is one of the strongest practical arguments for doing preference tuning with adapters: the memory cost of the reference model disappears entirely."}]},{"t":"h2","id":"the-failure-everyone-meets","runs":[{"text":"The failure everyone meets"}]},{"t":"p","runs":[{"text":"Watch the two reward statistics TRL logs: "},{"text":"rewards/chosen","code":true},{"text":" and "},{"text":"rewards/rejected","code":true},{"text":". What you expect is chosen rising and rejected falling."}]},{"t":"p","runs":[{"text":"What you frequently see is "},{"text":"both falling","b":true},{"text":", with the margin widening only because rejected falls faster."}]},{"t":"p","runs":[{"text":"This is a well-documented property of the objective, not a bug in your setup. The loss only cares about the difference, so reducing the probability of the chosen response is acceptable to it as long as the rejected one is reduced more. Taken far enough, the model assigns low probability to good answers and produces something neither chosen nor rejected: often shorter, blander, or degenerate."}]},{"t":"p","runs":[{"text":"Three mitigations:"}]},{"t":"ol","start":1,"items":[[{"text":"Watch the statistic and stop early.","b":true},{"text":" If "},{"text":"rewards/chosen","code":true},{"text":" has been falling for a few hundred steps, you have gone too far. This is the single most useful thing to monitor in a DPO run."}],[{"text":"Add an SFT term.","b":true},{"text":" A small negative-log-likelihood loss on the chosen response, mixed in with a coefficient around 0.1 ("},{"text":"rpo_alpha","code":true},{"text":" in TRL), anchors the chosen probability. This is standard practice now."}],[{"text":"Raise β.","b":true}]]},{"t":"h2","id":"length-again","runs":[{"text":"Length, again"}]},{"t":"p","runs":[{"text":"DPO reliably increases response length. Part of that is real (longer answers often are better) and part is an artefact of how the objective interacts with sequence probability, plus the length bias already sitting in whoever labelled your pairs."}]},{"t":"p","runs":[{"text":"So measure it. Log the mean token count of generated responses before and after. If the model got 40% longer and your win rate went up 6 points, you should find out how much of the 6 points survives a length-controlled comparison, because a judge that prefers longer answers will show you a gain that your users will not feel."}]},{"t":"h2","id":"the-whole-configuration","runs":[{"text":"The whole configuration"}]},{"t":"code","lang":"python","label":"Python","text":"from trl import DPOConfig, DPOTrainer\n\ncfg = DPOConfig(beta=0.1, rpo_alpha=0.1,\n                learning_rate=5e-6,        # much lower than SFT\n                num_train_epochs=1,\n                max_length=1024, max_prompt_length=512)"},{"t":"p","runs":[{"text":"Note the learning rate: "},{"text":"5e-6 to 5e-7","b":true},{"text":", one to two orders of magnitude below an SFT LoRA run. DPO moves a model that is already good, and moving it hard is how you break it. One epoch is usually right; a second often makes things worse."}]}],"check":{"question":[{"text":"Midway through a DPO run, rewards/chosen has fallen steadily while rewards/rejected has fallen faster, so the margin keeps widening and the loss keeps dropping. What does this mean?"}],"options":[{"text":[{"text":"The objective rewards only the difference, so the model makes good answers less likely too and drifts."}],"correct":true,"tempting":null},{"text":[{"text":"Training is healthy; only the margin matters to the objective, so the absolute levels are not informative."}],"correct":false,"tempting":[{"text":"States the objective’s indifference correctly and then draws the wrong practical conclusion, since the model’s output quality depends on those absolute levels."}]},{"text":[{"text":"The reference model is misconfigured, since correct reference log-probabilities would keep the chosen reward positive."}],"correct":false,"tempting":[{"text":"Blames configuration for a property of the objective that appears with a correctly frozen reference model."}]},{"text":[{"text":"Beta is too high, over-constraining the model to the reference and preventing chosen probabilities from rising."}],"correct":false,"tempting":[{"text":"Gets the direction of beta backwards; a high beta restrains movement away from the reference, which is what would prevent this drift."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"where-preference-data-comes-from","title":"Where preference data comes from","minutes":9},"next":{"slug":"the-dpo-family","title":"KTO, ORPO, SimPO and choosing among them","minutes":9}}}