{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-dpo-family","title":"KTO, ORPO, SimPO and choosing among them","url":"https://addaly.com/learn/fine-tuning/the-dpo-family","position":42,"of":84,"minutes":9,"answer":"Choose among the preference methods by what data you hold (KTO for unpaired thumbs, ORPO to merge SFT and preference into one reference-free stage, SimPO when length is inflating.","keyIdea":"Choose among the preference methods by what data you hold (KTO for unpaired thumbs, ORPO to merge SFT and preference into one reference-free stage, SimPO when length is inflating, IPO for noiseless constructed pairs) because their benchmark differences are smaller than the differences between datasets."},"body":{"format":"markdown","text":"## They differ in what data they need\n\nThe variants of DPO are usually presented as competing on benchmark scores. That is the least useful way to choose between them. The question that actually decides is **what data you have**, and on that axis they separate cleanly.\n\n## KTO: when your data is thumbs, not pairs\n\nKahneman-Tversky Optimisation takes *unpaired binary* labels: this response was good, that response was bad. No pairing required, and the two sets need not be the same size.\n\nThis matters because binary feedback is what real products collect. Every thumbs-up and thumbs-down button in your interface is producing KTO data and cannot produce DPO data without you constructing pairs artificially.\n\nThe method borrows from prospect theory: losses are weighted more heavily than equivalent gains, with a parameter controlling the asymmetry. Practically, if your positive and negative counts are imbalanced, you adjust the desirable and undesirable weights to compensate; TRL exposes these directly.\n\n**Choose KTO when:** you have thumbs data, or when constructing pairs would mean arbitrarily matching a good response to an unrelated bad one.\n\n## ORPO: one stage instead of two\n\nOdds Ratio Preference Optimisation folds SFT and preference learning into a single loss: a standard negative-log-likelihood term on the chosen response, plus an odds-ratio penalty that pushes down the rejected one.\n\nTwo consequences. **No reference model at all**, so the memory cost falls further. And **no separate SFT stage**, so you train once from a base model rather than twice.\n\nThat is a genuine simplification, and the reported results are competitive. The trade is control: with two stages you can evaluate the SFT model on its own and know whether your demonstrations worked before you add preferences on top. With ORPO, if the result is poor you have two hypotheses and one experiment.\n\n**Choose ORPO when:** you are starting from a base model with both demonstrations and preferences in hand, and you want one pipeline instead of two.\n\n## SimPO: reference-free, length-normalised\n\nSimPO removes the reference model by using the *average* log-probability per token as its implicit reward, and adds a target margin the chosen response must exceed.\n\nThe length normalisation is the interesting part. Because the reward is per token rather than per sequence, the objective stops mechanically favouring longer responses, which addresses one of DPO's most consistent artefacts at the level of the loss rather than by post-hoc filtering.\n\n**Choose SimPO when:** length inflation is your specific problem, or memory forbids a reference model.\n\n## IPO: when your labels are too confident\n\nDPO's log-sigmoid loss keeps rewarding a wider margin without limit, so on a dataset where the preferences are deterministic and noiseless it will drive the margin towards infinity: overfitting to the labels. IPO replaces the loss with a squared term that saturates at a target margin.\n\n**Choose IPO when:** your pairs are constructed rather than judged, so the labels carry no noise and the model can push arbitrarily hard on them.\n\n## CPO, RPO and the rest\n\nThere are more. They mostly combine the same three ingredients (a likelihood term on the chosen response, a penalty on the rejected one, and some treatment of the reference and of length) in different proportions. The differences between them on any given task are usually smaller than the difference made by the quality of the pairs.\n\n## A decision procedure\n\n1. **Do you have pairs or single judgements?** Single → KTO. Pairs → continue.\n2. **Have you already done SFT?** No, and you want one pipeline → ORPO. Yes → continue.\n3. **Is response length inflating?** Yes → SimPO, or DPO with a length-controlled evaluation.\n4. **Are your pairs constructed, with no label noise?** Yes → IPO.\n5. **Otherwise → DPO with `rpo_alpha=0.1`.** It is the best documented, the most widely used, and the one whose failure modes you can look up.\n\n## The caveat about comparisons\n\nThe published comparisons between these methods are largely run on instruction-following benchmarks scored by a model judge. Those judges have known length and style preferences, and the methods differ in exactly how they treat length. So the reported ranking is partly a measurement of how each method interacts with the evaluator.\n\nThis is not a reason to dismiss the results (they are careful and informative), but it is a reason not to pick a method because it is a point ahead on a leaderboard. Pick on what data you have, run one comparison on your own metric, and accept that the honest answer to \"which is best\" is that it has not been settled, and probably depends on the task."},"check":{"question":"A product has collected 40,000 thumbs-up and thumbs-down ratings on individual model responses, with no two responses to the same prompt. Which method fits, and why?","options":[{"text":"DPO, by randomly pairing a thumbs-up response with a thumbs-down response to synthesise the pairs it requires.","correct":false,"tempting":"Manufactures pairs across unrelated prompts, which breaks the assumption that both responses answer the same input."},{"text":"KTO, because it takes unpaired binary labels directly and does not require two responses to the same prompt.","correct":true},{"text":"ORPO, since it merges SFT and preference learning and therefore does not need paired data.","correct":false,"tempting":"Confuses removing the reference model and the separate stage with removing the pairing requirement, which ORPO still has."},{"text":"SimPO, because its reference-free length-normalised reward can score single responses without a comparison.","correct":false,"tempting":"Notes that SimPO scores a response without a reference model, but the objective still contrasts a chosen response against a rejected one."}]},"prev":{"slug":"dpo-from-first-principles","title":"DPO, from first principles","url":"https://addaly.com/learn/fine-tuning/dpo-from-first-principles"},"next":{"slug":"rlhf-and-ppo-honestly","title":"RLHF and PPO, honestly","url":"https://addaly.com/learn/fine-tuning/rlhf-and-ppo-honestly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'KTO, ORPO, SimPO and choosing among them', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-dpo-family (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"KTO, ORPO, SimPO and choosing among them","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"Choose among the preference methods by what data you hold (KTO for unpaired thumbs, ORPO to merge SFT and preference into one reference-free stage, SimPO when length is inflating, IPO for noiseless constructed pairs) because their benchmark differences are smaller than the differences between datasets.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"they-differ-in-what-data-they-need","runs":[{"text":"They differ in what data they need"}]},{"t":"p","runs":[{"text":"The variants of DPO are usually presented as competing on benchmark scores. That is the least useful way to choose between them. The question that actually decides is "},{"text":"what data you have","b":true},{"text":", and on that axis they separate cleanly."}]},{"t":"h2","id":"kto-when-your-data-is-thumbs-not-pairs","runs":[{"text":"KTO: when your data is thumbs, not pairs"}]},{"t":"p","runs":[{"text":"Kahneman-Tversky Optimisation takes "},{"text":"unpaired binary","i":true},{"text":" labels: this response was good, that response was bad. No pairing required, and the two sets need not be the same size."}]},{"t":"p","runs":[{"text":"This matters because binary feedback is what real products collect. Every thumbs-up and thumbs-down button in your interface is producing KTO data and cannot produce DPO data without you constructing pairs artificially."}]},{"t":"p","runs":[{"text":"The method borrows from prospect theory: losses are weighted more heavily than equivalent gains, with a parameter controlling the asymmetry. Practically, if your positive and negative counts are imbalanced, you adjust the desirable and undesirable weights to compensate; TRL exposes these directly."}]},{"t":"p","runs":[{"text":"Choose KTO when:","b":true},{"text":" you have thumbs data, or when constructing pairs would mean arbitrarily matching a good response to an unrelated bad one."}]},{"t":"h2","id":"orpo-one-stage-instead-of-two","runs":[{"text":"ORPO: one stage instead of two"}]},{"t":"p","runs":[{"text":"Odds Ratio Preference Optimisation folds SFT and preference learning into a single loss: a standard negative-log-likelihood term on the chosen response, plus an odds-ratio penalty that pushes down the rejected one."}]},{"t":"p","runs":[{"text":"Two consequences. "},{"text":"No reference model at all","b":true},{"text":", so the memory cost falls further. And "},{"text":"no separate SFT stage","b":true},{"text":", so you train once from a base model rather than twice."}]},{"t":"p","runs":[{"text":"That is a genuine simplification, and the reported results are competitive. The trade is control: with two stages you can evaluate the SFT model on its own and know whether your demonstrations worked before you add preferences on top. With ORPO, if the result is poor you have two hypotheses and one experiment."}]},{"t":"p","runs":[{"text":"Choose ORPO when:","b":true},{"text":" you are starting from a base model with both demonstrations and preferences in hand, and you want one pipeline instead of two."}]},{"t":"h2","id":"simpo-reference-free-length-normalised","runs":[{"text":"SimPO: reference-free, length-normalised"}]},{"t":"p","runs":[{"text":"SimPO removes the reference model by using the "},{"text":"average","i":true},{"text":" log-probability per token as its implicit reward, and adds a target margin the chosen response must exceed."}]},{"t":"p","runs":[{"text":"The length normalisation is the interesting part. Because the reward is per token rather than per sequence, the objective stops mechanically favouring longer responses, which addresses one of DPO’s most consistent artefacts at the level of the loss rather than by post-hoc filtering."}]},{"t":"p","runs":[{"text":"Choose SimPO when:","b":true},{"text":" length inflation is your specific problem, or memory forbids a reference model."}]},{"t":"h2","id":"ipo-when-your-labels-are-too-confident","runs":[{"text":"IPO: when your labels are too confident"}]},{"t":"p","runs":[{"text":"DPO’s log-sigmoid loss keeps rewarding a wider margin without limit, so on a dataset where the preferences are deterministic and noiseless it will drive the margin towards infinity: overfitting to the labels. IPO replaces the loss with a squared term that saturates at a target margin."}]},{"t":"p","runs":[{"text":"Choose IPO when:","b":true},{"text":" your pairs are constructed rather than judged, so the labels carry no noise and the model can push arbitrarily hard on them."}]},{"t":"h2","id":"cpo-rpo-and-the-rest","runs":[{"text":"CPO, RPO and the rest"}]},{"t":"p","runs":[{"text":"There are more. They mostly combine the same three ingredients (a likelihood term on the chosen response, a penalty on the rejected one, and some treatment of the reference and of length) in different proportions. The differences between them on any given task are usually smaller than the difference made by the quality of the pairs."}]},{"t":"h2","id":"a-decision-procedure","runs":[{"text":"A decision procedure"}]},{"t":"ol","start":1,"items":[[{"text":"Do you have pairs or single judgements?","b":true},{"text":" Single → KTO. Pairs → continue."}],[{"text":"Have you already done SFT?","b":true},{"text":" No, and you want one pipeline → ORPO. Yes → continue."}],[{"text":"Is response length inflating?","b":true},{"text":" Yes → SimPO, or DPO with a length-controlled evaluation."}],[{"text":"Are your pairs constructed, with no label noise?","b":true},{"text":" Yes → IPO."}],[{"text":"Otherwise → DPO with ","b":true},{"text":"rpo_alpha=0.1","b":true,"code":true},{"text":".","b":true},{"text":" It is the best documented, the most widely used, and the one whose failure modes you can look up."}]]},{"t":"h2","id":"the-caveat-about-comparisons","runs":[{"text":"The caveat about comparisons"}]},{"t":"p","runs":[{"text":"The published comparisons between these methods are largely run on instruction-following benchmarks scored by a model judge. Those judges have known length and style preferences, and the methods differ in exactly how they treat length. So the reported ranking is partly a measurement of how each method interacts with the evaluator."}]},{"t":"p","runs":[{"text":"This is not a reason to dismiss the results (they are careful and informative), but it is a reason not to pick a method because it is a point ahead on a leaderboard. Pick on what data you have, run one comparison on your own metric, and accept that the honest answer to “which is best” is that it has not been settled, and probably depends on the task."}]}],"check":{"question":[{"text":"A product has collected 40,000 thumbs-up and thumbs-down ratings on individual model responses, with no two responses to the same prompt. Which method fits, and why?"}],"options":[{"text":[{"text":"DPO, by randomly pairing a thumbs-up response with a thumbs-down response to synthesise the pairs it requires."}],"correct":false,"tempting":[{"text":"Manufactures pairs across unrelated prompts, which breaks the assumption that both responses answer the same input."}]},{"text":[{"text":"KTO, because it takes unpaired binary labels directly and does not require two responses to the same prompt."}],"correct":true,"tempting":null},{"text":[{"text":"ORPO, since it merges SFT and preference learning and therefore does not need paired data."}],"correct":false,"tempting":[{"text":"Confuses removing the reference model and the separate stage with removing the pairing requirement, which ORPO still has."}]},{"text":[{"text":"SimPO, because its reference-free length-normalised reward can score single responses without a comparison."}],"correct":false,"tempting":[{"text":"Notes that SimPO scores a response without a reference model, but the objective still contrasts a chosen response against a rejected one."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"dpo-from-first-principles","title":"DPO, from first principles","minutes":10},"next":{"slug":"rlhf-and-ppo-honestly","title":"RLHF and PPO, honestly","minutes":9}}}