KTO, ORPO, SimPO and choosing among them

They differ in what data they need

The variants of DPO are usually presented as competing on benchmark scores. That is the least useful way to choose between them. The question that actually decides is what data you have, and on that axis they separate cleanly.

KTO: when your data is thumbs, not pairs

Kahneman-Tversky Optimisation takes unpaired binary labels: this response was good, that response was bad. No pairing required, and the two sets need not be the same size.

This matters because binary feedback is what real products collect. Every thumbs-up and thumbs-down button in your interface is producing KTO data and cannot produce DPO data without you constructing pairs artificially.

The method borrows from prospect theory: losses are weighted more heavily than equivalent gains, with a parameter controlling the asymmetry. Practically, if your positive and negative counts are imbalanced, you adjust the desirable and undesirable weights to compensate; TRL exposes these directly.

Choose KTO when: you have thumbs data, or when constructing pairs would mean arbitrarily matching a good response to an unrelated bad one.

ORPO: one stage instead of two

Odds Ratio Preference Optimisation folds SFT and preference learning into a single loss: a standard negative-log-likelihood term on the chosen response, plus an odds-ratio penalty that pushes down the rejected one.

Two consequences. No reference model at all, so the memory cost falls further. And no separate SFT stage, so you train once from a base model rather than twice.

That is a genuine simplification, and the reported results are competitive. The trade is control: with two stages you can evaluate the SFT model on its own and know whether your demonstrations worked before you add preferences on top. With ORPO, if the result is poor you have two hypotheses and one experiment.

Choose ORPO when: you are starting from a base model with both demonstrations and preferences in hand, and you want one pipeline instead of two.

SimPO: reference-free, length-normalised

SimPO removes the reference model by using the average log-probability per token as its implicit reward, and adds a target margin the chosen response must exceed.

The length normalisation is the interesting part. Because the reward is per token rather than per sequence, the objective stops mechanically favouring longer responses, which addresses one of DPO’s most consistent artefacts at the level of the loss rather than by post-hoc filtering.

Choose SimPO when: length inflation is your specific problem, or memory forbids a reference model.

IPO: when your labels are too confident

DPO’s log-sigmoid loss keeps rewarding a wider margin without limit, so on a dataset where the preferences are deterministic and noiseless it will drive the margin towards infinity: overfitting to the labels. IPO replaces the loss with a squared term that saturates at a target margin.

Choose IPO when: your pairs are constructed rather than judged, so the labels carry no noise and the model can push arbitrarily hard on them.

CPO, RPO and the rest

There are more. They mostly combine the same three ingredients (a likelihood term on the chosen response, a penalty on the rejected one, and some treatment of the reference and of length) in different proportions. The differences between them on any given task are usually smaller than the difference made by the quality of the pairs.

A decision procedure

1. Do you have pairs or single judgements? Single → KTO. Pairs → continue.
2. Have you already done SFT? No, and you want one pipeline → ORPO. Yes → continue.
3. Is response length inflating? Yes → SimPO, or DPO with a length-controlled evaluation.
4. Are your pairs constructed, with no label noise? Yes → IPO.
5. Otherwise → DPO with rpo_alpha=0.1. It is the best documented, the most widely used, and the one whose failure modes you can look up.

The caveat about comparisons

The published comparisons between these methods are largely run on instruction-following benchmarks scored by a model judge. Those judges have known length and style preferences, and the methods differ in exactly how they treat length. So the reported ranking is partly a measurement of how each method interacts with the evaluator.

This is not a reason to dismiss the results (they are careful and informative), but it is a reason not to pick a method because it is a point ahead on a leaderboard. Pick on what data you have, run one comparison on your own metric, and accept that the honest answer to “which is best” is that it has not been settled, and probably depends on the task.