DPO, from first principles

The problem DPO removed

The classical approach to learning from preferences trains a separate reward model on the pairs, then uses reinforcement learning to maximise that reward while staying close to the original model. Four models in memory, an unstable optimisation, and a reward model that can be gamed.

Direct Preference Optimisation’s contribution is an algebraic observation: for that KL-constrained objective, the optimal policy has a closed form in terms of the reward, and that relationship can be inverted. The reward is therefore expressible in terms of the policy itself, which means you can optimise directly on preference pairs and never build a reward model at all.

The loss that falls out is a binary classification loss:

L = -log σ( β · [ log π(y_w|x)/π_ref(y_w|x)  −  log π(y_l|x)/π_ref(y_l|x) ] )

Read it in pieces:

• π is the model being trained, π_ref the frozen starting point.
• log π(y|x)/π_ref(y|x) is how much more likely your model has made a response than the reference did. This ratio is the implicit reward.
• The bracket is the reward of the chosen response minus the reward of the rejected one: the margin.
• σ and the negative log turn it into a loss that falls as the margin grows.
• β controls how far the model may move from the reference.

β, the one parameter

β is the KL penalty strength, and it is the only hyperparameter with a big effect.

• β = 0.1 is the standard default and a sensible start.
• Lower (0.01 to 0.05): the model is freer to move. Bigger behaviour change, more risk of drifting away from the coherent model you started with.
• Higher (0.3 to 0.5): strongly anchored to the reference. Safer, smaller effect.

If your DPO run produces a model that has become strange (repetitive, oddly formatted, degraded on things you did not touch) raise β before you change anything else.

The reference model, and a LoRA trick

DPO needs the reference model’s log-probabilities for both responses. Naively that means a second copy of the model in memory, doubling your weight footprint.

With LoRA you get it free. The reference model is the base model, and the base is right there under the adapter, so you compute reference log-probabilities by temporarily disabling the adapter:

with model.disable_adapter():
    ref_logps = compute_logps(model, batch)

TRL’s DPOTrainer does this automatically when it is given a PEFT model and no explicit ref_model. This is one of the strongest practical arguments for doing preference tuning with adapters: the memory cost of the reference model disappears entirely.

The failure everyone meets

Watch the two reward statistics TRL logs: rewards/chosen and rewards/rejected. What you expect is chosen rising and rejected falling.

What you frequently see is both falling, with the margin widening only because rejected falls faster.

This is a well-documented property of the objective, not a bug in your setup. The loss only cares about the difference, so reducing the probability of the chosen response is acceptable to it as long as the rejected one is reduced more. Taken far enough, the model assigns low probability to good answers and produces something neither chosen nor rejected: often shorter, blander, or degenerate.

Three mitigations:

1. Watch the statistic and stop early. If rewards/chosen has been falling for a few hundred steps, you have gone too far. This is the single most useful thing to monitor in a DPO run.
2. Add an SFT term. A small negative-log-likelihood loss on the chosen response, mixed in with a coefficient around 0.1 (rpo_alpha in TRL), anchors the chosen probability. This is standard practice now.
3. Raise β.

Length, again

DPO reliably increases response length. Part of that is real (longer answers often are better) and part is an artefact of how the objective interacts with sequence probability, plus the length bias already sitting in whoever labelled your pairs.

So measure it. Log the mean token count of generated responses before and after. If the model got 40% longer and your win rate went up 6 points, you should find out how much of the 6 points survives a length-controlled comparison, because a judge that prefers longer answers will show you a gain that your users will not feel.

The whole configuration

from trl import DPOConfig, DPOTrainer

cfg = DPOConfig(beta=0.1, rpo_alpha=0.1,
                learning_rate=5e-6,        # much lower than SFT
                num_train_epochs=1,
                max_length=1024, max_prompt_length=512)

Note the learning rate: 5e-6 to 5e-7, one to two orders of magnitude below an SFT LoRA run. DPO moves a model that is already good, and moving it hard is how you break it. One epoch is usually right; a second often makes things worse.