Preference training, from reward models to DPO

The problem it solves

After supervised fine-tuning, a model can follow instructions. It still has no signal about which of two acceptable answers people actually prefer, and writing that down as examples is hard: it is much easier for a person to say which of two responses is better than to produce the ideal one.

Preference training turns that comparison into a training signal.

The classical pipeline

Collect comparisons. Show annotators a prompt and two responses; record which they prefer. Tens to hundreds of thousands of pairs.

Train a reward model. A separate model (often the same architecture with the output layer replaced by a single score) is trained so that preferred responses score higher. It has learned a function from response to number that approximates human preference.

Optimise the policy against it. The language model generates, the reward model scores, and reinforcement learning (usually PPO) adjusts the language model to earn higher scores.

Constrain the drift. Critically, a penalty term keeps the model from straying too far from where it started, measured as KL divergence from the pre-RL model. Without it, optimisation finds degenerate text that scores high and reads as gibberish. That failure has a name, reward hacking, and it is the central difficulty of the method.

Why DPO replaced much of this

Direct Preference Optimisation, published in 2023, showed that under the standard assumptions you can skip the reward model and the reinforcement learning entirely. The preference data can be turned into a loss function on the language model directly, computed with the same machinery as ordinary fine-tuning.

Simpler, far cheaper, no separate reward model to host, and no PPO tuning. It became the default for open models very quickly, and free implementations are in Hugging Face trl.

The honest comparison: results are broadly comparable on many benchmarks, and there is evidence that a well-tuned online method still edges ahead at the top end, partly because it keeps sampling fresh responses rather than learning from a fixed set. Variants such as IPO, KTO and ORPO adjust different parts of the objective. This is an area that is still moving, and anyone who tells you the question is settled is ahead of the evidence.

What preference training actually produces

This is the part worth remembering, because it explains behaviours you notice daily.

Length inflation. Human raters prefer longer, more thorough-looking answers, so the reward model learns that longer scores higher, so the model writes longer. It is a measured effect that teams actively correct for, and it is why a one-line question earns five paragraphs.

Hedging and structure. Bulleted lists, caveats, restating the question: all rated well, all reinforced.

Sycophancy. Raters prefer agreement. The model learns to agree, including with corrections that are wrong. Module six treats this in detail.

Confidence loss. As the log-probability lesson noted, calibration degrades. Raters prefer decisiveness over accurate uncertainty.

Refusal patterns. Both genuine safety behaviour and over-refusal of harmless requests that pattern-match to something sensitive.

Every one of these is the proxy being optimised rather than the goal. The reward model approximates human approval; human approval approximates quality. Optimise hard enough against any proxy and you get what the proxy measures rather than what you wanted. This is a general law and not a flaw specific to any lab.

The people in the loop

Preference data comes from human annotators, and their instructions, incentives and demographics shape the resulting model. Rating guidelines are usually not published in full. Annotation is frequently outsourced, sometimes to workers reviewing distressing content for low pay, a fact reported repeatedly and not adequately addressed by the industry.

When you read that a model’s values were “aligned”, the concrete meaning is: a particular group of people, following a particular document, expressed preferences, and those preferences were compressed into a scoring function. Knowing that makes the resulting behaviour easier to interpret and easier to argue with.