RLHF and PPO, honestly

The pipeline that built the assistants

Reinforcement learning from human feedback is the method that turned base language models into the assistants people use. It has three stages:

1. Supervised fine-tuning on demonstrations, to get a model that follows instructions at all.
2. Reward model training. Collect preference pairs, train a model that takes a prompt and a response and returns a scalar. The standard loss is Bradley-Terry: maximise the probability that the chosen response scores above the rejected one. Architecturally the reward model is the language model with the token head replaced by a single linear output.
3. Policy optimisation. Use PPO to maximise the reward model’s score, with a KL penalty against the SFT model to stop the policy drifting into nonsense that happens to score well.

Stage three is where the difficulty lives.

Four models in memory

PPO needs:

• The policy being trained.
• The reference (frozen SFT model), for the KL penalty.
• The reward model, to score generated responses.
• A value model (critic), which estimates expected future reward to reduce gradient variance. It is usually initialised from the reward model, so it is another full-sized network.

Four networks, two of them trainable. For a 7B policy that is a memory bill roughly four times a standard fine-tune’s, plus a generation loop inside the training loop: each step generates responses, scores them, computes advantages, and takes several gradient passes over the same batch.

Why it is hard in practice

• It is a generation loop. Every step samples from the policy, so throughput is bounded by inference speed, not training speed. Runs take days rather than hours.
• It has many interacting hyperparameters. The KL coefficient, the clipping range, the number of PPO epochs per batch, the advantage normalisation, the value-loss coefficient. Getting these wrong produces collapse rather than mediocrity.
• The reward model is a model. It has its own errors, and the optimiser’s job is to find inputs that maximise it. It will find the errors. This is the reward hacking problem, and it has a lesson of its own next.
• Debugging is hard. When the KL rises and the reward rises and the outputs are worse, which of four networks is responsible?

Why almost nobody should do it

DPO and its relatives get most of the benefit for a fraction of the complexity, on datasets of the size a normal team can produce. That is not a slight on PPO: it is a statement about where the crossover lies. PPO earns its cost when you have a reward signal that is better than your preference pairs, which in practice means either an enormous quantity of preference data justifying a well-trained reward model, or a programmatic verifier. In the second case the modern answer is GRPO, which is the next lesson.

The honest recommendation: unless you are training a general-purpose assistant at scale, or you have a verifier and a reasoning task, DPO is where your preference effort belongs.

Why to understand it anyway

Three reasons it is worth the half hour.

It explains DPO. DPO is derived from exactly this objective: the KL-constrained reward maximisation above. Knowing what β is a penalty on, and what the reference model is anchoring against, is knowing PPO’s objective without the machinery.

It explains behaviour you see in every assistant. The hedging, the length, the refusal patterns, the characteristic enthusiasm: these are not pretraining artefacts. They are the fingerprints of a reward model that was trained on human preferences, with all the biases those preferences carry.

It is what GRPO removes. GRPO’s contribution is best understood as “PPO without the value model”, and that sentence only means something if you know what the value model was for.

If you do run it

TRL’s PPOTrainer implements the standard setup and is free. Sensible starting points from published practice: an initial KL coefficient around 0.05 with adaptive control, four PPO epochs per batch, generation at temperature 1.0 for exploration, and (most importantly) log the KL divergence from the reference and watch it above everything else. A KL that climbs steadily while reward climbs is the signature of a policy exploiting the reward model rather than improving. When you see that, stop; the checkpoint you want is several hundred steps back.