{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"rlhf-and-ppo-honestly","title":"RLHF and PPO, honestly","url":"https://addaly.com/learn/fine-tuning/rlhf-and-ppo-honestly","position":43,"of":84,"minutes":9,"answer":"RLHF's PPO stage holds four networks (policy, frozen reference, reward model and value critic) inside a generation loop, which is why DPO replaced it for most applied work.","keyIdea":"RLHF's PPO stage holds four networks (policy, frozen reference, reward model and value critic) inside a generation loop, which is why DPO replaced it for most applied work; but its objective is the one DPO was derived from, and its KL-versus-reward plot is the clearest picture of a policy exploiting its reward model."},"body":{"format":"markdown","text":"## The pipeline that built the assistants\n\nReinforcement learning from human feedback is the method that turned base language models into the assistants people use. It has three stages:\n\n1. **Supervised fine-tuning** on demonstrations, to get a model that follows instructions at all.\n2. **Reward model training.** Collect preference pairs, train a model that takes a prompt and a response and returns a scalar. The standard loss is Bradley-Terry: maximise the probability that the chosen response scores above the rejected one. Architecturally the reward model is the language model with the token head replaced by a single linear output.\n3. **Policy optimisation.** Use PPO to maximise the reward model's score, with a KL penalty against the SFT model to stop the policy drifting into nonsense that happens to score well.\n\nStage three is where the difficulty lives.\n\n## Four models in memory\n\nPPO needs:\n\n- The **policy** being trained.\n- The **reference** (frozen SFT model), for the KL penalty.\n- The **reward model**, to score generated responses.\n- A **value model** (critic), which estimates expected future reward to reduce gradient variance. It is usually initialised from the reward model, so it is another full-sized network.\n\nFour networks, two of them trainable. For a 7B policy that is a memory bill roughly four times a standard fine-tune's, plus a generation loop inside the training loop: each step generates responses, scores them, computes advantages, and takes several gradient passes over the same batch.\n\n## Why it is hard in practice\n\n- **It is a generation loop.** Every step samples from the policy, so throughput is bounded by inference speed, not training speed. Runs take days rather than hours.\n- **It has many interacting hyperparameters.** The KL coefficient, the clipping range, the number of PPO epochs per batch, the advantage normalisation, the value-loss coefficient. Getting these wrong produces collapse rather than mediocrity.\n- **The reward model is a model.** It has its own errors, and the optimiser's job is to find inputs that maximise it. It will find the errors. This is the reward hacking problem, and it has a lesson of its own next.\n- **Debugging is hard.** When the KL rises and the reward rises and the outputs are worse, which of four networks is responsible?\n\n## Why almost nobody should do it\n\nDPO and its relatives get most of the benefit for a fraction of the complexity, on datasets of the size a normal team can produce. That is not a slight on PPO: it is a statement about where the crossover lies. PPO earns its cost when you have a reward signal that is *better than your preference pairs*, which in practice means either an enormous quantity of preference data justifying a well-trained reward model, or a programmatic verifier. In the second case the modern answer is GRPO, which is the next lesson.\n\nThe honest recommendation: unless you are training a general-purpose assistant at scale, or you have a verifier and a reasoning task, DPO is where your preference effort belongs.\n\n## Why to understand it anyway\n\nThree reasons it is worth the half hour.\n\n**It explains DPO.** DPO is derived from exactly this objective: the KL-constrained reward maximisation above. Knowing what β is a penalty on, and what the reference model is anchoring against, is knowing PPO's objective without the machinery.\n\n**It explains behaviour you see in every assistant.** The hedging, the length, the refusal patterns, the characteristic enthusiasm: these are not pretraining artefacts. They are the fingerprints of a reward model that was trained on human preferences, with all the biases those preferences carry.\n\n**It is what GRPO removes.** GRPO's contribution is best understood as \"PPO without the value model\", and that sentence only means something if you know what the value model was for.\n\n## If you do run it\n\nTRL's `PPOTrainer` implements the standard setup and is free. Sensible starting points from published practice: an initial KL coefficient around 0.05 with adaptive control, four PPO epochs per batch, generation at temperature 1.0 for exploration, and (most importantly) **log the KL divergence from the reference and watch it above everything else**. A KL that climbs steadily while reward climbs is the signature of a policy exploiting the reward model rather than improving. When you see that, stop; the checkpoint you want is several hundred steps back."},"check":{"question":"In a PPO run the reward model's score climbs steadily while KL divergence from the reference model also climbs steadily. Why is this a warning rather than a success?","options":[{"text":"Rising KL means the value model has diverged from the reward model, so the advantage estimates are no longer valid.","correct":false,"tempting":"Attributes the KL term to a relationship between the two auxiliary models, when it measures the policy's distance from the frozen reference."},{"text":"KL should fall during training, and a rising value means the reference model was not properly frozen.","correct":false,"tempting":"Expects KL to fall, when some movement away from the reference is the entire point; the warning is in the magnitude and the pairing with reward."},{"text":"Rising KL means the policy has left the region the reward was fitted on, so the score reflects its errors.","correct":true},{"text":"A rising KL penalty subtracts from the reward, so the reported score is being inflated by the penalty term itself.","correct":false,"tempting":"Inverts the arithmetic: the penalty reduces the objective, so it cannot be what is inflating the reported reward."}]},"prev":{"slug":"the-dpo-family","title":"KTO, ORPO, SimPO and choosing among them","url":"https://addaly.com/learn/fine-tuning/the-dpo-family"},"next":{"slug":"verifiable-rewards-and-grpo","title":"Verifiable rewards and GRPO","url":"https://addaly.com/learn/fine-tuning/verifiable-rewards-and-grpo"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'RLHF and PPO, honestly', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/rlhf-and-ppo-honestly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"RLHF and PPO, honestly","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"RLHF’s PPO stage holds four networks (policy, frozen reference, reward model and value critic) inside a generation loop, which is why DPO replaced it for most applied work; but its objective is the one DPO was derived from, and its KL-versus-reward plot is the clearest picture of a policy exploiting its reward model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-pipeline-that-built-the-assistants","runs":[{"text":"The pipeline that built the assistants"}]},{"t":"p","runs":[{"text":"Reinforcement learning from human feedback is the method that turned base language models into the assistants people use. It has three stages:"}]},{"t":"ol","start":1,"items":[[{"text":"Supervised fine-tuning","b":true},{"text":" on demonstrations, to get a model that follows instructions at all."}],[{"text":"Reward model training.","b":true},{"text":" Collect preference pairs, train a model that takes a prompt and a response and returns a scalar. The standard loss is Bradley-Terry: maximise the probability that the chosen response scores above the rejected one. Architecturally the reward model is the language model with the token head replaced by a single linear output."}],[{"text":"Policy optimisation.","b":true},{"text":" Use PPO to maximise the reward model’s score, with a KL penalty against the SFT model to stop the policy drifting into nonsense that happens to score well."}]]},{"t":"p","runs":[{"text":"Stage three is where the difficulty lives."}]},{"t":"h2","id":"four-models-in-memory","runs":[{"text":"Four models in memory"}]},{"t":"p","runs":[{"text":"PPO needs:"}]},{"t":"ul","items":[[{"text":"The "},{"text":"policy","b":true},{"text":" being trained."}],[{"text":"The "},{"text":"reference","b":true},{"text":" (frozen SFT model), for the KL penalty."}],[{"text":"The "},{"text":"reward model","b":true},{"text":", to score generated responses."}],[{"text":"A "},{"text":"value model","b":true},{"text":" (critic), which estimates expected future reward to reduce gradient variance. It is usually initialised from the reward model, so it is another full-sized network."}]]},{"t":"p","runs":[{"text":"Four networks, two of them trainable. For a 7B policy that is a memory bill roughly four times a standard fine-tune’s, plus a generation loop inside the training loop: each step generates responses, scores them, computes advantages, and takes several gradient passes over the same batch."}]},{"t":"h2","id":"why-it-is-hard-in-practice","runs":[{"text":"Why it is hard in practice"}]},{"t":"ul","items":[[{"text":"It is a generation loop.","b":true},{"text":" Every step samples from the policy, so throughput is bounded by inference speed, not training speed. Runs take days rather than hours."}],[{"text":"It has many interacting hyperparameters.","b":true},{"text":" The KL coefficient, the clipping range, the number of PPO epochs per batch, the advantage normalisation, the value-loss coefficient. Getting these wrong produces collapse rather than mediocrity."}],[{"text":"The reward model is a model.","b":true},{"text":" It has its own errors, and the optimiser’s job is to find inputs that maximise it. It will find the errors. This is the reward hacking problem, and it has a lesson of its own next."}],[{"text":"Debugging is hard.","b":true},{"text":" When the KL rises and the reward rises and the outputs are worse, which of four networks is responsible?"}]]},{"t":"h2","id":"why-almost-nobody-should-do-it","runs":[{"text":"Why almost nobody should do it"}]},{"t":"p","runs":[{"text":"DPO and its relatives get most of the benefit for a fraction of the complexity, on datasets of the size a normal team can produce. That is not a slight on PPO: it is a statement about where the crossover lies. PPO earns its cost when you have a reward signal that is "},{"text":"better than your preference pairs","i":true},{"text":", which in practice means either an enormous quantity of preference data justifying a well-trained reward model, or a programmatic verifier. In the second case the modern answer is GRPO, which is the next lesson."}]},{"t":"p","runs":[{"text":"The honest recommendation: unless you are training a general-purpose assistant at scale, or you have a verifier and a reasoning task, DPO is where your preference effort belongs."}]},{"t":"h2","id":"why-to-understand-it-anyway","runs":[{"text":"Why to understand it anyway"}]},{"t":"p","runs":[{"text":"Three reasons it is worth the half hour."}]},{"t":"p","runs":[{"text":"It explains DPO.","b":true},{"text":" DPO is derived from exactly this objective: the KL-constrained reward maximisation above. Knowing what β is a penalty on, and what the reference model is anchoring against, is knowing PPO’s objective without the machinery."}]},{"t":"p","runs":[{"text":"It explains behaviour you see in every assistant.","b":true},{"text":" The hedging, the length, the refusal patterns, the characteristic enthusiasm: these are not pretraining artefacts. They are the fingerprints of a reward model that was trained on human preferences, with all the biases those preferences carry."}]},{"t":"p","runs":[{"text":"It is what GRPO removes.","b":true},{"text":" GRPO’s contribution is best understood as “PPO without the value model”, and that sentence only means something if you know what the value model was for."}]},{"t":"h2","id":"if-you-do-run-it","runs":[{"text":"If you do run it"}]},{"t":"p","runs":[{"text":"TRL’s "},{"text":"PPOTrainer","code":true},{"text":" implements the standard setup and is free. Sensible starting points from published practice: an initial KL coefficient around 0.05 with adaptive control, four PPO epochs per batch, generation at temperature 1.0 for exploration, and (most importantly) "},{"text":"log the KL divergence from the reference and watch it above everything else","b":true},{"text":". A KL that climbs steadily while reward climbs is the signature of a policy exploiting the reward model rather than improving. When you see that, stop; the checkpoint you want is several hundred steps back."}]}],"check":{"question":[{"text":"In a PPO run the reward model’s score climbs steadily while KL divergence from the reference model also climbs steadily. Why is this a warning rather than a success?"}],"options":[{"text":[{"text":"Rising KL means the value model has diverged from the reward model, so the advantage estimates are no longer valid."}],"correct":false,"tempting":[{"text":"Attributes the KL term to a relationship between the two auxiliary models, when it measures the policy’s distance from the frozen reference."}]},{"text":[{"text":"KL should fall during training, and a rising value means the reference model was not properly frozen."}],"correct":false,"tempting":[{"text":"Expects KL to fall, when some movement away from the reference is the entire point; the warning is in the magnitude and the pairing with reward."}]},{"text":[{"text":"Rising KL means the policy has left the region the reward was fitted on, so the score reflects its errors."}],"correct":true,"tempting":null},{"text":[{"text":"A rising KL penalty subtracts from the reward, so the reported score is being inflated by the penalty term itself."}],"correct":false,"tempting":[{"text":"Inverts the arithmetic: the penalty reduces the objective, so it cannot be what is inflating the reported reward."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"the-dpo-family","title":"KTO, ORPO, SimPO and choosing among them","minutes":9},"next":{"slug":"verifiable-rewards-and-grpo","title":"Verifiable rewards and GRPO","minutes":10}}}