{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"verifiable-rewards-and-grpo","title":"Verifiable rewards and GRPO","url":"https://addaly.com/learn/fine-tuning/verifiable-rewards-and-grpo","position":44,"of":84,"minutes":10,"answer":"GRPO drops PPO's value model by using a group of sampled responses as its own baseline, which makes verifiable-reward training affordable, but every group that scores identically produces no gradient.","keyIdea":"GRPO drops PPO's value model by using a group of sampled responses as its own baseline, which makes verifiable-reward training affordable, but every group that scores identically produces no gradient, and rejection-sampling fine-tuning on verified outputs is the cheaper baseline you should beat first."},"body":{"format":"markdown","text":"## When you can check the answer\n\nSome tasks have ground truth a program can confirm. A maths problem with a known result. Code that must pass tests. SQL that must return specific rows. JSON that must validate. A translation that must preserve a set of named entities.\n\nWhere that is true you have something better than a preference: a reward function that is free, instant, unlimited and cannot be flattered. This is reinforcement learning with verifiable rewards, and it is the setting where RL on language models has produced its clearest results.\n\n## GRPO: PPO without the critic\n\nPPO's value model exists to estimate how good a state is, so the advantage (how much better an action was than expected) can be computed. It is a second full-sized network to hold and train.\n\nGroup Relative Policy Optimisation replaces it with a sample average. For each prompt, generate a **group** of G responses. Score all of them. Then the advantage of each response is how it compares to its own group:\n\n```\nA_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G)\n```\n\nNo critic. The group provides the baseline. Memory falls from four networks to two (the policy and the frozen reference) plus a reward function that is usually a few lines of Python rather than a model at all.\n\nThis is the method behind the recent open reasoning models, where it was shown that a base model trained with rule-based rewards on maths and code develops longer, more deliberate chains of reasoning without ever being shown one.\n\n## Writing the reward\n\nIn TRL a reward is a callable returning a list of floats. Compose several:\n\n```python\ndef correctness(completions, answers, **kw):\n    return [2.0 if extract_final(c) == a else 0.0\n            for c, a in zip(completions, answers)]\n\ndef format_ok(completions, **kw):\n    return [0.5 if re.search(r\"<answer>.*?</answer>\", c, re.S) else 0.0\n            for c in completions]\n\n> Figure: PPO and GRPO, side by side. A two by two grid, rows PPO and GRPO, columns Networks held and The baseline: PPO and Networks held: 4; PPO and The baseline: a value model; GRPO and Networks held: 2; GRPO and The baseline: the group mean. Dropping the critic is what makes verifiable-reward training affordable. The cost is that every group whose responses score identically has zero variance and produces no gradient at all.\n\ntrainer = GRPOTrainer(model=model, reward_funcs=[correctness, format_ok],\n                      args=GRPOConfig(num_generations=8, beta=0.04,\n                                      learning_rate=1e-6))\n```\n\nThe small format reward is doing real work: early in training the model rarely gets the answer right, so a pure correctness reward is zero for every sample in the group, the standard deviation is zero, and the advantage is undefined or zero for all of them: no gradient. A shaping term that is achievable from the start keeps the group differentiated until correctness starts to fire.\n\nThat degenerate case (every response in a group scoring identically) is the single most common reason a GRPO run does nothing. Log the fraction of groups with zero variance. If it is high, your reward is too sparse or your sampling temperature is too low.\n\n## The costs, plainly\n\n**Generation dominates.** Every step generates G responses for every prompt. At G = 8 you are doing eight times the inference of a single-sample method, inside the training loop. Runs are measured in GPU-days for anything substantial.\n\n**It needs a verifier.** If you cannot check the answer programmatically, this whole lesson does not apply and you are back to preference methods.\n\n**Group size trades variance against cost.** G = 4 is cheap and noisy; G = 16 is stable and expensive. G = 8 is the common compromise.\n\n## The unsettled part\n\nThere is an open argument about what RL on verifiable rewards actually does.\n\nOne view: it teaches new reasoning capability, evidenced by models that solve problems the base model does not.\n\nThe other: it mostly *elicits* capability already present, by sharpening the model's distribution towards the reasoning paths it could already sometimes produce. The evidence offered is that while the trained model's single-sample accuracy rises sharply, its accuracy when allowed many attempts sometimes does not exceed the base model's: suggesting the set of solvable problems is unchanged and the reliability of reaching them has improved.\n\nThis is being actively argued with careful experiments on both sides, and the honest position is that it is not resolved. It matters practically: if RL sharpens rather than teaches, then a base model that cannot solve your problem at all, even occasionally across a hundred samples, is unlikely to be rescued by this method. Test that before committing GPU-weeks: sample your base model 100 times on 50 held-out problems and see whether the answer is ever right.\n\n## Where it fits for a normal team\n\nHonestly: rarely, and specifically. It is the right tool when you have a task with a cheap automatic verifier, a base model that can already do it sometimes, and a reason to want reliability rather than a new capability. Structured extraction that must validate, SQL generation against a schema you can execute, and constrained code generation are the realistic applied cases.\n\nFor everything else, SFT on verified outputs (generate many, keep the ones that pass, train on those) captures a large share of the benefit for a fraction of the cost, and needs no reinforcement learning at all. That technique is rejection-sampling fine-tuning, and it should be your baseline before any of this."},"check":{"question":"A GRPO run on maths problems uses a single reward: 1.0 for the correct final answer, 0.0 otherwise. Training makes no progress at all for thousands of steps. What is the mechanism?","options":[{"text":"Binary rewards cannot be normalised, so the advantage calculation divides by zero and the gradients are discarded.","correct":false,"tempting":"Correctly spots a division issue but misplaces it: binary rewards normalise fine whenever a group contains both outcomes."},{"text":"The learning rate is too low for a sparse reward, and a sparse signal needs a proportionally larger step to register.","correct":false,"tempting":"Reaches for a hyperparameter when the problem is that no usable gradient direction exists to scale up."},{"text":"Without a value model there is no baseline, so GRPO requires a dense reward and cannot work with binary correctness at all.","correct":false,"tempting":"Concludes binary rewards are incompatible, when they work well as soon as groups contain a mix of outcomes."},{"text":"Early on every response in a group is wrong, so rewards are equal and the advantage is zero for all.","correct":true}]},"prev":{"slug":"rlhf-and-ppo-honestly","title":"RLHF and PPO, honestly","url":"https://addaly.com/learn/fine-tuning/rlhf-and-ppo-honestly"},"next":{"slug":"reward-hacking","title":"Reward hacking","url":"https://addaly.com/learn/fine-tuning/reward-hacking"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Verifiable rewards and GRPO', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/verifiable-rewards-and-grpo (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Verifiable rewards and GRPO","courseTitle":"Fine-Tuning, and When Not To","module":{"index":5,"slug":"objectives-beyond-sft","title":"Objectives beyond next-token prediction"},"answer":null,"keyIdea":"GRPO drops PPO’s value model by using a group of sampled responses as its own baseline, which makes verifiable-reward training affordable, but every group that scores identically produces no gradient, and rejection-sampling fine-tuning on verified outputs is the cheaper baseline you should beat first.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Networks held","The baseline"],"rows":["PPO","GRPO"],"type":"matrix","cells":[["4","a value model"],["2","the group mean"]],"title":"PPO and GRPO, side by side","caption":"Dropping the critic is what makes verifiable-reward training affordable. The cost is that every group whose responses score identically has zero variance and produces no gradient at all.","cellLabels":[["policy, reference, reward, value","a second full-sized network to hold and train"],["policy and frozen reference","G sampled responses, scored and standardised"]]},"alt":"Figure: PPO and GRPO, side by side. A two by two grid, rows PPO and GRPO, columns Networks held and The baseline: PPO and Networks held: 4; PPO and The baseline: a value model; GRPO and Networks held: 2; GRPO and The baseline: the group mean. Dropping the critic is what makes verifiable-reward training affordable. The cost is that every group whose responses score identically has zero variance and produces no gradient at all."}},"blocks":[{"t":"h2","id":"when-you-can-check-the-answer","runs":[{"text":"When you can check the answer"}]},{"t":"p","runs":[{"text":"Some tasks have ground truth a program can confirm. A maths problem with a known result. Code that must pass tests. SQL that must return specific rows. JSON that must validate. A translation that must preserve a set of named entities."}]},{"t":"p","runs":[{"text":"Where that is true you have something better than a preference: a reward function that is free, instant, unlimited and cannot be flattered. This is reinforcement learning with verifiable rewards, and it is the setting where RL on language models has produced its clearest results."}]},{"t":"h2","id":"grpo-ppo-without-the-critic","runs":[{"text":"GRPO: PPO without the critic"}]},{"t":"p","runs":[{"text":"PPO’s value model exists to estimate how good a state is, so the advantage (how much better an action was than expected) can be computed. It is a second full-sized network to hold and train."}]},{"t":"p","runs":[{"text":"Group Relative Policy Optimisation replaces it with a sample average. For each prompt, generate a "},{"text":"group","b":true},{"text":" of G responses. Score all of them. Then the advantage of each response is how it compares to its own group:"}]},{"t":"code","lang":"","label":"Text","text":"A_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G)"},{"t":"p","runs":[{"text":"No critic. The group provides the baseline. Memory falls from four networks to two (the policy and the frozen reference) plus a reward function that is usually a few lines of Python rather than a model at all."}]},{"t":"p","runs":[{"text":"This is the method behind the recent open reasoning models, where it was shown that a base model trained with rule-based rewards on maths and code develops longer, more deliberate chains of reasoning without ever being shown one."}]},{"t":"h2","id":"writing-the-reward","runs":[{"text":"Writing the reward"}]},{"t":"p","runs":[{"text":"In TRL a reward is a callable returning a list of floats. Compose several:"}]},{"t":"code","lang":"python","label":"Python","text":"def correctness(completions, answers, **kw):\n    return [2.0 if extract_final(c) == a else 0.0\n            for c, a in zip(completions, answers)]\n\ndef format_ok(completions, **kw):\n    return [0.5 if re.search(r\"<answer>.*?</answer>\", c, re.S) else 0.0\n            for c in completions]\n\ntrainer = GRPOTrainer(model=model, reward_funcs=[correctness, format_ok],\n                      args=GRPOConfig(num_generations=8, beta=0.04,\n                                      learning_rate=1e-6))"},{"t":"p","runs":[{"text":"The small format reward is doing real work: early in training the model rarely gets the answer right, so a pure correctness reward is zero for every sample in the group, the standard deviation is zero, and the advantage is undefined or zero for all of them: no gradient. A shaping term that is achievable from the start keeps the group differentiated until correctness starts to fire."}]},{"t":"p","runs":[{"text":"That degenerate case (every response in a group scoring identically) is the single most common reason a GRPO run does nothing. Log the fraction of groups with zero variance. If it is high, your reward is too sparse or your sampling temperature is too low."}]},{"t":"h2","id":"the-costs-plainly","runs":[{"text":"The costs, plainly"}]},{"t":"p","runs":[{"text":"Generation dominates.","b":true},{"text":" Every step generates G responses for every prompt. At G = 8 you are doing eight times the inference of a single-sample method, inside the training loop. Runs are measured in GPU-days for anything substantial."}]},{"t":"p","runs":[{"text":"It needs a verifier.","b":true},{"text":" If you cannot check the answer programmatically, this whole lesson does not apply and you are back to preference methods."}]},{"t":"p","runs":[{"text":"Group size trades variance against cost.","b":true},{"text":" G = 4 is cheap and noisy; G = 16 is stable and expensive. G = 8 is the common compromise."}]},{"t":"h2","id":"the-unsettled-part","runs":[{"text":"The unsettled part"}]},{"t":"p","runs":[{"text":"There is an open argument about what RL on verifiable rewards actually does."}]},{"t":"p","runs":[{"text":"One view: it teaches new reasoning capability, evidenced by models that solve problems the base model does not."}]},{"t":"p","runs":[{"text":"The other: it mostly "},{"text":"elicits","i":true},{"text":" capability already present, by sharpening the model’s distribution towards the reasoning paths it could already sometimes produce. The evidence offered is that while the trained model’s single-sample accuracy rises sharply, its accuracy when allowed many attempts sometimes does not exceed the base model’s: suggesting the set of solvable problems is unchanged and the reliability of reaching them has improved."}]},{"t":"p","runs":[{"text":"This is being actively argued with careful experiments on both sides, and the honest position is that it is not resolved. It matters practically: if RL sharpens rather than teaches, then a base model that cannot solve your problem at all, even occasionally across a hundred samples, is unlikely to be rescued by this method. Test that before committing GPU-weeks: sample your base model 100 times on 50 held-out problems and see whether the answer is ever right."}]},{"t":"h2","id":"where-it-fits-for-a-normal-team","runs":[{"text":"Where it fits for a normal team"}]},{"t":"p","runs":[{"text":"Honestly: rarely, and specifically. It is the right tool when you have a task with a cheap automatic verifier, a base model that can already do it sometimes, and a reason to want reliability rather than a new capability. Structured extraction that must validate, SQL generation against a schema you can execute, and constrained code generation are the realistic applied cases."}]},{"t":"p","runs":[{"text":"For everything else, SFT on verified outputs (generate many, keep the ones that pass, train on those) captures a large share of the benefit for a fraction of the cost, and needs no reinforcement learning at all. That technique is rejection-sampling fine-tuning, and it should be your baseline before any of this."}]}],"check":{"question":[{"text":"A GRPO run on maths problems uses a single reward: 1.0 for the correct final answer, 0.0 otherwise. Training makes no progress at all for thousands of steps. What is the mechanism?"}],"options":[{"text":[{"text":"Binary rewards cannot be normalised, so the advantage calculation divides by zero and the gradients are discarded."}],"correct":false,"tempting":[{"text":"Correctly spots a division issue but misplaces it: binary rewards normalise fine whenever a group contains both outcomes."}]},{"text":[{"text":"The learning rate is too low for a sparse reward, and a sparse signal needs a proportionally larger step to register."}],"correct":false,"tempting":[{"text":"Reaches for a hyperparameter when the problem is that no usable gradient direction exists to scale up."}]},{"text":[{"text":"Without a value model there is no baseline, so GRPO requires a dense reward and cannot work with binary correctness at all."}],"correct":false,"tempting":[{"text":"Concludes binary rewards are incompatible, when they work well as soon as groups contain a mix of outcomes."}]},{"text":[{"text":"Early on every response in a group is wrong, so rewards are equal and the advantage is zero for all."}],"correct":true,"tempting":null}],"order":[1,0,2,3],"why":null},"prev":{"slug":"rlhf-and-ppo-honestly","title":"RLHF and PPO, honestly","minutes":9},"next":{"slug":"reward-hacking","title":"Reward hacking","minutes":9}}}