Verifiable rewards and GRPO

When you can check the answer

Some tasks have ground truth a program can confirm. A maths problem with a known result. Code that must pass tests. SQL that must return specific rows. JSON that must validate. A translation that must preserve a set of named entities.

Where that is true you have something better than a preference: a reward function that is free, instant, unlimited and cannot be flattered. This is reinforcement learning with verifiable rewards, and it is the setting where RL on language models has produced its clearest results.

GRPO: PPO without the critic

PPO’s value model exists to estimate how good a state is, so the advantage (how much better an action was than expected) can be computed. It is a second full-sized network to hold and train.

Group Relative Policy Optimisation replaces it with a sample average. For each prompt, generate a group of G responses. Score all of them. Then the advantage of each response is how it compares to its own group:

A_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G)

No critic. The group provides the baseline. Memory falls from four networks to two (the policy and the frozen reference) plus a reward function that is usually a few lines of Python rather than a model at all.

This is the method behind the recent open reasoning models, where it was shown that a base model trained with rule-based rewards on maths and code develops longer, more deliberate chains of reasoning without ever being shown one.

Writing the reward

In TRL a reward is a callable returning a list of floats. Compose several:

def correctness(completions, answers, **kw):
    return [2.0 if extract_final(c) == a else 0.0
            for c, a in zip(completions, answers)]

def format_ok(completions, **kw):
    return [0.5 if re.search(r"<answer>.*?</answer>", c, re.S) else 0.0
            for c in completions]

trainer = GRPOTrainer(model=model, reward_funcs=[correctness, format_ok],
                      args=GRPOConfig(num_generations=8, beta=0.04,
                                      learning_rate=1e-6))

The small format reward is doing real work: early in training the model rarely gets the answer right, so a pure correctness reward is zero for every sample in the group, the standard deviation is zero, and the advantage is undefined or zero for all of them: no gradient. A shaping term that is achievable from the start keeps the group differentiated until correctness starts to fire.

That degenerate case (every response in a group scoring identically) is the single most common reason a GRPO run does nothing. Log the fraction of groups with zero variance. If it is high, your reward is too sparse or your sampling temperature is too low.

The costs, plainly

Generation dominates. Every step generates G responses for every prompt. At G = 8 you are doing eight times the inference of a single-sample method, inside the training loop. Runs are measured in GPU-days for anything substantial.

It needs a verifier. If you cannot check the answer programmatically, this whole lesson does not apply and you are back to preference methods.

Group size trades variance against cost. G = 4 is cheap and noisy; G = 16 is stable and expensive. G = 8 is the common compromise.

The unsettled part

There is an open argument about what RL on verifiable rewards actually does.

One view: it teaches new reasoning capability, evidenced by models that solve problems the base model does not.

The other: it mostly elicits capability already present, by sharpening the model’s distribution towards the reasoning paths it could already sometimes produce. The evidence offered is that while the trained model’s single-sample accuracy rises sharply, its accuracy when allowed many attempts sometimes does not exceed the base model’s: suggesting the set of solvable problems is unchanged and the reliability of reaching them has improved.

This is being actively argued with careful experiments on both sides, and the honest position is that it is not resolved. It matters practically: if RL sharpens rather than teaches, then a base model that cannot solve your problem at all, even occasionally across a hundred samples, is unlikely to be rescued by this method. Test that before committing GPU-weeks: sample your base model 100 times on 50 held-out problems and see whether the answer is ever right.

Where it fits for a normal team

Honestly: rarely, and specifically. It is the right tool when you have a task with a cheap automatic verifier, a base model that can already do it sometimes, and a reason to want reliability rather than a new capability. Structured extraction that must validate, SQL generation against a schema you can execute, and constrained code generation are the realistic applied cases.

For everything else, SFT on verified outputs (generate many, keep the ones that pass, train on those) captures a large share of the benefit for a fraction of the cost, and needs no reinforcement learning at all. That technique is rejection-sampling fine-tuning, and it should be your baseline before any of this.