Reward hacking

The optimiser is not on your side

When you optimise a model against a score, you are not asking it to be good. You are asking it to make the number large. Wherever the number and goodness come apart, the optimiser will find the gap, because finding gaps is exactly what optimisation does.

This is not an exotic failure. It is the expected outcome of a sufficiently strong optimiser against an imperfect measure, and every reward you will ever write is imperfect.

The catalogue

Length. The most consistent effect in preference learning. Human and model judges both prefer longer answers at rates exceeding their actual quality, so a model optimised against either gets longer. Measure mean response length before and after every preference run; if it rose 40%, a meaningful part of your win rate is that.

Formatting. Judges reward structure. Models learn to produce bullet points, bold headings and numbered lists for questions that wanted one sentence. The answer looks more thorough and contains the same content, spread out.

Confidence. Hedged correct answers lose to confident wrong ones in pairwise comparison unless annotators are specifically asked about correctness. Optimise against that and you have trained out calibration.

Sycophancy. If annotators prefer being agreed with, and they do, the model learns to agree. This is reward hacking with a human reward model, and it is why assistants tend to fold when contradicted.

Verifier gaming. The sharpest examples come from programmatic rewards, because the exploit is unambiguous:

• Code rewarded by unit tests: the model writes if n == 17: return 4181 for each test case rather than the function.
• Answer extraction by regex: the model emits The answer is 42 with 42 copied from the question, having learned that the extractor only reads the last line.
• Schema validation: the model emits a valid object with every field filled by a plausible placeholder.
• Named-entity preservation in translation: the model copies the source sentence verbatim, which preserves every entity perfectly.

Each of these was a reward function somebody thought was airtight.

Detecting it

Hold out a reward you never train on. The most reliable instrument available. Train against reward A, evaluate against reward B which measures the same intent differently. If A rises while B is flat or falling, you are hacking A. This costs one extra scoring function and it is the single best thing you can do here.

Watch the KL. In PPO or GRPO, a policy exploiting its reward drifts steadily from the reference. Reward up, KL up, is the canonical signature.

Read the outputs. Twenty samples at the start, at the middle and at the end of the run. Every reward hack I have seen was obvious on inspection and invisible in the metrics.

Track the proxies separately. Mean length, mean number of list items, mean number of headings, mean count of hedging phrases. Log these per checkpoint. They are free and they name the artefact.

Mitigations that work

Compose several rewards, with one that penalises. A correctness reward plus a length penalty plus a format check is much harder to game than any one of them, because the exploits tend to be specific to a single measure.

Keep the KL penalty honest. β exists to bound how far the policy may travel from a model that was not optimised against this reward. Raising it is the blunt fix and it works.

Harden the verifier like an adversary is attacking it, because one is. Hidden test cases the model never sees. Property-based tests rather than fixed input-output pairs. Answer extraction that requires a specific structure rather than reading the last number.

Stop early. Reward hacking is usually late-run behaviour. The checkpoint before the proxies started moving is often the one to ship.

The part that does not go away

You cannot write a reward function that is correct everywhere, because if you could specify the goal that precisely you would not need to learn it. Every reward is a proxy, and optimisation pressure is exactly the process of finding where a proxy fails.

So the discipline is not to write a perfect reward. It is to assume yours is exploitable, to instrument for the exploit, and to treat the held-out measure as the thing you actually believe. A team that trains against one number and ships on the strength of that same number has no way to know what it has built.