Verifiable Rewards
Lesson 1 of 3 in RL for Reasoning Models.
Everything in RLHF hinges on one uncomfortable fact: the Reward model is a guess. It is a learned approximation of human preference, trained on a finite pile of Preference data, and the policy being optimized against it will eventually find the places where the guess is wrong. That is Reward hacking — and Gao, Schulman and Hilton (2022, arXiv:2210.10760) measured how reliably it arrives: push optimization hard enough and the true quality of outputs falls even as the proxy reward keeps climbing. Goodhart’s old proverb, running on GPUs: when a measure becomes a target, it ceases to be a good measure.
Now notice what happens in a domain where you don’t have to guess. A competition math problem has a final answer; a program either extracts it and matches it against the key, or it doesn’t. A coding task ships with a test suite; the generated function passes or fails. No annotator, no learned approximation, no proxy — a checker computes the reward directly from ground truth. Training against this kind of signal is often called reinforcement learning with verifiable rewards, and it is the unlock behind the current generation of reasoning models.
The recipe is almost embarrassingly simple to state. Take problems with checkable answers. Have the model generate attempts — full worked solutions, not just final answers. Score each attempt with the checker. Reinforce whatever the model did in the attempts that scored well. Repeat at scale. The model is free to discover any behavior that raises its hit rate — and as the next lesson shows, the behavior it discovers is thinking longer.
| Dimension | Preference reward (learned reward model) | Verifiable reward (programmatic checker) |
|---|---|---|
Signal source | Human comparisons distilled into a learned scorer — an approximation of what people prefer | A rule-based check against ground truth: answer matching, unit tests, a compiler, a game outcome |
Signal quality | Smooth but noisy; degrades off-distribution, exactly where optimization pushes the policy | Near-exact on what the checker covers; often binary, but grounded in the actual task |
Hackability | High — the proxy itself is the optimization target, so its errors become the policy’s strategy (Goodhart) | Lower, not zero — weak test suites, format exploits, and checker bugs are the remaining attack surface |
Domains covered | Anything humans can judge: tone, helpfulness, style, safety, taste — the whole open-ended space | Only where correctness is checkable: math with known answers, code with tests, formal proofs, some games |
Cost to scale | Expensive — every training signal ultimately traces back to paid human comparison labor | Cheap once the checker and problem set exist — sample as many attempts as compute allows |
The open, documented example is DeepSeek-R1 (DeepSeek-AI 2025, arXiv:2501.12948) — worth studying precisely because the report says what was done. It describes an experiment called R1-Zero: take a base model, skip supervised fine-tuning entirely, and run large-scale RL against rule-based rewards — an accuracy reward that checks the final answer, plus a format reward that asks the model to put its reasoning inside designated thinking tags. According to the report, reasoning behavior emerged from that pressure alone. The report also states the team deliberately avoided a neural reward model for this stage, citing the risk of reward hacking in large-scale RL — the lesson of the preference era, applied.
R1-Zero’s raw output had problems the report is candid about — poor readability and mixed languages inside its reasoning — so the released R1 pipeline adds a small amount of cold-start fine-tuning data before RL, and further rounds afterward to restore general assistant behavior. The pattern to remember: verifiable-reward RL builds the reasoning; the surrounding stages make it usable.
One honesty note before we go further: proprietary reasoning models publish far less. Where a provider’s public report describes its training, you can repeat what it says, hedged; beyond that, how any closed model was trained is unverifiable from the outside. This module leans on R1 because it is the case with receipts.
Key terms: Reasoning model, Chain-of-thought (CoT), Reward model, Reward hacking, Preference data
The RL loop when the checker is the reward
The loop, one pass at a time: draw a problem from the training set, sample a group of complete attempts from the current policy, run the checker on each, and update the policy to raise the probability of the token sequences that scored well. Because the checker is a program, this scales with compute rather than with annotator hours — the constraint becomes the supply of good problems with reliable checkers, not the supply of human judgments.
The R1 report describes using GRPO (group relative policy optimization) as the optimizer. The idea that matters for intuition: classic RLHF-era methods in the PPO family (Schulman et al. 2017, arXiv:1707.06347) train a separate value model to estimate how well the policy was expected to do, and reinforce attempts that beat that estimate. GRPO drops the value model and uses the group itself as the baseline — each attempt’s advantage is how much better or worse it scored than the average of its own group. Solve a problem your siblings failed, get pushed up; fail one they solved, get pushed down. One fewer large network to train, and a baseline that is always exactly on-distribution.
Two caveats keep this honest. First, a binary reward is sparse: on problems the policy never solves, every attempt scores zero, the group baseline is zero, and there is no gradient signal — which is why problem difficulty has to roughly track policy ability as training progresses. Second, verifiable does not mean perfectly specified: if the test suite is weak, RL will find the solution that passes the tests rather than the one that is correct — the same Goodhart pressure, one level down.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.