RLVR: rewards you can verify
Human preference labels are noisy and expensive. For math, code, and structured tasks there is a better oracle: check the answer. A python interpreter, a unit-test suite, or an exact-answer comparator gives a reward signal that is deterministic, cheap, and impossible to charm with fluent prose.
The Oracle Set
Math: exact-match or SymPy equivalence. Code: compile + hidden tests. Agentic tasks: did the tool call succeed? These are rule-based rewards (RLVR).
Format Matters
Answers must be extractable — 'put the final number in \boxed{}'. A correct answer in the wrong format earns partial or zero reward, so format compliance gets its own reward component.
No Reward Model
Because rewards are computed, not learned, there is no reward model to over-optimize. The failure mode shifts from reward hacking to overfitting the train problem distribution.
Binary correctness dominates; a small format reward shapes parseability without letting the model profit from well-formatted wrong answers.
- DeepSeek-R1 and OpenAI's o-series are the canonical demonstrations: RL on verifiable rewards alone elicits long self-correcting chains of thought.
- Infrastructure is the hard part: sandboxed code execution at thousands of rollouts/step, with timeout and memory limits per sample.