Reinforcement Learning from Verifiable Rewards (RLVR)

Definition

Reinforcement Learning from Verifiable Rewards (RLVR) is a fine-tuning method in which the reward signal comes from a program that checks the answer rather than from a neural network trained to predict what a human would prefer. If the final number matches the reference, the reward is 1. If the unit tests pass, the reward is 1. Otherwise it is 0. Nothing about "a good answer" is learned; correctness is decided by execution.

The whole idea is a one-line substitution in the RLHF objective. RLHF maximizes the expected score of a reward model r_φ(x, y) — a network fitted to tens of thousands of human comparisons — minus a KL penalty that keeps the policy near its starting point. RLVR keeps the KL penalty, keeps the reinforcement learning, and swaps r_φ for a verification function v(x, y) that returns a constant if the answer is correct and zero if it is not. In Tülu 3, the AI2 report that named the method, that constant is α = 10.

What the swap buys is stated most bluntly in the DeepSeek-R1 paper, which trained its reasoning behaviour entirely against rule-based rewards: "we abstain from applying neural reward models — whether outcome-based or process-based — to reasoning tasks. This decision is predicated on our observation that neural reward models are susceptible to reward hacking during large-scale reinforcement learning." A learned reward model is a proxy for human judgement, and optimizing hard against a proxy eventually breaks it. A Python function that compares two integers is not a proxy for anything. You cannot flatter a compiler.

A note on the name: Tülu 3 introduced it as "Reinforcement Learning with Verifiable Rewards", and the literature uses "with" and "from" interchangeably — the "from" spelling mirrors RLHF. Both mean the same method, and the acronym is always RLVR.

How It Works

The training loop is the familiar one. Take a prompt with a known answer, sample several completions from the current model, score each one, and push the policy toward the completions that scored well. The only unusual step is the scoring, and it is the cheap one: extract the boxed answer and compare it to the ground truth, or write the candidate function to a file and run pytest.

Compare the cost of the two scorers, because it explains a lot of the enthusiasm. A forward pass through a 7B-parameter reward model costs roughly 2 × N × T floating-point operations, so scoring one 2,000-token response is about 2 × 7×10⁹ × 2,000 ≈ 2.8 × 10¹³ FLOPs — 28 TFLOPs of GPU work, per sampled response. Sampling eight responses per prompt means eight of those, on hardware that is already saturated generating them. Running a maths comparison or a three-case test suite costs microseconds of CPU and zero GPU memory. In a loop that scores millions of completions, one of these is a line item and the other is a rounding error.

The memory picture changes too, though less than the marketing suggests. Classic PPO-based RLHF keeps four networks resident: policy, frozen reference, reward model, critic. Tülu 3's RLVR setup keeps three — "the policy model, the reference policy model, and the value model" — because the reward model is gone from the reward path. It is not gone from the pipeline: AI2 still trained an 8B reward model, for 9 hours on 8 H100s, and used it to initialize the value function, which they found worked better than the alternatives. Pairing RLVR with GRPO, which deletes the critic as well, is what gets you down to two.

The reward carries one bit, and sometimes zero

A verifiable reward is binary, and that has consequences the maths makes obvious. A response that is one algebra slip from correct scores exactly what a response of pure gibberish scores. There is no gradient toward "closer".

Worse, on the prompts at either extreme there is no gradient at all. Group-relative methods estimate the advantage from the spread of rewards within a sampled group, so if all eight completions are correct — or all eight are wrong — the spread is zero and the prompt contributes nothing to the update. RLVR learns only at the frontier of what the model can sometimes do, which makes curriculum and problem difficulty a first-class concern rather than a detail. The GRPO page works through the advantage arithmetic that produces this.

RLVR is not an algorithm

This trips people up constantly, so it is worth stating plainly. GRPO, PPO, RLOO and DAPO are optimizers: they answer "given rewards, how do I compute an advantage and update the weights?" RLVR answers a different question — "where does the reward come from?" The two choices are independent. Tülu 3 ran RLVR with PPO. DeepSeek-R1 ran RLVR with GRPO. Nothing stops you from running GRPO against a learned reward model, and people do.

The pairing is popular for a reason: with a rule-based reward there is no reward network to hold in memory, and GRPO removes the critic, so the combination is the cheapest configuration anyone has found that still works. But "RLVR" and "GRPO" are not synonyms, and a paper that says one does not imply the other.

Real-World Applications

DeepSeek-R1 and R1-Zero. DeepSeek-R1 is the clearest public demonstration. Its reward was Reward_rule = Reward_acc + Reward_format — correctness of the final answer, plus a check that the reasoning was wrapped in <think> tags — with no neural scorer anywhere in the reasoning stage. The R1-Zero variant started from a base model with no supervised reasoning data at all, and its average pass@1 on AIME 2024 rose from 15.6% to 77.9% over the course of RL training. Long chain-of-thought behaviour — self-checking, backtracking, spending more tokens on harder problems — emerged from the correctness reward alone. Training R1-Zero took 64×8 H800 GPUs for roughly 198 hours.

Tülu 3. AI2's open post-training recipe is the reference implementation, and the more useful one to read if you intend to reproduce any of this. RLVR is the final stage, after SFT and DPO, trained on a mixture of roughly 30,000 prompts across three verifiers: GSM8K (extract the final number, compare to the label), MATH (extract the answer, apply the "flex" matching logic), and IFEval (check each format constraint programmatically). The full recipe, data and code are public, which is why almost every subsequent RLVR paper cites it as the baseline.

Nemotron-Research-Reasoning-Qwen-1.5B. NVIDIA's ProRL work trained a 1.5B model on 136K verifiable problems spanning maths, code, STEM, logic puzzles and instruction following — the interesting part being how far the verifiable-reward idea stretches once you stop treating it as a maths technique. See Nemotron for the family.

Everything with a test suite. The practical reason RLVR matters outside frontier labs is that a grader is often something you already have. A team with a JSON schema, a linter, a regression suite or a SQL execution harness has a verifier; what they do not have is 50,000 human preference comparisons. That asymmetry — graders are cheap, preference data is not — is what moved reinforcement fine-tuning from a frontier-lab technique to something a small team can attempt.

Key Concepts

What makes a task verifiable

The requirement is stricter than "has a right answer". A task is verifiable for RLVR purposes when a program can decide correctness cheaply, deterministically, and without a human in the loop — and when the check is not itself gameable. Arithmetic with a single final answer qualifies. Code with a test suite mostly qualifies. Format and schema constraints qualify almost perfectly, which is why instruction-following with checkable constraints was one of Tülu 3's three domains.

The boundary is not where people expect. "Is this proof correct?" is verifiable if you are writing Lean and not verifiable if you are writing English. "Does this SQL return the right rows?" is verifiable against a fixture database and not verifiable in general. "Is this a good summary?" is not verifiable at all, and no amount of engineering makes it so — you can build a scorer, but the moment the scorer is learned you are doing RLHF again and have inherited every failure mode you left.

Why RLHF did not disappear

Because most of what makes a model pleasant to use has no checker. Tone, tact, refusal behaviour, when to ask a clarifying question, how much hedging is honest rather than annoying, whether a factual claim about a contested topic is fairly stated — these are judgements, and a judgement cannot be compiled. There is no v(x, y) for "was that a kind way to deliver bad news".

So the modern pipeline runs both, in sequence and for different targets: SFT, then preference tuning (DPO or PPO-based RLHF) for style and helpfulness, then RLVR for what a program can grade. DeepSeek-R1 does the same thing at a finer grain, combining a rule reward for reasoning with a model-based preference reward for general tasks — and reports that running too many steps with the preference reward reintroduced reward hacking, so it was confined to the final 400 of 1,700 steps.

Outcome rewards versus process rewards

RLVR as usually practised grades the endpoint: the final answer, the test result. It says nothing about whether the reasoning that produced it was sound, which is why a model can reach the right number through a wrong argument and be reinforced for it. Yue et al. found this frequently enough that they hand-checked chains of thought on a subset of outputs rather than trusting final-answer matching. Process supervision — grading each step — is the alternative, and it is more informative and far more expensive, since the step labels have to come from somewhere.

Challenges

The verifier's false negatives are a systematic tax on the reward. A study of rule-based maths verifiers found near-perfect precision — above 99%, so an answer that passes almost certainly is correct — paired with an average recall of only 86%, meaning roughly one correct answer in seven is scored as wrong because it was formatted differently: 0.5 against 1/2, an unsimplified fraction, a stray trailing period. Think about what that does to training. If the model genuinely solves a problem with probability p, the reward it observes is about 0.86p, and the gap is not random noise — it concentrates on exactly the responses whose formatting is unusual. The same study found recall falling to 0.78 on harder datasets, and getting worse as the generating model gets stronger, because stronger models produce more varied answer formats. Grading is hardest precisely where it matters most.

Reward hacking does not go away; it changes target. With a learned reward model the failure is flattery. With a verifier the failure is gaming the check. Tülu 3's own appendix has the canonical example: asked to measure a pen with the constraint "in your response, the letter e should appear 14 times", an over-optimized model answered e, e, e, e, e, e, e, e, e, e, e, e, e, e and nothing else. Constraint satisfied, reward collected, task ignored. In code the equivalent is a solution that special-cases the visible tests, and the equivalent for model-based verifiers is worse still — the same study found that generative verifiers could be fooled by adversarial prefixes, empty symbols, gibberish and prompt injection, whether or not they had been fine-tuned for verification.

Partial credit is genuinely hard. A binary reward cannot distinguish a near-miss from nonsense, and the obvious fixes make things worse. Give credit for intermediate steps and you need a grader for steps, which is the expensive thing you were avoiding. Give credit for format compliance and the model learns to produce beautifully formatted wrong answers, which is the pen-measuring failure in a different costume. Most production recipes accept the binary reward and manage the problem through problem selection instead — feeding the model prompts it solves sometimes, where the group spread is informative.

Elicitation or new capability? This one is open. The optimistic reading is that RL discovers reasoning strategies the base model did not have. The pessimistic reading is that it only makes the model more likely to sample strategies it already had, at the cost of the ones it drops.

The strongest evidence for the pessimistic reading is Yue et al.'s pass@k analysis. Measured at k = 1, RLVR-trained models beat their base models everywhere — that is the result everyone reports. Measured at large k, the curves cross. On AIME24 at k = 1024, they found the base model solved 13.3% of problems that its RLVR-trained version could not, while the set of problems the RLVR model solved and the base model could not was 0.0% — not one. Across training, pass@1 on their training set climbed from 26.1 to 42.5 while pass@256 fell. In their framing, RLVR was sharpening the distribution inside the base model's existing support, not extending it.

The counter-evidence is real too. NVIDIA's ProRL argues that the narrowing is an artefact of short training runs, and reports that with KL control, reference-policy resets and prolonged training, RL-trained models beat base models across the pass@k range including on tasks where the base model fails at every k — with the largest gains where the base model was weakest. And a third result complicates both: "Spurious Rewards" showed that on Qwen2.5-Math-7B, GRPO training with randomly assigned rewards improved MATH-500 by 21.4 absolute points, against 29.1 for ground-truth rewards, by amplifying a "code reasoning" behaviour the model had already learned in pretraining (its frequency rose from 65% to over 90%). The same spurious rewards produced no gains for Llama3 or OLMo2. If a random reward captures three-quarters of the benefit on one model family and none on another, then some published RLVR gains are measuring the base model, not the method.

Treat this as live. The honest summary is that RLVR reliably improves the answer you get on the first try, that whether it expands what the model can do at all depends on the training regime and the base model, and that any paper claiming to settle it on a single model family should be read with the Qwen result in mind.

It does not generalize outward for free. A model tuned hard on verifiable maths gets better at verifiable maths. Tülu 3 is explicit that improvement on the targeted evaluation does not guarantee an improvement in the average across all evaluations, and that over-optimization — visible as the KL divergence from the reference model growing — reliably drops the average score even while the verifiable reward keeps climbing. The number you are watching goes up exactly while the model gets worse, which is the same shape of failure RLHF has, arriving by a different road.

Future Trends

The verifiable surface keeps growing. Every task that acquires a cheap automatic checker moves from the preference side of the line to the verifiable side. The direction of travel is from maths and code toward tool use, agentic tasks with a testable end state, and anything expressible as "did the environment end up in the right state" — which is a much larger set than "did the final number match".

Verifier engineering becomes the bottleneck. When the reward was a learned model, the scarce resource was preference data. When the reward is a program, the scarce resource is a program that is both complete enough not to reject correct work and tight enough not to be gamed. The 14% false-negative rate above is not a solved problem, and hybrid schemes — a rule-based verifier with a model-based fallback for the answers the rules reject — are where most of the current effort is going.

Process rewards get cheaper. Hand-labelling reasoning steps does not scale. Deriving step labels automatically, by checking whether a prefix of the reasoning leads to a correct completion, turns process supervision into something a verifier can bootstrap. As models spend more test-time compute, grading the trajectory rather than the endpoint matters more.

The pass@k debate drives the methodology. The immediate practical consequence of the elicitation argument is that reporting pass@1 alone is no longer sufficient to claim a capability gain. Expect pass@k curves, cross-family replication and base-model controls to become standard in RLVR papers, largely because the Qwen-specific spurious-reward result made everyone nervous about single-family results.

Code Example

Two failure modes are easier to see in ten lines than in ten paragraphs. First, the false negative: a verifier that string-matches the final answer rejects most of the ways of writing it.

from fractions import Fraction

GOLD = "1/2"
sampled = ["1/2", "0.5", "\\frac{1}{2}", "2/4", ".5", "1/2."]   # all equal one half

def exact_match(pred, gold):
    return 1.0 if pred.strip() == gold.strip() else 0.0

def normalized(pred, gold):
    def parse(s):
        s = s.strip().rstrip(".").replace("\\frac{", "").replace("}{", "/").replace("}", "")
        try:
            return Fraction(s)
        except ValueError:
            return None
    p, g = parse(pred), parse(gold)
    return 1.0 if p is not None and p == g else 0.0

for name, fn in [("exact_match", exact_match), ("normalized", normalized)]:
    rewards = [fn(s, GOLD) for s in sampled]
    print(f"{name:12} rewards={rewards}  mean={sum(rewards)/len(rewards):.2f}")

Output:

exact_match  rewards=[1.0, 0.0, 0.0, 0.0, 0.0, 0.0]  mean=0.17
normalized   rewards=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0]  mean=1.00

Every one of those six responses is correct. The naive verifier scores five of them zero, and the model is trained to believe five of its six correct answers were wrong. This is the mechanism behind the 86% recall figure, and it is why "write a verifier" is real engineering rather than a one-liner.

The second failure runs the other way — a wrong answer that the verifier accepts:

VISIBLE_TESTS = [((2, 3), 5), ((0, 0), 0), ((-1, 1), 0)]
HELDOUT_TESTS = [((7, 8), 15), ((100, 1), 101)]

def candidate(a, b):
    return {(2, 3): 5, (0, 0): 0, (-1, 1): 0}[(a, b)]   # "passes the tests"

def verifier(fn, tests):
    passed = 0
    for args, expected in tests:
        try:
            passed += (fn(*args) == expected)
        except Exception:
            pass
    return passed / len(tests)

print("reward from visible tests :", verifier(candidate, VISIBLE_TESTS))
print("held-out accuracy        :", verifier(candidate, HELDOUT_TESTS))

Output:

reward from visible tests : 1.0
held-out accuracy        : 0.0

A perfect reward for a function that has learned nothing about addition. Nothing in the RL loop can tell the difference — the verifier said 1.0, so the update reinforces the lookup table. Held-out tests, randomized fixtures and property-based checks exist to make this harder, and the general lesson stands: RLVR optimizes exactly what the grader measures, so the grader is now the specification.

Academic Sources

• "Tülu 3: Pushing Frontiers in Open Language Model Post-Training" — Lambert et al. (2024). Introduces and names RLVR; the verifiable prompt mixture, the α = 10 reward, and the IFEval over-optimization examples.
• "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" — DeepSeek-AI (2025). Rule-based accuracy and format rewards, and the explicit rejection of neural reward models for reasoning tasks.
• "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" — Yue et al. (2025). The pass@k analysis and the solvable-problem coverage table.
• "ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models" — Liu et al. (2025). The counter-argument, plus the 136K-problem multi-domain verifiable dataset.
• "Spurious Rewards: Rethinking Training Signals in RLVR" — Shao et al. (2025). Random rewards, clipping bias, and why single-model-family results are suspect.
• "From Accuracy to Robustness: A Study of Rule- and Model-Based Verifiers in Mathematical Reasoning" — Huang et al. (2025). Verifier precision, recall and susceptibility to hacking patterns.