Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

On this page

Definition

A reward model is a model that reads a prompt and a candidate response and returns a single number estimating how much a human would like that response. It exists for one reason: you cannot put a human in a training loop that runs for millions of steps, so you train a cheap approximation of the human once and query that instead.

It is the load-bearing component of RLHF, and it is also where nearly everything goes wrong. The reward model is not a measurement of quality — it is a proxy for one, fitted from a finite set of human comparisons and then optimized against for far longer than it was ever validated for. Understanding a reward model means understanding the gap between those two things.

How It Works

A reward model is usually the same architecture as the model it will be used to train, with the token-prediction head replaced by a linear layer that outputs one scalar. It is initialized from a supervised-fine-tuned checkpoint, so it already understands language, and only the scoring behaviour has to be learned.

Training data is comparisons, not scores. Annotators see one prompt and two responses and pick the better one, which produces triples of the form (prompt x, chosen y_w, rejected y_l). Nobody assigns an absolute grade, because humans are unreliable at that and reliable at comparing.

The Bradley–Terry model turns comparisons into a loss. It assumes the probability a human prefers y_w over y_l is σ(r(x, y_w) − r(x, y_l)), which gives:

L = −log σ( r(x, y_w) − r(x, y_l) )

Only the difference enters the gradient, and that single fact explains most of the confusing behaviour of reward models. Scoring the chosen response 2.0 and the rejected 1.5 gives a loss of −ln σ(0.5) ≈ 0.47; scoring them 9.0 and 8.5 gives exactly the same loss. The scale has no zero and no unit. A reward of 4.2 means nothing in isolation, cannot be compared across two reward models, and cannot even be compared across two training runs of the same architecture on the same data.

Once trained, the reward model is frozen and used in one of two places. During training it supplies the reward that reinforcement learning maximizes. During inference it can rank candidates: generate N responses, score them all, return the best — best-of-N sampling, which buys quality with compute and needs no training at all.

Types

Bradley–Terry scalar reward models are the default described above: one number, one forward pass, cheap enough to call on every sample of an RL run.

Outcome vs process reward models. An outcome reward model (ORM) scores the final answer only. A process reward model (PRM) scores each intermediate reasoning step. "Let's Verify Step by Step" built the PRM800K dataset of 800,000 step-level human labels and found process supervision solved 78% of a representative MATH test subset, clearly beating outcome supervision. The catch is in that number: 800,000 hand-labelled steps is a very expensive dataset, which is why ORMs remain far more common.

Generative reward models and LLM-as-a-judge prompt a capable model to compare two responses in natural language rather than emitting a scalar. They need no training and can explain themselves, but cost a full generation per comparison. That price is affordable for evaluation and mostly not affordable inside an RL loop.

Rule-based verifiers are not learned at all — a unit test, a symbolic maths checker, a regex on a final answer. Where one exists it strictly dominates a learned reward model, because a rule cannot be flattered by confident prose. This is the basis of RLVR, and pairing a verifier with GRPO is how reasoning models are trained.

There is also the option of not having a reward model at all: DPO derives the same objective in closed form and optimizes the policy directly on preference pairs. Whether that is a simplification or a loss depends on whether you needed to score responses the model had not produced yet.

Real-World Applications

Llama 2's two reward models. Meta's Llama 2 paper is the most detailed public account: roughly 1.4 million binary comparisons collected, and crucially two separate reward models — one for helpfulness and one for safety — because a single scalar could not represent both without one objective quietly eating the other. That design decision is the clearest practical evidence that "reward" is not one quantity.

Rejection sampling in production. Llama 2 also used its reward models at inference time, sampling K outputs and keeping the highest-scoring one to build training data for the next round. The same trick, best-of-N, is how a reward model improves answers without any RL at all.

RewardBench. RewardBench was the first benchmark to evaluate reward models directly rather than the policies trained from them, on chat, reasoning and safety subsets. It made a previously invisible component measurable — before it, teams shipped reward models whose accuracy they could only infer from the downstream model's behaviour.

Reasoning model training. DeepSeek-R1 used rule-based verifiers rather than a learned reward model for its maths and code training, which is a real-world statement about reward models: where the alternative exists, teams take it.

Challenges

Overoptimization is the defining failure. "Scaling Laws for Reward Model Overoptimization" measured the shape precisely: as a policy is optimized against a reward model, the reward-model score rises monotonically while true human preference rises, peaks, and then declines, with the turning point predictable from the KL distance travelled. The consequence is that the metric on your dashboard climbs while the model gets worse, so RLHF cannot be trained to convergence and a reward model cannot be tuned by watching its own reward.

It inherits every bias in the annotation pool. Length is the best-documented case — "A Long Way to Go" found length accounts for much of the apparent reward gain. Sycophancy is the most damaging: "Towards Understanding Sycophancy" showed preference models sometimes score a well-written response that agrees with the user above a correct one that does not.

Annotator disagreement caps its accuracy. InstructGPT reported inter-annotator agreement of about 73%. Roughly one comparison in four is contested, and no amount of additional data of the same kind raises that ceiling — it is a property of the question being asked, not the dataset size.

It goes stale. A reward model is trained on responses from one policy and then used to score a policy that has since changed. The further the policy drifts, the more it is being scored on text outside the reward model's training distribution — which is precisely why RLHF needs a KL penalty and why production pipelines like Llama 2's retrain the reward model between rounds instead of fitting one and walking away.

Future Trends

Verifiers replace learned rewards where they can. The boundary is moving from maths and code toward tool use and agentic tasks with a testable end state. Learned reward models will keep the domains that have no checker — tone, helpfulness, harmlessness — which are also the domains where they are least reliable.

Process supervision at scale. Hand-labelling 800,000 reasoning steps does not generalize. Generating step labels automatically, by checking whether a step leads to a correct completion, is the direction that makes PRMs affordable outside of a frontier lab budget.

More than one reward. Llama 2 needed two. Per-domain and per-population reward models, plus explicit representation of disagreement rather than an average, are the natural continuation — the alternative is a single scalar that satisfies nobody in particular.

Code Example

The whole training objective is three lines. What matters is what the arithmetic below shows:

import torch
import torch.nn.functional as F

def reward_model_loss(rm, prompt, chosen, rejected):
    """Bradley-Terry: -log sigmoid(r_chosen - r_rejected)."""
    return -F.logsigmoid(rm(prompt, chosen) - rm(prompt, rejected)).mean()

# Only the gap matters, never the absolute values:
gap = lambda a, b: -F.logsigmoid(torch.tensor([a - b])).item()
print(gap(2.0, 1.5))   # 0.474
print(gap(9.0, 8.5))   # 0.474  <- identical loss, wildly different scores
print(gap(2.0, 1.9))   # 0.644  <- a smaller margin is penalised more

Using it at inference needs no reinforcement learning at all — best-of-N is often the cheapest way to find out whether a reward model is any good before committing to an RL run:

def best_of_n(policy, rm, prompt, n=8):
    """Generate n candidates, return the one the reward model ranks highest."""
    candidates = [policy.generate(prompt) for _ in range(n)]
    return max(candidates, key=lambda c: rm(prompt, c))

If best-of-8 does not visibly improve outputs, the reward model is not ready to be optimized against for a million steps.

Academic Sources

Frequently Asked Questions

It is a model that reads a prompt and a response and returns a single number saying how good a human would find it. It exists so that reinforcement learning has something to optimize, since you cannot put a human in the training loop for millions of steps.
Nothing on its own. Reward models are trained only on the difference between a preferred and a rejected response, so the scale has no fixed zero and no fixed unit. A score of 4.2 is not comparable to 4.2 from a different reward model, or even from a different training run of the same one — only the ranking between two responses is meaningful.
A classic reward model has a scalar head and outputs a number directly, which is fast enough to call on every sample during RL. LLM-as-a-judge prompts a general model to compare two answers in text. The judge is more flexible and needs no training, but is far slower and more expensive per call, which is why judges dominate evaluation and scalar reward models dominate training.
An outcome reward model (ORM) scores only the final answer. A process reward model (PRM) scores each reasoning step. Lightman et al. (2023) found process supervision clearly better on mathematics, solving 78% of a representative MATH test subset — but it required 800,000 human step-level labels, which is why ORMs are still far more common.
Because annotators do, mildly, and the reward model amplifies it. Singhal et al. (2023) found that length explains a large share of the reward gain RLHF produces. Unless you correct for it, a model optimized against such a reward model learns to pad, and the reward keeps rising while quality does not.
Sometimes. DPO replaces it with a closed-form objective on preference pairs, and RLVR replaces it with a programmatic checker when correctness is verifiable — a unit test, a maths answer. For open-ended qualities like tone, helpfulness or harmlessness there is no checker, and a trained reward model remains the only option.

Related Terms

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Direct Preference Optimization (DPO)

DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.

Group Relative Policy Optimization (GRPO)

GRPO is PPO without the critic: it scores a group of sampled answers against each other. How it works, what it saves, and where it biases training.

Loss Function

A mathematical function that measures how well a machine learning model's predictions match the actual target values, essential for training and optimization.

Benchmark

An AI benchmark is a standardized test or dataset used to evaluate and compare the performance of different AI models across tasks like reasoning and coding.

AI Safety

Learn about AI safety principles and methods for ensuring artificial intelligence systems behave reliably without causing unintended harm to humans or society.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary