Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Published July 21, 2026 Last reviewed July 24, 2026

Definition

A reward model is a model that reads a prompt and a candidate response and returns a single
number estimating how much a human would like that response. It exists for one reason: you
cannot put a human in a training loop that runs for millions of steps, so you train a cheap
approximation of the human once and query that instead.

It is the load-bearing component of
RLHF, and it is also where nearly
everything goes wrong. The reward model is not a measurement of quality — it is a proxy for
one, fitted from a finite set of human comparisons and then optimized against for far longer than
it was ever validated for. Understanding a reward model means understanding the gap between those
two things.

How It Works

A reward model is usually the same architecture as the model it will be used to train, with the
token-prediction head replaced by a linear layer that outputs one scalar. It is initialized from a
supervised-fine-tuned checkpoint, so it already understands language, and only the scoring
behaviour has to be learned.

Training data is comparisons, not scores. Annotators see one prompt and two responses and pick the
better one, which produces triples of the form (prompt x, chosen y_w, rejected y_l). Nobody
assigns an absolute grade, because humans are unreliable at that and reliable at comparing.

The Bradley–Terry model turns comparisons into a loss. It assumes the probability a human prefers
y_w over y_l is σ(r(x, y_w) − r(x, y_l)), which gives:

L = −log σ( r(x, y_w) − r(x, y_l) )

Only the difference enters the gradient, and that single fact explains most of the confusing
behaviour of reward models. Scoring the chosen response 2.0 and the rejected 1.5 gives a loss of
−ln σ(0.5) ≈ 0.47; scoring them 9.0 and 8.5 gives exactly the same loss. The scale has no zero
and no unit. A reward of 4.2 means nothing in isolation, cannot be compared across two reward
models, and cannot even be compared across two training runs of the same architecture on the same
data.

Once trained, the reward model is frozen and used in one of two places. During training it
supplies the reward that reinforcement learning maximizes.
During inference it can rank candidates: generate N responses, score them all, return the best
— best-of-N sampling, which buys quality with compute and needs no training at all.

Types

Bradley–Terry scalar reward models are the default described above: one number, one forward
pass, cheap enough to call on every sample of an RL run.

Outcome vs process reward models. An outcome reward model (ORM) scores the final answer only.
A process reward model (PRM) scores each intermediate reasoning step. “Let’s Verify Step by
Step” built the PRM800K dataset of 800,000 step-level human
labels and found process supervision solved 78% of a representative MATH test subset, clearly
beating outcome supervision. The catch is in that number: 800,000 hand-labelled steps is a very
expensive dataset, which is why ORMs remain far more common.

Generative reward models and LLM-as-a-judge prompt a capable model to compare two responses in
natural language rather than emitting a scalar. They need no training and can explain themselves,
but cost a full generation per comparison. That price is affordable for evaluation and mostly not
affordable inside an RL loop.

Rule-based verifiers are not learned at all — a unit test, a symbolic maths checker, a regex
on a final answer. Where one exists it strictly dominates a learned reward model, because a rule
cannot be flattered by confident prose. This is the basis of RLVR, and pairing a verifier with
GRPO is how reasoning models are trained.

There is also the option of not having a reward model at all:
DPO derives the same objective in closed form and
optimizes the policy directly on preference pairs. Whether that is a simplification or a loss
depends on whether you needed to score responses the model had not produced yet.

Real-World Applications

Llama 2’s two reward models. Meta’s Llama 2 paper is the
most detailed public account: roughly 1.4 million binary comparisons collected, and crucially
two separate reward models — one for helpfulness and one for safety — because a single scalar
could not represent both without one objective quietly eating the other. That design decision is
the clearest practical evidence that “reward” is not one quantity.

Rejection sampling in production. Llama 2 also used its reward models at inference time,
sampling K outputs and keeping the highest-scoring one to build training data for the next round.
The same trick, best-of-N, is how a reward model improves answers without any RL at all.

RewardBench. RewardBench was the first benchmark to
evaluate reward models directly rather than the policies trained from them, on chat, reasoning and
safety subsets. It made a previously invisible component measurable — before it, teams shipped
reward models whose accuracy they could only infer from the downstream model’s behaviour.

Reasoning model training. DeepSeek-R1 used rule-based verifiers rather
than a learned reward model for its maths and code training, which is a real-world statement about
reward models: where the alternative exists, teams take it.

Challenges

Overoptimization is the defining failure. “Scaling Laws for Reward Model
Overoptimization” measured the shape precisely: as a policy is
optimized against a reward model, the reward-model score rises monotonically while true human
preference rises, peaks, and then declines, with the turning point predictable from the KL
distance travelled. The consequence is that the metric on your dashboard climbs while the model
gets worse, so RLHF cannot be trained to convergence and a reward model cannot be tuned by
watching its own reward.

It inherits every bias in the annotation pool. Length is the best-documented case —
“A Long Way to Go” found length accounts for much of the
apparent reward gain. Sycophancy is the most damaging: “Towards Understanding
Sycophancy” showed preference models sometimes score a
well-written response that agrees with the user above a correct one that does not.

Annotator disagreement caps its accuracy. InstructGPT reported inter-annotator agreement of
about 73%. Roughly one comparison in four is contested, and no amount of additional data of the
same kind raises that ceiling — it is a property of the question being asked, not the dataset size.

It goes stale. A reward model is trained on responses from one policy and then used to score a
policy that has since changed. The further the policy drifts, the more it is being scored on text
outside the reward model’s training distribution — which is precisely why RLHF needs a KL penalty
and why production pipelines like Llama 2’s retrain the reward model between rounds instead of
fitting one and walking away.

Future Trends

Verifiers replace learned rewards where they can. The boundary is moving from maths and code
toward tool use and agentic tasks with a testable end state. Learned reward models will keep the
domains that have no checker — tone, helpfulness, harmlessness — which are also the domains where
they are least reliable.

Process supervision at scale. Hand-labelling 800,000 reasoning steps does not generalize.
Generating step labels automatically, by checking whether a step leads to a correct completion,
is the direction that makes PRMs affordable outside of a frontier lab budget.

More than one reward. Llama 2 needed two. Per-domain and per-population reward models, plus
explicit representation of disagreement rather than an average, are the natural continuation —
the alternative is a single scalar that satisfies nobody in particular.

Code Example

The whole training objective is three lines. What matters is what the arithmetic below shows:

import torch
import torch.nn.functional as F

def reward_model_loss(rm, prompt, chosen, rejected):
    """Bradley-Terry: -log sigmoid(r_chosen - r_rejected)."""
    return -F.logsigmoid(rm(prompt, chosen) - rm(prompt, rejected)).mean()

# Only the gap matters, never the absolute values:
gap = lambda a, b: -F.logsigmoid(torch.tensor([a - b])).item()
print(gap(2.0, 1.5))   # 0.474
print(gap(9.0, 8.5))   # 0.474  <- identical loss, wildly different scores
print(gap(2.0, 1.9))   # 0.644  <- a smaller margin is penalised more

Using it at inference needs no reinforcement learning at all — best-of-N is often the cheapest way
to find out whether a reward model is any good before committing to an RL run:

def best_of_n(policy, rm, prompt, n=8):
    """Generate n candidates, return the one the reward model ranks highest."""
    candidates = [policy.generate(prompt) for _ in range(n)]
    return max(candidates, key=lambda c: rm(prompt, c))

If best-of-8 does not visibly improve outputs, the reward model is not ready to be optimized
against for a million steps.

Academic Sources

• “Deep reinforcement learning from human preferences” —
  Christiano et al. (2017). Reward modeling from comparisons.
• “Fine-Tuning Language Models from Human Preferences” —
  Ziegler et al. (2019). Reward models for language.
• “Training language models to follow instructions with human
  feedback” — Ouyang et al. (2022). InstructGPT; annotator
  agreement figures.
• “Llama 2: Open Foundation and Fine-Tuned Chat Models” —
  Touvron et al. (2023). Two reward models, 1.4M comparisons, rejection sampling.
• “Scaling Laws for Reward Model Overoptimization” — Gao,
  Schulman & Hilton (2022).
• “A Long Way to Go: Investigating Length Correlations in RLHF”
  — Singhal et al. (2023).
• “Towards Understanding Sycophancy in Language Models” —
  Sharma et al. (2023).
• “Let’s Verify Step by Step” — Lightman et al. (2023).
  Process reward models and PRM800K.
• “RewardBench: Evaluating Reward Models for Language
  Modeling” — Lambert et al. (2024).