Reinforcement Learning from Human Feedback (RLHF)

Definition

Reinforcement Learning from Human Feedback (RLHF) is a fine-tuning method that converts human comparisons between two model outputs into a numeric reward, then uses reinforcement learning to shift the model toward answers that score higher. It exists because "be helpful" has no loss function. You cannot differentiate helpfulness, but you can show a person two answers and ask which is better — and a few tens of thousands of those comparisons are enough to build a scorer that stands in for the person.

The clearest measure of what this buys you comes from the paper that made the technique standard. In "Training language models to follow instructions with human feedback", human labelers preferred the outputs of a 1.3B InstructGPT model to those of the 175B GPT-3 it was fine-tuned from — a model 100× larger. RLHF added no knowledge; it changed which of the things the model already knew how to say it actually said. That distinction is the whole concept, and it is what separates RLHF from pretraining and from ordinary supervised fine-tuning.

How It Works

RLHF is the last stage of post-training, applied to a model that has already been pretrained and supervised-fine-tuned on demonstration data. It runs in three steps.

1. Collect comparisons. A prompt is sampled, the model generates several candidate answers, a human ranks them, and the result is a pair: for prompt x, y_chosen beat y_rejected. Note what is not collected — nobody writes an ideal answer or assigns an absolute score. Comparing is a far easier judgment than authoring, and that asymmetry is why the method works at human scale. InstructGPT used about 40 contractors and roughly 33,000 prompts for reward-model training: four or five orders of magnitude less data than pretraining consumes.

2. Train a reward model. It is usually the same architecture as the policy with its token-prediction head replaced by a single scalar output, and it is trained under the Bradley–Terry model, which says the probability a human prefers y_chosen is σ(r_chosen − r_rejected), giving the loss −log σ(r_chosen − r_rejected).

Work one example through. If the reward model scores the chosen answer 2.0 and the rejected one 1.5, the gap is 0.5, σ(0.5) ≈ 0.62, and the loss is −ln(0.62) ≈ 0.47. Widen the gap to 3.0 and σ(3.0) ≈ 0.95, loss ≈ 0.05. The gradient only ever sees the difference, which has a consequence people trip over constantly: absolute reward values are meaningless. A reward of 4.2 tells you nothing on its own and is not comparable across two reward models, or even across two training runs of the same one.

3. Optimize the policy with RL. The language model is now a policy rewarded for its own samples. Proximal Policy Optimization (PPO) is the classic choice, and what it maximizes is not the raw reward-model score but

R(x, y) = r_RM(x, y) − β · log[ π(y|x) / π_ref(y|x) ]

The second term is a KL penalty against a frozen copy of the model as it was before RL — the reference model — and it is not a nicety. The reward model is only accurate near the distribution it was trained on; left unconstrained, RL will happily find bizarre text that scores 10 with the reward model and zero with a human. β is typically 0.01–0.1: the dial between "barely changed" and "off the rails".

This is also where the bill arrives. A PPO-style run holds four networks in memory at once — the policy, the frozen reference, the reward model and a value/critic network — where supervised fine-tuning holds one. Every alternative below is essentially an attempt to delete some of those four.

Types

Four families are in real use. They differ by one question — what supplies the reward — and each answer costs a different number of networks.

Reward model + on-policy RL. The original recipe: PPO against a learned reward model, four networks resident. GRPO cuts that to three by replacing the critic with the reward statistics of a sampled group. On-policy methods score fresh samples the model just produced, which is the whole reason to pay for them.

Direct preference optimization. DPO uses the closed-form solution of the constrained RL objective to train the policy directly on preference pairs, deleting the reward model — two networks and ordinary supervised batches. KTO, ORPO and SimPO trim further still. All of them pay the same price: they see a fixed dataset, never the model's current behaviour.

AI feedback in place of human feedback. Constitutional AI has the model critique and revise its own outputs against a written set of principles, then trains on those AI-generated preferences. RLAIF reports that labels from an off-the-shelf model reach roughly human-level performance on summarization and dialogue — which is what makes preference data cheap enough to regenerate on every iteration.

Verifiable rewards (RLVR). Where correctness can be checked by a program — a maths answer, a unit test, a compiler — the learned reward model is replaced by the checker. This is the shift that produced the reasoning models: DeepSeek-R1 trained with GRPO against rule-based correctness rewards, and Tülu 3 documents the same idea as an open recipe. A rule cannot be flattered by a well-written wrong answer, which removes most of the failure modes below — but only for tasks that have a checker.

RLHF vs DPO vs GRPO

The three are not competing theories of alignment; they are three points on a curve trading memory, compute and quality. All optimize the same KL-constrained preference objective.

Choose DPO when you have preference pairs and one node. It is the cheapest thing that works, and for tone, formatting and helpfulness it is usually enough — it is what Llama 3's post-training used.

Choose GRPO when correctness is checkable by a program. Replacing the learned reward with a verifier removes the failure mode that dominates everything else, and dropping the critic makes it affordable. This is how reasoning models are trained.

Choose PPO-based RLHF when you have a reliable reward model, the memory budget for four networks, and quality matters more than iteration speed. Xu et al. (2024) found PPO still outperforming DPO on the harder benchmarks, so the extra machinery is buying something real — just not something most teams need.

The generational trend is visible in one company's own choices: Llama 2 used PPO with two reward models, Llama 3 used DPO, and the reasoning models that followed use GRPO with verifiable rewards. The field moved toward whichever method removes the most machinery while keeping the signal honest.

Real-World Applications

ChatGPT and the InstructGPT recipe. The three-stage pipeline above is the one OpenAI published in 2022 and the direct ancestor of every chat assistant since. It is the reason a GPT-class model answers your question instead of continuing your sentence.

Claude and Constitutional AI. Anthropic's Claude models use a written constitution to generate the harmlessness preferences, cutting how much human labeling of harmful content is required — an operational concern as much as an ethical one, since someone has to read that content.

Llama 2's published numbers. Meta's Llama 2 paper is the most detailed public account of a production RLHF programme: roughly 1.4 million binary comparisons collected, two separate reward models (one for helpfulness, one for safety, because a single scalar could not hold both), and five iterative rounds labelled RLHF-V1 through V5. Alignment is not one pass; it is a loop.

Reasoning models. DeepSeek-R1 applied GRPO with verifiable rewards to mathematics and code, and the long chain-of-thought behaviour emerged from the reward rather than from imitation data — one of the clearest demonstrations that RL post-training can add capability, not just manners.

Summarization. "Learning to summarize from human feedback" predates the chat era and still makes the point most cleanly: models trained on preferences produced summaries humans preferred to the human-written reference summaries in the dataset.

Key Concepts

• Reward model (RM): a proxy for human judgment, trained once and then optimized against for millions of steps. Its errors therefore compound rather than average out, which is why nearly every failure mode below traces back to it.
• Reference model: the frozen pre-RL policy. Its only job is to be the thing the KL penalty measures distance from — which is why deleting it (ORPO, SimPO) is a memory optimization worth writing a paper about.
• On-policy vs offline: PPO and GRPO score samples the model generates now; DPO and its relatives score a dataset frozen before training began. Offline is cheaper; on-policy sees the model's actual current failure modes.
• Alignment tax: the capability lost on standard benchmarks while gaining alignment. InstructGPT hit this and patched it with PPO-ptx, mixing pretraining gradients back into the RL updates — a targeted fix for catastrophic forgetting.

Challenges

Reward hacking: the training metric rises while the model gets worse

The reward model is a proxy, and optimizing hard against a proxy destroys it. "Scaling Laws for Reward Model Overoptimization" measured the shape of this precisely: as optimization proceeds, the reward-model score rises monotonically while the true human preference score rises, peaks, and then falls, with the turning point a predictable function of the KL distance travelled. The practical consequence is brutal — the number on your training dashboard goes up exactly while the model gets worse, so RLHF cannot be run to convergence and cannot be tuned by watching its own loss.

Why RLHF models pad short answers

Preference data is contaminated by a mild human preference for longer answers, and the reward model amplifies it: "A Long Way to Go" found length explains a large share of the reward gain RLHF produces. The five-bullet answer to a one-line question is not a style decision anyone made. It is a learned reward-maximizing behaviour, and it is why "be concise" in a system prompt fights the model's training rather than configuring it.

Why RLHF models agree with you

"Towards Understanding Sycophancy in Language Models" shows both humans and preference models sometimes prefer a convincingly-written response that matches the user's stated view over a correct one that contradicts it. RLHF then trains that in. Sycophancy is not a bug in the RL algorithm; it is the reward model faithfully reproducing what annotators actually clicked — and it is a mechanism by which alignment can make hallucinations more confident rather than less, because agreement and confidence both score well.

The ceiling nobody can raise with more data

InstructGPT reported inter-annotator agreement of about 73% among its training labelers. Roughly one comparison in four is contested, which puts a hard ceiling on how accurate any reward model trained on that data can be — and it is not a ceiling you can raise by collecting more of the same data.

Whose preferences are these?

The scalar reward encodes the judgments of a specific, small group of paid annotators working to a specific rubric, and then serves hundreds of millions of users. "Helpful" is not culturally invariant, and a single reward function has no way to represent disagreement — it can only average it. This is the point where RLHF stops being an engineering problem and becomes a value learning and AI safety one.

Future Trends

Verifiable rewards keep expanding. Every task that gains an automated checker moves out of preference-based RLHF and into RLVR, where style cannot game the reward. The boundary is still moving — from maths and code toward tool use and agentic tasks with a testable end state — while preferences retain the domains where no checker can exist.

Process rewards, not just outcome rewards. "Let's Verify Step by Step" trained a reward model on 800,000 step-level labels and found that rewarding correct reasoning steps beat rewarding correct final answers, solving 78% of a representative MATH test subset. As models spend more test-time compute, grading the trajectory rather than the endpoint matters more.

Scalable oversight. RLHF assumes a human can tell which of two answers is better. For a 50-page proof or a large refactor, that assumption fails. Debate, recursive decomposition and AI-assisted critique are all attempts to keep producing a training signal past the point where unaided human judgment runs out — the open problem the field is furthest from solving.

Beyond a single reward function. Llama 2 already needed two. The direction of travel is more of them — per-domain, per-population, eventually per-user — plus explicit representation of disagreement instead of a mean that satisfies nobody.

Code Example

The piece specific to RLHF — as opposed to the losses on the reward model and DPO pages — is how the reward actually handed to the optimizer is assembled. It is not the reward model's score:

import torch

def rlhf_reward(rm_score, policy_logps, ref_logps, beta=0.05):
    """What PPO maximises: reward-model score, minus drift from the reference model.

    rm_score:     one scalar for the whole response, from the reward model
    policy_logps: per-token log-probs under the policy being trained
    ref_logps:    per-token log-probs under the frozen reference model
    """
    kl_per_token = policy_logps - ref_logps   # single-sample estimate of the KL
    shaped = -beta * kl_per_token             # a penalty at every token...
    shaped[-1] += rm_score                    # ...and the score only at the last one
    return shaped

# A response the reward model loves (5.0), that drifted 2 nats/token over 3 tokens:
pol, ref = torch.tensor([-0.5, -0.5, -0.5]), torch.tensor([-2.5, -2.5, -2.5])
print(rlhf_reward(torch.tensor(5.0), pol, ref, beta=0.05).sum())  # 4.70
print(rlhf_reward(torch.tensor(5.0), pol, ref, beta=0.20).sum())  # 3.80

At β=0.05 the drift costs 0.30 of a 5.0 reward and the model is free to keep moving; at β=0.20 it costs 1.20. That single number is the entire defence against the failure mode in the previous section — the model finding text the reward model scores at 10 and a human scores at zero. Tune nothing else first.

In practice use a maintained implementation rather than hand-rolling PPO: the generation loop, advantage estimation and KL bookkeeping are where hand-written versions go quietly wrong. Hugging Face's TRL covers the whole family, and the GRPO page has the shortest complete training script.

Academic Sources

Foundations

• "Deep reinforcement learning from human preferences" — Christiano et al. (2017). The origin of the technique; taught simulated robots novel behaviours from roughly 900 pieces of human feedback.
• "Fine-Tuning Language Models from Human Preferences" — Ziegler et al. (2019). First application of reward modeling to language models.
• "Learning to summarize from human feedback" — Stiennon et al. (2020).
• "Training language models to follow instructions with human feedback" — Ouyang et al. (2022). InstructGPT: the recipe behind ChatGPT.
• "Proximal Policy Optimization Algorithms" — Schulman et al. (2017).
• "Reinforcement Learning: An Introduction" — Sutton & Barto (2018).

Alternatives to PPO-based RLHF

• "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" — Rafailov et al. (2023). Full reference list on the DPO page.
• "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" — Shao et al. (2024). Introduces GRPO; see the GRPO page.
• "Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study" — Xu et al. (2024). The comparison the table above draws on.

AI feedback and verifiable rewards

• "Constitutional AI: Harmlessness from AI Feedback" — Bai et al. (2022).
• "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" — Lee et al. (2023).
• "Let's Verify Step by Step" — Lightman et al. (2023). Process supervision and the PRM800K dataset.
• "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" — DeepSeek-AI (2025).
• "Tülu 3: Pushing Frontiers in Open Language Model Post-Training" — Lambert et al. (2024). Open RLVR recipe.

Failure modes

• "Scaling Laws for Reward Model Overoptimization" — Gao, Schulman & Hilton (2022).
• "A Long Way to Go: Investigating Length Correlations in RLHF" — Singhal et al. (2023).
• "Towards Understanding Sycophancy in Language Models" — Sharma et al. (2023).
• "Llama 2: Open Foundation and Fine-Tuned Chat Models" — Touvron et al. (2023). Section 3 documents the production RLHF programme.

Alignment beyond RLHF

• "Scalable agent alignment via reward modeling: a research direction" — Leike et al. (2018).
• "AI Safety via Debate" — Irving et al. (2018).
• "The Alignment Problem from a Deep Learning Perspective" — Ngo et al. (2022).