Direct Preference Optimization (DPO)
DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.
Definition
Direct Preference Optimization (DPO) trains a language model on human preference pairs using an
ordinary supervised loss — no reward model, no reinforcement learning, no sampling loop. It gets
the same objective that RLHF optimizes,
and reaches it by algebra rather than by search.
The insight is worth stating precisely, because it is the whole method: the KL-constrained
objective that PPO-based RLHF maximizes has a closed-form optimal policy. If you know the
optimum in closed form, you can invert the relationship, express the reward in terms of the policy
that would be optimal for it, and substitute it into the preference loss. The
reward model does not need to be trained — it needs to be eliminated.
The practical consequence is a large one. PPO-based RLHF holds four networks in memory and runs a
generation loop; DPO holds two, one of which is frozen and can be precomputed, and trains on fixed
batches like any supervised job. That is why DPO went from a 2023 paper to the default alignment
method for open-weight models within about a year.
How It Works
Start from what RLHF is actually maximizing — reward minus a KL penalty against the frozen
reference model π_ref:
max_π E[ r(x, y) ] − β · KL( π(y|x) ‖ π_ref(y|x) )
This has a known solution. The optimal policy re-weights the reference model by the exponentiated
reward:
π*(y|x) = (1 / Z(x)) · π_ref(y|x) · exp( r(x, y) / β )
Rearrange for the reward:
r(x, y) = β · log[ π*(y|x) / π_ref(y|x) ] + β · log Z(x)
The partition function Z(x) is intractable — it sums over every possible response — and this is
where the trick lands. The Bradley–Terry preference loss depends only on the difference of two
rewards for the same prompt, and β · log Z(x) depends only on x. It cancels. What is left is
a loss with no reward model in it:
L_DPO = −log σ( β·log[π(y_w|x)/π_ref(y_w|x)] − β·log[π(y_l|x)/π_ref(y_l|x)] )
Read the shape of it: this is exactly the reward model’s Bradley–Terry loss, with the scalar
reward replaced by β times a log-ratio of policies. The model is the reward model, which is
what the paper’s subtitle means by “your language model is secretly a reward model”.
Two consequences follow directly. First, β is the same KL knob as in PPO — around 0.1 in
practice — only now it is a coefficient in a loss rather than a penalty on a reward. Second, the
gradient acts on a margin: it increases log π(y_w|x) relative to log π(y_l|x). Nothing in
the objective requires the chosen response’s probability to actually rise.
Types
Standard DPO is the loss above: preference pairs, a frozen reference model, one supervised
pass.
Iterative or online DPO breaks the biggest limitation by regenerating the data. Sample fresh
responses from the current policy, get preferences on those, run a DPO round, repeat. This
recovers much of what on-policy RL provides, at the cost of needing a preference source in the
loop — usually an AI judge rather than humans.
Reference-free variants attack the remaining memory cost.
ORPO folds a preference term into the supervised fine-tuning
loss so alignment happens in a single stage with no reference model at all.
SimPO replaces the log-ratio with a length-normalized average
log-probability and adds a target margin.
Unpaired variants. KTO learns from individual
thumbs-up/thumbs-down labels instead of pairs, drawing on prospect theory. This matters
operationally more than theoretically: production feedback arrives as single ratings on single
responses, not as neat comparisons.
Real-World Applications
Zephyr-7B was the demonstration that changed practice. Zephyr
applied DPO to a Mistral base model using AI-generated preferences from UltraFeedback, with no
human annotation in the loop at all, and matched much larger chat models on MT-Bench. It made
preference alignment something a small team could run.
Llama 3. The Llama 3 Herd of Models documents DPO as the
preference-optimization method in Meta’s post-training pipeline, chosen over PPO explicitly for
compute efficiency and better behaviour at scale — a notable reversal from Llama 2, which used
PPO-based RLHF with two reward models. See Llama for the model family.
Tülu 3. AI2’s open post-training recipe uses DPO as its
preference stage before adding reinforcement learning with verifiable rewards on top — a
representative modern pipeline, where DPO handles style and helpfulness and RL handles what can be
checked.
Everyday fine-tuning. DPO is the reason a team with a single GPU node and a few thousand
preference pairs can align a small model at all. The PPO alternative was never available at that
budget.
Challenges
It is offline, and that is the real ceiling. DPO scores a dataset frozen before training
began, so it never sees what the model is doing wrong now. “Understanding the performance gap
between online and offline alignment algorithms” isolates this
as the main source of the difference, and “Is DPO Superior to PPO for LLM
Alignment?” found PPO outperforming DPO on harder benchmarks
including competitive code generation. Cheaper is not the same as better, and on the tasks where
alignment is hardest the gap shows up.
Likelihood displacement. Because the loss only enforces a margin, DPO can push down the
probability of the chosen response as long as it pushes the rejected one down faster.
“Unintentional Unalignment” documents this and shows it can
move probability mass to responses that were in neither column — including unsafe ones. The loss
curve looks healthy throughout. Log the chosen response’s log-probability as a training metric,
because the loss will not tell you.
Distribution mismatch with the preference data. DPO assumes the pairs are broadly on-policy for
the model being trained. Preferences collected from a different model push the policy toward text
it would not have produced, which is one reason a DPO run on someone else’s open preference dataset
often underperforms the same run on data sampled from your own checkpoint.
β is doing more work than it looks. It is the entire KL constraint. Set too low, the model
drifts and degenerates; too high and nothing changes. Unlike PPO there is no separate KL term to
monitor, so the diagnostic that would tell you which failure you are in is not there by default.
Future Trends
Online is winning back ground. Iterative DPO, and RL methods that removed PPO’s cost in a
different way — notably GRPO — are converging on
the same conclusion: the value of scoring fresh samples was mostly what PPO was buying, and it is
worth paying something for.
Preference data becomes synthetic by default. Zephyr showed AI-generated preferences work well
enough. The remaining question is not whether to use them, but how to keep a model from
optimizing against its own judge’s blind spots.
Preferences retreat to what cannot be checked. As verifiable rewards absorb maths, code and
tool use, DPO’s territory narrows to tone, formatting, refusal behaviour and helpfulness — the
things no unit test can score, and the things it was always best at.
Code Example
The loss is short enough to read in full, and reading it is the fastest way to see why likelihood
displacement happens:
import torch.nn.functional as F
def dpo_loss(policy_chosen_logps, policy_rejected_logps,
ref_chosen_logps, ref_rejected_logps, beta=0.1):
"""Sequence-level log-probs in, scalar loss out. No reward model anywhere."""
chosen_logratio = policy_chosen_logps - ref_chosen_logps
rejected_logratio = policy_rejected_logps - ref_rejected_logps
# Only the MARGIN is optimised — nothing requires chosen_logratio to rise.
return -F.logsigmoid(beta * (chosen_logratio - rejected_logratio)).mean()
In practice use TRL, which handles the reference-model caching, length normalization and the
metrics that matter:
from trl import DPOConfig, DPOTrainer
trainer = DPOTrainer(
model=model, # ref_model=None reuses the frozen adapter base
args=DPOConfig(output_dir="dpo-out", beta=0.1),
train_dataset=dataset, # columns: prompt, chosen, rejected
processing_class=tokenizer,
)
trainer.train()
TRL logs logps/chosen alongside the loss. Watch it: a run where the loss falls steadily while
logps/chosen also falls is displacing likelihood, not learning preferences.
Academic Sources
- “Direct Preference Optimization: Your Language Model is Secretly a Reward
Model” — Rafailov et al. (2023). The original derivation. - “Training language models to follow instructions with human
feedback” — Ouyang et al. (2022). The PPO-based baseline DPO
replaces. - “Zephyr: Direct Distillation of LM Alignment” — Tunstall et
al. (2023). DPO on AI-generated preferences. - “The Llama 3 Herd of Models” — Grattafiori et al. (2024).
DPO in a frontier post-training pipeline. - “Is DPO Superior to PPO for LLM Alignment? A Comprehensive
Study” — Xu et al. (2024). - “Understanding the performance gap between online and offline alignment
algorithms” — Tang et al. (2024). - “Unintentional Unalignment: Likelihood Displacement in Direct Preference
Optimization” — Razin et al. (2024). - “KTO: Model Alignment as Prospect Theoretic Optimization” —
Ethayarajh et al. (2024). - “ORPO: Monolithic Preference Optimization without Reference
Model” — Hong et al. (2024). - “SimPO: Simple Preference Optimization with a Reference-Free
Reward” — Meng et al. (2024).
Common questions
What is DPO in simple terms?
DPO is a way to train a language model on human preference pairs using an ordinary supervised loss, with no reward model and no reinforcement learning. It works because the RLHF objective has a closed-form solution that lets you express the reward in terms of the model itself, so the reward model can be substituted away instead of trained.
DPO vs RLHF — what is the actual difference?
DPO is a form of RLHF in the broad sense; the difference is what it removes. Classic PPO-based RLHF trains a separate reward model and then runs reinforcement learning with four networks in memory. DPO trains the policy directly from the same preference pairs with two networks, and the second one can be precomputed. It is much simpler to run and much harder to get wrong.
Is DPO better than PPO?
It is cheaper and more reliable to run; it is not uniformly better. Xu et al. (2024) found PPO can outperform DPO on harder benchmarks, and Tang et al. (2024) traced the gap to DPO being offline — it only ever sees a fixed dataset, never the model's current mistakes. Most teams start with DPO and move to PPO or GRPO only when they need the last few points.
What does the beta parameter do in DPO?
Beta controls how far the trained model may drift from the reference model — it is the same KL constraint that appears in PPO-based RLHF, folded into the loss. Typical values are around 0.1. Lower beta permits larger changes and more degeneration risk; higher beta keeps the model close to where it started.
Why does DPO sometimes make the preferred answers less likely?
Because the loss only cares about the gap between chosen and rejected. Pushing the rejected response's probability down satisfies the objective just as well as pulling the chosen one up, and in practice both can fall together — a failure mode documented as likelihood displacement by Razin et al. (2024). Monitoring the chosen response's log-probability during training, not just the loss, catches it.
Do I still need a reference model with DPO?
Yes for standard DPO — it appears in the loss as the baseline the KL constraint measures against — but it is frozen, so its log-probabilities can be computed once and cached rather than kept resident. Variants such as ORPO and SimPO remove the reference model entirely, at the cost of the constraint it provided.