Direct Preference Optimization (DPO)

DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.

On this page

Definition

Direct Preference Optimization (DPO) trains a language model on human preference pairs using an ordinary supervised loss — no reward model, no reinforcement learning, no sampling loop. It gets the same objective that RLHF optimizes, and reaches it by algebra rather than by search.

The insight is worth stating precisely, because it is the whole method: the KL-constrained objective that PPO-based RLHF maximizes has a closed-form optimal policy. If you know the optimum in closed form, you can invert the relationship, express the reward in terms of the policy that would be optimal for it, and substitute it into the preference loss. The reward model does not need to be trained — it needs to be eliminated.

The practical consequence is a large one. PPO-based RLHF holds four networks in memory and runs a generation loop; DPO holds two, one of which is frozen and can be precomputed, and trains on fixed batches like any supervised job. That is why DPO went from a 2023 paper to the default alignment method for open-weight models within about a year.

How It Works

Start from what RLHF is actually maximizing — reward minus a KL penalty against the frozen reference model π_ref:

max_π  E[ r(x, y) ] − β · KL( π(y|x) ‖ π_ref(y|x) )

This has a known solution. The optimal policy re-weights the reference model by the exponentiated reward:

π*(y|x) = (1 / Z(x)) · π_ref(y|x) · exp( r(x, y) / β )

Rearrange for the reward:

r(x, y) = β · log[ π*(y|x) / π_ref(y|x) ] + β · log Z(x)

The partition function Z(x) is intractable — it sums over every possible response — and this is where the trick lands. The Bradley–Terry preference loss depends only on the difference of two rewards for the same prompt, and β · log Z(x) depends only on x. It cancels. What is left is a loss with no reward model in it:

L_DPO = −log σ( β·log[π(y_w|x)/π_ref(y_w|x)] − β·log[π(y_l|x)/π_ref(y_l|x)] )

Read the shape of it: this is exactly the reward model's Bradley–Terry loss, with the scalar reward replaced by β times a log-ratio of policies. The model is the reward model, which is what the paper's subtitle means by "your language model is secretly a reward model".

Two consequences follow directly. First, β is the same KL knob as in PPO — around 0.1 in practice — only now it is a coefficient in a loss rather than a penalty on a reward. Second, the gradient acts on a margin: it increases log π(y_w|x) relative to log π(y_l|x). Nothing in the objective requires the chosen response's probability to actually rise.

Types

Standard DPO is the loss above: preference pairs, a frozen reference model, one supervised pass.

Iterative or online DPO breaks the biggest limitation by regenerating the data. Sample fresh responses from the current policy, get preferences on those, run a DPO round, repeat. This recovers much of what on-policy RL provides, at the cost of needing a preference source in the loop — usually an AI judge rather than humans.

Reference-free variants attack the remaining memory cost. ORPO folds a preference term into the supervised fine-tuning loss so alignment happens in a single stage with no reference model at all. SimPO replaces the log-ratio with a length-normalized average log-probability and adds a target margin.

Unpaired variants. KTO learns from individual thumbs-up/thumbs-down labels instead of pairs, drawing on prospect theory. This matters operationally more than theoretically: production feedback arrives as single ratings on single responses, not as neat comparisons.

Real-World Applications

Zephyr-7B was the demonstration that changed practice. Zephyr applied DPO to a Mistral base model using AI-generated preferences from UltraFeedback, with no human annotation in the loop at all, and matched much larger chat models on MT-Bench. It made preference alignment something a small team could run.

Llama 3. The Llama 3 Herd of Models documents DPO as the preference-optimization method in Meta's post-training pipeline, chosen over PPO explicitly for compute efficiency and better behaviour at scale — a notable reversal from Llama 2, which used PPO-based RLHF with two reward models. See Llama for the model family.

Tülu 3. AI2's open post-training recipe uses DPO as its preference stage before adding reinforcement learning with verifiable rewards on top — a representative modern pipeline, where DPO handles style and helpfulness and RL handles what can be checked.

Everyday fine-tuning. DPO is the reason a team with a single GPU node and a few thousand preference pairs can align a small model at all. The PPO alternative was never available at that budget.

Challenges

It is offline, and that is the real ceiling. DPO scores a dataset frozen before training began, so it never sees what the model is doing wrong now. "Understanding the performance gap between online and offline alignment algorithms" isolates this as the main source of the difference, and "Is DPO Superior to PPO for LLM Alignment?" found PPO outperforming DPO on harder benchmarks including competitive code generation. Cheaper is not the same as better, and on the tasks where alignment is hardest the gap shows up.

Likelihood displacement. Because the loss only enforces a margin, DPO can push down the probability of the chosen response as long as it pushes the rejected one down faster. "Unintentional Unalignment" documents this and shows it can move probability mass to responses that were in neither column — including unsafe ones. The loss curve looks healthy throughout. Log the chosen response's log-probability as a training metric, because the loss will not tell you.

Distribution mismatch with the preference data. DPO assumes the pairs are broadly on-policy for the model being trained. Preferences collected from a different model push the policy toward text it would not have produced, which is one reason a DPO run on someone else's open preference dataset often underperforms the same run on data sampled from your own checkpoint.

β is doing more work than it looks. It is the entire KL constraint. Set too low, the model drifts and degenerates; too high and nothing changes. Unlike PPO there is no separate KL term to monitor, so the diagnostic that would tell you which failure you are in is not there by default.

Future Trends

Online is winning back ground. Iterative DPO, and RL methods that removed PPO's cost in a different way — notably GRPO — are converging on the same conclusion: the value of scoring fresh samples was mostly what PPO was buying, and it is worth paying something for.

Preference data becomes synthetic by default. Zephyr showed AI-generated preferences work well enough. The remaining question is not whether to use them, but how to keep a model from optimizing against its own judge's blind spots.

Preferences retreat to what cannot be checked. As verifiable rewards absorb maths, code and tool use, DPO's territory narrows to tone, formatting, refusal behaviour and helpfulness — the things no unit test can score, and the things it was always best at.

Code Example

The loss is short enough to read in full, and reading it is the fastest way to see why likelihood displacement happens:

import torch.nn.functional as F

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    """Sequence-level log-probs in, scalar loss out. No reward model anywhere."""
    chosen_logratio = policy_chosen_logps - ref_chosen_logps
    rejected_logratio = policy_rejected_logps - ref_rejected_logps
    # Only the MARGIN is optimised — nothing requires chosen_logratio to rise.
    return -F.logsigmoid(beta * (chosen_logratio - rejected_logratio)).mean()

In practice use TRL, which handles the reference-model caching, length normalization and the metrics that matter:

from trl import DPOConfig, DPOTrainer

trainer = DPOTrainer(
    model=model,                       # ref_model=None reuses the frozen adapter base
    args=DPOConfig(output_dir="dpo-out", beta=0.1),
    train_dataset=dataset,             # columns: prompt, chosen, rejected
    processing_class=tokenizer,
)
trainer.train()

TRL logs logps/chosen alongside the loss. Watch it: a run where the loss falls steadily while logps/chosen also falls is displacing likelihood, not learning preferences.

Academic Sources

Frequently Asked Questions

DPO is a way to train a language model on human preference pairs using an ordinary supervised loss, with no reward model and no reinforcement learning. It works because the RLHF objective has a closed-form solution that lets you express the reward in terms of the model itself, so the reward model can be substituted away instead of trained.
DPO is a form of RLHF in the broad sense; the difference is what it removes. Classic PPO-based RLHF trains a separate reward model and then runs reinforcement learning with four networks in memory. DPO trains the policy directly from the same preference pairs with two networks, and the second one can be precomputed. It is much simpler to run and much harder to get wrong.
It is cheaper and more reliable to run; it is not uniformly better. Xu et al. (2024) found PPO can outperform DPO on harder benchmarks, and Tang et al. (2024) traced the gap to DPO being offline — it only ever sees a fixed dataset, never the model's current mistakes. Most teams start with DPO and move to PPO or GRPO only when they need the last few points.
Beta controls how far the trained model may drift from the reference model — it is the same KL constraint that appears in PPO-based RLHF, folded into the loss. Typical values are around 0.1. Lower beta permits larger changes and more degeneration risk; higher beta keeps the model close to where it started.
Because the loss only cares about the gap between chosen and rejected. Pushing the rejected response's probability down satisfies the objective just as well as pulling the chosen one up, and in practice both can fall together — a failure mode documented as likelihood displacement by Razin et al. (2024). Monitoring the chosen response's log-probability during training, not just the loss, catches it.
Yes for standard DPO — it appears in the loss as the baseline the KL constraint measures against — but it is frozen, so its log-probabilities can be computed once and cached rather than kept resident. Variants such as ORPO and SimPO remove the reference model entirely, at the cost of the constraint it provided.

Related Terms

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Group Relative Policy Optimization (GRPO)

GRPO is PPO without the critic: it scores a group of sampled answers against each other. How it works, what it saves, and where it biases training.

Fine-tuning (FT)

What fine-tuning is, when to use it instead of RAG or prompting, and the methods that matter in 2026 — from full fine-tuning to LoRA, QLoRA and DPO.

Loss Function

A mathematical function that measures how well a machine learning model's predictions match the actual target values, essential for training and optimization.

Large Language Model (LLM)

Large language models like GPT-5.6, Claude, and Gemini trained on massive text data, capable of understanding and generating human language.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary