What is DPO (direct preference optimization) in LLMs?

Also called DPO, DPO training

Definition

Direct preference optimization (DPO) is a fine-tuning method that trains a language model on pairs of preferred and rejected responses with a simple classification loss, without the reward model and reinforcement learning that RLHF needs.

Direct preference optimization (DPO) is a method for aligning a language model with preferences by training it directly on pairs of responses to the same prompt, one preferred and one rejected. Rafailov et al. (2023) introduced it as a simpler alternative to RLHF: it uses the same kind of preference data but needs no separate reward model and no reinforcement learning loop. Hugging Face's TRL library implements it as DPOTrainer, so teams can run preference tuning on an open-weights model with the same tooling they use for supervised fine-tuning.

How DPO works#

DPO works by raising the probability of the preferred response relative to the rejected one, measured against a frozen reference model, usually the supervised fine-tuned starting point. A parameter called beta controls how far the trained model may drift from that reference. The loss is a binary classification loss on each pair, so training resembles ordinary supervised fine-tuning.

Each training example holds a prompt and two completions. In the conversational format that the TRL DPOTrainer accepts, a record looks like this:

{
  "prompt": [{"role": "user", "content": "Summarize this ticket in one sentence."}],
  "chosen": [{"role": "assistant", "content": "The customer was charged twice and wants one refund."}],
  "rejected": [{"role": "assistant", "content": "Here is a summary of the ticket you shared with me."}]
}

DPO rewrites the RLHF objective so that the optimal policy has a closed-form solution. The model's own log-probability ratio against the reference then acts as an implicit reward, the idea behind the paper's subtitle, Your Language Model is Secretly a Reward Model. The authors' pipeline samples two responses per prompt, collects a human preference label for each pair, and then minimizes the DPO loss, starting the reference from the supervised fine-tuned model when one exists.

In TRL, the reference defaults to the model's state before DPO training starts, and beta defaults to 0.1; higher values keep the model closer to the reference. The trainer also exposes variants through loss_type, including IPO and a length-normalized loss from the SimPO paper.

DPO vs RLHF#

DPO and RLHF train on the same kind of preference data but optimize differently. RLHF first fits a reward model to the preferences, then uses reinforcement learning, typically PPO, to maximize that reward while the model generates new outputs. DPO skips both steps and updates the model directly from a fixed dataset, which makes it simpler and cheaper to run.

RLHF with PPO DPO
Reward model Trained separately on preferences None; the log-probability ratio is an implicit reward
Sampling during training Yes, the model generates new outputs No, it trains on a fixed offline dataset
Optimization Reinforcement learning Classification loss on each pair

The authors report that DPO exceeded PPO-based RLHF at controlling the sentiment of generations, and matched or improved response quality on summarization and single-turn dialogue, in experiments on models of up to 6 billion parameters. Because DPO learns only from the pairs it receives, audit a sample of pairs before training, whether humans or an LLM judge produced the labels.

RAG vs fine-tuning explains when preference tuning, or any change to model weights, is the right lever.

Cite this definition

The AI Engineering Editors. "What is DPO (direct preference optimization) in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/direct-preference-optimization

Title
What is DPO (direct preference optimization) in LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM models and benchmarks hub · Updated