---
title: "What is DPO (direct preference optimization) in LLMs?"
description: "Direct preference optimization (DPO) fine-tunes an LLM on pairs of preferred and rejected answers without a reward model. How DPO works, and DPO vs RLHF."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/direct-preference-optimization"
markdown_url: "https://theaiengineering.com/glossary/direct-preference-optimization.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Direct preference optimization","DPO","DPO training"]
citation: "The AI Engineering Editors. \"What is DPO (direct preference optimization) in LLMs?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/direct-preference-optimization"
---

# What is DPO (direct preference optimization) in LLMs?

Also called DPO, DPO training.

> **Definition of Direct preference optimization:** Direct preference optimization (DPO) is a fine-tuning method that trains a language model on pairs of preferred and rejected responses with a simple classification loss, without the reward model and reinforcement learning that RLHF needs.

Direct preference optimization (DPO) is a method for aligning a language model with preferences by training it directly on pairs of responses to the same prompt, one preferred and one rejected. [Rafailov et al. (2023)](https://arxiv.org/abs/2305.18290) introduced it as a simpler alternative to [RLHF](https://theaiengineering.com/glossary/rlhf): it uses the same kind of preference data but needs no separate reward model and no reinforcement learning loop. Hugging Face's TRL library implements it as `DPOTrainer`, so teams can run preference tuning on an [open-weights model](https://theaiengineering.com/glossary/open-weights-model) with the same tooling they use for supervised fine-tuning.

## How DPO works

DPO works by raising the probability of the preferred response relative to the rejected one, measured against a frozen reference model, usually the supervised fine-tuned starting point. A parameter called beta controls how far the trained model may drift from that reference. The loss is a binary classification loss on each pair, so training resembles ordinary supervised [fine-tuning](https://theaiengineering.com/glossary/fine-tuning).

Each training example holds a prompt and two completions. In the conversational format that the [TRL DPOTrainer](https://huggingface.co/docs/trl/dpo_trainer) accepts, a record looks like this:

```json
{
  "prompt": [{"role": "user", "content": "Summarize this ticket in one sentence."}],
  "chosen": [{"role": "assistant", "content": "The customer was charged twice and wants one refund."}],
  "rejected": [{"role": "assistant", "content": "Here is a summary of the ticket you shared with me."}]
}
```

DPO rewrites the RLHF objective so that the optimal policy has a closed-form solution. The model's own log-probability ratio against the reference then acts as an implicit reward, the idea behind the paper's subtitle, *Your Language Model is Secretly a Reward Model*. The authors' pipeline samples two responses per prompt, collects a human preference label for each pair, and then minimizes the DPO loss, starting the reference from the supervised fine-tuned model when one exists.

In TRL, the reference defaults to the model's state before DPO training starts, and beta defaults to 0.1; higher values keep the model closer to the reference. The trainer also exposes variants through `loss_type`, including IPO and a length-normalized loss from the SimPO paper.

## DPO vs RLHF

DPO and RLHF train on the same kind of preference data but optimize differently. RLHF first fits a reward model to the preferences, then uses reinforcement learning, typically PPO, to maximize that reward while the model generates new outputs. DPO skips both steps and updates the model directly from a fixed dataset, which makes it simpler and cheaper to run.

| | RLHF with PPO | DPO |
|---|---|---|
| Reward model | Trained separately on preferences | None; the log-probability ratio is an implicit reward |
| Sampling during training | Yes, the model generates new outputs | No, it trains on a fixed offline dataset |
| Optimization | Reinforcement learning | Classification loss on each pair |

The authors report that DPO exceeded PPO-based RLHF at controlling the sentiment of generations, and matched or improved response quality on summarization and single-turn dialogue, in experiments on models of up to 6 billion parameters. Because DPO learns only from the pairs it receives, audit a sample of pairs before training, whether humans or an [LLM judge](https://theaiengineering.com/glossary/llm-as-a-judge) produced the labels.

[RAG vs fine-tuning](https://theaiengineering.com/articles/rag-vs-fine-tuning) explains when preference tuning, or any change to model weights, is the right lever.

## Related terms

- [RLHF](https://theaiengineering.com/glossary/rlhf): RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.
- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [LoRA](https://theaiengineering.com/glossary/lora): LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.
- [Pretraining](https://theaiengineering.com/glossary/pretraining): Pretraining is the first stage of training a language model, in which it learns to predict the next token across a large unlabeled corpus of text and code, producing a general-purpose base model.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Open-weights model](https://theaiengineering.com/glossary/open-weights-model): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license.

## Guides that cover direct preference optimization

- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is DPO (direct preference optimization) in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/direct-preference-optimization

- Title: What is DPO (direct preference optimization) in LLMs?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/direct-preference-optimization
- Markdown: https://theaiengineering.com/glossary/direct-preference-optimization.md
