---
title: "What is RLHF? Reinforcement learning from human feedback"
description: "RLHF (reinforcement learning from human feedback) trains LLMs on human preference rankings via a reward model. How RLHF works, and RLHF vs DPO and RLVR."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/rlhf"
markdown_url: "https://theaiengineering.com/glossary/rlhf.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["RLHF","reinforcement learning from human feedback","preference tuning"]
citation: "The AI Engineering Editors. \"What is RLHF? Reinforcement learning from human feedback.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rlhf"
---

# What is RLHF? Reinforcement learning from human feedback

Also called reinforcement learning from human feedback, preference tuning.

> **Definition of RLHF:** RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.

RLHF, or reinforcement learning from human feedback, is the post-training technique that turned pretrained [large language models](https://theaiengineering.com/glossary/large-language-model) into instruction-following assistants. People compare model responses, a reward model learns to predict their preferences, and the language model is optimized to produce responses the reward model scores highly. The method builds on [Christiano et al. (2017)](https://arxiv.org/abs/1706.03741), and OpenAI applied it to GPT-3 in the [InstructGPT paper](https://arxiv.org/abs/2203.02155) (Ouyang et al., 2022).

## How RLHF works

InstructGPT describes three stages:

**Figure 1: The three stages of RLHF as described in InstructGPT.**

Three stages left to right: supervised fine-tuning on labeler-written answers, a reward model trained on labeler rankings, then PPO reinforcement learning against that reward.

Components:

- **1. SFT**: Labelers write ideal answers
- **2. Reward model**: Labelers rank several outputs
- **3. RL (PPO)**: Model samples, reward model scores

Connections:

- 1. SFT → 2. Reward model
- 2. Reward model → 3. RL (PPO)

First comes supervised [fine-tuning](https://theaiengineering.com/glossary/fine-tuning) on human-written demonstrations. Second, a reward model is trained on human rankings of several outputs for the same prompt. Third, reinforcement learning (PPO in InstructGPT) optimizes the model against that reward, with a penalty for drifting too far from the supervised model. The headline result: human evaluators preferred outputs from the 1.3B-parameter InstructGPT model over outputs from the 175B-parameter GPT-3, despite the 100-fold difference in size.

## RLHF vs DPO

[Direct Preference Optimization](https://theaiengineering.com/glossary/direct-preference-optimization) ([Rafailov et al., 2023](https://arxiv.org/abs/2305.18290)) uses the same kind of preference data but drops the separate reward model and the reinforcement learning loop. DPO optimizes the model directly with a simple classification loss, which makes training simpler and more stable. The authors report that DPO matches or beats PPO-based RLHF on sentiment control, summarization, and single-turn dialogue.

## RLHF vs RLVR

Reinforcement learning with verifiable rewards (RLVR), a name introduced in the [Tülu 3 paper](https://arxiv.org/abs/2411.15124) (2024), replaces the learned reward model with a programmatic check, such as whether a math answer is correct or whether code passes its tests. Verifiable rewards are central to training [reasoning models](https://theaiengineering.com/glossary/reasoning-model): DeepSeek-R1 used rule-based rewards for answer accuracy and output format. RLHF remains the method for qualities with no ground truth, such as helpfulness and tone.

A related variant, RLAIF (reinforcement learning from AI feedback), named in Anthropic's 2022 [Constitutional AI paper](https://arxiv.org/abs/2212.08073), uses model-generated preference labels in place of human ones, the same idea behind [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) evaluation. For how reinforcement learning shapes current reasoning models, see the [reasoning models explainer](https://theaiengineering.com/articles/reasoning-models).

## Related terms

- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [LoRA](https://theaiengineering.com/glossary/lora): LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.
- [Direct preference optimization](https://theaiengineering.com/glossary/direct-preference-optimization): Direct preference optimization (DPO) is a fine-tuning method that trains a language model on pairs of preferred and rejected responses with a simple classification loss, without the reward model and reinforcement learning that RLHF needs.

## Guides that cover rlhf

- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is RLHF? Reinforcement learning from human feedback." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rlhf

- Title: What is RLHF? Reinforcement learning from human feedback
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/rlhf
- Markdown: https://theaiengineering.com/glossary/rlhf.md
