What is RLHF? Reinforcement learning from human feedback

Also called reinforcement learning from human feedback, preference tuning

Definition

RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.

RLHF, or reinforcement learning from human feedback, is the post-training technique that turned pretrained large language models into instruction-following assistants. People compare model responses, a reward model learns to predict their preferences, and the language model is optimized to produce responses the reward model scores highly. The method builds on Christiano et al. (2017), and OpenAI applied it to GPT-3 in the InstructGPT paper (Ouyang et al., 2022).

How RLHF works#

InstructGPT describes three stages:

Three stages left to right: supervised fine-tuning on labeler-written answers, a reward model trained on labeler rankings, then PPO reinforcement learning against that reward. 1. SFTLabelers writeideal answers2. Reward modelLabelers rankseveral outputs3. RL (PPO)Model samples,reward model scores
Figure 1: The three stages of RLHF as described in InstructGPT.

First comes supervised fine-tuning on human-written demonstrations. Second, a reward model is trained on human rankings of several outputs for the same prompt. Third, reinforcement learning (PPO in InstructGPT) optimizes the model against that reward, with a penalty for drifting too far from the supervised model. The headline result: human evaluators preferred outputs from the 1.3B-parameter InstructGPT model over outputs from the 175B-parameter GPT-3, despite the 100-fold difference in size.

RLHF vs DPO#

Direct Preference Optimization (Rafailov et al., 2023) uses the same kind of preference data but drops the separate reward model and the reinforcement learning loop. DPO optimizes the model directly with a simple classification loss, which makes training simpler and more stable. The authors report that DPO matches or beats PPO-based RLHF on sentiment control, summarization, and single-turn dialogue.

RLHF vs RLVR#

Reinforcement learning with verifiable rewards (RLVR), a name introduced in the Tülu 3 paper (2024), replaces the learned reward model with a programmatic check, such as whether a math answer is correct or whether code passes its tests. Verifiable rewards are central to training reasoning models: DeepSeek-R1 used rule-based rewards for answer accuracy and output format. RLHF remains the method for qualities with no ground truth, such as helpfulness and tone.

A related variant, RLAIF (reinforcement learning from AI feedback), named in Anthropic's 2022 Constitutional AI paper, uses model-generated preference labels in place of human ones, the same idea behind LLM-as-a-judge evaluation. For how reinforcement learning shapes current reasoning models, see the reasoning models explainer.

Cite this definition

The AI Engineering Editors. "What is RLHF? Reinforcement learning from human feedback." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rlhf

Title
What is RLHF? Reinforcement learning from human feedback
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM models and benchmarks hub · Updated