What is RLHF? Reinforcement learning from human feedback
Also called reinforcement learning from human feedback, preference tuning
Definition
RLHF (reinforcement learning from human feedback) is a fine-tuning method that trains a reward model on human rankings of model outputs, then optimizes the language model with reinforcement learning to score highly.
RLHF, or reinforcement learning from human feedback, is the post-training technique that turned pretrained large language models into instruction-following assistants. People compare model responses, a reward model learns to predict their preferences, and the language model is optimized to produce responses the reward model scores highly. The method builds on Christiano et al. (2017), and OpenAI applied it to GPT-3 in the InstructGPT paper (Ouyang et al., 2022).
How RLHF works#
InstructGPT describes three stages:
First comes supervised fine-tuning on human-written demonstrations. Second, a reward model is trained on human rankings of several outputs for the same prompt. Third, reinforcement learning (PPO in InstructGPT) optimizes the model against that reward, with a penalty for drifting too far from the supervised model. The headline result: human evaluators preferred outputs from the 1.3B-parameter InstructGPT model over outputs from the 175B-parameter GPT-3, despite the 100-fold difference in size.
RLHF vs DPO#
Direct Preference Optimization (Rafailov et al., 2023) uses the same kind of preference data but drops the separate reward model and the reinforcement learning loop. DPO optimizes the model directly with a simple classification loss, which makes training simpler and more stable. The authors report that DPO matches or beats PPO-based RLHF on sentiment control, summarization, and single-turn dialogue.
RLHF vs RLVR#
Reinforcement learning with verifiable rewards (RLVR), a name introduced in the Tülu 3 paper (2024), replaces the learned reward model with a programmatic check, such as whether a math answer is correct or whether code passes its tests. Verifiable rewards are central to training reasoning models: DeepSeek-R1 used rule-based rewards for answer accuracy and output format. RLHF remains the method for qualities with no ground truth, such as helpfulness and tone.
A related variant, RLAIF (reinforcement learning from AI feedback), named in Anthropic's 2022 Constitutional AI paper, uses model-generated preference labels in place of human ones, the same idea behind LLM-as-a-judge evaluation. For how reinforcement learning shapes current reasoning models, see the reasoning models explainer.
Related terms
Guides that cover rlhf
Cite this definition
The AI Engineering Editors. "What is RLHF? Reinforcement learning from human feedback." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rlhf
- Title
- What is RLHF? Reinforcement learning from human feedback
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated