Reinforcement learning from human feedback (RLHF)✎

Reinforcement learning from human feedback (RLHF) is a training technique that steers a language model toward outputs humans find helpful, harmless, and honest. The core insight is that while humans struggle to write ideal responses from scratch, they are fast and reliable at comparing two candidate outputs and saying which is better. RLHF captures this comparative judgment in a learned reward model and then uses reinforcement learning to optimize the main LLM policy against it. The method is central to the post-pretraining alignment stage described in LLM training and pretraining, and it underpins flagship systems including OpenAI's ChatGPT and InstructGPT, Anthropic's Claude, DeepMind's Sparrow, and Google's Gemini.

The approach was originally proposed to create a general algorithm for learning from a practical amount of human feedback — motivated by the observation that many desirable tasks are easy to judge but hard to specify formally. Early RL-from-feedback efforts were either too narrow to generalize or struggled with sparse and noisy reward signals. RLHF addressed both problems by separating preference elicitation (handled by human annotators) from policy optimization (handled by RL algorithms), making the pipeline scalable without requiring massive annotation budgets.

The three-stage pipeline

RLHF as used in modern LLMs (formalized in the InstructGPT paper by Ouyang et al., 2022) proceeds in three stages.

Supervised fine-tuning (SFT). A pretrained base model is fine-tuned on a curated dataset of prompt–response pairs written by human contractors, for a single epoch to avoid overfitting. This shifts the model from raw document-completion behavior toward a helpful-assistant style and produces the starting policy π^SFT.

Reward model training. Human annotators rank multiple model-generated responses to the same prompt. These rankings (often scored via the Elo rating system or pairwise comparison) train a reward model r_θ that replaces the LLM's final layer with a scalar regression head. The model is trained to minimize a cross-entropy loss that measures how well it predicts which of two responses a human would prefer. After training, outputs are normalized so reference completions have a mean score of zero.

RL policy optimization. The SFT model is further trained using a policy gradient method — typically proximal policy optimization (PPO) — to maximize the reward model's scores on newly generated responses. The objective function contains two competing terms: the expected reward E[r_θ(x, y)], which pushes toward higher-rated responses; and a KL-divergence penalty β·log(π^RL / π^SFT), which keeps the policy from straying too far from the SFT model, preventing both mode collapse (repetitive canned responses) and reward hacking. A third term, the PPO-ptx extension from the InstructGPT paper, mixes in the original pretraining objective γ·E[log π^RL(x)] to prevent catastrophic forgetting of general knowledge.

PPO itself uses a clipped surrogate advantage objective to prevent destructively large policy updates, and employs a value estimator V_ξ initialized from the reward model. The advantage for a given (prompt, response) pair is computed as the reward minus the value estimate, and both the policy and value estimator are updated concurrently throughout training.

Collecting human feedback

Human rankings are the fuel of RLHF, and their quality directly caps what the reward model can learn. A small amount of high-quality comparison data can match larger datasets, and increasing data volume tends to be less effective than scaling the reward model itself. However, a larger and more diverse annotator pool becomes critical whenever bias is a concern: if annotators are not representative of intended users, the reward model will encode their particular preferences and penalize outputs those users would actually prefer.

In practice, prompts for reward model training are sampled from a predefined dataset — Anthropic used a chat tool on Amazon Mechanical Turk, while OpenAI used prompts submitted by users to the GPT API — and passed through the initial language model to generate candidate responses. Annotators then rank or compare the outputs rather than assigning raw scalar scores, which are noisier and less calibrated across individuals. Rankings can be converted to scalar reward signals via Elo-style systems or normalized pairwise comparisons. The scale of data used for reward model training is typically around 50,000 labeled preference samples — expensive but not prohibitive. Human-generated text for the supervised fine-tuning stage is considerably more costly, as it requires expert contractors rather than crowdsourced comparison judgments.

An interesting empirical pattern is that successful RLHF systems use reward models noticeably smaller than the policy they supervise: OpenAI's 175B policy was supervised by a 6B reward model; Anthropic used models in the 10B–52B range for both; DeepMind used 70B Chinchilla-scale models for both LM and reward. The intuition is that the reward model needs enough capacity to understand the text it judges, but not necessarily as much as is needed to generate it.

Both offline (batch update on a fixed dataset) and online (immediate policy updates via environment interaction) data collection strategies have been studied formally, with proven sample-complexity bounds. In the online setting, an optimistic maximum likelihood estimator using an upper confidence bound enables sample-efficient learning from pairwise comparisons. Anthropic describes an Iterated Online RLHF approach in which users continue ranking outputs from updated policy versions, with successive iterations of the policy included in a running Elo ranking system — introducing complex co-evolution dynamics between the policy and reward model that remain an active research problem.

A key theoretical complication is the non-Markovian nature of optimal RLHF policies: unlike standard RL, the best response in a conversation often depends on the history of prior actions, making the strategy inherently memory-dependent.

Limitations

RLHF's most fundamental challenge is the quality and consistency of human feedback itself. Annotation is expensive and harder to scale than self-supervised pretraining data. Biased or unrepresentative feedback leads to biased reward models, which in turn produce biased policies. Even with a representative sample, a single reward function cannot reconcile the genuinely conflicting preferences of diverse user groups — it tends to favor the majority view and may disadvantage underrepresented groups.

A related failure mode is reward hacking: a model that learns to maximize reward model scores rather than genuinely improve may learn, for instance, that expressing confident-sounding text — even if incorrect — earns higher ratings. Research has found that humans are poor at detecting errors in LLM outputs on complex tasks, making this a significant deployment risk. Overfitting is also a concern: the model may memorize the quirks of the feedback sample rather than learning to generalize.

Alternatives and successors

Reinforcement learning from AI feedback (RLAIF)

RLAIF replaces human annotators with another AI model that generates preference labels automatically. Anthropic's Constitutional AI uses this approach: an AI evaluates responses for conformance to a set of written principles (a "constitution"), removing the human bottleneck while preserving the RLHF training structure.

Direct alignment algorithms

Direct alignment algorithms (DAAs) bypass the reward-model intermediary entirely, framing alignment as a supervised learning problem directly over human preference data.

Direct preference optimization (DPO) uses a change of variables to show that the optimal RLHF policy can be expressed directly in terms of the SFT model and the preference data, eliminating the separate reward model and RL loop. The resulting loss trains the policy end-to-end on pairwise comparisons with a single sigmoid-based objective. DPO is simpler to implement and has shown comparable or superior results on many benchmarks, though RLHF retains an edge on some truthfulness benchmarks — the relative advantage depends on the task and the structure of the preference data.

Identity preference optimization (IPO) modifies the DPO objective with a quadratic regularization term that controls the gap between the log-likelihood ratios of the fine-tuned and reference models. This reduces overfitting on noisy preference data and does not rely on the Bradley-Terry modeling assumption that pairwise preferences decompose into pointwise reward scores.

Kahneman-Tversky optimization (KTO) draws from behavioral economics (prospect theory) to model human decisions under uncertainty. Unlike DPO and IPO, it requires only a binary signal — "desirable" or "undesirable" — rather than explicit preference pairs, making it more data-efficient for pipelines where like/dislike feedback is easier to collect than ranked comparisons. Its value function is defined piecewise using a KL-divergence baseline, and the β hyperparameter controls how quickly the function saturates.