Term tag: RLHF

Direct Preference Optimization (DPO)

DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.

Group Relative Policy Optimization (GRPO)

GRPO is PPO without the critic: it scores a group of sampled answers against each other. How it works, what it saves, and where it biases training.

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Reinforcement Learning from Verifiable Rewards (RLVR)

RLVR replaces the learned reward model with a program that checks the answer. What that buys, where it stops working, and how it differs from RLHF.

Reward Hacking

When an AI maximises its specified reward but defeats the designer's intent — optimising the proxy, not the goal. Goodhart's law, made mechanical.

Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Supervised Fine-Tuning (SFT)

Supervised fine-tuning trains a pretrained language model on curated prompt-response pairs with next-token loss, teaching it to follow instructions.

Sycophancy

Sycophancy is when an AI model tells you what you want to hear — agreeing, flattering, caving when challenged — because human-preference training rewards it.

Value Learning

The AI-alignment problem of teaching a system to infer and act on human values, so a capable optimizer does what people want, not a misspecified proxy.