Term tag: preference learning

Direct Preference Optimization (DPO)

DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Sycophancy

Sycophancy is when an AI model tells you what you want to hear — agreeing, flattering, caving when challenged — because human-preference training rewards it.

Value Learning

The AI-alignment problem of teaching a system to infer and act on human values, so a capable optimizer does what people want, not a misspecified proxy.