Term tag: AI alignment

AI Safety

AI safety is three fields, not one: present-day reliability, alignment, and catastrophic risk. What each actually claims, and why they get confused.

Artificial Superintelligence (ASI)

Artificial superintelligence: what it would mean, why the intelligence explosion argument is contested, and what labs and regulators actually do about it.

Direct Preference Optimization (DPO)

DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.

Group Relative Policy Optimization (GRPO)

GRPO is PPO without the critic: it scores a group of sampled answers against each other. How it works, what it saves, and where it biases training.

Jailbreak (Red-Teaming)

A jailbreak is an input that steers an AI past its safety training to produce content it was built to refuse. How they work, and how they are defended.

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Reinforcement Learning from Verifiable Rewards (RLVR)

RLVR replaces the learned reward model with a program that checks the answer. What that buys, where it stops working, and how it differs from RLHF.

Reward Hacking

When an AI maximises its specified reward but defeats the designer's intent — optimising the proxy, not the goal. Goodhart's law, made mechanical.

Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Supervised Fine-Tuning (SFT)

Supervised fine-tuning trains a pretrained language model on curated prompt-response pairs with next-token loss, teaching it to follow instructions.

Sycophancy

Sycophancy is when an AI model tells you what you want to hear — agreeing, flattering, caving when challenged — because human-preference training rewards it.

Value Learning

The AI-alignment problem of teaching a system to infer and act on human values, so a capable optimizer does what people want, not a misspecified proxy.