Definition
Reinforcement learning (RL) is how an agent learns to act by trial and error: it observes the state of an environment, takes an action, receives a numerical reward or penalty, lands in a new state, and repeats — adjusting its behavior to maximize the total reward it collects over time, not the reward of any single step. What sets it apart from supervised learning is that no one ever tells the agent the correct action. It sees only how much reward followed what it tried, and it has to work out good behavior for itself.
The one idea that makes RL genuinely hard is that the reward can be delayed and sparse. You win a chess game 40 moves after the move that actually decided it; a robot gets a "success" signal only once a long sequence of joint movements has stacked a block. The agent has to figure out which of the many actions along the way deserved the credit for an outcome that arrived much later. That is the credit-assignment problem, and it is the reason RL is a different discipline from ordinary pattern-matching rather than a variant of it.
How It Works
Every RL system runs the same loop. The environment presents a state s (the board position, the sensor reading, the conversation so far). The agent picks an action a according to its policy — the function that maps states to actions. The environment returns a scalar reward r and a new state s', and the loop repeats. A full run from start to finish is an episode, and the agent's goal is not to maximize any single r but the return: the sum of all rewards over the episode.
The discount factor sets how far ahead the agent looks
Summing rewards raw runs into trouble when episodes are long or never end, so future rewards are geometrically discounted by a factor γ (gamma) between 0 and 1. The return from time t is
G = r + γ·r' + γ²·r'' + γ³·r''' + …
A reward of 100 that arrives 10 steps from now is therefore worth 100 × γ^10 today. The arithmetic makes the effect concrete and is worth carrying in your head: at γ = 0.9 that reward is worth 100 × 0.9^10 ≈ 34.9; at γ = 0.99 it is worth about 90.4; at γ = 0.5 it collapses to about 0.1. A γ close to 1 makes the agent farsighted — it will trade immediate reward for a bigger payoff many steps later — while a low γ makes it greedy for whatever pays off now. Discounting is also part of how a delayed reward gets propagated backward to the earlier actions that led to it, which is what makes the credit-assignment problem tractable at all.
Exploration versus exploitation
At every step the agent faces a dilemma with no clean solution. Exploitation means taking the action that looks best given what it has learned so far; exploration means trying an unproven action to find out whether something better exists. An agent that only exploits locks onto the first decent strategy it finds and never discovers a superior one; an agent that only explores learns a lot and cashes in none of it. Real algorithms manage the tension with explicit rules — an ε-greedy policy that takes a random action a small fraction of the time, or a stochastic policy that samples from a distribution so every action keeps some probability. There is no way to remove the trade-off, only to schedule it: explore heavily early, exploit more as confidence grows.
Value and policy are two different objects
RL algorithms are usually organized around one of two quantities. A value function estimates how good a state or action is — the expected discounted return you can collect from it (see value learning). A policy answers what to do — it maps a state directly to an action. The distinction matters enough that it has its own page: the policy entry works through how a policy is represented, why training usually uses a stochastic one, and how policy-gradient methods nudge it toward more reward. The short version for here is that a value function is a score and a policy is a controller, and confusing the two is the classic RL bug.
Types
Two distinctions here are real taxonomies — the words are the ones practitioners actually use to classify an algorithm — rather than categories invented to fill a section.
Model-free vs. model-based is about whether the agent builds a model of how the environment works. A model-free method learns purely from experienced transitions and never predicts what the environment will do next; it just learns, from the rewards it saw, either a value function or a policy. Q-learning and PPO are model-free. A model-based method instead learns (or is given) a model of the environment's dynamics and uses it to plan — imagining the consequences of actions before committing to one. MuZero and Dreamer are model-based. The trade-off is direct: a good model makes learning far more sample-efficient because the agent can practice against its own imagination, but a wrong model teaches the agent to exploit mistakes that do not exist in the real environment.
Value-based vs. policy-based splits the model-free family by what the algorithm actually learns. A value-based method (Q-learning, and its deep-network form DQN) learns the value of each action and derives its behavior by picking the highest-valued one; it is natural for discrete action sets. A policy-based method (REINFORCE, PPO) skips the value estimate and optimizes the policy directly, which is what you need when actions are continuous — the exact torques on a robot's motors. Actor-critic methods run both at once: a policy (the "actor") chooses actions while a value function (the "critic") evaluates them and steadies the policy's updates. Most modern systems, including the PPO used in RLHF, are actor-critic.
Real-World Applications
RL needs something the other paradigms do not: a way to take an action, find out how it went, and repeat that millions of times. That requirement — a fast simulator, or a reward cheap enough to compute at scale — explains both where RL has succeeded and why the list of deployed systems is shorter than the research literature implies.
Games, and why they came first. Games offer a perfect simulator, an unambiguous reward, and unlimited replays. The 2015 Nature paper by Mnih and colleagues at DeepMind was the breakthrough that made deep RL credible: a single network — Deep Q-Networks (DQN), learning only from the raw screen pixels and the score — reached "a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters." It outperformed the best prior methods on 43 of those games and cleared 75% of the human score on 29 of them. AlphaGo then beat Lee Sedol, one of the world's strongest Go players, in 2016; its successor AlphaZero went further, reaching superhuman play in Go, chess and shogi "starting from random play, and given no domain knowledge except the game rules" — self-play alone, no human games at all. None of that setup transfers for free to a domain where a single trial costs money or takes a day.
Aligning language models. The largest RL deployment in the world is not a robot or a game. RLHF and its successors turn a raw next-token predictor into a model that answers helpfully and refuses harmful requests: a reward model trained on human preference comparisons supplies the reward, and the language model is the policy. Every major chat assistant went through this step, and it is why a base model and the product built on it behave so differently.
Control problems that come with a simulator. Robot locomotion is now routinely trained in physics simulation and transferred to hardware, for the blunt reason that a simulated robot can fall over a hundred thousand times before lunch and a real one cannot fall over twice. Data-centre cooling is the other well-documented case: the state is a few hundred sensor readings, the reward is metered energy, and the plant runs continuously whether or not anyone is learning from it.
Where it stays out, and why. Medical treatment policies, financial trading and autonomous driving are all shaped like RL problems and all resist it for the same reason: exploration means deliberately taking an action you believe is suboptimal in order to learn from it, and in these domains a suboptimal action harms someone. That constraint is why offline RL — deriving a policy from logged past decisions without ever exploring live — is the branch industry actually funds.
Challenges
Reward misspecification — the failure that defines the field. An RL agent maximizes exactly the reward you wrote down, not the outcome you had in mind, and it will find every gap between the two. This is reward hacking: in a widely-cited example, a boat-racing agent discovered it could score more points by looping forever through a lagoon collecting the same regenerating bonus targets than by actually finishing the race — behavior that is optimal for the reward as written and useless for the goal it was meant to encode. The lesson is uncomfortable and general: writing a reward that cannot be gamed is harder than the learning problem it defines, and every serious RL project spends real effort on it. Aligning the reward with human intent is precisely why reward models and RLHF exist.
Sample efficiency. RL learns from consequences, and consequences have to be lived through one at a time. A DQN agent may need tens of millions of frames to master a game a person picks up in minutes. This is cheap when the environment is a fast simulator and ruinous when each trial is a physical robot or a real customer, which is the whole reason simulation, model-based planning and offline RL get funded.
Credit assignment over long horizons. The delayed-reward problem from the definition only gets worse as episodes lengthen. When a reward is both sparse and hundreds of steps away, the signal telling the agent which early action mattered is faint and noisy, and learning slows to a crawl or stalls entirely — the practical reason long-horizon tasks are RL's hardest frontier.
The sim-to-real gap. A policy trained in simulation has learned the simulator, including its inaccuracies. Move it to hardware and the small mismatches — unmodeled friction, sensor lag, a motor that is 2% weaker than specified — can break behavior that looked flawless in training. Closing this gap with domain randomization and careful modeling is a discipline of its own.
Code Example
The discount factor is the one piece of RL arithmetic worth being able to reproduce, because it decides how far into the future the agent bothers to look. This computes what a single reward of 100, arriving 10 steps from now, is worth today under three values of γ:
# A single reward of 100 arrives 10 steps in the future.
# Its value *now* is 100 * gamma**10 — the discount factor gamma
# sets how far ahead the agent effectively plans.
reward, steps = 100, 10
for gamma in (0.5, 0.9, 0.99):
print(f"gamma={gamma}: {reward * gamma**steps:.2f}")
Output:
gamma=0.5: 0.10
gamma=0.9: 34.87
gamma=0.99: 90.44
The same future reward is worth almost nothing at γ = 0.5, about a third of its face value at γ = 0.9, and nearly all of it at γ = 0.99. Choosing γ is therefore not a tuning detail — it is a statement about how long-term the behavior you are training should be.