Definition
In reinforcement learning, a policy is the function that maps a state (or observation) to an action — it is the agent's behavior, the thing training optimizes. Written π (the Greek letter "pi"), it takes the situation the agent is in and returns either a single action — a deterministic policy, π(s) = a — or a probability distribution over the available actions — a stochastic policy, π(a|s) — from which the agent samples the move it actually makes.
Everything else in a reinforcement learning system exists to improve this one function. The reward signal, the value estimates, the replay buffers and the gradient updates are all machinery for answering a single question: in this state, what should the agent do? The policy is the answer, and the quality of the whole system is exactly the quality of the policy it produces. That is why "learning" in RL literally means changing the policy until the actions it prescribes earn more reward over time.
How It Works
A policy sits at the centre of the agent–environment loop. At each step the environment hands the agent a state s; the agent consults its policy to pick an action a; the environment responds with a new state and a scalar reward. The policy never sees the reward directly — reward shapes the policy only indirectly, through the training algorithm that adjusts it. This separation matters: the policy is a pure "what to do" map, and the "how good was that" judgement lives in a separate object (the value function, discussed below).
A stochastic policy is a distribution that sums to one
Consider a tiny gridworld where the agent can move up, down, left, or right. A stochastic policy for the current cell is nothing more than four numbers — one probability per action — that must add up to 1.0. For example:
π(up | s) = 0.10 π(down | s) = 0.60 π(left | s) = 0.20 π(right | s) = 0.10
total = 1.00
To act, the agent draws one action from this distribution: down is most likely (60% of the time) but not certain. That residual randomness is the whole reason stochastic policies exist. A deterministic policy that always picks the single best-looking action can never discover that a different action was better — it stops gathering evidence about the alternatives. Because a stochastic policy assigns non-zero probability to every action, it keeps trying them occasionally, and this built-in exploration is what lets the agent escape a locally-good-but-globally-wrong habit. It also makes the policy differentiable in a convenient way, which is what the next section relies on.
Representing the policy: a table, then a network
For a problem small enough to enumerate — a handful of states, a handful of actions — the policy can be a plain lookup table: one row per state, storing either the chosen action or the action probabilities. Tabular policies are exact and easy to reason about, but they do not scale: a chess or Go position, a camera frame, or a paragraph of text has far too many possible states to list.
For those, the policy is a neural network — the "policy network." The state goes in as a vector; the network's output layer produces the action (or, via a softmax, the action probabilities). The network's weights are the policy's parameters, usually written θ, so "improving the policy" becomes "adjusting θ." When people say a large language model is "the policy" in RLHF, this is exactly what they mean: the model maps a state (the prompt so far) to an action (a distribution over the next token), and its billions of weights are θ.
Policy-gradient methods change θ in the direction of more reward
Because a stochastic policy is a differentiable function of θ, you can improve it with gradient ascent. The policy-gradient family (REINFORCE, and the actor-critic methods built on it such as PPO) nudges the parameters in the direction that makes high-return actions more probable, using an update of the form:
θ ← θ + α · G · ∇θ log π(a | s)
where α is a learning rate and G is the return that followed taking action a in state s. Read plainly: if an action was followed by good outcomes (G positive), push θ so that action's probability goes up; if it was followed by bad outcomes, push it down. The ∇θ log π term just says "in whichever direction most efficiently raises that action's probability."
Here is one concrete step. Start from a uniform softmax policy over our four gridworld actions, so each action has probability 0.25 and the four preferences θ are all 0. The agent takes right, and it works out well, giving return G = 1; use learning rate α = 0.5. The gradient of the log-probability raises the taken action's preference by (1 − 0.25) and lowers each of the other three by their probability, so the preferences become θ_right = +0.375 and θ_up = θ_down = θ_left = −0.125. Pushing those back through the softmax gives the new policy:
before: up 0.25 down 0.25 left 0.25 right 0.25 (sum 1.00) after: up 0.215 down 0.215 left 0.215 right 0.355 (sum 1.00)
One rewarded step moved right from 0.25 to about 0.35 and shaved the other three from 0.25 to about 0.215 — and, because it is still a probability distribution, the four numbers still sum to 1.0. Do this millions of times and the policy concentrates probability on the actions that pay off. PPO (Proximal Policy Optimization), the method used to fine-tune most RLHF language models, is this same idea with a safety rail: it clips the update so a single batch cannot move the policy too far from the previous one in a single step, which stops the destructively large jumps that plain policy gradients are prone to.
Types
Two distinctions here are genuine typologies — words practitioners actually use — rather than invented categories.
Deterministic vs. stochastic is about the output of π. A deterministic policy, π(s) = a, commits to one action per state; it is common for the final, deployed policy of a continuous-control robot, where you want repeatable behavior. A stochastic policy, π(a|s), outputs a distribution and samples; it is what you almost always train, because the sampling supplies exploration and makes the policy gradient well-defined. Methods are often named for this: DDPG and TD3 learn deterministic policies, while PPO and SAC learn stochastic ones.
On-policy vs. off-policy is about which policy generated the data you learn from. An on-policy method (REINFORCE, PPO, SARSA) can only improve the policy that collected the current batch of experience; once the policy changes, that data is stale and is thrown away, which is part of why these methods are sample-hungry. An off-policy method (Q-learning, DDPG) can improve a target policy using data gathered by a different behavior policy — for instance, a replay buffer full of experience from older, worse policies — which reuses data far more efficiently at the cost of extra stability engineering.
Real-World Applications
AlphaGo and AlphaZero (DeepMind). These systems pair a policy network — which proposes promising moves given a board position — with a value network that scores positions. The policy network is trained by self-play and reinforcement, and it is the component that actually plays: at inference the search is guided by the move probabilities the policy emits. This is the canonical case of a policy network mapping a huge state space (a board) to an action (a move).
InstructGPT and ChatGPT (OpenAI). After RLHF, the language model is the policy. Its state is the conversation so far; its action is a distribution over the next token; PPO adjusts the model's weights to raise the probability of responses that a separate reward model scores highly. Newer alignment methods reframe the same optimization: Direct Preference Optimization updates the policy directly from preference pairs without a separate reward model in the loop, and Group Relative Policy Optimization estimates the update from a group of sampled responses rather than a learned value baseline — but in every case the object being optimized is still the policy π(next token | prompt).
Dexterous robotic manipulation (OpenAI's Dactyl). A policy trained in simulation controlled a physical robot hand to reorient objects, mapping sensor observations (the state) to motor commands (the action). Training in simulation and transferring the resulting policy to real hardware is a standard pattern precisely because the policy is a self-contained function you can lift out and deploy.
Across all three, the pattern is identical and worth internalizing: whatever the agent is — a Go player, a chatbot, a robot hand — the policy is the part you keep. Everything else was scaffolding used to build it.
Key Concepts
The policy is not the value function — and confusing them is the classic bug
This is the single most important thing to get right. A policy answers what to do: it is a map from states to actions. A value function answers how good is this: it estimates the expected future return of a state (written V(s)) or of a state–action pair (written Q(s, a)). One is a controller; the other is a score. They have different types: a policy returns an action (or a distribution over actions), a value function returns a number.
They are related — an optimal value function implies a greedy policy ("in each state, take the action with the highest Q"), and many algorithms learn both at once — but they are not interchangeable. In an actor-critic method the "actor" is the policy and the "critic" is the value function, and they are literally two separate networks with two separate jobs: the critic evaluates, the actor acts. Treating a value estimate as if it were an action, or expecting a policy to tell you how much reward to expect, is a category error that produces silent, hard-to-debug failures. When you read a paper or a codebase, the first question to settle is always: is this object a policy or a value function?
Exploration lives in the policy
Because the policy is what chooses actions, it is also where exploration is (or is not) built in. A purely greedy deterministic policy explores nothing. Stochastic policies explore by sampling; ε-greedy schemes explore by occasionally overriding the policy with a random action; entropy bonuses explore by explicitly rewarding the policy for keeping its distribution spread out rather than collapsing to a single action too early. All of these are decisions about the policy's behavior, not about the reward or the value function.
Challenges
Telling policy and value apart in practice. As above, the most common conceptual error is conflating the two objects. In actor-critic code the symptom is subtle: gradients that should update the critic get applied to the actor, or an advantage (a value-based quantity) is fed where an action is expected, and the agent trains but never improves. The fix is discipline about types, not more compute.
Policy-gradient variance. The plain REINFORCE update multiplies the log-probability gradient by the raw return G, and returns are noisy, so the update direction jitters from episode to episode. This high variance is the practical reason actor-critic methods exist: subtracting a value-function baseline from G leaves the average update unchanged but shrinks its variance, so training is far less erratic. It is also why a single rewarded step, like the one worked above, should be read as a nudge in a very noisy average rather than a reliable improvement on its own.
On-policy sample cost. Methods like PPO must discard their experience every time the policy changes, so they can require enormous numbers of environment interactions. This is cheap in a fast simulator and ruinously expensive on a physical robot, which is the whole motivation for off-policy methods and for sim-to-real transfer.
Moving the policy too far, too fast. Because the data distribution the policy is trained on is generated by that same policy, a large update can push the policy into a region where its own recently collected data no longer applies, and performance collapses. Preventing exactly this is what PPO's clipping and TRPO's trust region are for — they are not accuracy tricks, they are guardrails on how far a single update is allowed to move the behavior.