Group Relative Policy Optimization (GRPO)

GRPO is PPO without the critic: it scores a group of sampled answers against each other. How it works, what it saves, and where it biases training.

On this page

Definition

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for language models that removes PPO's value network. Instead of training a second model to predict how good a response should be, it samples a group of responses to the same prompt and judges each one against the group's own average.

Introduced in DeepSeekMath in 2024 and made famous by DeepSeek-R1, it is the reason large-scale RL post-training became practical outside frontier labs. The core observation is almost embarrassing in hindsight: the critic exists to answer "was this answer better than expected?", and if you sample eight answers to the same question, you can just look.

How It Works

PPO — the algorithm behind classic RLHF — computes an advantage, how much better an action was than expected, using a learned value function. For language models this is the awkward part: the critic is a second network the size of the policy, it must estimate per-token value from a reward that only arrives at the end of the sequence, and it is the component most likely to be the reason a run diverges.

GRPO estimates the same quantity from statistics. For a prompt x, sample G completions, score them all, then normalize each reward against the group:

Â_i = ( r_i − mean(r_1 … r_G) ) / std(r_1 … r_G)

Every token in completion i gets that same advantage. An answer above the group mean is reinforced, one below is discouraged, and the amount is scaled by how spread out the group was. The clipped policy-gradient update and the KL penalty against a frozen reference model are kept from PPO — only the critic is gone.

Now the arithmetic that explains why anyone cares. A PPO run on a 7B model keeps four networks resident: policy, frozen reference, reward model, and critic — about 28B parameters of weights, plus optimizer state for the two being trained. GRPO drops the critic: three networks, ~21B. When the reward is a rule rather than a learned model, the reward network goes too: two networks, ~14B — half of PPO's footprint, and only one of them is being trained.

The cost lands elsewhere. You generate G completions per prompt instead of one, so rollout compute scales roughly linearly with the group size — DeepSeekMath used G = 64, and 8 to 16 is typical for smaller runs. GRPO trades memory for inference. On hardware where generation is cheap and memory is the binding constraint, that is a very good trade, which is exactly the situation most teams are in.

One property falls out of the formula and is easy to miss: if every completion in a group receives the same reward, the standard deviation is zero, the advantage is zero, and the prompt contributes no gradient at all. Questions the model always gets right and questions it always gets wrong both teach it nothing. GRPO learns only at the frontier of what the model can sometimes do.

Real-World Applications

DeepSeek-R1 and R1-Zero. DeepSeek-R1 trained with GRPO against rule-based correctness rewards on maths and code. The R1-Zero variant is the more striking result: starting from a base model with no supervised reasoning data, long chain-of-thought behaviour — self-checking, backtracking, spending more tokens on harder problems — emerged from the reward alone. That is the clearest public evidence that RL post-training adds capability rather than only manners.

DeepSeekMath. The original paper applied GRPO to mathematical reasoning and reported substantial gains on GSM8K and MATH over the supervised baseline, which is what established the method before R1 made it famous.

Tülu 3. AI2's open post-training recipe uses reinforcement learning with verifiable rewards as its final stage, documenting the whole pipeline — the practical reference for reproducing this outside a frontier lab.

Everyday use through TRL. Hugging Face's GRPOTrainer accepts an arbitrary Python function as the reward, which is what put GRPO in reach of teams aligning small models on domain-specific correctness — format compliance, schema validity, passing tests.

Challenges

The normalizations are biased. "Understanding R1-Zero-Like Training: A Critical Perspective" showed that dividing by the group standard deviation and by response length skews optimization toward longer responses, most visibly on questions the model answers incorrectly — the model learns that when it is unsure, writing more is rewarded. Their variant, Dr. GRPO, removes both normalizations. If you have wondered why an RL-trained reasoning model rambles most on the problems it fails, this is a large part of the answer.

Entropy collapse. Policies trained with GRPO tend to lose sampling diversity, converging on one way of answering and losing the exploration that made the group comparison informative in the first place. DAPO documents this at scale and proposes decoupled clipping bounds and dynamic sampling — the practical fix being to drop groups whose rewards are all identical, which contribute nothing anyway.

Reward hacking has not gone away, it has moved. With a learned reward model the failure is flattery; with a rule-based verifier it is gaming the checker — matching the answer format without the reasoning, exploiting a lenient regex, or in code tasks writing something that passes the visible tests. The reward is harder to flatter but not impossible to game.

Group size is a real cost, not a hyperparameter to max out. Generation dominates the compute budget of a GRPO run. Doubling G doubles rollout time to reduce variance in an estimate that is already reasonable at 8–16, which is a poor exchange in most settings.

It needs a scoreable task. GRPO's advantage estimate requires meaningful variance in reward within a group. For open-ended generation with a noisy learned reward, groups often come back nearly tied and the signal is thin — which is why GRPO's successes are concentrated in maths, code and other domains with a crisp notion of correct.

Future Trends

Verifiable rewards keep expanding. The boundary is moving from maths and code toward tool use, agentic task completion and anything with a testable end state. GRPO is the algorithm that made this affordable, so its reach grows with the set of tasks a program can grade.

Critic-free is becoming the default. GRPO, RLOO and their relatives all reach the same conclusion by different routes: for language models the learned value function costs more than it returns. DPO arrives at the same place from the other direction, deleting the critic and the reward model together. Expect the variants to keep multiplying and the critic not to come back.

Debiased variants consolidate. Dr. GRPO and DAPO fix distinct problems in the original formulation, and the corrections are cheap enough that they will likely be folded into default implementations rather than remaining separate methods.

Code Example

The advantage computation is the entire idea, and it is six lines:

import torch

def group_advantages(rewards, normalize_std=True):
    """rewards: shape (G,) — one scalar per sampled completion of the same prompt."""
    centered = rewards - rewards.mean()
    if normalize_std:                       # Dr. GRPO drops this division
        centered = centered / (rewards.std() + 1e-8)
    return centered                         # broadcast to every token of its completion

print(group_advantages(torch.tensor([1., 1., 0., 0., 1., 0., 1., 0.])))
# tensor([ 0.93,  0.93, -0.93, -0.93,  0.93, -0.93,  0.93, -0.93])

print(group_advantages(torch.tensor([1., 1., 1., 1., 1., 1., 1., 1.])))
# tensor([0., 0., 0., 0., 0., 0., 0., 0.])   <- all correct: no gradient, prompt wasted

The second case is the one to internalize: a prompt the model already solves every time contributes nothing. Curriculum matters more in GRPO than in most RL setups.

A complete run with a verifiable reward is short:

from trl import GRPOConfig, GRPOTrainer

def reward_correct(completions, answers, **kwargs):
    """Any Python function can be the reward — no reward model needed."""
    return [1.0 if a in c else 0.0 for c, a in zip(completions, answers)]

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_correct,
    args=GRPOConfig(
        output_dir="grpo-out",
        num_generations=8,   # G: group size — rollout cost scales with this
        beta=0.04,           # KL coefficient against the frozen reference model
    ),
    train_dataset=train_dataset,
)
trainer.train()

Academic Sources

Frequently Asked Questions

GRPO is a reinforcement learning algorithm that removes PPO's value network. Instead of training a second model to predict how good a response should be, it samples a group of answers to the same prompt and judges each one against the group's own average. If an answer beats the group average it is reinforced; if it falls below, it is discouraged.
PPO needs a critic — a separate network the size of the policy that estimates expected reward. GRPO replaces it with the mean and standard deviation of rewards within a sampled group. That drops a PPO run from four resident networks to three, or to two when the reward is a rule rather than a model, but requires generating several completions per prompt instead of one.
No. GRPO needs a reward, which can come from anything — a trained reward model, a unit test, a maths answer checker, or a regex. Its best-known uses pair it with rule-based verifiable rewards, which is what made DeepSeek-R1's training possible without a learned scorer at all.
Somewhere between 8 and 64. The group's mean and standard deviation are estimated from those samples, so too few makes the advantage noisy while too many multiplies generation cost linearly. DeepSeekMath used 64; 8 is a common starting point for smaller experiments.
In memory, clearly — it deletes a network the size of the policy and the trickiest part of PPO to tune. In compute, it depends: you now generate G completions per prompt rather than one, so rollout cost scales with the group size. GRPO moves cost from memory to inference, which is a good trade on modern hardware but not a free lunch.
Dividing the advantage by the group's standard deviation and by response length introduces biases. Liu et al. (2025) showed these push the model toward longer responses, particularly on questions it gets wrong, and proposed removing both normalizations — a variant known as Dr. GRPO.

Related Terms

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Direct Preference Optimization (DPO)

DPO trains a model on human preferences without a reward model or reinforcement learning: the derivation, the memory it saves, and where it loses to PPO.

Reinforcement Learning (RL)

A learning paradigm where agents learn to make decisions by interacting with an environment to maximize cumulative reward.

Policy

A strategy, rule, or function that guides decision-making processes, commonly used in reinforcement learning to determine agent actions.

Chain-of-Thought (CoT)

A model writing out its reasoning before answering. Once a prompting trick, now trained in by RL, billed as output tokens, and frequently unfaithful.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary