GRPO: group-relative advantage, no critic
PPO's value model exists to reduce the variance of a single rollout's return estimate. GRPO deletes it: for each prompt, sample a group of G responses, then use the group's own reward statistics as the baseline. Advantages become a simple normalization inside the group.
Group Baseline
For each prompt, sample G completions, compute rewards, and set A_i = (r_i − mean) / (std + ε). The group mean replaces the learned value function.
What You Save
No critic model in memory, no value-function training instability, no GAE hyperparameters. The memory bill drops by one full model.
What You Pay
G× more rollouts per prompt per step. Compute moves from parameters (critic) to sampling (rollouts) — usually a good trade for reasoning workloads.
Deleting a 7B critic saves ~112 GB of training state — often the difference between a feasible and infeasible reasoning-RL run.
- DeepSeekMath introduced GRPO; DeepSeek-R1 scaled it to emergent long reasoning with groups of 8–64 samples.
- Watch group reward variance: when every sample in a group is right (or wrong), advantages are ~0 and the batch contributes nothing — curriculum on prompt difficulty matters.