POST-TRAINING & DISTILLATION

MODULE 5: REASONING & VERIFIABLE REWARDS · SCENE 14

GRPO: Group-Relative Advantages, No Critic

Sampling a group, normalizing rewards within it, and deleting the value network from the memory bill.

SAMPLE ir = 1.0A = 1.00 ↑ REINFORCECRITIC DELETED · SAVES ~112 GB
GROUP REWARDS (μ = 0.51)1.0+1.00.0-1.01.0+1.01.0+1.00.0-1.00.1-0.81.0+1.00.0-1.0

Advantage is a z-score inside the sampled group: the group mean replaces the learned value function. Compute moves from a critic model to extra rollouts.

ARCHITECTURE & FORMULATIONModule 5: Reasoning & Verifiable Rewards

GRPO: group-relative advantage, no critic

PPO's value model exists to reduce the variance of a single rollout's return estimate. GRPO deletes it: for each prompt, sample a group of G responses, then use the group's own reward statistics as the baseline. Advantages become a simple normalization inside the group.

Group Baseline

For each prompt, sample G completions, compute rewards, and set A_i = (r_i − mean) / (std + ε). The group mean replaces the learned value function.

What You Save

No critic model in memory, no value-function training instability, no GAE hyperparameters. The memory bill drops by one full model.

What You Pay

G× more rollouts per prompt per step. Compute moves from parameters (critic) to sampling (rollouts) — usually a good trade for reasoning workloads.

MATHEMATICAL FORMULATION · GROUP ADVANTAGE & THE MEMORY BILL
A_i = (r_i − μ_group) / (σ_group + ε); saved_GB = critic_params_B × 16

Deleting a 7B critic saves ~112 GB of training state — often the difference between a feasible and infeasible reasoning-RL run.

REAL-WORLD PRODUCTION ENGINEERING
  • DeepSeekMath introduced GRPO; DeepSeek-R1 scaled it to emergent long reasoning with groups of 8–64 samples.
  • Watch group reward variance: when every sample in a group is right (or wrong), advantages are ~0 and the batch contributes nothing — curriculum on prompt difficulty matters.
← 13 · RLVR: Rewards You Can Verify
15 · Chain-of-Thought & Test-Time Compute →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]