Skip to content

Course 3: Post-Training & On-Policy Distillation

Paper coverage: Section 5.1 specialist training · Section 5.1.2 OPD · reasoning-effort modes Lab: posttrain/opd.py · exp11

The pipeline replaced mixed RL entirely [PAPER Section 5.1]

Base model
   ↓  SFT on domain data          (per domain: math, coding, agent, instruction-following)
Domain specialists
   ↓  GRPO reinforcement learning  (domain reward models)
RL specialists                    ← three reasoning-effort modes each
   ↓  multi-teacher ON-POLICY DISTILLATION
Unified model

More than ten teacher models consolidate into one student. The critical substitution: wherever prior DeepSeek pipelines mixed RL policies, V4 distills instead.

Why reverse KL, why on-policy [PAPER Eq 29]

\[\mathcal{L}_{\text{OPD}}(\theta)=\sum_i w_i\cdot D_{\mathrm{KL}}\big(\pi_\theta \,\|\, \pi_{E_i}\big)\]

Three design choices carry the weight:

  1. Student-generated trajectories: teachers score states the student actually visits (on-policy). Off-policy teacher states would supervise a distribution the student never occupies.
  2. Reverse KL direction \(\mathrm{KL}(\pi_\theta\|\pi_E)\) is mode-seeking: per context the student commits to the relevant expert's behavior instead of averaging all experts into mush.
  3. Full-vocabulary logits (not sampled-token KL estimates): lower-variance gradients; the token-level shortcut caused instability in prior work.

Teacher scheduling at scale [PAPER Section 5.2.2, see infrastructure course]

Weights offloaded to storage; only last-layer hidden states cached; logits rebuilt through the head on demand; samples ordered by teacher index so exactly one head resides in memory.

Reasoning efforts [PAPER Section 5.1.1]

Three modes (Non-think / Think High / Think Max) produced by training separate specialists under different RL length penalties and context windows, integrated via <think> scaffolding. Effort is a trained property, not an inference knob.

The lab found the sharp edge [MEASURED, exp11]

Our toy consolidation (two synthetic domains, matched budgets) reproduced two failure modes worth knowing before touching OPD:

  1. Greedy scheduling collapses. Routing each rollout to its most-likely teacher created a feedback ratchet: student drifts toward domain A → A wins scheduling → B decays (loss 2.08 → 3.11). Balanced round-robin scheduling eliminated the collapse.
  2. Consolidation must clear its overhead. With specialist advantage of only ~0.24 nats, distilled students did not beat matched-budget SFT continuation. OPD's economics require specialists strong enough that logit-level transfer outweighs scheduling noise.

Both findings are consistent with, and explain the necessity of, the paper's heavy teacher-scheduling engineering.