On-Policy Distillation: toy pipeline & honest negative result (Exp11)¶
Paper Section 5.1–5.2 · Code: src/deepseek_v4_lab/posttrain/opd.py
Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
Why this exists
Problem: many RL-strengthened domain specialists must consolidate into one deployable model without losing their strengths.
Response: reverse-KL on-policy distillation: students sample trajectories, teachers score full vocabularies, with dedicated teacher scheduling.
New cost: scheduling instability under mode-seeking objectives; teacher/rollout expense; weak specialists give noisy gradients.
Our experiment: Exp11 reproduces the greedy-scheduling collapse, then documents an honest negative result vs matched-budget SFT [MEASURED, incl. negative].
Exchange rate: DISTILLATION buys consolidation of specialists; spends teacher/rollout cost and schedule stability.
Implemented [IMPLEMENTED]¶
- Reverse KL objective
KL(π_θ ∥ π_E)per Eq 29, differentiable, non-negative, zero at identity (unit-tested) - Student-generated rollouts (true on-policy sampling loop)
- Teacher scheduling: per-rollout argmax-likelihood routing AND balanced round-robin variant
- Specialist training on disjoint synthetic domains; union-SFT baseline; matched-budget SFT-continuation control
Toy-scale result. Direction is informative; production magnitude is not.
Measured findings [MEASURED]¶
1. Greedy scheduling collapses under reverse KL¶
argmax-likelihood routing drove teacherA_frac → 100% by step 60: student
rollouts drift toward one domain, that teacher keeps winning, the other
domain decays (loss 3.11 vs baseline 2.08). Round-robin scheduling
eliminates the collapse. This failure mode is a real property of
scheduling + mode-seeking objectives, plausibly part of why the paper
invests in dedicated full-vocab teacher-scheduling infrastructure (Section 5.2.2).
2. Toy-scale consolidation does NOT beat more SFT (negative result)¶
| model | domainA | domainB |
|---|---|---|
| baseline union-SFT | 2.064 | 2.174 |
| +SFT continuation (matched budget) | 2.084 | 2.063 |
| +OPD (balanced, CE anchor 0.5) | 2.256 | 2.156 |
| specialist oracle | 1.821 | 1.820 |
OPD did not beat the matched-budget control. Consistent explanation [INFERRED]: the paper's OPD consolidates >10 RL-strengthened specialists whose advantage over base is large and dense in logit space; our specialists exceed base by only ~0.24 nats of CE, so reverse-KL gradients are weak relative to optimization noise. We did not verify the paper's claim; we verified our implementation mechanics and mapped one genuine failure mode.
What transfers from this phase¶
- Working reverse-KL/scheduling/rollout machinery for future experiments
- A concrete demonstration that consolidation quality depends on specialist strength × schedule stability, the operational content of PAPER Section 5.2.2
Limitations¶
- Two synthetic domains; no reward-model stage; token-level KL not used (we went straight to full-vocab, matching the paper's final choice).