Course 2: Pre-Training & Long Context¶
Paper coverage: Section 4 Lab: million-token economics · exp05
Data construction [Section 4.1]¶
On top of V3's corpus: stronger filtering of templated/auto-generated web content (anti model-collapse), enlarged math/code cores, bigger multilingual coverage for long-tail culture knowledge, and a deliberate shift toward long documents (papers, technical reports). The data regime must contain contexts worth compressing. Tokenizer stays at 128K vocab with new context-construction special tokens; token-splitting and FIM inherited from V3; documents packed to minimize truncation; sample-level attention masking replaces document masking.
Totals: 32T tokens (Flash), 33T (Pro) [PAPER Section 4.2.2].
Training setups [Section 4.2.2]¶
| knob | Flash | Pro |
|---|---|---|
| optimizer | Muon (matrices) + AdamW (embeddings/head/RMSNorm) | same |
| Muon | momentum .95 · wd .1 · update RMS 0.18 | same |
| AdamW | β=(0.9, 0.95) · ε=1e-20 · wd 0.1 | same |
| peak → end LR | 2.7e-4 → 2.7e-5 (cosine) | 2.0e-4 → 2.0e-5 |
| max batch | 75.5M tokens | 94.4M tokens |
The long-context curriculum is the section's core lesson:
seq length: 4K ──▶ 16K ──▶ 64K ──▶ 1M
attention: dense (first 1T tokens) ──▶ sparse top-k from the 64K stage
Sparsity is introduced gradually: a short indexer-warmup phase precedes full sparse training. Our exp02 independently rediscovered why: a densely-trained indexer has zero selection skill; the STE sparse phase is what teaches it. Curriculum is load-bearing [MEASURED].
Balancing knobs: router-bias speed 1e-3, sequence-wise balance loss weight 1e-4, MTP weight 0.3 → 0.1 at LR decay.
Mitigating instability [Section 4.2.3]¶
Trillion-param MoE training hit loss spikes tied to MoE outliers, with routing amplifying them. Two empirical fixes (mechanism still not understood; the paper says so):
- Anticipatory Routing: routing indices computed from θ_{t−Δt} instead of θₜ, decoupling backbone and router updates. ~20% wall-clock overhead when active; auto-enabled after spike detection, auto-disabled later.
- SwiGLU Clamping: linear component clamped to [−10, 10]; gate component capped at 10.
Evaluations [Section 4.3]¶
Base models compared against V3.2-Base across world knowledge / reasoning / coding-math / long context. Gap ≤ 0.3 counts as tie. V4-Flash-Base surpasses V3.2-Base on most benchmarks despite fewer activated parameters; V4-Pro-Base leads across all four axes.