Course 4d: Post-Training Infrastructure [Section 5.2]

Five systems that make Section 5.1's pipeline runnable at million-token scale.

Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger

FP4 quantization-aware training [Section 5.2.1]

MXFP4 (E2M1) over 1×32 tiles, applied to MoE expert weights and the CSA indexer QK path. Pipeline: FP32 master → quantize FP4 → lossless dequant to FP8 for compute (lossless because E4M3's two extra exponent bits absorb sub-block scale ratios, verified empirically). Backward is STE to master weights; rollouts use native FP4 so training sampling matches deployment exactly.

Index scores quantized FP32→BF16: 2× top-k selector speedup at 99.7% recall [PAPER].

Lab: quant/fake_quant.py implements E2M1 grid + tile scales + STE; exp10 shows QAT-FP4 ≈ fp32-finetune control while PTQ-FP4 never reaches baseline [MEASURED].

Teacher scheduling for full-vocabulary OPD [Section 5.2.2]

Materializing >100k-vocab logits × >10 teachers is prohibitive. Answers: offload teacher weights to distributed storage; cache only last-layer hidden states; rebuild logits through the prediction head on demand; order samples by teacher index so one head is resident; exact KL computed in a fused kernel.

Preemptible rollout service [Section 5.2.3]

Cluster scheduler may preempt anything. Per-request token-granular WAL: every generated token appends immediately; preemption pauses engine + saves KV caches; resume continues decoding; fatal errors re-prefill from WAL.

The correctness argument deserves quoting: regenerating from scratch is mathematically incorrect, because shorter responses survive interruption more often; regeneration biases training toward short outputs.

Lab echo: agents/trajectory.py proves byte-identical recovery after os._exit(137) mid-trajectory, with non-idempotent commands fast-forwarded rather than re-executed [MEASURED exp12].

Million-token RL data path [Section 5.2.4]

Rollout records split into lightweight metadata (global shuffling/packing decisions) + heavy per-token fields (shared-memory loader, released at mini-batch granularity); on-device mini-batch count adapts to trade throughput against I/O overlap.

DSec sandbox [Section 5.2.5]

Rust trio (Apiserver/Edge/Watcher) over 3FS; one Python SDK across four execution substrates (function-call pool, container, Firecracker microVM, QEMU fullVM). Layered images: EROFS read-only layers for containers, overlaybd for microVMs. Every sandbox keeps a globally ordered trajectory log enabling client fast-forward (skip non-idempotent side effects), fine-grained provenance, and deterministic replay; the same properties our toy runner tests.