Skip to content

Evidence Ledger

Every major claim made anywhere in this repo must carry one tag:

Tag Meaning
PAPER Explicitly stated in arXiv:2606.19348v1 (section noted)
DERIVED Mathematically derived from paper-stated facts
IMPLEMENTED Reproduced in src/deepseek_v4_lab (test-linked)
MEASURED Experimentally observed in experiments/ (artifact-linked)
INFERRED Reasoned interpretation, not established
UNKNOWN Paper does not establish; we don't either (yet)

Rule: never turn "the paper reports" into "our experiment proves."

Ledger

Scale & headline efficiency

  • PAPER V4-Pro 1.6T/49B, V4-Flash 284B/13B, both 1M-token context (Abstract, Section 4.2.1)
  • PAPER At 1M ctx: Pro = 27% FLOPs & 10% KV vs V3.2; Flash = 10% & 7% (Section 1)
  • PAPER KV ≈ 2% of BF16-GQA8(head 128) baseline at 1M ctx (Section 2.3.4)

mHC

  • PAPER Update: X_{l+1} = B_l X_l + C_l F_l(A_l X_l); X ∈ R^{n_hc×d} (Section 2.2, Eq 1)
  • PAPER B_l constrained to Birkhoff polytope (doubly stochastic); ⇒ ‖B_l‖₂ ≤ 1 ⇒ non-expansive; ℳ closed under multiplication (Section 2.2, Eq 2)
  • PAPER Projection via Sinkhorn-Knopp: M⁰=exp(B̃), alternate row/col normalize, t_max=20 (Section 2.2, Eq 8)
  • PAPER A=σ(Ã), C=2σ(C̃), non-negative bounded, avoids signal cancellation (Section 2.2, Eqs 6–7)
  • PAPER Dynamic params: RMSNorm(vec(X)) → W·x̂ + static bias S, gated by small-init α (Section 2.2, Eqs 3–5)
  • DERIVED Doubly stochastic ⇒ rows/cols sum to 1 ⇒ spectral norm ≤ 1 follows from Gershgorin/√(mn)-bound on non-negative matrices
  • MEASURED (planned Phase 5) unconstrained HC destabilizes at depth on toy stacks

CSA

  • PAPER Dual series C^a,C^b = H·W^{aKV,bKV}; weights Z^a,Z^b = H·W^{aZ,bZ} (Section 2.3.1, Eqs 9–10)
  • PAPER Entry i mixes 2m raw entries (own block via a-series, preceding block via b-series) softmax-normalized with learnable positional biases; overlapping windows ⇒ net 1/m compression (Eqs 11–12)
  • PAPER Lightning indexer: low-rank q (c^Q=h·W^DQ; q^I=c^Q·W^IUQ), per-head weights w=h·W^w, score I_{t,s}=Σ_h w_h·ReLU(q^I_h·K^IComp_s), top-k selects compressed blocks (Eqs 13–17)
  • PAPER Query attends only to blocks s < ⌊t/m⌋ (strict causality) (Section 2.3.1)
  • PAPER Core attention is MQA: compressed entry serves as both K and V (Eqs 18–19)
  • PAPER Grouped output projection: n_h heads → g groups → d_g intermediates → d (Section 2.3.1)
  • PAPER Indexer computed in FP4 during deployment; QK path FP4 under QAT (Section 2.3.4, Section 5.2.1)
  • PAPER FP4 indexer + BF16 index scores: 2× top-k selector speedup, 99.7% recall (Section 5.2.1)

HCA

  • PAPER Same machinery, m′ ≫ m (128 in both models), no overlap, dense attention over all compressed entries (Section 2.3.2, Eqs 20–26)

Shared attention details

  • PAPER RMSNorm on each query head and on compressed KV entries just before core attention, prevents exploding logits (Section 2.3.3)
  • PAPER Partial RoPE on last 64 dims of q/KV; outputs additionally rotated by position −i so they carry relative positions (Section 2.3.3)
  • PAPER Sliding-window branch: n_win=128 uncompressed recent KV entries concatenated into core attention (Section 2.3.3)
  • PAPER Attention sink: learnable z′_h adds exp(z′_h) to softmax denominator (Eq 27)
  • PAPER Mixed KV storage: BF16 for RoPE dims + FP8 elsewhere ⇒ ~half cache size (Section 2.3.4)

Muon

  • PAPER Algorithm 1: momentum buffer, Nesterov trick (μM_t+G_t orthogonalized), update O = NS(M)·sqrt(max(n,m))·γ; decoupled wd: W ← W(1−ηλ) − ηO
  • PAPER Hybrid Newton-Schulz: M₀=M/‖M‖_F; 8 iters (a,b,c)=(3.4445,−4.7750,2.0315) then 2 iters (2,−1.5,0.5) (Section 2.4)
  • PAPER AdamW retained for embeddings, prediction head, RMSNorm weights, mHC static biases/gates (Section 2.4, Section 4.2.2)
  • PAPER Momentum 0.95, wd 0.1, update RMS rescaled to 0.18; no QK-Clip needed because q/KV are RMSNorm'd (Section 2.4, Section 4.2.2)
  • PAPER NS iterations stable in BF16 matmuls (Section 3.4.1)

MoE

  • PAPER DeepSeekMoE retained; affinity switched Sigmoid → Sqrt(Softplus(·)); aux-loss-free balancing + sequence-wise balance loss weight 1e-4; bias update speed 1e-3; hash routing in first 3 MoE layers (Section 2.1, Section 4.2.2)
  • UNKNOWN Why hash routing for early layers (not explained)

Infrastructure

  • PAPER Per-layer: total comm time < compute time ⇒ full overlap leaves compute-bound (Section 3.1)
  • PAPER Overlap condition: C/B ≤ V_comp/V_comm; for V4-Pro 6hd FLOPs vs 3h bytes ⇒ C/B ≤ 2d = 6144 FLOPs/Byte (Section 3.1)
  • PAPER Wave-based expert scheduling; 1.50–1.73× general, 1.96× RL-rollout speedup; MegaMoE open-sourced (Section 3.1)
  • PAPER Contextual parallelism two-stage: ship last m entries to next rank, local compress to fixed s/m+1, all-gather, fused select-and-pad (Section 3.4.3)
  • PAPER KV cache: state cache (SWA + not-yet-compressible tails, fixed-size per request) + classical blocks covering lcm(m,m′) tokens ⇒ k₁=lcm/m CSA, k₂=lcm/m′ HCA entries/block (Section 3.5.1)
  • PAPER On-disk: compressed entries persisted; prefix reused up to last complete block; SWA restore via full-caching / periodic-checkpoint(p) / zero-caching (recompute last n_win·L) (Section 3.5.2)
  • PAPER Batch invariance: no split-KV; dual-kernel decoding with identical accumulation order; DeepGEMM instead of cuBLAS (Section 3.3)
  • PAPER Determinism: SM-local grad buffers + deterministic global reduce (attn bwd); token-order preprocessing + rank-buffer isolation (MoE bwd) (Section 3.3)
  • PAPER mHC engineering: fused kernels + selective recomputation ⇒ 6.7% wall-time overhead on 1F1B pipeline (Section 3.4.2)
  • PAPER Tensor-level activation checkpointing via TorchFX-traced minimal recomputation subgraphs (Section 3.4.4)
  • PAPER Muon ZeRO: knapsack buckets ≤5 matrices/rank (<10% padding); MoE grads synced in BF16 w/ stochastic rounding via all-to-all + FP32 local sums (Section 3.4.1)

Training

  • PAPER 32T (Flash) / 33T (Pro) tokens; seq len staged 4K→16K→64K→1M; dense warmup 1T tokens, sparse attention introduced at 64K stage (Section 4.2.2)
  • PAPER LR: Flash peak 2.7e-4 → 2.7e-5 cosine; batch → 75.5M tokens; sample-level attention masking (Section 4.2.2, Section 4.1)
  • PAPER Instability: spikes tied to MoE outliers; mitigated by anticipatory routing (indices from θ_{t−Δt}; ~20% overhead, auto-triggered post-spike) and SwiGLU clamping (linear ∈ [−10,10], gate cap 10) (Section 4.2.3)
  • PAPER MTP loss weight 0.3 → 0.1 at LR decay (Section 4.2.2)

Post-training

  • PAPER Pipeline: domain specialists (SFT → GRPO RL) then unified model via multi-teacher OPD; mixed RL entirely replaced (Section 5.1)
  • PAPER OPD objective: L = Σᵢ wᵢ·KL(π_θ ∥ π_Eᵢ) reverse-KL on student-generated trajectories (Eq 29)
  • PAPER Full-vocabulary logit distillation chosen over token-level KL estimates (lower variance, stable gradients) (Section 5.1.2)
  • PAPER Teacher scheduling: offloaded weights, cached last-layer hidden states, logits reconstructed via head on-the-fly, samples ordered by teacher index (one head in memory) (Section 5.2.2)
  • PAPER Three reasoning-effort modes via distinct RL length penalties/context windows/ formats (Section 5.1.1)
  • PAPER FP4 QAT: MXFP4(E2M1) 1×32 tiles; FP32 master → FP4 → lossless dequant to FP8 compute; STE backward; native FP4 at rollout (Section 5.2.1)

Agents

  • PAPER Rollout service: token-granular WAL + saved KV cache; regenerating from scratch is mathematically incorrect (length bias toward short sequences) (Section 5.2.3)
  • PAPER DSec: Apiserver/Edge/Watcher; 4 substrates (function-call pool, container, Firecracker microVM, QEMU fullVM) behind one SDK; layered EROFS/overlaybd image loading; globally ordered trajectory logs enabling fast-forward, provenance, deterministic replay (Section 5.2.5)

Open questions we may resolve experimentally (toy scale)

  1. Does unconstrained HC actually destabilize small deep stacks? (Phase 5)
  2. How does retrieval recall degrade with m′ in HCA at toy scale? (Phase 3)
  3. Where is the exposed/hidden communication crossover for toy expert counts? (Phase 6)