Skip to content

MoE + Expert-Parallel Overlap (Exp08)

Paper Section 2.1, Section 3.1 · Code: src/deepseek_v4_lab/moe/, src/deepseek_v4_lab/dist/

Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger

Why this exists

Problem: MoE dispatch/combine moves every token twice per layer; naively the network stalls the GPUs while mail travels.

Response: wave pipelining that overlaps dispatch, expert compute, and combine, justified by the paper's measurement that intra-layer communication fits inside computation.

New cost: utilization depends on the FLOPs-per-Byte ratio of the interconnect; below crossover, bandwidth dominates everything.

Our experiment: the Exp08 simulator reproduces saturation at the C/B crossover and exposed-comm growth below it [MEASURED, simulator].

Exchange rate: BANDWIDTH buys throughput; spends engineering complexity and a hard utilization ceiling.

Implemented [IMPLEMENTED]

  • DeepSeekMoE: shared experts (always active) + fine-grained routed experts, top-k gating over Sqrt(Softplus(·)) affinities [PAPER Section 2.1 V4 change], auxiliary-loss-free balancing via per-expert bias updates, and a hash router for early-layer emulation [PAPER Section 2.1].
  • Overlap simulator: dispatch/compute/combine volumes per token-expert pair (6·d·d_ff FLOPs vs 3·d bytes [PAPER Section 3.1]); computes exposed vs hidden communication, utilization, speedup, and the full-hiding crossover bandwidth C/B = V_comp/V_comm.

Measured

  • Routing: bias balancing keeps expert load from collapsing onto the initially-dominant expert (utilization history in artifacts) [MEASURED].
  • Crossover at Flash-like dims: 6·2048·2048 / (3·2048·1.5avg-bytes) ≈ 4096 FLOPs/Byte: same structure as the paper's 6144 for Pro's d=7168 [DERIVED; PAPER states the law, we instantiate it].
  • Speedup saturates at 2× as bandwidth crosses the crossover; below it, exposed comm grows linearly [MEASURED via simulator].

Limitations

  • Simulator models steady-state wave pipelining abstractly (no per-wave variance, no memory contention).
  • Utilization tracked for a single MoE layer on a toy task.