Expert parallelism & Mixture of Experts (MoE)
Mixture of Experts (MoE) replaces dense FFN layers with multiple specialized 'expert' sub-networks, using a gating router to send each token to only top-K experts (e.g., 2 out of 64). Expert Parallelism places different experts on different GPUs, using All-to-All communication to route tokens to their target expert.
Sub-Linear Compute Scaling
A 380B parameter MoE model (like DeepSeek-V3 or Mixtral) might activate only 21B-37B parameters per token, achieving high model capacity with low FLOP cost.
Router Load Balancing
If all tokens pick the same expert, that GPU bottlenecks while others idle. Routers use auxiliary load-balancing losses to distribute tokens evenly.
All-to-All Bottleneck
Expert parallelism shifts the bottleneck to network interconnects, as tokens must be routed to expert GPUs and gathered back.
For a 382B model with 6B shared and 64 experts picking top-2: Active = 6 + 2 × (376 / 64) = 17.75B active parameters per token.
- DeepSeek-V3 uses Auxiliary-loss-free Load Balancing and Multi-head Latent Attention (MLA) to achieve extreme MoE efficiency at scale.
- DeepSpeed-MoE and vLLM optimize MoE kernel execution with custom grouped GEMMs.