Network Incast, Microbursts & Packet Spraying
Network incast occurs when hundreds or thousands of GPU senders simultaneously transmit data packets to a single destination switch port during AllGather or ReduceScatter collectives. This instant microburst overwhelms switch egress buffers, triggering tail latency inflation and packet drops.
The Incast Phenomenon
In synchronized distributed training, all workers complete backward computation at nearly the same millisecond and initiate collective communications simultaneously, generating massive synchronized traffic surges.
Hash Polarization in ECMP
Equal-Cost Multi-Path (ECMP) hashes flow 5-tuples (IP, Port) to select uplinks. Because long-lived AI collective flows share identical tuple structures, multiple heavy flows collide on the same physical link while adjacent links remain completely idle.
Packet Spraying & Adaptive Routing
Adaptive routing switches break large flows into fine-grained packets or flowlets and spray them across all available parallel links based on real-time link queue depths, reordering packets in hardware at the receiving NIC.
When k = 64 nodes transmit at 400 Gbps into a single 400 Gbps egress port, buffer queue depth expands at 63 * 50 GB/s = 3.15 TB/s. A standard 32 MB switch buffer overflows in ~10 microseconds unless flow control kicks in.
- Enable Adaptive Routing (AR) in InfiniBand switches or Dynamic Load Balancing (DLB) on RoCE fabrics to eliminate ECMP hash polarization.
- Use NCCL_BUFFSIZE=4194304 (4 MB) or 8 MB to optimize DMA ring buffers for high-bandwidth burst absorbing.
- Implement staggered collective initiation or priority queueing to smooth incast peaks during MoE (Mixture of Experts) all-to-all dispatches.