Collectives: Bandwidth-Optimal Ring All-Reduce
Synchronizing gradients across thousands of GPUs requires efficient collective communication algorithms. Ring All-Reduce achieves optimal communication efficiency independent of cluster size.
Two-Phase Communication
1. Scatter-Reduce: Each GPU sends a gradient chunk to its logical neighbor in a ring for N-1 steps, computing partial sums. 2. All-Gather: Each GPU passes fully reduced chunks around the ring for N-1 steps.
Bandwidth Optimality
Total data transferred per GPU is precisely 2 * ((N-1)/N) * S bytes, where S is tensor size. As rank count N grows large, data transferred approaches 2S bytes — constant regardless of cluster scale!
Interconnect Dependency
Ring All-Reduce requires high ring interconnect bandwidth (NVLink within node, InfiniBand/RoCE between nodes) to prevent network stragglers.
Proving that per-GPU network transfer volume saturates at 2x tensor size even on 10,000+ GPU nodes.
- NCCL (NVIDIA Collective Communications Library) automatically configures tree or ring topologies based on physical NVLink and InfiniBand detection.