Topology-Aware Placement & Switch-Hop Minimization
A distributed job running across 64 nodes can be placed within a single spine switch block or scattered across multiple datacenter pods. Topology-aware schedulers understand the physical network hierarchy and pack communicating ranks to minimize switch hops and cross-tier link contention.
Switch Hierarchy Latency Penalty
Intra-rack communication (Leaf hop) takes ~1 microsecond. Inter-rack spine communication (Spine hop) takes ~2.5 microseconds. Inter-pod core communication (Core hop) takes 5+ microseconds with higher risk of link oversubscription.
Hierarchical 3D Parallelism Mapping
Map Tensor Parallelism (TP=8) strictly inside the node (NVLink). Map Pipeline Parallelism (PP) to adjacent nodes on the same Leaf switch. Map Data Parallelism (DP) across Spine switches where latency is least critical.
Bin-Packing Algorithms
Topology schedulers use tree-based best-fit bin-packing to allocate nodes belonging to the smallest common ancestor switch sub-tree.
Where w_h is the latency penalty weight per switch tier. Scattering ranks across core switches increases average hop count from 1 to 5, resulting in a 35%–50% slowdown in communication-heavy AllGather phases.
- Use the Kubernetes Topology-Aware Scheduling plugin or Slurm --switches=N@timeout flag to enforce strict rack locality.
- Generate a physical cluster network topology graph (topology.xml) and pass it to NCCL via NCCL_TOPO_DUMP_FILE for optimal ring formation.
- Never allow multi-tenant fragmentations to break contiguous node blocks; run periodic cluster defragmentation during low-load windows.