NVLink Mesh & NVSwitch Scale-Up Architecture
Inside an 8-GPU scale-up server (such as HGX H100/H200 or B200), traditional PCIe bus interconnects are completely inadequate for distributed tensor-parallel GEMM operations. NVLink provides high-bandwidth, low-latency, point-to-point links interconnected via on-board NVSwitch ASICs, turning 8 physical GPUs into a single unified shared-memory domain with up to 3.6 TB/s bisection bandwidth.
Full-Mesh vs Switched Topology
Early NVLink generations used direct point-to-point mesh routing, limiting scale-up to 4 or 8 GPUs with non-uniform hop latencies. NVSwitch introduces an on-node crossbar switch that delivers flat, uniform, non-blocking single-hop latency across all 8 GPUs simultaneously.
Load/Store Memory Semantics
NVLink operates with native hardware-level memory read/write semantics rather than packetized network socket protocols. A GPU thread can directly dereference a global memory pointer located in another GPU's HBM with ~100ns latency.
Scale-Up vs Scale-Out Boundary
The physical baseboard marks the fundamental boundary in distributed training. Intra-node communication over NVLink is 10–20x faster than inter-node communication over InfiniBand or RoCE v2, dictating where Tensor Parallelism (intra-node) stops and Pipeline/Data Parallelism (inter-node) begins.
For an 8-GPU H100 baseboard with 18 NVLink-4 links per GPU running at 50 GB/s bidirectional (25 GB/s each direction), bisection bandwidth across any bisection cut equals 4 GPUs * 18 links * 50 GB/s = 3.6 TB/s total bandwidth.
- Hardware NVLink CRC counters must be monitored via nvidia-smi nvlink -s; replay bursts indicate degraded PCB traces or thermal expansion issues.
- Tensor Parallelism (TP) degree should never exceed the number of GPUs connected via NVLink (typically TP <= 8). Splitting TP across InfiniBand causes severe communication stalls.
- NVSwitch SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) offloads AllReduce reductions directly into the switch hardware.