WORLD 06 INFRASTRUCTURE
Trace demand through the systems that keep it alive.
From NVLink mesh silicon to 100kW+ liquid-cooled datacenter power envelopes across 6 modules: high-radix fat-trees, RoCE v2 flow control, GPUDirect Storage, gang schedulers, straggler elimination, and FinOps unit economics.
JOURNEY · 6 MODULES · 24 SCENES
From silicon trace to cluster power grid.
Silicon & Node Architecture
The physical baseboard: NVLink meshes, PCIe Gen5 bottlenecks, and HBM3e bandwidth physics.
01 · SILICON FABRIC
NVLink mesh & NVSwitch scale-up
Inside the 8-GPU baseboard: how full-mesh NVLink provides 900 GB/s to 3.6 TB/s bisection bandwidth, blurring the boundary between chip and bus.
OPEN EXHIBIT →02 · HOST BUS
PCIe Gen5 vs CXL host-to-device limits
Host CPU-to-GPU transfer bottlenecks, bounce buffers, pinned memory semantics, and how CXL cache coherency rewires memory pooling.
OPEN EXHIBIT →03 · MEMORY WALL
HBM3e bandwidth saturation & roofline knee
The arithmetic intensity threshold where high-bandwidth memory reaches saturation and GPU compute switches from memory-bound to compute-bound.
OPEN EXHIBIT →04 · NODE ANATOMY
The modern AI server: HGX baseboard layout
8x GPUs, dual x86/ARM host sockets, 8x CX-7 NICs, BlueField-3 DPUs, and PCIe Gen5 retimers configured for zero-contention scale-up.
OPEN EXHIBIT →Interconnects & Cluster Topologies
High-radix Fat-Trees, Rail-Optimized fabrics, InfiniBand vs RoCE, and collective communication physics.
05 · FABRIC PROTOCOLS
InfiniBand vs RoCE v2: Lossless transport mechanics
Credit-based flow control vs PFC/ECN pause frames, packet drops, tail latency spikes, and tuning headroom to prevent deadlocks.
OPEN EXHIBIT →06 · TOPOLOGY
Rail-optimized Fat-Tree & Dragonfly+ fabrics
Grouping GPU ordinal ranks onto dedicated spine rails to eliminate cross-rail traffic during tensor and pipeline parallel collectives.
OPEN EXHIBIT →07 · COLLECTIVES
Ring vs Tree AllReduce: Bandwidth vs latency physics
Why ring topologies saturate maximum wire bandwidth for large tensors while binomial/double-binary trees win on latency for small payloads.
OPEN EXHIBIT →08 · CONGESTION
Network incast, microbursts & packet spraying
When thousands of GPUs send AllGather traffic simultaneously: buffer bloat, incast packet drops, and dynamic adaptive packet spraying.
OPEN EXHIBIT →Distributed Storage & Checkpointing
GPUDirect Storage, asynchronous checkpointing pipelines, Daly interval optimization, and IO starvation.
09 · STORAGE IO
GPUDirect Storage (GDS) vs POSIX bounce buffers
Bypassing the CPU host memory and OS page cache with direct DMA from NVMe-oF flash into GPU HBM, slashing CPU utilization from 60% to 2%.
OPEN EXHIBIT →10 · CHECKPOINTING
Daly's formula: Optimal checkpoint frequency
Balancing the I/O cost of writing state against the compute waste of re-executing lost steps under stochastic cluster failure rates.
OPEN EXHIBIT →11 · DATA PIPELINE
Petabyte data streaming & worker prefetching
Preventing GPU starvation during multimodal pre-training: chunked shard streaming, ring buffer prefetching, and asynchronous Ray/DALI workers.
OPEN EXHIBIT →12 · RECOVERY
Checkpoint blast-radius & sub-minute MTTR
Resuming 16,384 GPU ranks without metadata server stampedes: distributed rank sharding, peer-to-peer weight broadcasting, and fast initialization.
OPEN EXHIBIT →Cluster Scheduling & Orchestration
Gang scheduling, topology-aware placement, straggler elimination, and multi-tenant hardware partitioning.
13 · ORCHESTRATION
Gang scheduling & all-or-nothing allocation
Why distributed AI jobs cannot tolerate partial scheduling: deadlock prevention, Kueue/Volcano admission controllers, and Slurm reservations.
OPEN EXHIBIT →14 · PLACEMENT
Topology-aware packing & switch-hop minimization
Bin-packing ranks to ensure high-communication tensor parallel domains share the same NVLink baseboard and leaf switch, avoiding spine bottlenecks.
OPEN EXHIBIT →15 · DIAGNOSTICS
Straggler detection & collective barrier lag
How a single degraded GPU running 5% slower stalls the entire 10,000-GPU synchronous collective barrier, and how NCCL flight recorders catch it.
OPEN EXHIBIT →16 · MULTI-TENANCY
MIG hardware slicing vs time-sliced virtualization
Partitioning an H100 into up to 7 isolated GPU instances with dedicated SMs, memory controllers, and L2 cache crossbars for guaranteed QoS.
OPEN EXHIBIT →Reliability, Thermal & Power Grid
Silent Data Corruption (SDC), 100kW+ liquid-cooled rack envelopes, autonomous node draining, and PUE economics.
17 · RELIABILITY
Silent Data Corruption (SDC) & transient bit-flips
Cosmic rays, voltage droops, and undetected matrix math errors: why gradient norms explode and how hardware checksums & periodic canary audits catch them.
OPEN EXHIBIT →18 · THERMAL & POWER
100kW+ rack envelopes & direct-to-chip liquid cooling
Managing 1,000W TDP silicon: cold plates, coolant distribution units (CDUs), facility water loops, and thermal throttling hysteresis curves.
OPEN EXHIBIT →19 · AUTONOMOUS OPS
Predictive health checks & autonomous node draining
Detecting PCIe bus error bursts, uncorrectable ECC trends, and optical transceiver degradation before a hard node crash destroys the training run.
OPEN EXHIBIT →20 · SUSTAINABILITY
Datacenter PUE & temporal compute shifting
Power Usage Effectiveness dynamics: shifting non-urgent batch pre-training jobs across global datacenters to follow green grid energy supply curves.
OPEN EXHIBIT →FinOps & Capacity Economics
Model FLOPs Utilization (MFU), cluster TCO, spot vs reserved arbitrage, and cost per million tokens.
21 · EFFICIENCY
Model FLOPs Utilization (MFU) vs MBU accounting
Measuring true hardware efficiency: separating kernel execution time, communication overhead, bubble time, and memory bandwidth limits.
OPEN EXHIBIT →22 · FINOPS
Cluster CapEx vs OpEx: Amortized TCO breakdown
Silicon depreciation, optical transceiver replacement rates, power contracts, datacenter shell leases, and the hidden cost of idle GPU hours.
OPEN EXHIBIT →23 · UNIT ECONOMICS
Cost per million tokens: Pretraining vs serving
Calculating the exact dollar cost of a 15-trillion token pretraining run, and mapping infrastructure efficiency directly to token margins.
OPEN EXHIBIT →24 · CAPACITY
Spot instance arbitrage & zero-overhead recovery
Slashing compute bills by 60–70% using preemptible spot instances paired with sub-30-second asynchronous snapshotting and dynamic rank rejoining.
OPEN EXHIBIT →