INFRASTRUCTURE

MODULE 2: INTERCONNECTS & CLUSTER TOPOLOGIES · SCENE 06

Rail-optimized Fat-Tree & Dragonfly+ fabrics

Grouping GPU ordinal ranks onto dedicated spine rails to eliminate cross-rail traffic during tensor and pipeline parallel collectives.

OVERSUBSCRIPTION · 1.00:1 (NON-BLOCKING)
SPINE 0SPINE 1SPINE 2SPINE 3RAIL LEAF 0RAIL LEAF 1RAIL LEAF 2RAIL LEAF 3GPU ORDINAL 0-7 ATTACHED STRICTLY TO RAIL LEAF 0-7 · ZERO CROSS-RAIL CONGESTION
Total Bisection Bandwidth: 25.6 Tbps across spine plane.
ARCHITECTURE & FORMULATIONModule 2: Interconnects & Cluster Topologies

Rail-Optimized Fat-Tree & Dragonfly+ Fabrics

In multi-thousand GPU clusters, network cabling architecture determines whether collective communication scales or collapses under congestion. Rail-optimized fat-tree networks map each GPU ordinal position to an independent leaf switch rail, perfectly mirroring the communication structure of 3D parallel distributed training.

Rail Optimization Principle

In a cluster of 8-GPU nodes, all GPU 0s connect to Leaf Switch Rail 0, all GPU 1s to Leaf Switch Rail 1, and so on. In Data Parallel and Pipeline Parallel collectives, communication flows exclusively within individual rails, eliminating cross-rail contention.

Non-Blocking Fat-Tree (1:1 Oversubscription)

A non-blocking Fat-Tree provides equal uplink and downlink bandwidth at every switch tier, guaranteeing full bisection bandwidth regardless of traffic permutation across nodes.

Dragonfly+ Topologies

Dragonfly+ organizes switches into groups with dense intra-group optical links and sparse global inter-group links, reducing optical transceiver and cable counts by 30–40% compared to traditional 3-tier fat trees.

MATHEMATICAL FORMULATION · FAT-TREE OVERSUBSCRIPTION RATIO
R_oversub = (N_downlinks × BW_down) / (N_uplinks × BW_up) [1.0 = Non-Blocking]

A leaf switch with 32x 400Gbps downlinks to servers and 16x 400Gbps uplinks to spines has an oversubscription ratio of 2:1 (0.5 bandwidth availability), reducing inter-leaf collective throughput by 50% under worst-case all-to-all permutations.

REAL-WORLD PRODUCTION ENGINEERING
  • Maintain a 1:1 non-blocking ratio for the compute fabric. Oversubscription is only acceptable on separate storage and management networks.
  • Label all fiber cables by Node ID and GPU Rail index; cross-cabling GPU 0 into Rail 1 causes severe silent performance degradation in NCCL communicators.
  • Use optical cable monitors (DDM) to continuously log optical power levels (Tx/Rx power in dBm) to catch failing lasers before link drops occur.
← 05 · InfiniBand vs RoCE v2: Lossless transport mechanics
07 · Ring vs Tree AllReduce: Bandwidth vs latency physics →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]