INFRASTRUCTURE

MODULE 2: INTERCONNECTS & CLUSTER TOPOLOGIES · SCENE 05

InfiniBand vs RoCE v2: Lossless transport mechanics

Credit-based flow control vs PFC/ECN pause frames, packet drops, tail latency spikes, and tuning headroom to prevent deadlocks.

REQUIRED PFC HEADROOM · 102.5 KB / Port

Native InfiniBand (NDR / XDR)

Uses link-layer credit-based flow control. Senders only transmit when switch receive buffers have open credits. Zero drops by construction, zero PFC pause storms.

RoCE v2 Lossless Ethernet

Relies on PFC (802.1Qbb) + ECN/DCQCN. Requires precise headroom buffer reservation of 102.5 KB to prevent packet drops before PAUSE frames halt the sender.

SENDER NIC400 Gbps Output Queue100m Fiber (500ns RTT delay)SWITCH INGRESS BUFFERPFC Headroom: 102.5 KB
ARCHITECTURE & FORMULATIONModule 2: Interconnects & Cluster Topologies

InfiniBand vs RoCE v2 Transport Mechanics

Distributed deep learning requires lossless network fabrics. Unlike standard TCP/IP where packet drops are resolved via slow retransmission timeouts, AI fabrics use either native InfiniBand (with hardware credit-based flow control) or RoCE v2 (RDMA over Converged Ethernet with Priority Flow Control and Explicit Congestion Notification).

Hardware Credit-Based Flow Control (InfiniBand)

InfiniBand switches maintain credit buffers for each receiver. A sender only transmits packets when receiver buffer credits are available, guaranteeing zero packet drops in the fabric with microsecond latency.

Priority Flow Control (PFC) & ECN (RoCE v2)

RoCE v2 runs on Ethernet. Priority Flow Control (IEEE 802.1Qbb) sends PAUSE frames when switch ingress buffers cross high watermarks. Explicit Congestion Notification (ECN / RFC 3168) marks packets (CE) to trigger sender-side rate reduction (DCQCN).

PFC Deadlocks & Headroom Sizing

If PAUSE frames propagate upstream in a circular dependency across multiple switches, the entire fabric experiences a PFC deadlock, freezing all training jobs cluster-wide until switches are rebooted.

MATHEMATICAL FORMULATION · PFC HEADROOM BUFFER REQUIREMENT
B_headroom = (2 × τ_prop + τ_switch) × BW + Margin_jitter

For a 100-meter fiber cable (500ns propagation each way) running 400 Gbps (50 GB/s) with 400ns switch processing time, buffer headroom must be >= (1400ns * 50 GB/s) * 1.5 = ~105 KB per port to prevent packet drops before the PAUSE frame halts the sender.

REAL-WORLD PRODUCTION ENGINEERING
  • In RoCE v2 clusters, enable Dynamic ECN tuning and monitor PFC pause frame counters; nonzero PFC storms require immediate buffer threshold adjustments.
  • InfiniBand Subnet Manager (OpenSM) topology routing should use fat-tree routing (ftree) or adaptive routing (ar) to ensure collision-free collective paths.
  • Use rdma-core and ibv_rc_pingpong to verify raw RDMA latency (< 1.5 microseconds) before starting multi-node training.
← 04 · The modern AI server: HGX baseboard layout
06 · Rail-optimized Fat-Tree & Dragonfly+ fabrics →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]