TRAINING AT SCALE

MODULE 4: DISTRIBUTED INFRASTRUCTURE · SCENE 17

An assembly line of layers

Pipeline parallelism streams micro-batches through stage slices — and pays a bubble tax.

MICRO-BATCHES (m)PIPELINE BUBBLE IDLE: 42.9%

1F1B schedules interleave forward and backward micro-batches to cap activation memory while keeping pipeline bubble idle time small.

ARCHITECTURE & FORMULATIONModule 4: Distributed Infrastructure & 3D Parallelism

Pipeline Parallelism & 1F1B Scheduling

Pipeline Parallelism (PP) splits model layers sequentially across GPU stages (e.g. layers 1-20 on Stage 0, 21-40 on Stage 1). Micro-batches flow through stages sequentially like a physical assembly line.

The Pipeline Bubble

Naive pipeline schedules cause GPUs to sit idle waiting for activations from upstream stages or gradients from downstream stages. Bubble fraction = (p - 1) / (m + p - 1) where p is stages and m is micro-batches.

1F1B (One Forward, One Backward) Schedule

After a warmup phase, each GPU stage alternates executing 1 micro-batch forward pass followed by 1 micro-batch backward pass, capping activation memory while keeping pipeline bubbles small.

Point-to-Point Interconnect

Unlike TP or DP, PP only communicates activations and gradients between adjacent stage nodes using direct peer-to-point network calls.

MATHEMATICAL FORMULATION · PIPELINE BUBBLE IDLE TIME FRACTION
Bubble_Fraction = ( Stages_p - 1 ) / ( MicroBatches_m + Stages_p - 1 )

Increasing micro-batch count m relative to stage count p drastically reduces idle GPU dead time.

REAL-WORLD PRODUCTION ENGINEERING
  • Interleaved 1F1B (Megatron) assigns multiple non-contiguous layer chunks to each stage, reducing bubble size by another 2x.
← 16 · Slice the matrix itself
18 · Sixteen bits, carefully placed →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]