INFERENCE ENGINEERING

SCENE 20 / 24 · DISTRIBUTED INFERENCE & INFRASTRUCTURE SCALE

Slice weights, chain layers

Tensor parallelism inside a node, pipeline parallelism across it.

TP RANKSTP4 · 35 GB / GPUPIPE BUBBLE · 24 ms/stage
TENSOR PARALLEL — one layer, sliced 4× inside the nodePIPELINE PARALLEL — layers chained across GPUsL0–26L26–53L53–80L80–106

Tensor parallel splits each matmul and needs NVLink-class interconnect; pipeline parallel chains layers and hides bubbles with micro-batches. Together they fit big models and multiply effective bandwidth per token.

ARCHITECTURE & FORMULATIONModule 7: Distributed Inference & Infrastructure Scale

Tensor vs Pipeline parallelism: split weights or chain layers

When a model exceeds the memory or bandwidth of a single GPU, parallelism splits the workload. Tensor Parallelism (TP) splits individual weight matrices inside each layer across GPUs within a node (requiring fast NVLink all-reduce). Pipeline Parallelism (PP) assigns sets of layers to different GPUs across nodes, using micro-batching to hide pipeline bubbles.

Tensor Parallelism (TP)

Matrix multiplications are partitioned across GPU column/row splits. Requires 2 All-Reduce communications per Transformer layer, making NVLink (900 GB/s) mandatory.

Pipeline Parallelism (PP)

Layers are split across GPUs (e.g., layers 1-16 on GPU 0, 17-32 on GPU 1). Micro-batches pass activations sequentially, introducing pipeline fill/drain bubbles.

Hybrid Strategy (TP+PP)

Large clusters use intra-node TP (e.g., TP=8 on one node) and inter-node PP (e.g., PP=4 across nodes) to balance communication latency.

MATHEMATICAL FORMULATION · PIPELINE BUBBLE FRACTION
Bubble_fraction = (PP − 1) / (PP − 1 + MicroBatches)

With PP=4 and 16 micro-batches, the pipeline bubble is (3 / 18) ≈ 16.6% idle time. Increasing micro-batches reduces bubble overhead.

REAL-WORLD PRODUCTION ENGINEERING
  • Megatron-LM and vLLM combine TP and PP to serve 70B to 405B models across multi-node GPU clusters.
  • Sequence Parallelism (SP) is often added on top of TP to split LayerNorm and Dropout activations, reducing VRAM footprint further.
← 19 · Draft cheaply, verify once
21 · Route to a few experts →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]