Tensor vs Pipeline parallelism: split weights or chain layers
When a model exceeds the memory or bandwidth of a single GPU, parallelism splits the workload. Tensor Parallelism (TP) splits individual weight matrices inside each layer across GPUs within a node (requiring fast NVLink all-reduce). Pipeline Parallelism (PP) assigns sets of layers to different GPUs across nodes, using micro-batching to hide pipeline bubbles.
Tensor Parallelism (TP)
Matrix multiplications are partitioned across GPU column/row splits. Requires 2 All-Reduce communications per Transformer layer, making NVLink (900 GB/s) mandatory.
Pipeline Parallelism (PP)
Layers are split across GPUs (e.g., layers 1-16 on GPU 0, 17-32 on GPU 1). Micro-batches pass activations sequentially, introducing pipeline fill/drain bubbles.
Hybrid Strategy (TP+PP)
Large clusters use intra-node TP (e.g., TP=8 on one node) and inter-node PP (e.g., PP=4 across nodes) to balance communication latency.
With PP=4 and 16 micro-batches, the pipeline bubble is (3 / 18) ≈ 16.6% idle time. Increasing micro-batches reduces bubble overhead.
- Megatron-LM and vLLM combine TP and PP to serve 70B to 405B models across multi-node GPU clusters.
- Sequence Parallelism (SP) is often added on top of TP to split LayerNorm and Dropout activations, reducing VRAM footprint further.