Pipeline Parallelism & 1F1B Scheduling
Pipeline Parallelism (PP) splits model layers sequentially across GPU stages (e.g. layers 1-20 on Stage 0, 21-40 on Stage 1). Micro-batches flow through stages sequentially like a physical assembly line.
The Pipeline Bubble
Naive pipeline schedules cause GPUs to sit idle waiting for activations from upstream stages or gradients from downstream stages. Bubble fraction = (p - 1) / (m + p - 1) where p is stages and m is micro-batches.
1F1B (One Forward, One Backward) Schedule
After a warmup phase, each GPU stage alternates executing 1 micro-batch forward pass followed by 1 micro-batch backward pass, capping activation memory while keeping pipeline bubbles small.
Point-to-Point Interconnect
Unlike TP or DP, PP only communicates activations and gradients between adjacent stage nodes using direct peer-to-point network calls.
Increasing micro-batch count m relative to stage count p drastically reduces idle GPU dead time.
- Interleaved 1F1B (Megatron) assigns multiple non-contiguous layer chunks to each stage, reducing bubble size by another 2x.