Pipeline Parallelism and Its Bubbles
Lesson 4 of 5 in Distributed Training.
The third axis cuts the model the way you would cut a book: by chapters. Pipeline parallelism assigns consecutive layers to different devices — device 1 holds the first quarter of the stack, device 2 the next quarter, and so on. Each group is a stage, and activations flow from stage to stage like parts down an assembly line.
The economics are attractive. Per-device memory falls roughly linearly with the number of stages, because each device stores only its own layers’ weights and optimizer states. And the communication is remarkably cheap: unlike tensor parallelism’s all-reduces inside every layer, a pipeline sends only the activations at each stage boundary, point-to-point, once on the way forward and once (as gradients) on the way back. That modest, localized traffic is exactly what slower links between servers can handle — pipeline parallelism is the axis that crosses node boundaries gracefully.
The catch appears the moment you run it naively. Feed one batch into the pipeline and stage 2 sits idle while stage 1 computes, stage 3 waits on stage 2, and during the backward pass the whole sequence plays out in reverse. At any instant, most of your expensive cluster is doing nothing. The idle time has a name that will follow you through every systems paper: the pipeline bubble.
Heatmap with four rows (pipeline stages 1 to 4, each holding a quarter of the layers) and seven columns (time steps). Filled cells form a diagonal band: stage 1 is busy in steps 1 through 4, stage 2 in steps 2 through 5, stage 3 in steps 3 through 6, and stage 4 in steps 4 through 7. The empty triangular corners at top-right and bottom-left are the pipeline bubble, where devices sit idle.
GPipe (Huang et al. 2018) supplied the standard fix: don’t send the batch through whole — chop it into micro-batches and stream them. While stage 2 works on micro-batch 1, stage 1 is already computing micro-batch 2. The pipeline fills like a bucket brigade, all stages work simultaneously through the middle of the step, and gradients from all micro-batches accumulate before one synchronized optimizer update — so the math stays identical to ordinary training. The bubble does not disappear — the ramp-in and ramp-out are unavoidable — but it shrinks as the micro-batch count grows.
Micro-batching has its own bill, though: activations for every in-flight micro-batch must be held until its backward pass arrives. GPipe’s answer was re-materialization — activation checkpointing: store only each stage’s boundary activations, throw the rest away, and recompute them during the backward pass. Spend FLOPs, save memory. Later schedules (the “one-forward-one-backward” family) reorder work to start backward passes earlier and cap how many micro-batches are ever in flight — the details vary, but every schedule is negotiating the same triangle of bubble, memory, and recompute.
The bubble math, worked
For a pipeline with K stages and m micro-batches per step, the bubble occupies a fraction of the ideal step time of roughly (K − 1) / (m + K − 1) — the GPipe paper derives this as O((K − 1)/(m + K − 1)) and reports the overhead becomes negligible once m ≥ 4·K, helped by recomputation overlapping the bubble (Huang et al. 2018).
Work the toy numbers from the figure: K = 4, m = 4 gives 3/7 ≈ 43% idle — catastrophic. Raise m to 16 and it is 3/19 ≈ 16%; m = 32 gives 3/35 ≈ 9%; the m ≥ 4K rule (m = 16 here) is the knee of that curve. Two design pressures follow directly. First, deep pipelines (large K) need proportionally more micro-batches to stay efficient, and the micro-batch supply is capped by the global batch size — which optimization, not systems, gets to choose. Second, this is why nobody uses pipeline parallelism at a degree higher than they must: every added stage deepens the ramp that must be amortized.
One more subtlety: micro-batches divide the batch, so each micro-batch is smaller — and stages must still be load-balanced so no single slow stage stretches every time slot. An unbalanced split (one stage with heavier layers) widens every column of the grid to the slowest stage’s pace: the pipeline is only as fast as its fattest chapter.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.