Distributed Training

No single device fits the model or the data: data, tensor, and pipeline parallelism — and how real runs combine all three.

Content last verified 2026-09.

Lessons

1. Why One GPU Is Never Enough
2. Data Parallelism and Sharded Optimizers
3. Tensor Parallelism: Splitting the Math
4. Pipeline Parallelism and Its Bubbles
5. Putting It Together: 3D Parallelism

Sources

• Rajbhandari et al. (2019) — ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Shoeybi et al. (2019) — Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Huang et al. (2018) — GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Micikevicius et al. (2017) — Mixed Precision Training