Distributed training, Spot, and checkpoints
Lesson 3 of 4 in Training and Tuning on SageMaker AI.
When one instance stops being enough, distributed training splits the work — and the exam only asks that you split it the right way. Data parallelism: every GPU holds a full copy of the model, each sees a different shard of the batch, gradients synchronize after each step. It is the default answer when the model fits on one GPU but training is too slow. Model parallelism: the model itself is partitioned across GPUs because its weights and activations cannot fit in one GPU's memory — the signature of large language models. SageMaker ships managed libraries for both, and script mode runs the frameworks' native equivalents.
| Strategy | Shards what | Pick when | Watch out |
|---|---|---|---|
Data parallel | The batch — model copied to every GPU | Model fits on one GPU; you need faster epochs over lots of data | Gradient sync is network-heavy; scale instances, watch throughput |
Model parallel | The model — layers/tensors split across GPUs | The model is too large for a single GPU's memory | More complex, slower per step — use only when memory forces it |
The other efficiency lever is Managed Spot Training: run the same job on spare EC2 capacity at a deep discount. The catch — Spot capacity can be reclaimed mid-job. So Spot jobs must be interruption-tolerant: enable checkpointing to S3, and set a maximum wait time (at least the maximum runtime) so SageMaker can pause and resume when capacity returns. A restarted job reads the last checkpoint and continues; without checkpoints, an interruption means starting from scratch — cheap compute, expensive restarts.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.