What a Training Cluster Demands
Lesson 2 of 3 in In Production: Should You Ever Pre-train?.
Suppose the tree really did exit at “pre-train.” What did you just sign up for? Not “a lot of GPUs” — something stranger: thousands of accelerators pretending to be one computer, for weeks or months, without dropping the act. A frontier-scale run is a single synchronous job. Every optimizer step, every participating device must finish its share and exchange results before anyone takes the next step; the data-parallel replicas must average gradients, the tensor-parallel shards must exchange activations layer by layer, the pipeline stages must hand tensors down the line. One slow device slows all of them. One dead device stops the job.
That is why the people who run these clusters obsess about a component that never appears in the marketing photo: the interconnect. The arithmetic lives in the accelerators, but the schedule lives in the network between them — links inside each node, and the fabric between nodes. When the fabric is slow relative to the compute, accelerators finish their local math and wait, and the ratio of useful math to peak capability — MFU — sags. At scale you are not buying FLOPs; you are buying FLOPs that can talk to each other fast enough to matter.
Stack diagram of a training cluster in four layers, bottom to top: an accelerator layer of thousands of GPUs or TPUs grouped into nodes; an interconnect layer of intra-node links and an inter-node fabric, marked as the usual bottleneck; a storage and data-pipeline layer streaming tokenized corpus shards in and checkpoints out; and an orchestration layer of schedulers, health checks, restart automation, and an on-call team.
Key terms: Model FLOPs utilization (MFU), Data parallelism, Checkpoint, Loss spike, FLOPs
The second thing scale changes is failure arithmetic. Individually, modern nodes are reliable. But a synchronous job inherits the failure rate of every participant: if any node dies, the whole run stops. Put enough nodes under one job and interruptions shift from “incident” to “weather” — something that happens routinely and must be engineered around, not apologized for. That is why checkpointing discipline is a first-class design axis: how often you save the full training state, how fast you can write it, how fast you can restore and refill the data pipeline to the exact batch where you stopped. Save too rarely and each failure burns hours of paid compute; save too often and the saving itself eats the schedule. The public OPT logbook (Zhang et al. 2022) put this on the record: a large run’s history is a history of restarts — hardware failures, loss spikes, rollbacks — managed by a team, in shifts.
Which is the third demand: people. A run needs data engineers keeping shard streams and the Data mixture honest, systems engineers tuning parallelism and hunting stragglers, researchers reading the Loss curve and deciding whether that spike heals or needs a rollback, and an on-call rotation for the 3 a.m. page — because the meter never stops running. The cluster bill is the visible line item; the standing team is the one that surprises organizations that thought they were buying hardware.
Two back-of-envelope laws of cluster life
Failure scaling. If each node fails independently at some average rate, a synchronous job over k nodes is interrupted at roughly k times that rate — mean time between job interruptions shrinks like 1/k. A node that fails once a year sounds superb; two thousand of them under one job means an expected interruption every few hours from that cause alone, before software, network, and storage add theirs. Nothing about better hardware repeals this; it only moves the constant. The engineering responses all follow from the formula: checkpoint at intervals sized to the expected interruption rate, keep warm spare nodes to swap in, and make restore time — load state, rewind the data loader, rejoin the collective — a tuned path, not an afterthought.
Communication scaling. In synchronous data parallelism, each step ends by averaging gradients: every replica must send and receive on the order of one model’s worth of gradient data per step, however large the batch. Compute per device rises with its share of the batch; gradient traffic does not shrink with it. Overlapping communication behind the backward pass hides much of the cost — until model size, step time, and fabric bandwidth fall out of balance, and the interconnect sets the pace. Tensor parallelism is stricter still: activations cross the wire inside every layer, every microbatch, which is why it is confined to the fastest links — usually within a node — while pipeline and data parallelism span the slower fabric between nodes. The 3D-parallelism layouts of the distributed-training module are exactly this: matching each parallelism’s traffic pattern to the bandwidth tier that can afford it.
In production
Every mechanism in this lesson is a purchasable surface on the clouds — and the questions to ask a provider are the mechanisms wearing product names.
AWS
On AWS, the interconnect concern becomes cluster placement and high-bandwidth, RDMA-class networking between GPU nodes — you ask how nodes are placed relative to each other and what the fabric delivers at your job’s scale. Failure arithmetic becomes health checks, node replacement, and auto-resume in SageMaker’s managed training: the platform detects a failed node, swaps it, and restarts from your last checkpoint. Storage becomes S3 as the checkpoint sink and corpus home, with throughput to the cluster as the number to verify.
Azure
Azure’s GPU training fleets are built around RDMA-class interconnect between nodes, and Azure Machine Learning runs distributed jobs with health monitoring and restart-from-checkpoint patterns. The questions are the same mechanisms: what bandwidth ties nodes together at my scale, what happens automatically when a node dies mid-step, and how fast can Blob Storage absorb a full checkpoint and stream shards back after a restore.
Google Cloud
Google’s TPU pods make the lesson vivid because the interconnect is part of the accelerator’s architecture: pods are designed as tightly coupled slices, and you provision a topology, not a pile of chips. Ask how slices recover from hardware faults, what the resume story is, and how Cloud Storage sustains checkpoint writes at your cadence. On GPU fleets the same placement, fabric, and auto-repair questions apply as anywhere.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.