Training Infrastructure on the Clouds
Lesson 3 of 3 in In Production: Should You Ever Pre-train?.
If your organization does train — from scratch or continued — you will almost certainly do it on rented infrastructure, and all three hyperscalers sell it in managed form. At the level that stays true across hardware generations: AWS offers managed training through SageMaker — jobs that provision clusters, wire up distribution, watch node health, and resume from checkpoints — on GPU fleets and on purpose-built training accelerators AWS designs itself. Azure runs distributed training through Azure Machine Learning on GPU clusters built around high-bandwidth RDMA-class interconnect, with the identity, network isolation, and compliance surface of the rest of Azure attached. Google Cloud offers TPU training — accelerators and interconnect co-designed as pods you provision by topology — alongside GPU fleets, with the deepest assumptions about the software stack (TPUs reward compiler-first frameworks that emit XLA).
The trade-offs are the same three on every cloud. Managed versus assembled: managed training buys you the orchestration layer from last lesson — provisioning, health checks, restart-from-checkpoint — at the cost of fitting your job into the platform’s shapes; assembling your own cluster from raw instances returns the control and the pager. Capacity: accelerators at cluster scale are a reservation negotiation, not an API call — large blocks are booked ahead, and how far ahead, in what shapes, on what terms, is often the actual constraint on when your run can start. Data gravity: a trillion-token corpus and a steady stream of checkpoints live in object storage next to the cluster; moving either across clouds mid-project is painful enough that the storage decision quietly becomes the cloud decision.
| Dimension | Ask AWS | Ask Azure | Ask Google Cloud |
|---|---|---|---|
Capacity & reservations | What cluster sizes can SageMaker training actually schedule for me, and how far ahead must a large block be reserved — on GPUs versus your own training accelerators? | What GPU cluster shapes can Azure ML schedule at my scale, and what do cluster-scale reservations look like in my region? | What TPU slice topologies (or GPU clusters) can I reserve, on what lead time, and in which regions? |
Interconnect | How are my nodes placed relative to each other, and what inter-node bandwidth does the fabric sustain at my job size? | Which cluster shapes come with RDMA-class interconnect end to end, and is it uniform across the whole allocation? | What does the pod topology I am buying mean for tensor- versus data-parallel traffic in my planned layout? |
Failure & resume | What does managed training detect and replace automatically, and what is the measured resume time from my checkpoint size? | What health monitoring and restart behavior does Azure ML provide for multi-node jobs, and where does my restart logic take over? | How do slices behave on hardware fault — what is replaced, what restarts, and what does my framework need to handle? |
Data & checkpoints | What throughput can S3 sustain to my cluster for shard streaming, and how long does a full checkpoint take to write and restore? | Same questions against Blob Storage — sustained read for the data pipeline, burst write for checkpoints at my cadence. | Same questions against Cloud Storage — and does the TPU data path change how I should shard the corpus? |
Software stack | Which distribution strategies does the managed layer configure for me, and which do I bring (my own launcher, my own parallelism)? | What does Azure ML assume about my training framework, and how much of the job definition is mine to own? | How much of my stack must be XLA-friendly to earn the TPU’s economics — and what is the porting cost from my current codebase? |
In production
What “managed training” actually manages, cloud by cloud — and the line where the platform’s responsibility ends and yours begins.
AWS
SageMaker training jobs take a container and a cluster spec and handle provisioning, inter-node wiring, health monitoring, node replacement, and restart-from-checkpoint; purpose-built training accelerators exist as an AWS-designed alternative to GPUs, trading ecosystem breadth for a price-performance argument you should evaluate on your own workload. What stays yours on any path: the corpus and its mixture, the parallelism layout, the hyperparameters, the loss curve, and the judgment call when it spikes.
Azure
Azure Machine Learning runs multi-node distributed jobs on GPU clusters with RDMA-class interconnect, providing scheduling, health monitoring, and integration with Azure’s identity and network-isolation machinery — often the deciding surface for regulated buyers whose sovereignty requirements stop at infrastructure rather than reaching the weights. The training science remains entirely yours; the platform keeps nodes alive, not loss curves healthy.
Google Cloud
Cloud TPU training provisions pod slices where accelerator and interconnect arrive as one co-designed unit — you choose a topology rather than assembling a network — and managed layers handle scheduling and fault recovery. The distinctive trade-off is software: TPU economics reward XLA-compiled stacks, so porting cost is part of the price. Across all three clouds the boundary is identical: platforms manage machines; convergence is never a managed service.
And that is the pre-training domain, closed. You can read a data pipeline, an objective, a scaling-law plot (the Scaling Law Plotter at /tools/scaling-law-plotter will stay useful), a FLOPs budget, a parallelism layout, a loss-spike postmortem — and now a build-versus-adapt decision — the way this domain has insisted throughout: mechanisms first, numbers cited or not stated at all.
But notice what all of it produces: a Base model — a formidable text-completer with no instructions followed, no refusals, no conversation. Every path through this module ends at the same door, whether you pre-trained from scratch, continued on an open-weights base, or wisely did neither: the model that people actually use is made in the next stage. That is the Post-training & Alignment domain — SFT, RLHF, DPO, and reasoning training; how a predictor becomes an assistant.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.