When Runs Go Wrong
Lesson 3 of 3 in Mixed Precision and Stability.
A pre-training run is a months-long computation on thousands of accelerators, and at that scale failure is not an event — it is a rate. Two families of trouble dominate. Numerics: the loss, after weeks of smooth decline, suddenly jumps — a Loss spike — and either recovers on its own or keeps climbing into divergence. Hardware: training is synchronous, every device must finish every step, so a single failed GPU, flaky link, or bad memory module stalls the entire cluster. Multiply a small per-device failure rate by thousands of devices and weeks of runtime, and the mean time between interruptions shrinks from months to hours.
None of this is speculation, because some teams published their diaries. The OPT team (Zhang et al. 2022) released a public logbook alongside the model, recording mid-run hardware swaps, restarts, and instabilities as they happened; the BLOOM project kept similar public chronicles. The honest picture from those documents: a big training run is not one clean curve but a sequence of segments stitched together at checkpoints. Planning for restarts is not pessimism — it is the baseline engineering posture.
Line chart of training loss versus tokens seen, in billions. One line shows a run whose loss falls smoothly from 4.0 to about 2.6, then spikes sharply upward to 3.8 around 93 to 96 billion tokens. A second line branches from the 90-billion-token checkpoint, skipping the suspect batches: it continues smoothly downward to about 2.4, as if the spike never happened.
The playbook that emerges from those logbooks is remarkably consistent. Checkpoint regularly — and remember from last lesson that a resumable checkpoint is the full ledger, weights plus optimizer state. On a spike, wait a moment (many spikes self-recover), and if it does not: roll back to a checkpoint from before the spike, skip the data batches the run was about to consume, and continue. The PaLM team (Chowdhery et al. 2022) described exactly this — restart from an earlier checkpoint, skip the batches around the spike — and noted that the same data did not re-trigger a spike when replayed later in training, pointing at an interaction between particular batches and the particular optimizer state of that moment rather than at simply “bad data”. Why spikes happen remains only partly understood; what to do about them is settled practice. On divergence — the loss climbs and keeps climbing — roll back further and change something real: lower the Learning rate, tighten gradient clipping. A plain replay of a diverging run mostly diverges again.
The early-warning instrument is the gradient-norm plot, watched alongside the loss: it often twitches before the loss curve shows anything. And the quiet enabler of the whole playbook is determinism — a run that can replay its exact data order from any checkpoint is a run you can debug; one that cannot is a run you can only restart and hope.
| Failure | What the metrics show | Standard first response |
|---|---|---|
Transient loss spike | Loss jumps after a smooth decline; the gradient norm often twitches first; may self-recover within a stretch of steps | Watch it. If it recovers, keep going; if not, roll back to a pre-spike Checkpoint and skip the batches from the suspect window |
Divergence | Loss climbs after a spike and keeps climbing; gradient norms stay elevated | Roll back further and change the dynamics — lower the Learning rate, tighten gradient clipping. A plain replay usually diverges again |
NaN loss | Loss becomes NaN in a single step; on FP16, the grad-scaler log often shows repeated scale halvings just before | A numeric overflow somewhere. Resume from the last clean checkpoint; if on FP16, suspect range — BF16 removes the 65,504 ceiling |
Node or link failure | Throughput drops to zero — synchronous training means everyone waits for the missing rank | Automated detection, swap in a healthy node, restart from the newest checkpoint. Lost work = everything since that checkpoint |
Silent data corruption | The hardest case: numbers go wrong with no error raised. Large fleet operators have documented rare CPUs/accelerators that miscompute | Suspect it when failures defy the other rows; countermeasures are redundancy and checksum audits, plus rollback once located |
In production
Checkpointing is where training stability meets the storage bill. The state to save is the full training ledger — weights plus optimizer state, on the order of 12–16 bytes per parameter for mixed-precision Adam (Rajbhandari et al. 2019) — so one checkpoint of a large model weighs from hundreds of gigabytes to terabytes. The dial every team turns: checkpoint too rarely and each failure burns hours of cluster time redoing lost work; too often and you pay in storage capacity, write bandwidth, and steps spent waiting on I/O.
AWS
The durable sink is object storage — Amazon S3 — with the common pattern being a two-tier write: checkpoints land fast on cluster-local or parallel file storage (FSx for Lustre), then copy asynchronously to S3 so the synchronous training step never waits on a multi-terabyte upload. Lifecycle rules do the retention math: keep the last few checkpoints dense for quick restart, thin older ones to every Nth for rollback depth, expire the rest. Managed training-cluster offerings add the other half of the loop — detecting a failed node, swapping in a spare, and resuming automatically from the newest checkpoint — a mechanism exactly as valuable as your cadence is short.
Azure
On Azure the same mechanism runs through Blob Storage as the durable tier, typically fronted by fast scratch or parallel file storage for the hot write path in Azure Machine Learning training jobs. The cadence arithmetic is worth doing explicitly: expected lost work per failure is about half the checkpoint interval, so multiply your measured cluster failure rate by half the interval and by the cluster’s hourly compute cost, then compare against the storage and pause cost of writing more often. The interval falls out of measured numbers — it is an engineering decision, not a habit.
Google Cloud
On Google Cloud, Cloud Storage is the durable sink for runs on Vertex AI training or TPU pods, and the write itself is the part to engineer: sharded checkpoints — each data-parallel or Fully sharded data parallel (FSDP) rank writing its own shard concurrently — turn one enormous serial write into many parallel ones, shrinking the pause a checkpoint imposes on training. Retention needs rollback depth, not just recency: a loss spike is often noticed well after it began, and if only the newest checkpoint survives, the state you need to roll back to may already be gone. Keep a ladder of older checkpoints, not a single latest copy.
Key terms: Mixed precision, BF16 (bfloat16), Loss spike, Checkpoint, Learning rate
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.