From FLOPs to GPU-Hours
Lesson 2 of 3 in Compute Budgets.
C ≈ 6·N·D tells you what the run needs. It says nothing about what you must buy, because no machine delivers its advertised speed on real training work. Every accelerator ships with a peak FLOPs-per-second figure — a spec-sheet maximum, achieved by an ideal benchmark kernel at one particular numeric precision. A training run is not an ideal kernel. It stalls on memory traffic, waits on other GPUs, synchronizes gradients across a network, sits in pipeline bubbles, and re-runs forward math to save memory.
The honest efficiency metric is MFU — Model FLOPs Utilization: the useful model FLOPs your run completes per second (the 6·N·D kind), divided by the hardware’s peak. MFU is honest because it charges you for every inefficiency at once — kernel overheads, communication, stragglers, recomputation — by crediting only math that trains the model. A run at 0.40 MFU converts 40% of what you paid for into learning; the rest is friction.
Bar chart of GPU-hours versus MFU for a fixed compute budget on a hypothetical accelerator with a one-petaFLOP-per-second peak: about 47,000 GPU-hours at 0.50 MFU, 58,000 at 0.40, 78,000 at 0.30, and 117,000 at 0.20.
Where does the missing fraction of peak go? Four sinks take most of it: memory traffic (weights and activations shuttling between memory and compute units while the math sits idle), communication (gradient synchronization in Data parallelism, activation shuffles in Tensor parallelism — the distributed-training module dissects both), bubbles and stragglers (pipeline stages and whole-cluster synchronization waiting on the slowest worker), and recomputation (forward math re-executed to save memory — real FLOPs the hardware performs that MFU refuses to credit; a cousin metric, hardware FLOPs utilization, does count them, which is why HFU always reads higher than MFU for the same run).
And MFU is only the first leak between FLOPs and dollars. The second is failure. At thousands-of-devices scale, hardware faults are routine, so a run that needs 100,000 GPU-hours of math will buy more than 100,000 — some hours go to crashes, restarts, and replaying work since the last Checkpoint. That leak is priced in the next lesson.
From FLOPs to dollars — with the two leaks
- Compute budget C
From C ≈ 6·N·D — a property of the model and dataset, before any hardware is chosen.
- Leak 1: divide by peak × MFU
Peak is precision-dependent spec-sheet fine print; MFU is what your engineering actually achieves. Everything below peak is money spent on friction.
- Ideal GPU-hours
Device-hours of machine time if nothing ever failed: C ÷ (peak × MFU) ÷ 3,600.
- Leak 2: add failures, restarts, replayed work
Crashes discard all work since the last checkpoint; restarts idle the whole fleet while state reloads. The bigger the cluster, the more often something breaks.
- Bought GPU-hours ÷ fleet size = calendar time
The same GPU-hours spread across more devices finish sooner — if MFU survives the extra communication that a bigger fleet requires.
- Dollars: GPU-hours × rate + storage + network
The hourly rate depends on how capacity was procured — the next lesson’s subject.
Key terms: FLOPs, Model FLOPs utilization (MFU), Compute-optimal, Mixed precision, Checkpoint
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.