What a Run Costs — the Mechanism

Lesson 3 of 3 in Compute Budgets.

Dollars enter through a short formula with long footnotes:

cost ≈ GPU-hours × hourly rate + storage + networking + retries

The first term dominates, but every term hides decisions. The rate depends on how capacity was bought — on-demand list price, a negotiated long-term commitment, or hardware you own and amortize — and the same GPU-hours can differ by multiples across those choices. Storage is no afterthought at training scale: the corpus, its preprocessed shards, and a months-long trail of large checkpoints all live on high-throughput storage. Networking meters the movement — corpus into the cluster, checkpoints out, replication across zones. And retries is the honest name for everything that never made the final loss curve: crashed jobs, loss spikes rolled back, and the ablation runs that chose the hyperparameters in the first place.

Bar chart of an invented training-budget split: main training run 55%, experiments and ablations 20%, failures and restarts 10%, storage 6%, evaluation runs 5%, networking 4%.

A deliberately toy decomposition of a pre-training budget. The proportions are invented for teaching — real splits vary by team and are rarely published — but the categories are the ones every honest accounting contains: the headline run is not the whole bill. (illustrative — source: Category structure informed by the public OPT-175B logbook (Zhang et al. 2022))

Which hourly rate did they assume?

Public estimates usually multiply GPU-hours by an on-demand list price — the most expensive way anyone buys serious capacity. Labs train on negotiated commitments or owned hardware, where the effective rate is a fraction of list. Same GPU-hours, wildly different dollars.

Which FLOPs did they count?

The final run’s 6·N·D, or everything around it? Ablations, failed attempts, and restarts can add a large fraction on top of the headline compute — and outside estimators cannot see any of it.

Which MFU did they assume?

Outsiders must guess the utilization. Assuming 0.3 versus 0.5 MFU moves the estimated GPU-hours — and thus the estimate — by roughly 1.7× on its own, before any other assumption differs.

What did they leave out entirely?

Storage, networking, evaluation, idle reserved capacity between runs, and the engineers’ salaries. Whether those belong in “the cost of the run” is a framing choice — which is exactly why two good-faith analyses of the same run can differ by several times. Read the assumptions, not the headline.

Checkpointing is insurance — priced like insurance

A Checkpoint is a periodic snapshot of model and optimizer state, and its economics are pure expected value. Checkpoint every T hours, and a crash costs on average T/2 hours of replayed fleet-time plus the restart; checkpoint more often, and you pay more write-stalls and storage instead. The larger the fleet, the more expensive every lost hour — which is why checkpoint frequency tightens as clusters grow. The OPT-175B team’s public logbook (Zhang et al. 2022) is the classic open account of what this looks like in practice: a training diary punctuated by hardware failures and restarts from checkpoints. And the insurance pays double: a run derailed by a Loss spike can be rolled back to a healthy checkpoint and resumed — the mixed-precision-and-stability module picks that story up.

In production

The single biggest lever on the dollars line is not technical — it is how the capacity was procured. Every major cloud prices the identical GPU-hour differently depending on commitment, and a training cluster has a shape (thousands of co-located accelerators on a fast interconnect, held for weeks) that on-demand pools are not built to supply.

AWS

On AWS the mechanism is the trade between on-demand and reserved capacity. On-demand buys flexibility a pre-training run mostly cannot exercise — the job cannot pause when capacity gets scarce mid-run — while capacity reservations and committed-spend agreements trade a fixed term for a lower effective rate and, just as importantly, a guarantee that a co-located, interconnected block of accelerators exists when the run starts. Interruptible spot-style capacity is cheap precisely because it can vanish; it fits aggressively checkpointed ablations, not the headline run.

Azure

Azure exposes the same trade through reservations and committed capacity for GPU virtual-machine families. The discount is the visible part; the operational part is placement — a training run needs its accelerators inside one high-bandwidth interconnect scope, and that topology is what a reservation actually secures. Quota and capacity approvals belong in the budget conversation from day one: the calendar time to obtain a large contiguous block can exceed the time needed to prepare the data.

Google Cloud

Google Cloud’s version is committed-use agreements and capacity reservations, with the same mechanism underneath: committed capacity is cheaper per hour and — decisive for training — schedulable, while on-demand availability of large contiguous accelerator blocks is never guaranteed. Whatever the cloud, the budgeting conclusion is identical: the hourly rate in your spreadsheet is a procurement outcome, not a constant, and the reserved-versus-on-demand decision moves the total more than most single engineering optimizations.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.