FLOPs Accounting: C ≈ 6·N·D

Lesson 1 of 3 in Compute Budgets.

Every question in this module reduces to one number: how many floating-point operations (FLOPs) does the training run need? Get that number and everything downstream — GPU-hours, calendar time, dollars — is multiplication and division. Miss it and no spreadsheet can save you.

The estimate is almost embarrassingly simple. A transformer with N parameters trained on D tokens costs approximately

C ≈ 6 · N · D FLOPs

— six floating-point operations per parameter, per token: about 2 in the forward pass and about 4 in the backward pass (Kaplan et al. 2020, §2.1). This one line is the accounting backbone of the scaling-law literature: when a paper calls a model Compute-optimal, C ≈ 6·N·D is the C being optimized.

Log–log line chart of training FLOPs versus model size at a fixed 2-trillion-token dataset. Points at 1, 3, 7, 13, 30, and 70 billion parameters give 1.2×10²², 3.6×10²², 8.4×10²², 1.56×10²³, 3.6×10²³, and 8.4×10²³ FLOPs respectively, forming a straight line.

Training compute C = 6·N·D at a fixed dataset of D = 2×10¹² tokens, for model sizes from 1B to 70B parameters. Each point is the arithmetic C = 6 × N × 2×10¹²; the line is straight on log–log axes because C is linear in N at fixed D. Derived from the C ≈ 6·N·D approximation of Kaplan et al. (2020), §2.1. (calculated — source: Kaplan et al. (2020) — Scaling Laws for Neural Language Models, §2.1)

Raw FLOP counts are unwieldy — nobody budgets in units of 10²². The scaling-law papers work in a friendlier unit: the petaflop/s-day (PF-day) — one machine sustaining 10¹⁵ FLOPs per second for 24 hours, i.e. 10¹⁵ × 86,400 ≈ 8.64×10¹⁹ FLOPs. Our 7B-on-2T example is 8.4×10²² ÷ 8.64×10¹⁹ ≈ 970 PF-days. The unit’s virtue is its shape: machine rate × time is how compute is actually bought, so a PF-day figure already whispers what kind of cluster, and what kind of calendar, a run implies.

Two pieces of fine print before you wield the formula in the wild: Kaplan et al. count N as non-embedding parameters, and C ≈ 6·N·D counts only the model’s core multiply–accumulate work. The deep dive below itemizes exactly what the 6 covers — and what it quietly ignores.

Where the 6 comes from

Forward pass ≈ 2·N FLOPs per token. Nearly all of a transformer’s work is matrix multiplication, and in a matrix multiply each weight participates in one multiply and one add per input — two FLOPs per parameter per token.

Backward pass ≈ 4·N FLOPs per token. Backpropagation computes two gradients at every layer: one with respect to the activations (to keep the chain rule moving backward) and one with respect to the weights (to actually learn). Each costs about as much as the forward pass, so backward ≈ 2 × forward. Total: 2 + 4 = 6 FLOPs per parameter per token, times D tokens (Kaplan et al. 2020, §2.1).

What the approximation knowingly ignores:

  • Attention’s context-dependent work. Computing Attention scores scales with sequence length, not parameter count; Kaplan et al. treat it as a small correction at typical training context lengths relative to model width. Push contexts very long and it stops being negligible.
  • Embedding and unembedding lookups — de-minimis for large models.
  • The optimizer step — a handful of FLOPs per parameter per update, dwarfed by the per-token forward/backward work.
  • Recomputation. Memory-constrained runs often recompute activations during the backward pass (activation checkpointing), re-spending forward FLOPs the formula never counted. The hardware does that work; C ≈ 6·N·D pretends it didn’t happen. The gap between “useful model FLOPs” and “FLOPs the hardware executed” is exactly where the next lesson’s MFU metric lives.

One habit worth stealing from the papers: because C ≈ 6·N·D binds three quantities, fixing any two determines the third. That is what makes the compute-optimal question — for a fixed C, how to split it between N and D — well-posed at all. The Scaling Law Plotter does this arithmetic live against the published fits.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.