Number Formats: FP32, FP16, BF16
Lesson 1 of 3 in Mixed Precision and Stability.
Every number inside a training run — every weight, every gradient, every activation — is a floating-point number stored in a fixed budget of bits. Multiply the budget by the count and precision becomes a systems decision: a model with 70 billion parameters carries hundreds of gigabytes of training state, and every step moves those bytes through memory and across the network. Halve the bits per number and you halve that traffic — and modern accelerators run 16-bit matrix math at a multiple of their FP32 throughput, so the halving is also a speedup. The peak FLOPs figures on accelerator spec sheets — the denominators of your Model FLOPs utilization (MFU) — assume these low-precision paths.
A float is binary scientific notation, and its bits split into three fields. One sign bit. Some exponent bits, which set the range — how large and how small a magnitude the format can represent at all. And some mantissa bits, which set the precision — how many significant digits survive within that range. The entire drama of training precision is the tug-of-war between those two fields inside a fixed 16-bit budget.
Bar chart of bit allocations across three floating-point formats. FP32 has 8 exponent bits and 23 mantissa bits; FP16 has 5 exponent bits and 10 mantissa bits; BF16 has 8 exponent bits and 7 mantissa bits. Each format also carries one sign bit. The chart shows BF16 matching FP32 in exponent width while having the fewest mantissa bits.
FP32 (32 bits: 1 sign, 8 exponent, 23 mantissa) was deep learning’s default: range to about 3.4×10³⁸, roughly seven significant decimal digits. Nobody doubts it is enough — the question is whether it is necessary, at double the memory and a fraction of the matrix-math throughput.
There are two ways to cut a float to 16 bits, and they fail in opposite directions. FP16 (1/5/10) keeps a healthy mantissa but starves the exponent: five bits give it a largest finite value of 65,504 and normal numbers down to only about 6.1×10⁻⁵. BF16 — brain floating point, from the Google Brain team — makes the opposite bet (1/8/7): keep FP32’s full 8-bit exponent, so range runs to ~3.4×10³⁸ exactly as in FP32, and accept a mantissa of seven bits, roughly two to three decimal digits.
Training rewards the BF16 bet, because the two failure modes are not symmetric. Gradients and activations in a large model span many orders of magnitude, and a single value outside FP16’s range becomes infinity, turns into NaN, and poisons the run — a catastrophic, one-way failure. Losing low-order digits, by contrast, adds a little rounding noise to a process that is already noisy: gradients arrive from randomly sampled batches, and small errors largely average out across millions of updates. Range failures kill; precision failures blur. BF16 buys immunity from the killer by paying in blur.
| Format | Bits (sign/exp/mantissa) | Largest finite value | Smallest positive normal | ≈ decimal digits | Where you meet it |
|---|---|---|---|---|---|
FP32 | 1 / 8 / 23 | ≈3.4×10³⁸ | ≈1.2×10⁻³⁸ | ~7 | Master weights, optimizer state, and sensitive ops (Softmax, normalization, loss) |
FP16 | 1 / 5 / 10 | 65,504 | ≈6.1×10⁻⁵ (subnormals reach ≈6×10⁻⁸) | ~3 | The original mixed-precision training format — workable only with loss scaling (next lesson) |
BF16 | 1 / 8 / 7 | ≈3.4×10³⁸ | ≈1.2×10⁻³⁸ | ~2–3 | The default 16-bit compute format for training on modern accelerators |
Why FP16 overflows where BF16 does not
The exponent field stores a power of two, so its width sets the representable magnitudes almost by itself. Five exponent bits give FP16 normal exponents from −14 to +15 (the extreme codes are reserved for subnormals and for infinity/NaN), so its largest finite value is (2 − 2⁻¹⁰) × 2¹⁵ = 65,504. Eight exponent bits give BF16 — like FP32 — exponents from −126 to +127, a ceiling near 3.4×10³⁸. Between five bits and eight bits lies a factor of about 10³⁴ in headroom.
Training traffic actually visits FP16’s edges from both sides. On the high side, intermediate activations and pre-softmax Attention scores can exceed 65,504 in a large model — one such value becomes infinity, the next operation turns it into NaN, and the NaN propagates through the whole computation graph within a step. On the low side, Micikevicius et al. (2017) profiled gradient histograms and found much of the distribution sitting below FP16’s representable range — values that silently flush to zero, which is why FP16 training needs the loss-scaling trick you will meet next lesson.
BF16’s seven mantissa bits mean a relative rounding step of about 2⁻⁸ — roughly 0.4% per stored value, coarse enough that you would never bank in it. Training tolerates it because minibatch noise in the gradients is typically larger than the rounding noise, and because the recipe never lets rounding accumulate where it matters: the authoritative weights stay in FP32. The same trade is now being pushed one step further — newer accelerators offer 8-bit floating-point paths for parts of training and inference, with even less mantissa and even more of the burden shifted onto careful scaling.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.