# BF16 vs FP16

BF16 (bfloat16) and FP16 are both 16-bit floating-point formats. They allocate their bits differently:

- **FP32** — 1 sign, 8 exponent, 23 mantissa
- **FP16** — 1 sign, 5 exponent, 10 mantissa
- **BF16** — 1 sign, 8 exponent, 7 mantissa

FP16 keeps FP32's mantissa precision and halves the exponent range. BF16 keeps FP32's exponent range and halves the mantissa precision. The two are not interchangeable.

## Why FP16 stalled training

The original move when GPUs added 16-bit float arithmetic was to use FP16 because it preserved the more "visible" property — significant digits. In practice, neural-network training cares about *dynamic range* far more than precision:

- Gradient descent visits magnitudes around $10^{-7}$ to $10^{-8}$. FP16's 5-bit exponent represents values down to ~$6 \times 10^{-5}$ in normal range, with subnormals reaching $\sim 6 \times 10^{-8}$. Gradients smaller than that flush to zero, and the weight update silently does nothing.
- Backpropagation makes small corrections that depend on small weight differences staying representable. FP16 underflows them.
- Activation functions can produce extreme values mid-training. FP16's max representable value is ~$6 \times 10^4$; an inner-product overflow lands at $+\infty$, which then propagates as NaN through the loss.

The early workaround was **Mixed Precision** — store weights in FP32, do most arithmetic in FP16, keep a master copy in FP32, do loss scaling to push gradient magnitudes into FP16's representable range. Real but fragile, and it *increased* total memory usage compared to pure FP32 because of the master copies.

## What BF16 changed

Google's BF16 keeps FP32's exponent (same range from ~$10^{-38}$ to ~$10^{38}$) and trims the mantissa from 23 to 7 bits. Significant-digit precision drops but the gradient flow stays alive without loss scaling. Training proceeds with the same loop you'd write for FP32, just with 16-bit weights.

The implication for inference: every modern open-weight model is published in BF16, not FP16. When you see a model size like "Qwen 3.6 35B = 70 GB at BF16," that's 35B × 2 bytes. The same model in FP16 is *also* 70 GB but with different precision behavior, and the FP16 weights would have been impossible to train in the first place.

## Why this still matters for local LLMs

The choice of BF16 affects what quantization does. Standard llama.cpp quantization formats target the BF16 baseline: KLD measurements (the comparison Vyacheslav's [[habr-local-llm-quantization-deep-dive|deep-dive]] uses) are taken against BF16, not against FP16 or FP32. When you read "this quant is +2.1% degraded vs the reference," the reference is the BF16 release.

The 70 GB figure also drives the architectural decisions further down — at 35B BF16, the smallest disk footprint that retains most quality is ~12 GB (UD-Q2_K_XL). That five-and-a-half-times compression is what makes consumer-GPU inference of a 35B model possible at all.

## Cross-references

- [[llm-quantization]] — what comes after BF16 in the size-reduction chain
- [[habr-local-llm-quantization-deep-dive]] — Vyacheslav's primer where the FP16-vs-BF16 history is laid out
- [[mixture-of-experts]] — MoE total parameter counts are quoted in BF16-equivalent terms