Map

BF16 vs FP16

Wiki conceptllmnumericstraining ↳ show in map Markdown
title
BF16 vs FP16
type
concept
summary
BF16 keeps FP32's 8-bit exponent and shrinks the mantissa; FP16 shrinks the exponent and keeps the mantissa — for neural training, range beats precision
tags
llm, numerics, training
sources
habr-local-llm-quantization-deep-dive
created
2026-05-13
updated
2026-05-13

BF16 (bfloat16) and FP16 are both 16-bit floating-point formats. They allocate their bits differently:

  • FP32 — 1 sign, 8 exponent, 23 mantissa
  • FP16 — 1 sign, 5 exponent, 10 mantissa
  • BF16 — 1 sign, 8 exponent, 7 mantissa

FP16 keeps FP32's mantissa precision and halves the exponent range. BF16 keeps FP32's exponent range and halves the mantissa precision. The two are not interchangeable.

Why FP16 stalled training

The original move when GPUs added 16-bit float arithmetic was to use FP16 because it preserved the more "visible" property — significant digits. In practice, neural-network training cares about dynamic range far more than precision:

  • Gradient descent visits magnitudes around $10^{-7}$ to $10^{-8}$. FP16's 5-bit exponent represents values down to ~$6 \times 10^{-5}$ in normal range, with subnormals reaching $\sim 6 \times 10^{-8}$. Gradients smaller than that flush to zero, and the weight update silently does nothing.
  • Backpropagation makes small corrections that depend on small weight differences staying representable. FP16 underflows them.
  • Activation functions can produce extreme values mid-training. FP16's max representable value is ~$6 \times 10^4$; an inner-product overflow lands at $+\infty$, which then propagates as NaN through the loss.

The early workaround was Mixed Precision — store weights in FP32, do most arithmetic in FP16, keep a master copy in FP32, do loss scaling to push gradient magnitudes into FP16's representable range. Real but fragile, and it increased total memory usage compared to pure FP32 because of the master copies.

What BF16 changed

Google's BF16 keeps FP32's exponent (same range from ~$10^{-38}$ to ~$10^{38}$) and trims the mantissa from 23 to 7 bits. Significant-digit precision drops but the gradient flow stays alive without loss scaling. Training proceeds with the same loop you'd write for FP32, just with 16-bit weights.

The implication for inference: every modern open-weight model is published in BF16, not FP16. When you see a model size like "Qwen 3.6 35B = 70 GB at BF16," that's 35B × 2 bytes. The same model in FP16 is also 70 GB but with different precision behavior, and the FP16 weights would have been impossible to train in the first place.

Why this still matters for local LLMs

The choice of BF16 affects what quantization does. Standard llama.cpp quantization formats target the BF16 baseline: KLD measurements (the comparison Vyacheslav's deep-dive uses) are taken against BF16, not against FP16 or FP32. When you read "this quant is +2.1% degraded vs the reference," the reference is the BF16 release.

The 70 GB figure also drives the architectural decisions further down — at 35B BF16, the smallest disk footprint that retains most quality is ~12 GB (UD-Q2_K_XL). That five-and-a-half-times compression is what makes consumer-GPU inference of a 35B model possible at all.

Cross-references

Linked from