NVFP4 buys a 6x smaller rounding error than MXFP4 for a quarter of a bit

1. Representation

A floating point number is made of three parts: a sign, an exponent and a mantissa. It works like scientific notation in base 2. For example, $-12$ is stored as $-12=(-1{)}^1\times 1.5\times 2^3$. Figure 1 lets you flip the bits of a few real formats and see the value they build.

The sign gives the direction of the number. Here the sign bit is $s=1$, so the number is negative, while $+12$ would have $s=0$. The exponent gives the scale, which sets the range the number falls in. With $e=3$, the number lies in $[2^3,2^4)=[8,16)$, and $e=10$ would move it up to $[2^{10},2^{11})=[1024,2048)$. The mantissa gives the detail, meaning where exactly the number sits inside that range. With $m=1.5$, it sits halfway through $[8,16)$, which gives $1.5\times 8=12$.

The number of mantissa bits decides how many values each range can hold. With $3$ mantissa bits (as in E4M3), $[8,16)$ is split into $2^3=8$ steps ($8,9,10,\dots ,15$), so $12.4$ cannot be stored exactly and is rounded to $12$. With a single mantissa bit (as in FP4 E2M1), each range holds only $2^1=2$ values, so $[8,16)$ would contain just $8$ and $12$. In real FP4 the largest range is $[4,8)$, which holds only $4$ and $6$. In short, the exponent places the number at the right order of magnitude, and the mantissa pins it down within that magnitude.

FP32 E8M23 BF16 E8M7 FP16 E5M10 FP8 E4M3 FP4 E2M1

? Explain how -12 becomes (-1)^1 x 1.5 x 2^3

Try a number -12 12.4 0.3 9.5

Float explorer

×

How -12 becomes (-1)^1 x 1.5 x 2^3

The sign is the easy part. The other two factors come from one idea: every number is some power of two, times a bit more.

Step 1: find the power of two just below 12. The powers of two are 1, 2, 4, 8, 16, 32, ... The largest one that does not exceed 12 is 8 = 2^3, so the exponent is 3. That is what "e = 3 places the number in [8, 16)" means: 12 lives between 8 and the next power, 16.

Step 2: measure how far past 8 the number is, as a multiplier. 12 / 8 = 1.5. That 1.5 is the mantissa. It is always between 1 and 2, because the number sits between one power of two and the next: a mantissa of 1 means exactly 8, 1.99 means just under 16, and 1.5 means halfway, which is 12.

Put together, 1.5 x 8 = 12 and 8 = 2^3, so 12 = 1.5 x 2^3. Add the sign: -12 = (-1)^1 x 1.5 x 2^3.

Why split it this way. The exponent gives the order of magnitude, cheaply. The mantissa gives the detail within that magnitude, and that is where the bits get spent. With 3 mantissa bits, the span from 8 to 16 is cut into 8 steps of 1, so 8, 9, 10, ..., 15 exist and 12 is one of them. With 1 mantissa bit the same span is cut in 2, so only 8 and 12 exist. Either way 12 survives, but 12.4 falls between steps and gets rounded.

How the bits store it. The exponent is not stored as 3 directly. The format adds a fixed bias so that negative exponents fit in an unsigned field. FP32 has a bias of 127, so the stored exponent is 130, which is why the explorer says "exponent 130 minus bias 127 gives 3". The mantissa stores only the fraction past the leading 1, so 1.5 is stored as 0.5: a single 1 in the first fraction bit, then zeros. That is the exponent field 10000010 and the mantissa field 10000000... you see above.

Try it: 12 then 8 (only the mantissa changes), then 16 (the exponent ticks up and the mantissa resets).

1.2 Common formats

The three formats most used in deep learning divide their bits differently between range and detail, as shown in Figure 2. FP32 (E8M23) is the full-precision reference, reaching about $3.4\times 10^{38}$ with roughly $7$ decimal digits of detail. FP16 (E5M10) halves the size and keeps a good amount of detail by cutting the exponent down to $5$ bits. Its range stops at $65504$, so large values overflow and tiny gradients underflow unless the loss is rescaled. BF16 (E8M7) makes the opposite choice. It keeps the $8$ exponent bits of FP32, and with them the same range, but only $7$ mantissa bits, about $2$ to $3$ decimal digits. Since training cares more about range than about fine detail, BF16 became the standard training format.

Precision grows linearly with the number of mantissa bits (Figure 3), and each extra bit adds about ${\log }_{10}2\approx 0.3$ decimal digits. Range, on the other hand, does not follow this line. BF16 has fewer digits than FP16 but a far larger maximum, because its bits went to the exponent rather than the mantissa.

1.3 MXFP4: microscaling FP4

MXFP4 (Microscaling FP4) is a 4-bit format standardized by the Open Compute Project (OCP).¹ Each value is stored in FP4 (E2M1), with $1$ sign bit, $2$ exponent bits and $1$ mantissa bit, as shown in Figure 4. With so few bits, FP4 alone can only represent the magnitudes $0,0.5,1,1.5,2,3,4,6$, while real tensors often span many orders of magnitude. The key idea of MXFP4 is therefore scaling. Rather than treating each value on its own, it cuts the tensor into blocks of $32$ consecutive values that share one scale factor. Values are divided by the scale before being rounded to FP4 and multiplied back when read, so $\hat{x}=x_{\text{E2M1}}\times s_{\text{block}}$.

The scale is stored in E8M0, an $8$-bit format made only of exponent bits, so it must be a power of two. The standard picks $s_{\text{block}}=2^{\lfloor {\log }_2{\max }_i|x_i|\rfloor -2}$, which places the largest value of the block near the top of the FP4 range. Power-of-two scales are very cheap in hardware, since multiplying by $2^k$ only shifts the exponent, and sharing one scale across $32$ values adds just $8/32=0.25$ bit per value, for about $4.25$ bits in total.

For example, take a block containing $0.3$, $-1.2$, $2.7$ and $9.5$. Its largest magnitude is $9.5$ and $\lfloor {\log }_{2}9.5\rfloor =3$, so the scale is $2^{3-2}=2$. Dividing by $2$ gives $0.15$, $-0.6$, $1.35$ and $4.75$, which round to the FP4 values $0$, $-0.5$, $1.5$ and $4$. Multiplying back by $2$ gives $0$, $-1$, $3$ and $8$. The large values survive roughly intact, but $0.3$ disappears entirely and $9.5$ comes back as $8$. Both errors trace back to the two limits of MXFP4: blocks of $32$ values, where one large value dictates the scale for all the others, and a scale restricted to powers of two. NVFP4 was designed to fix both.

1.4 NVFP4

NVFP4 is NVIDIA’s 4-bit format, introduced with the Blackwell GPUs.² Its values are still FP4 (E2M1) codes, so the difference lies entirely in how they are scaled, which happens on two levels. First, the tensor is cut into blocks of $16$ consecutive values, and each block gets its own scale stored in FP8 (E4M3). Second, the whole tensor gets one FP32 scale that brings its values into the range the E4M3 block scales can express. A value is then recovered as $\hat{x}=x_{\text{E2M1}}\times s_{\text{block}}\times s_{\text{tensor}}$ (Figure 5).

The block scale is chosen so that the largest value of the block lands on the largest FP4 magnitude, $s_{\text{block}}={\max }_i|x_i|/6$. For example, a block whose largest value is $13.3$ gets $s_{\text{block}}=13.3/6\approx 2.22$, which E4M3 stores as $2.25$. The value is then encoded as $13.3/2.25\approx 5.91$, rounded to $6$, and decoded as $6\times 2.25=13.5$, an error of only $0.2$. The storage cost stays small, since one $8$-bit scale shared by $16$ values adds $8/16=0.5$ bit per value, for about $4.5$ bits in total.

1.5 NVFP4 vs MXFP4

NVFP4 builds on MXFP4 and fixes its two main limits (Figure 6). The first is the block size. MXFP4 shares one scale across $32$ values, so a single large value forces a large scale on the whole block and crushes its small values toward zero. NVFP4 halves the block to $16$ values, which gives each scale a more local view of the data and leaves less disparity inside each block.

The second is the scale format. Because E8M0 can only represent powers of two, the ideal scale $2.22$ from the example above must become either $2$ or $4$. With $2$, the value $13.3$ becomes $6.65$ and is clipped to $6$, giving $12$. With $4$, it becomes $3.33$ and is rounded to $3$, which also gives $12$. Either way the error is $1.3$, against $0.2$ with the E4M3 scale of NVFP4. E4M3 covers a much smaller range than E8M0, however, which is why NVFP4 adds the FP32 per-tensor scale on top: the tensor scale restores the global range, while the block scales handle local precision. The price is a slightly larger footprint of about $4.5$ bits per value, against $4+8/32=4.25$ for MXFP4.

2. NVFP4 training recipe

In Pretraining Large Language Models with NVFP4,² NVIDIA pretrained a $12$B-parameter model on $10$ trillion tokens in NVFP4 and almost exactly matched the same model trained in FP8, with $62.58\%$ against $62.62\%$ on MMLU-Pro. Simply switching every layer to FP4 makes training diverge, though. The result relies on four ingredients, each fixing a specific way in which $4$ bits break training.

2.1 Keep sensitive layers in higher precision

Not all layers are equally fragile. The layers at the ends of the network, and above all the last ones that turn the internal representation into the final prediction, need more range and detail than FP4 can offer. The recipe therefore keeps about $16\%$ of the linear layers in BF16, namely the first $2$ blocks and the last $8$ blocks of the $12$B model, and runs everything else in NVFP4. The price is small, since most of the compute still happens in $4$ bits, and it removes the main source of divergence.

2.2 2D block scaling for the weights

During training, each weight matrix $W$ is used twice: as $W$ in the forward pass and as its transpose $W^T$ in the backward pass. With the usual NVFP4 blocks of $1\times 16$ values, a block that is a row of $W$ becomes a column of $W^T$, so the two passes would group, scale and round the weights differently. The gradient would then be computed for slightly different weights than the ones actually used in the forward pass. The fix is to scale the weights in square blocks of $16\times 16$ values (Figure 7). A square block remains the same square after a transpose, so both passes see exactly the same quantized weights. Activations and gradients keep the standard $1\times 16$ blocks.

2.3 Random Hadamard transforms against outliers

Since a block has only one scale, a single outlier sets that scale for all $16$ values and pushes the small ones to zero. A Hadamard transform is a rotation that mixes the values of a block together, spreading the energy of the outlier across all of them. For example, with $4$ values, $[8,0.5,0.5,0.5]$ becomes $[4.75,3.75,3.75,3.75]$. The largest value drops from $16$ times the others to about $1.3$ times, which FP4 represents much better. The rotation is exactly reversible, so no information is lost, and random signs are added to the matrix so that no pattern in the data lines up badly with it. The recipe uses $16\times 16$ transforms and applies them only to the inputs of the weight-gradient computation (Wgrad), the only place where they helped. On the forward pass, they brought no gain.

2.4 Stochastic rounding on gradients

Normal rounding always sends a value to its nearest neighbor, which creates a bias for gradients. FP4 cannot represent $0.2$, so round-to-nearest turns it into $0$ every single time, and small updates silently vanish. Stochastic rounding instead picks one of the two neighbors at random, with the closer one being more likely. Here $0.2$ becomes $0.5$ with probability $0.4$ and $0$ with probability $0.6$, so its average is exactly $0.5\times 0.4=0.2$. Each individual value is noisier, but the errors cancel out over many steps instead of piling up in one direction. The recipe applies it only to gradients. On the forward pass the extra noise would hurt more than the bias does, so weights and activations keep round-to-nearest.

Glossary

• Precision: number of bits used to store a number. Fewer bits means less memory and faster math, but also less information per value.
• FP (floating point): number format made of a sign $s$, an exponent $e$ and a mantissa fraction $f$, with value $x=(-1{)}^s\times 2^{e-b}\times (1+f)$, where $b$ is the bias.
• Sign bit: a single bit giving the direction of the number, with $s=0$ for positive and $s=1$ for negative.
• Exponent: bits giving the scale $2^e$ of the number. More exponent bits means a wider dynamic range.
• Mantissa: bits giving the detail inside a given scale. With $k$ mantissa bits, each range $[2^e,2^{e+1})$ is split into $2^k$ evenly spaced values.
• Bias: constant $b$ subtracted from the stored exponent so that both large numbers and small numbers (negative exponents) can be represented.
• ExMy notation: shorthand for a format with $x$ exponent bits and $y$ mantissa bits, plus the sign bit. For example, E4M3 uses $1+4+3=8$ bits.
• FP32: the $32$-bit float (E8M23), considered standard “full” precision.
• BF16 (Brain Float 16): the $16$-bit format from Google Brain (E8M7). It keeps the same $8$ exponent bits as FP32, and therefore the same range, but has much less detail.
• FP16: the $16$-bit IEEE float (E5M10), which has more detail than BF16 but a much smaller range.
• FP8: the $8$-bit floats, mainly E4M3, which favors detail with a maximum of about $448$, and E5M2, which trades detail for range.³
• FP4 (E2M1): a $4$-bit float with $1$ sign, $2$ exponent and $1$ mantissa bit, which can only represent the magnitudes $0,0.5,1,1.5,2,3,4,6$.
• E8M0: an $8$-bit format made only of exponent bits, so it can only represent powers of two ($2^e$). It is used as the MXFP4 scale.
• Dynamic range: ratio $x_{\max }/x_{\min }$ between the largest and smallest non-zero values a format (or a tensor) covers.
• Exponent vs mantissa trade-off: with a fixed bit budget, every bit given to the exponent (range) is taken away from the mantissa (detail), and vice versa.
• Quantization: converting values from a high-precision format to a lower-precision one, for example from BF16 to FP4. Dequantization is the reverse operation.
• Scale factor: number $s$ that values are divided by before quantization, so that they fit the representable range of the small format, with $q=\mathrm{round}(x/s)$ and later $x\approx q\times s$.
• Block (micro) scaling: splitting a tensor into small blocks of consecutive values, each with its own shared scale factor, instead of using one scale for the whole tensor.
• MXFP4 (Microscaling FP4): OCP standard 4-bit format that stores FP4 (E2M1) values in blocks of $32$, each block sharing one power-of-two (E8M0) scale.
• NVFP4: NVIDIA 4-bit format that stores FP4 (E2M1) values in blocks of $16$, gives each block an FP8 (E4M3) scale, and adds one FP32 scale per tensor.
• Per-tensor scale: single FP32 scale applied to a whole tensor in NVFP4, which brings the tensor into the range where the E4M3 block scales can work.
• Hierarchical (two-level) scaling: the NVFP4 scheme combining the FP32 per-tensor scale with the E4M3 per-block scales, so a value is recovered as $\hat{x}=x_{\text{E2M1}}\times s_{\text{block}}\times s_{\text{tensor}}$.
• Outlier: value much larger in magnitude than the rest of its block. It forces a large scale, which crushes the other values toward zero.
• Sensitive layers: layers where low precision hurts accuracy the most, such as the last layers of the network, which are therefore kept in higher precision (BF16).
• 2D block scaling: scaling weights with 2D blocks of $16\times 16$ instead of 1D blocks, so the weight quantizes the same way in the forward pass and in the backward pass, where it is transposed.
• Random Hadamard Transform (RHT): orthogonal rotation $x^{\prime }=Hx$, using a Hadamard matrix $H$ with random signs, applied before quantization. It spreads outliers across many values so the block distribution becomes smoother, and it can be undone exactly with $x=H^T{x}^{\prime }$.
• Round-to-nearest: deterministic rounding to the closest representable value. Small updates can be systematically lost, which makes it biased.
• Stochastic rounding: rounding up or down at random, with the closer neighbor being more likely. For example, $2.3$ rounds to $2$ with probability $0.7$ and to $3$ with probability $0.3$, so the expected result is exactly $2.3$. This unbiasedness matters for gradients.
• Gradient: direction and size of the weight update computed during backpropagation.
• Mixed precision: using different precisions for different parts of training, for example FP4 matrix multiplications, BF16 sensitive layers, and FP32 master weights and optimizer states.

Footnotes

1. Open Compute Project, OCP Microscaling Formats (MX) Specification, Version 1.0, 2023; Rouhani et al., Microscaling Data Formats for Deep Learning, 2023. ↩

2. NVIDIA, Pretraining Large Language Models with NVFP4, 2025. ↩ ↩²

3. Micikevicius et al., FP8 Formats for Deep Learning, 2022. ↩