codeloopback
Guide contents

Lesson 18 of 73

Number Formats: FP32, FP16, BF16, FP8, INT8, INT4, and FP4

A language model is a very large collection of numbers. These numbers are called parameters or weights. An 8-billion-parameter model has 8 billion of them, and the computer has to keep every one in memory. How many bytes each number takes decides how big the model file is, how much GPU memory it needs, and how fast it can generate text.

A computer cannot store a real number exactly in a fixed number of bits. It stores an approximation, and a number format is the set of rules for that approximation. A format with more bits is more accurate but costs more memory. A format with fewer bits is smaller but rougher. This lesson builds each common format in Java so the reader can see exactly which values it can hold, which values it rounds, and which values it cannot hold at all.

After this lesson, a reader can look at a format name such as BF16, E4M3, or MXFP4 and say how many bits it uses, how large and how small a number it can hold, how precise it is, and how many bytes per parameter it costs.

The table lists the formats in the order the lesson covers them. "Layout" means how many bits go to the sign, the exponent, and the mantissa. Those three parts are explained in Example 1.

Format Bits Layout (sign, exponent, mantissa) Example
FP32 32 1, 8, 23 1, 2
FP16 16 1, 5, 10 3, 4
BF16 16 1, 8, 7 5, 6, 7
FP8 E4M3 8 1, 4, 3 8, 10
FP8 E5M2 8 1, 5, 2 9
INT8 8 integer with a separate scale 11
INT4 4 integer, two per byte 12
FP4 E2M1 4 1, 2, 1 13
MXFP4 4.25 per value E2M1 values, one power-of-two scale per 32 14
NVFP4 4.5 per value E2M1 values, one FP8 scale per 16, one FP32 scale per tensor 15

Java has the float type, which is FP32. Since Java 20 it also has methods that convert to and from FP16. The other formats do not exist in Java, so this lesson implements them with bit operations. Bit operations (>>>, &, |, <<) shift and mask the individual bits of an integer, and they are the tool for building a format by hand.

Example 1: the bits of a float

Scientific notation writes a number such as 6,500 as 6.5 times 10^3. A floating-point number does the same thing in base 2: it stores a sign, a power of two, and the leading digits. The three stored parts have names:

  • The sign is one bit: 0 for positive, 1 for negative.
  • The exponent says which power of two to use.
  • The mantissa (also called the fraction) holds the digits after the leading 1.

In FP32 the 32 bits are split like this:

Text
FP32, 32 bits
 S  EEEEEEEE  MMMMMMMMMMMMMMMMMMMMMMM
 1     8                23
sign exponent          mantissa

A normal value equals (-1)^sign * (1 + mantissa / 2^23) * 2^(exponent - 127). The number 127 is the bias. The exponent field is an unsigned number from 0 to 255, so it cannot be negative on its own. Subtracting 127 lets the same field mean powers of two from about -126 to +127, so both very small and very large values fit.

Here is the formula applied by hand to -2.5. Write -2.5 as -1.25 times 2^1. The sign is 1. The exponent is 1, so the stored field is 1 + 127 = 128. The mantissa is 0.25, so the stored field is 0.25 * 2^23 = 2,097,152, which is 0x200000 in hexadecimal.

The program below splits three floats into these fields and rebuilds each value from them. Look at the sign, exp, and mantissa columns, and check that rebuilt matches the original value.

Java
import java.util.Locale;

public class FloatBits {
    public static void main(String[] args) {
        for (float f : new float[] {1.0f, -2.5f, 0.1f}) {
            int bits = Float.floatToRawIntBits(f);
            int sign = bits >>> 31;
            int exp = (bits >>> 23) & 0xFF;
            int mant = bits & 0x7FFFFF;
            double rebuilt = (sign == 1 ? -1 : 1) * (1 + mant / 8388608.0) * Math.pow(2, exp - 127);
            System.out.printf(Locale.ROOT, "%5.1f  bits %08X  sign %d  exp %3d (2^%d)  mantissa %06X  rebuilt %s%n",
                    f, bits, sign, exp, exp - 127, mant, rebuilt);
        }
    }
}
Output
  1.0  bits 3F800000  sign 0  exp 127 (2^0)  mantissa 000000  rebuilt 1.0
 -2.5  bits C0200000  sign 1  exp 128 (2^1)  mantissa 200000  rebuilt -2.5
  0.1  bits 3DCCCCCD  sign 0  exp 123 (2^-4)  mantissa 4CCCCD  rebuilt 0.10000000149011612

For 1.0, the exponent field is 127, which means 2^0, and the mantissa is zero. For -2.5, the sign is 1, the exponent field is 128 (2^1), and the mantissa is 0x200000, which is a quarter of 2^23. That matches the hand calculation above.

0.1 shows a limit of every binary format. The mantissa has a repeating pattern CCCC... because 0.1 has no finite binary expansion, in the same way that 1/3 has no finite decimal expansion. The stored value is the nearest float, which is slightly above 0.1 (the rebuilt value is 0.10000000149...).

The leading 1 of 1 + mantissa / 2^23 is not stored. It is implied for every normal number, so no bit is spent on it. This gives 24 bits of precision from 23 stored bits.

Example 2: precision and the spacing of floats

A format with a fixed number of bits can only hold a fixed number of different values. Those values are not spread evenly on the number line. Near zero they are packed closely, and for large numbers they are far apart. The distance between one float and the next is called the unit in the last place (ULP). Java's Math.ulp(x) returns it.

The reason the gap grows is the exponent. Between 1 and 2 there are 2^23 evenly spaced floats. Between 2 and 4 there are also 2^23, but they cover twice the distance, so each gap is twice as wide. Every doubling of the value doubles the gap.

The program prints the exact value stored for 0.1f, then the gaps at 1, 1,000, 1,000,000, and 16,777,216. The last line adds 1 to 16,777,216 and checks whether anything changed.

Java
import java.math.BigDecimal;

public class FloatPrecision {
    public static void main(String[] args) {
        System.out.println("0.1f is exactly " + new BigDecimal(0.1f));
        for (float x : new float[] {1f, 1000f, 1_000_000f, 16_777_216f}) {
            System.out.println("ulp(" + x + ") = " + Math.ulp(x));
        }
        System.out.println("16777216f + 1f == 16777216f: " + (16_777_216f + 1f == 16_777_216f));
    }
}
Output
0.1f is exactly 0.100000001490116119384765625
ulp(1.0) = 1.1920929E-7
ulp(1000.0) = 6.1035156E-5
ulp(1000000.0) = 0.0625
ulp(1.6777216E7) = 2.0
16777216f + 1f == 16777216f: true

The gap at 1.0 is 2^-23, about 1.2e-7. This means FP32 keeps about 7 significant decimal digits. At one million, neighboring floats are 0.0625 apart, so a float cannot tell 1,000,000.00 from 1,000,000.03. At 2^24 (16,777,216) the gap is 2, so adding 1 lands halfway between two floats and rounds back to the same value. The last line prints true: the addition changed nothing.

What a floating-point format fixes is relative precision. Every value is stored with about the same number of significant bits, whatever its size. A small number gets a small absolute error and a large number gets a large absolute error, but the error is always a similar fraction of the value.

Example 3: FP16 rounding, maximum, and overflow

FP32 uses 4 bytes per number. Halving that to 2 bytes halves the memory. FP16 (IEEE half precision) is the 16-bit float. It has 1 sign bit, 5 exponent bits with bias 15, and 10 mantissa bits:

Text
FP16, 16 bits
 S  EEEEE  MMMMMMMMMM
 1    5        10

Java 20 added two methods for it. Float.floatToFloat16(float) returns the 16 bits as a short. It rounds to the nearest representable value, and when a value is exactly halfway between two candidates it picks the one whose last bit is 0 (this rule is called "ties to even"). Float.float16ToFloat(short) converts back.

The program converts eight values and prints, for each, the 16 bits in hexadecimal and the value after the round trip. Compare the original and the round-trip columns to see the rounding. Look at the last three rows, which are at and above the maximum. The final line measures the gap between 1.0 and the next FP16 value.

Java
import java.util.Locale;

public class Fp16Basics {
    static float roundTrip(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }

    public static void main(String[] args) {
        for (float x : new float[] {0.1f, 1f / 3, 3.14159265f, 1000.3f, 65504f, 65519f, 65520f, 100000f}) {
            short h = Float.floatToFloat16(x);
            System.out.printf(Locale.ROOT, "%12.7f -> bits %04X -> %s%n", x, h & 0xFFFF, roundTrip(x));
        }
        System.out.println("ulp at 1.0 in FP16: " + (roundTrip(1f + 1f / 1024) - 1f));
    }
}
Output
   0.1000000 -> bits 2E66 -> 0.099975586
   0.3333333 -> bits 3555 -> 0.33325195
   3.1415927 -> bits 4248 -> 3.140625
1000.2999878 -> bits 63D1 -> 1000.5
65504.0000000 -> bits 7BFF -> 65504.0
65519.0000000 -> bits 7BFF -> 65504.0
65520.0000000 -> bits 7C00 -> Infinity
100000.0000000 -> bits 7C00 -> Infinity
ulp at 1.0 in FP16: 9.765625E-4

FP16 has 11 bits of precision (10 stored plus the implied 1), so it keeps about 3 decimal digits. Pi becomes 3.140625, and 1000.3 becomes 1000.5, because the gap near 1000 is 0.5. The last line confirms the gap at 1.0 is 2^-10, about 0.000977.

The maximum is the interesting part. The largest finite FP16 value is 0x7BFF, which equals (2 - 2^-10) * 2^15 = 65504. The next step would be 2^16 = 65536, and the midpoint between 65504 and 65536 is 65520. A value below the midpoint, such as 65519, rounds down to 65504. A value at or above it, such as 65520, rounds to infinity (0x7C00). When a finite number becomes infinity because it is too big, that is called overflow.

Weights are usually small numbers and fit FP16 well. Activations (the intermediate values a model computes while running) can exceed 65504, and Example 7 shows what that does.

Example 4: FP16 subnormals and underflow

Overflow is the problem at the top of the range. The mirror problem is at the bottom: numbers so small that the format cannot hold them. The smallest normal FP16 value is 2^-14, about 6.1e-5. Below that, the format switches to subnormal numbers. In a subnormal number the exponent field is 0, there is no implied leading 1, and the value is mantissa * 2^-24.

Subnormals fill the gap between the smallest normal number and zero with evenly spaced values. The spacing is a fixed 2^-24 (about 6e-8), so the smaller the number, the fewer significant bits it keeps.

The program prints the two boundaries, then converts six small values and prints the bits, the stored value, and the relative error (the error as a percentage of the true value). Watch how the error column changes as the input gets smaller.

Java
import java.util.Locale;

public class Fp16Subnormal {
    static float rt(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }

    public static void main(String[] args) {
        System.out.println("smallest normal    2^-14 = " + rt((float) Math.pow(2, -14)));
        System.out.println("smallest subnormal 2^-24 = " + rt((float) Math.pow(2, -24)));
        for (float x : new float[] {1e-4f, 1e-5f, 1e-6f, 1e-7f, 3e-8f, 2.9e-8f}) {
            float y = rt(x);
            short h = Float.floatToFloat16(x);
            System.out.printf(Locale.ROOT, "%.1e -> bits %04X -> %.6e  relative error %6.2f%%%n",
                    x, h & 0xFFFF, y, 100 * Math.abs(y - x) / x);
        }
    }
}
Output
smallest normal    2^-14 = 6.1035156E-5
smallest subnormal 2^-24 = 5.9604645E-8
1.0e-04 -> bits 068E -> 1.000166e-04  relative error   0.02%
1.0e-05 -> bits 00A8 -> 1.001358e-05  relative error   0.14%
1.0e-06 -> bits 0011 -> 1.013279e-06  relative error   1.33%
1.0e-07 -> bits 0002 -> 1.192093e-07  relative error  19.21%
3.0e-08 -> bits 0001 -> 5.960464e-08  relative error  98.68%
2.9e-08 -> bits 0000 -> 0.000000e+00  relative error 100.00%

1e-4 is a normal number and keeps full precision (0.02% error). From 1e-5 down, the values are subnormal and the exponent bits are 00000. The relative error grows as the value shrinks, because every step is the same size, 2^-24. Here is the arithmetic:

Text
1e-7 / 2^-24 = 1e-7 / 5.96e-8 = 1.68 steps -> rounds to 2 steps
2 steps = 2 * 5.96e-8 = 1.19e-7   (error 19%)

3e-8 is slightly more than half of one step, so it rounds up to one step (error 99%). 2.9e-8 is less than half of one step, so it rounds to zero. A nonzero value that becomes zero is called underflow.

This matters in training. Gradients (the signals that tell a model how to change its weights) are often tiny. Mixed-precision training in FP16 multiplies the loss by a large factor, called "loss scaling", to push small gradients up out of this underflow range.

Example 5: BF16 as the top half of FP32

FP16 overflows at 65504, which caused trouble for large models. BF16 ("brain float 16", introduced by Google) fixes this with a different split of the same 16 bits. It keeps FP32's 8-bit exponent and cuts the mantissa to 7 bits:

Text
FP32:  S EEEEEEEE MMMMMMMMMMMMMMMMMMMMMMM
BF16:  S EEEEEEEE MMMMMMM

Because the layout matches FP32 in the first 16 bits, a BF16 value is the top 16 bits of an FP32 value. Converting BF16 up to FP32 is a shift left by 16 bits. Converting FP32 down to BF16 means dropping the low 16 bits, and there are two ways to do it:

  • Truncate: throw the low 16 bits away.
  • Round to nearest even: add a rounding constant first, then drop the low bits.

The rounding formula in the program is the one that ggml (the C library under llama.cpp) uses in ggml_compute_fp32_to_bf16, and that gguf-py uses in its BF16 class. It adds 0x7FFF plus the lowest kept bit, then shifts right by 16. Adding 0x7FFF carries into the kept bits exactly when the dropped part is more than half. The lowest kept bit makes exact halves round to the even neighbor.

The program converts six values both ways. Compare the "trunc value" and "rne value" (round to nearest even) columns.

Java
import java.util.Locale;

public class Bf16Convert {
    static int truncate(float f) { return Float.floatToRawIntBits(f) >>> 16; }

    static int roundNearestEven(float f) {
        int bits = Float.floatToRawIntBits(f);
        if ((bits & 0x7FFFFFFF) > 0x7F800000) return (bits >>> 16) | 64;     // NaN stays NaN (quiet)
        return (bits + 0x7FFF + ((bits >>> 16) & 1)) >>> 16;
    }

    static float toFloat(int bf16) { return Float.intBitsToFloat(bf16 << 16); }

    public static void main(String[] args) {
        float[] xs = {3.14159265f, 0.1f, 1.00390625f, 1.01171875f, 65520f, 3.0e38f};
        System.out.println("value          trunc bits  trunc value   rne bits  rne value");
        for (float x : xs) {
            int t = truncate(x), r = roundNearestEven(x);
            System.out.printf(Locale.ROOT, "%-14s %04X        %-12s  %04X      %s%n",
                    Float.toString(x), t, Float.toString(toFloat(t)), r, Float.toString(toFloat(r)));
        }
    }
}
Output
value          trunc bits  trunc value   rne bits  rne value
3.1415927      4049        3.140625      4049      3.140625
0.1            3DCC        0.099609375   3DCD      0.100097656
1.0039062      3F80        1.0           3F80      1.0
1.0117188      3F81        1.0078125     3F82      1.015625
65520.0        477F        65280.0       4780      65536.0
3.0E38         7F61        2.990763E38   7F62      3.0040553E38

BF16 has 8 bits of precision (7 stored), so its step at 1.0 is 2^-7 = 0.0078125. Now follow the tie cases:

  • 1.00390625 is exactly halfway between 1.0 and 1.0078125. Round-to-nearest-even picks 1.0, because its last mantissa bit is 0 (even).
  • 1.01171875 is exactly halfway between 1.0078125 (last bit 1, odd) and 1.015625 (last bit 0, even). Rounding goes up to 1.015625, while truncation goes down to 1.0078125.
  • For 0.1, truncation gives 0.0996 and rounding gives 0.1001. The rounded value is closer to 0.1.

Truncation always moves values toward zero, which is a systematic bias. Over millions of weights, that bias adds up, so converters round. The last two rows show BF16's main advantage: 65520 and 3.0e38 are stored without overflow, because the exponent range equals FP32's.

Example 6: range against precision, FP16 and BF16 side by side

FP16 and BF16 spend the same 16 bits differently. FP16 spends 10 bits on the mantissa, so it has more precision (finer steps). BF16 spends 8 bits on the exponent, so it has more range (a wider span from smallest to largest). Neither is better in general. Each fails in a different place.

The program stores the same six values in both formats and prints the relative error, or the words overflow and underflow when the value cannot be stored. Read the table row by row.

Java
import java.util.Locale;

public class RangeVsPrecision {
    static float fp16(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }
    static float bf16(float x) {
        int b = Float.floatToRawIntBits(x);
        return Float.intBitsToFloat(((b + 0x7FFF + ((b >>> 16) & 1)) >>> 16) << 16);
    }
    static String err(float x, float y) {
        if (Float.isInfinite(y)) return "overflow";
        if (y == 0 && x != 0) return "underflow";
        return String.format(Locale.ROOT, "%.4f%%", 100 * Math.abs(y - x) / Math.abs(x));
    }

    public static void main(String[] args) {
        System.out.println("value        FP16          error      BF16          error");
        for (float x : new float[] {1e-8f, 1e-6f, 0.1f, 1000.5f, 70000f, 1e10f}) {
            float a = fp16(x), b = bf16(x);
            System.out.printf(Locale.ROOT, "%-12s %-13s %-10s %-13s %s%n",
                    Float.toString(x), Float.toString(a), err(x, a), Float.toString(b), err(x, b));
        }
    }
}
Output
value        FP16          error      BF16          error
1.0E-8       0.0           underflow  1.0011718E-8  0.1172%
1.0E-6       1.013279E-6   1.3279%    9.983778E-7   0.1622%
0.1          0.099975586   0.0244%    0.100097656   0.0977%
1000.5       1000.5        0.0000%    1000.0        0.0500%
70000.0      Infinity      overflow   70144.0       0.2057%
1.0E10       Infinity      overflow   9.999221E9    0.0078%

Look at the rows in groups:

  • 1.0E-8 and 1.0E-6 are tiny. FP16 underflows to zero for the first and has 1.33% error for the second. BF16 stores both with error below 0.2%.
  • 0.1 and 1000.5 are in the comfortable middle. FP16 is more accurate here: 1000.5 is exact in FP16 but becomes 1000.0 in BF16.
  • 70000 and 1.0E10 are large. FP16 overflows to infinity for both. BF16 stores them with small errors.

For values between about 6e-5 and 65504, FP16 is up to 8 times more precise than BF16 (8 times because it has 3 more mantissa bits, and 2^3 is 8). Outside that window FP16 fails. BF16 covers the same range as FP32 (about 1e-38 to 3e38) with a nearly constant relative error below 0.4%.

Large models are trained in BF16 because an overflowing activation breaks the whole model, while slightly coarser values usually do not.

Example 7: a large activation that overflows FP16 but not BF16

Some models produce a few very large activation values in specific channels. These are often called activation outliers, and they can reach magnitudes in the hundreds or thousands. Outliers are a real problem for FP16 because computing with them involves squares and sums, which grow quickly.

One place where this happens is RMSNorm, from lesson 8. RMSNorm (root mean square normalization) rescales a vector so its values have a typical size of about 1. To do that, it first adds up the square of every value in the vector, then takes the square root of the average.

The program builds a 4,096-value activation. Most values are ordinary numbers between -1.5 and 1.5. One value, at index 1234, is an outlier of 300. The program adds up the squares in three ways: in FP32, in FP16, and in BF16. In the FP16 and BF16 versions it rounds after every multiplication and every addition, the way a computation kernel that works in that format would.

First, the expected sizes. The ordinary values have an average square of 1, so 4,096 of them add up to about 4,096. The outlier alone adds 300 * 300 = 90,000.

Java
import java.util.Locale;

public class Fp16Overflow {
    static float fp16(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }
    static float bf16(float x) {
        int b = Float.floatToRawIntBits(x);
        return Float.intBitsToFloat(((b + 0x7FFF + ((b >>> 16) & 1)) >>> 16) << 16);
    }

    public static void main(String[] args) {
        float[] x = new float[4096];
        for (int i = 0; i < x.length; i++) x[i] = (i % 7 - 3) * 0.5f;    // ordinary values in [-1.5, 1.5]
        x[1234] = 300f;                                                    // one outlier channel

        float s32 = 0, s16 = 0, sbf = 0;
        for (float v : x) {
            s32 += v * v;
            s16 = fp16(s16 + fp16(v * v));
            sbf = bf16(sbf + bf16(v * v));
        }
        float n = x.length;
        System.out.printf(Locale.ROOT, "sum of squares FP32 = %.1f%n", s32);
        System.out.println("sum of squares FP16 = " + s16);
        System.out.println("sum of squares BF16 = " + sbf);
        System.out.printf(Locale.ROOT, "RMS FP32 = %.4f, FP16 = %s, BF16 = %.4f%n",
                Math.sqrt(s32 / n), Math.sqrt(s16 / n), Math.sqrt(sbf / n));
        System.out.println("normalized x[0] with FP16 RMS = " + (x[0] / (float) Math.sqrt(s16 / n)));
    }
}
Output
sum of squares FP32 = 94097.0
sum of squares FP16 = Infinity
sum of squares BF16 = 91136.0
RMS FP32 = 4.7930, FP16 = Infinity, BF16 = 4.7170
normalized x[0] with FP16 RMS = -0.0

The FP32 sum is 94,097, which is close to 4,096 + 90,000, as expected. The FP16 sum is infinity. As soon as the running total passes 65,504, which happens when the outlier's square (90,000) is added, the next rounding produces infinity. Once the sum is infinity, the RMS is infinity, and every normalized value is x / infinity, which is zero. The last line shows -0.0: the layer's output is destroyed.

BF16 keeps a finite sum, 91,136, which is about 3% below the true value. This is because each addition rounds to 8 bits of precision, and the rounding errors add up. The RMS is about 1.6% low, which a model usually tolerates.

Engines avoid the problem by accumulating in FP32 or wider, even when the stored data is FP16 or BF16. ggml's CPU backend sums the squares for RMSNorm in double (its ggml_float type). Its dot-product kernels accumulate in FP32 or wider, and in integers for quantized blocks.

Example 8: FP8 E4M3, the encoder and decoder

The next step down is 8 bits per number. FP8 formats store a number in one byte. Two formats are used for inference. Both are defined in the Open Compute Project (OCP) 8-bit floating point specification and are implemented in NVIDIA Hopper and Blackwell GPUs.

The name E4M3 says how the byte is split: 4 exponent bits (E4) and 3 mantissa bits (M3), plus 1 sign bit:

Text
E4M3, 8 bits
 S  EEEE  MMM
 1   4     3

The exponent bias is 7. The format has some special rules:

  • It has no infinity. The code S.1111.111 (all exponent and mantissa bits set) is NaN (not a number). Every other code with exponent 15 is a normal number. This extends the maximum to 1.75 * 2^8 = 448.
  • Subnormals (exponent 0) have the value mantissa / 8 * 2^-6.

A worked example: the code 0x38 is 0 0111 000. The exponent field is 7, so the power is 2^(7-7) = 1. The mantissa is 0, so the value is 1.0. The largest code, 0x7E, is 0 1111 110. The exponent field is 15 (power 2^8 = 256) and the mantissa is 6, so the value is (1 + 6/8) * 256 = 448.

With only 256 codes, the program decodes every code by formula. To encode, it tries all 256 codes and picks the nearest finite value, with ties going to the code whose last bit is 0 (round to nearest even). The program prints the maximum, the smallest normal and subnormal values, the grid of values between 0.5 and 2, and the result of encoding seven values.

Java
import java.util.Locale;

public class Fp8E4M3 {
    static float decode(int code) {
        int s = (code >> 7) & 1, e = (code >> 3) & 0xF, m = code & 7;
        if (e == 15 && m == 7) return Float.NaN;
        float v = (e == 0) ? m / 8f * (float) Math.pow(2, -6)
                           : (1 + m / 8f) * (float) Math.pow(2, e - 7);
        return s == 1 ? -v : v;
    }

    static int encode(float x) {                  // nearest finite value, ties to even, saturating
        int best = 0;
        float bestErr = Float.MAX_VALUE;
        for (int c = 0; c < 256; c++) {
            float v = decode(c);
            if (Float.isNaN(v)) continue;
            float err = Math.abs(v - x);
            if (err < bestErr || (err == bestErr && (c & 1) == 0 && (best & 1) == 1)) { best = c; bestErr = err; }
        }
        return best;
    }

    public static void main(String[] args) {
        System.out.println("max = " + decode(0x7E) + ", min normal = " + decode(0x08) + ", min subnormal = " + decode(0x01));
        StringBuilder grid = new StringBuilder("grid from 0.5 to 2:");
        for (int c = 0x30; c <= 0x40; c++) grid.append(' ').append(decode(c));
        System.out.println(grid);
        for (float x : new float[] {1.06f, 1.07f, 0.3f, 300f, 500f, -2.2f, 0.001f}) {
            int c = encode(x);
            System.out.printf(Locale.ROOT, "%7.3f -> 0x%02X -> %s%n", x, c, decode(c));
        }
    }
}
Output
max = 448.0, min normal = 0.015625, min subnormal = 0.001953125
grid from 0.5 to 2: 0.5 0.5625 0.625 0.6875 0.75 0.8125 0.875 0.9375 1.0 1.125 1.25 1.375 1.5 1.625 1.75 1.875 2.0
  1.060 -> 0x38 -> 1.0
  1.070 -> 0x39 -> 1.125
  0.300 -> 0x2A -> 0.3125
300.000 -> 0x79 -> 288.0
500.000 -> 0x7E -> 448.0
 -2.200 -> 0xC1 -> -2.25
  0.001 -> 0x01 -> 0.001953125

The first line shows the range: the largest value is 448, the smallest normal is 2^-6 = 0.015625, and the smallest subnormal is 2^-9 = 0.001953125.

The grid line shows the spacing. Between 1 and 2 there are 8 values, 0.125 apart. Between 0.5 and 1 there are 8 values, 0.0625 apart. Every power of two holds 8 steps, so the gap doubles each time the value doubles. The relative step is 12.5% at worst, so E4M3 keeps about one significant decimal digit.

Now the encoded values:

  • 1.060 is 0.06 from 1.0 and 0.065 from 1.125, so it becomes 1.0. 1.070 is closer to 1.125, so it becomes 1.125.
  • Between 256 and 448 the step is 32, so 300 becomes 288.
  • 500 is beyond the range. It saturates to 448, which means it is clamped to the largest value instead of becoming NaN. The OCP specification defines both a saturating conversion and one that produces NaN for out-of-range values. Inference libraries use the saturating one together with a scale (Example 10).
  • 0.001 is a subnormal. It is below the smallest subnormal's midpoint region and rounds up to the smallest one, 2^-9.

The search over 256 codes is slow, but it is clearly correct. That makes it a useful reference for testing a fast encoder that uses bit manipulation.

Example 9: FP8 E5M2, more range and coarser steps

E5M2 is the other FP8 format. It trades one mantissa bit for one exponent bit: 1 sign bit, 5 exponent bits with bias 15, and 2 mantissa bits.

Text
E5M2, 8 bits
 S  EEEEE  MM
 1    5     2

It follows IEEE rules, the same rules as FP16: exponent 31 means infinity (when the mantissa is 0) or NaN (otherwise). The maximum finite value is 1.75 * 2^15 = 57344. The first 8 bits of an FP16 value, the sign, all 5 exponent bits, and the top 2 mantissa bits, form an E5M2 byte, so E5M2 is the top byte of an FP16 value.

The program decodes the format and encodes the same seven values as Example 8, so the two outputs can be compared. The last line checks the top-byte relation for 1.5.

Java
import java.util.Locale;

public class Fp8E5M2 {
    static float decode(int code) {
        int s = (code >> 7) & 1, e = (code >> 2) & 0x1F, m = code & 3;
        if (e == 31) return m == 0 ? (s == 1 ? Float.NEGATIVE_INFINITY : Float.POSITIVE_INFINITY) : Float.NaN;
        float v = (e == 0) ? m / 4f * (float) Math.pow(2, -14)
                           : (1 + m / 4f) * (float) Math.pow(2, e - 15);
        return s == 1 ? -v : v;
    }

    static int encode(float x) {                  // nearest finite value, ties to even, saturating
        int best = 0;
        float bestErr = Float.MAX_VALUE;
        for (int c = 0; c < 256; c++) {
            float v = decode(c);
            if (Float.isNaN(v) || Float.isInfinite(v)) continue;
            float err = Math.abs(v - x);
            if (err < bestErr || (err == bestErr && (c & 1) == 0 && (best & 1) == 1)) { best = c; bestErr = err; }
        }
        return best;
    }

    public static void main(String[] args) {
        System.out.println("max = " + decode(0x7B) + ", min normal = " + decode(0x04) + ", min subnormal = " + decode(0x01));
        StringBuilder grid = new StringBuilder("grid from 0.5 to 2:");
        for (int c = 0x38; c <= 0x40; c++) grid.append(' ').append(decode(c));
        System.out.println(grid);
        for (float x : new float[] {1.06f, 1.07f, 0.3f, 300f, 500f, -2.2f, 0.001f}) {
            int c = encode(x);
            System.out.printf(Locale.ROOT, "%7.3f -> 0x%02X -> %s%n", x, c, decode(c));
        }
        short h = Float.floatToFloat16(1.5f);
        System.out.printf("FP16 bits of 1.5 = %04X, top byte = %02X = E5M2 code of 1.5 (%02X)%n",
                h, (h >> 8) & 0xFF, encode(1.5f));
    }
}
Output
max = 57344.0, min normal = 6.1035156E-5, min subnormal = 1.5258789E-5
grid from 0.5 to 2: 0.5 0.625 0.75 0.875 1.0 1.25 1.5 1.75 2.0
  1.060 -> 0x3C -> 1.0
  1.070 -> 0x3C -> 1.0
  0.300 -> 0x35 -> 0.3125
300.000 -> 0x5D -> 320.0
500.000 -> 0x60 -> 512.0
 -2.200 -> 0xC0 -> -2.0
  0.001 -> 0x14 -> 9.765625E-4
FP16 bits of 1.5 = 3E00, top byte = 3E = E5M2 code of 1.5 (3E)

E5M2 has 4 steps per power of two instead of 8. Compare with Example 8:

  • 1.07 rounds to 1.0 (E4M3 gave 1.125), and -2.2 rounds to -2.0 (E4M3 gave -2.25).
  • 300 becomes 320 instead of 288, because the steps at that size are 64 apart.
  • 500 is stored as 512 instead of saturating, and 0.001 is a normal number instead of a subnormal.

In exchange for coarser steps, E5M2 covers about 1.5e-5 to 57,344. The usual split, described in NVIDIA's Transformer Engine documentation, is E4M3 for weights and activations, where precision matters more, and E5M2 for gradients in training, where range matters more.

The last line, FP16 bits of 1.5 = 3E00, top byte = 3E, shows that the E5M2 code of 1.5 is the high byte of the FP16 bits. E5M2 is FP16 with the low byte removed, the same relation that BF16 has with FP32.

Example 10: scaling a tensor into FP8 range

A byte can only hold 256 different values, and E4M3's values sit between about 0.002 and 448. Real weights are often much smaller than that, for example around 0.002. Those weights would land in the subnormal region, where steps are large relative to the values, and many would round to zero.

The fix is a scale. A scale is a single float stored next to the encoded values. Before encoding, each value is divided by the scale. After decoding, each value is multiplied by the scale. If the scale is chosen well, the divided values use the format's full range.

A tensor is a block of numbers, such as all the weights of one layer. A per-tensor scale for E4M3 is usually amax / 448, where amax is the largest absolute value in the tensor. After dividing, the largest value becomes exactly 448, the largest code.

The program creates 1,000 small random weights (standard deviation 0.002), then encodes them twice: once directly, and once with a scale. It reports the mean absolute error and, for the unscaled version, how many values became zero.

Java
import java.util.Locale;
import java.util.Random;

public class Fp8Scaling {
    static float decode(int code) {
        int s = (code >> 7) & 1, e = (code >> 3) & 0xF, m = code & 7;
        if (e == 15 && m == 7) return Float.NaN;
        float v = (e == 0) ? m / 8f * (float) Math.pow(2, -6) : (1 + m / 8f) * (float) Math.pow(2, e - 7);
        return s == 1 ? -v : v;
    }
    static int encode(float x) {
        int best = 0; float bestErr = Float.MAX_VALUE;
        for (int c = 0; c < 256; c++) {
            float v = decode(c);
            if (Float.isNaN(v)) continue;
            float err = Math.abs(v - x);
            if (err < bestErr || (err == bestErr && (c & 1) == 0 && (best & 1) == 1)) { best = c; bestErr = err; }
        }
        return best;
    }

    public static void main(String[] args) {
        Random rnd = new Random(1);
        float[] w = new float[1000];
        for (int i = 0; i < w.length; i++) w[i] = (float) (rnd.nextGaussian() * 0.002);   // small weights
        float amax = 0;
        for (float v : w) amax = Math.max(amax, Math.abs(v));
        float scale = amax / 448f;

        double errRaw = 0, errScaled = 0; int zerosRaw = 0;
        for (float v : w) {
            float raw = decode(encode(v));
            float scaled = decode(encode(v / scale)) * scale;
            if (raw == 0) zerosRaw++;
            errRaw += Math.abs(raw - v); errScaled += Math.abs(scaled - v);
        }
        System.out.printf(Locale.ROOT, "amax = %.6f, scale = %.3e%n", amax, scale);
        System.out.printf(Locale.ROOT, "no scale:   mean abs error %.3e, values stored as 0: %d of %d%n",
                errRaw / w.length, zerosRaw, w.length);
        System.out.printf(Locale.ROOT, "with scale: mean abs error %.3e%n", errScaled / w.length);
    }
}
Output
amax = 0.007843, scale = 1.751e-05
no scale:   mean abs error 4.923e-04, values stored as 0: 378 of 1000
with scale: mean abs error 3.647e-05

The first line shows the largest weight, 0.007843, and the scale 0.007843 / 448 = 1.751e-05.

Without a scale, the weights sit in E4M3's subnormal range, where steps are 0.00195, about the size of the weights themselves. 378 of the 1,000 weights become zero, and the mean error is 4.9e-4. With the scale, the largest weight maps to 448 and the others spread over the normal range with the full 3 bits of mantissa precision. The mean error drops to 3.6e-5, which is more than 13 times smaller.

The scale costs 4 bytes for the whole tensor, which is almost nothing. Lesson 23 covers per-channel and dynamic scales in FP8 serving.

Example 11: INT8 with a scale, and an integer dot product

Integer formats work differently from floats. INT8 stores whole numbers from -128 to 127 in one byte. To store real numbers, the program uses a scale again. Each value is divided by the scale and rounded to the nearest integer:

Text
q = round(x / scale),   scale = amax / 127

Choosing scale = amax / 127 makes the largest value map to 127. To get a value back, multiply the integer by the scale. The spacing is the same everywhere (one scale unit), unlike floats, where the spacing grows with the value.

By hand, with amax = 2.40: the scale is 2.40 / 127 = 0.018898, and the value 0.50 becomes round(0.50 / 0.018898) = round(26.46) = 26.

INT8 is useful for inference speed. A dot product multiplies two vectors element by element and adds the results. It is the core operation of a model. With INT8, the multiplications and additions happen on integers, which CPUs and GPUs do quickly. The program keeps the running sum in an int, then multiplies the final integer by both scales once at the end.

The program does this for two 8-element vectors. It prints the INT8 codes, the scales, the integer sum, the INT8 dot product, and the exact float dot product. The last line checks that an int accumulator cannot overflow for 4,096 elements.

Java
import java.util.Locale;

public class Int8Dot {
    static float scaleOf(float[] v) {
        float amax = 0;
        for (float x : v) amax = Math.max(amax, Math.abs(x));
        return amax / 127f;
    }
    static byte[] quantize(float[] v, float scale) {
        byte[] q = new byte[v.length];
        for (int i = 0; i < v.length; i++) q[i] = (byte) Math.round(v[i] / scale);
        return q;
    }

    public static void main(String[] args) {
        float[] a = {0.50f, -1.20f, 0.03f, 2.40f, -0.75f, 1.10f, -2.00f, 0.25f};
        float[] b = {1.00f, 0.40f, -0.90f, 0.10f, 0.60f, -1.50f, 0.20f, 0.80f};
        float sa = scaleOf(a), sb = scaleOf(b);
        byte[] qa = quantize(a, sa), qb = quantize(b, sb);

        int acc = 0;
        for (int i = 0; i < a.length; i++) acc += qa[i] * qb[i];       // integer multiply-add
        float approx = acc * sa * sb;

        float exact = 0;
        for (int i = 0; i < a.length; i++) exact += a[i] * b[i];
        System.out.println("qa = " + java.util.Arrays.toString(qa));
        System.out.println("qb = " + java.util.Arrays.toString(qb));
        System.out.printf(Locale.ROOT, "scales %.6f %.6f, int sum %d%n", sa, sb, acc);
        System.out.printf(Locale.ROOT, "int8 dot = %.5f, float dot = %.5f%n", approx, exact);
        System.out.println("max |sum| for n=4096: " + 127L * 127 * 4096 + " (int max " + Integer.MAX_VALUE + ")");
    }
}
Output
qa = [26, -64, 2, 127, -40, 58, -106, 13]
qb = [85, 34, -76, 8, 51, -127, 17, 68]
scales 0.018898 0.011811, int sum -9426
int8 dot = -2.10389, float dot = -2.06700
max |sum| for n=4096: 66064384 (int max 2147483647)

Both vectors use their own scale (0.018898 and 0.011811). The largest value of each vector becomes 127 (a) or -127 (b). The integer sum is -9426. Multiplying by both scales gives -9426 * 0.018898 * 0.011811 = -2.10389.

The integer sum is exact. The only errors come from rounding each value to its INT8 code. Each code is off by up to half a step (0.0094 for a, 0.0059 for b), and those errors add up over the products, so the result differs from the float dot product (-2.06700) by about 1.8%.

The last line shows why an int is enough. Even if every product were 127 * 127, 4,096 of them add up to 66 million, far below the int maximum of 2.1 billion. AVX-512 VNNI (on x86 CPUs) and sdot (on ARM CPUs) are instructions that multiply groups of 8-bit values and add them into 32-bit lanes in one step.

Math.round rounds halves up (toward positive infinity), which differs from C's roundf (halves away from zero) for negative halves. Lesson 21 uses a rounding helper that matches ggml exactly.

Example 12: INT4, two values in one byte

Going from 8 to 4 bits means each value has only 16 possible codes. INT4 stores integers in 4 bits. A group of 4 bits is called a nibble. Java has no 4-bit type, so two nibbles are packed into one byte: one in the low 4 bits and one in the high 4 bits.

Text
byte:  [ high nibble | low nibble ]
bits:   7 6 5 4        3 2 1 0

There are two ways to read a nibble as a number:

  • Signed (two's complement): the nibble holds -8 to 7. The top bit of the nibble is the sign. For example, 1000 is -8 and 1111 is -1.
  • Unsigned with an offset: the nibble holds 0 to 15, and the reader subtracts 8 to get -8 to 7. This is the convention ggml uses for Q4_0, its 4-bit block format, which is built in lesson 21.

The program packs pairs of values and unpacks them both ways. The first three lines use the signed form. The last two lines use the offset form, and show the nibbles before and after subtracting 8.

Java
public class Int4Pack {
    static byte pack(int lo, int hi) { return (byte) ((lo & 0x0F) | ((hi & 0x0F) << 4)); }
    static int loSigned(byte b) { return (b << 28) >> 28; }             // sign-extend bits 0..3
    static int hiSigned(byte b) { return (b << 24) >> 28; }             // sign-extend bits 4..7
    static int loUnsigned(byte b) { return b & 0x0F; }
    static int hiUnsigned(byte b) { return (b >> 4) & 0x0F; }

    public static void main(String[] args) {
        int[][] pairs = {{-8, 7}, {-1, 1}, {3, -5}};
        for (int[] p : pairs) {
            byte b = pack(p[0], p[1]);
            System.out.printf("signed (%2d, %2d) -> byte %02X -> signed (%2d, %2d)%n",
                    p[0], p[1], b & 0xFF, loSigned(b), hiSigned(b));
        }
        // ggml-style: store q + 8 in 0..15, subtract 8 when reading
        int[][] values = {{-8, 7}, {-1, 1}};
        for (int[] v : values) {
            byte b = (byte) ((v[0] + 8) | ((v[1] + 8) << 4));
            System.out.printf("offset (%2d, %2d) -> byte %02X -> nibbles (%2d, %2d) -> minus 8 (%2d, %2d)%n",
                    v[0], v[1], b & 0xFF, loUnsigned(b), hiUnsigned(b), loUnsigned(b) - 8, hiUnsigned(b) - 8);
        }
        System.out.println("8 values in " + 8 / 2 + " bytes; 4096 values in " + 4096 / 2 + " bytes");
    }
}
Output
signed (-8,  7) -> byte 78 -> signed (-8,  7)
signed (-1,  1) -> byte 1F -> signed (-1,  1)
signed ( 3, -5) -> byte B3 -> signed ( 3, -5)
offset (-8,  7) -> byte F0 -> nibbles ( 0, 15) -> minus 8 (-8,  7)
offset (-1,  1) -> byte 97 -> nibbles ( 7,  9) -> minus 8 (-1,  1)
8 values in 4 bytes; 4096 values in 2048 bytes

Follow the first pair (-8, 7) by hand. -8 & 0x0F is 1000 (8). 7 is 0111. The high nibble goes to bits 4 to 7, so the byte is 0111 1000 = 0x78. Reading it back, the low nibble 1000 is sign-extended to -8, and the high nibble 0111 is 7.

Sign extension shifts the nibble to the top of a 32-bit int and back with >>, which copies the sign bit into the empty positions. The offset form avoids sign handling: the stored nibble is 0 to 15, and the reader subtracts 8. For the offset pair (-8, 7), the stored nibbles are 0 and 15, and the byte is 0xF0.

Both forms encode the same 16 values with different bit patterns. For example, -8 is 1000 in two's complement and 0000 in the offset form. A reader must therefore use the same convention as the writer, or every value will be wrong. The last line shows the saving: 4,096 values take 2,048 bytes.

Example 13: FP4 E2M1, all 16 values

A 4-bit format has only 16 codes, so it is possible to list every value. FP4 E2M1 is a 4-bit float with 1 sign bit, 2 exponent bits (bias 1), and 1 mantissa bit, as defined in the OCP Microscaling (MX) specification.

Text
E2M1, 4 bits
 S  EE  M
 1   2  1

With exponent 0, the value is subnormal: mantissa * 0.5. Otherwise it is (1 + mantissa / 2) * 2^(exponent - 1). There is no infinity and no NaN. For example, code 5 is 0 10 1: exponent 2, mantissa 1, so the value is 1.5 * 2^1 = 3.0.

The program prints all 16 codes with their values. It also compares them with the kvalues_fp4 table in ggml's ggml-common.h. That table stores each value doubled as an integer, so that it fits in an int8_t.

Java
public class Fp4E2M1 {
    static float decode(int code) {
        int s = (code >> 3) & 1, e = (code >> 1) & 3, m = code & 1;
        float v = (e == 0) ? m * 0.5f : (1 + m / 2f) * (float) Math.pow(2, e - 1);
        return s == 1 ? -v : v;
    }

    public static void main(String[] args) {
        int[] ggmlDoubled = {0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12};   // kvalues_fp4
        boolean allMatch = true;
        for (int c = 0; c < 16; c++) {
            float v = decode(c);
            String bits = String.format("%4s", Integer.toBinaryString(c)).replace(' ', '0');
            System.out.printf("code %2d  %s  %5s%n", c, bits, v);
            allMatch &= (v * 2 == ggmlDoubled[c]);
        }
        System.out.println("matches ggml kvalues_fp4 / 2: " + allMatch);
    }
}
Output
code  0  0000    0.0
code  1  0001    0.5
code  2  0010    1.0
code  3  0011    1.5
code  4  0100    2.0
code  5  0101    3.0
code  6  0110    4.0
code  7  0111    6.0
code  8  1000   -0.0
code  9  1001   -0.5
code 10  1010   -1.0
code 11  1011   -1.5
code 12  1100   -2.0
code 13  1101   -3.0
code 14  1110   -4.0
code 15  1111   -6.0
matches ggml kvalues_fp4 / 2: true

The positive values are 0, 0.5, 1, 1.5, 2, 3, 4, 6. The steps are fine near zero (0.5 apart) and coarse near the top (2 apart between 4 and 6). This suits weights, which cluster around zero. Code 8 is negative zero, which equals zero in use.

All 16 codes match ggml's table divided by 2, so the last line prints true. ggml stores the values doubled (0, 1, 2, 3, 4, 6, 8, 12) so that they fit in an int8_t table, and folds the factor of one half into the scale.

With only 8 distinct positive values and a maximum of 6, E2M1 alone cannot represent real weights. Every use of E2M1 pairs it with a scale that is shared by a small block of values. The next two examples show the two main ways to do that.

Example 14: MXFP4, 32 values with one power-of-two scale

MXFP4 is a block format from the OCP Microscaling specification. It groups 32 E2M1 values and gives the group one shared scale. The group is called a block. The scale is an E8M0 number: 8 exponent bits, no mantissa, and no sign. It is always a power of two, 2^(e - 127), so it is stored in one byte.

How is the scale chosen? The block's largest absolute value, amax, should fit near the top of E2M1's range. E2M1's largest power of two is 4, which is 2^2. Following the specification and ggml's quantize_row_mxfp4_ref, the scale exponent is therefore floor(log2(amax)) - 2. After the scale is picked, each value takes the nearest of the 16 E2M1 values multiplied by the scale.

The program quantizes one block of 32 random values. It prints the largest value, the scale, the largest value the block can represent (6 times the scale), the 32 codes in hexadecimal, the mean error, and the storage cost.

Java
import java.util.Locale;
import java.util.Random;

public class Mxfp4Block {
    static final float[] E2M1 = {0, 0.5f, 1, 1.5f, 2, 3, 4, 6, -0f, -0.5f, -1, -1.5f, -2, -3, -4, -6};

    static int nearest(float x, float d) {                 // first index wins on ties, as in ggml
        int best = 0; float bestErr = Math.abs(E2M1[0] * d - x);
        for (int i = 1; i < 16; i++) {
            float err = Math.abs(E2M1[i] * d - x);
            if (err < bestErr) { best = i; bestErr = err; }
        }
        return best;
    }

    public static void main(String[] args) {
        Random rnd = new Random(3);
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) rnd.nextGaussian();
        float amax = 0;
        for (float v : x) amax = Math.max(amax, Math.abs(v));

        int e = (int) (Math.floor(Math.log(amax) / Math.log(2)) - 2 + 127);   // E8M0 byte
        float d = (float) Math.pow(2, e - 127);
        double err = 0;
        StringBuilder codes = new StringBuilder();
        for (int i = 0; i < 32; i++) {
            int c = nearest(x[i], d);
            codes.append(Integer.toHexString(c));
            err += Math.abs(E2M1[c] * d - x[i]);
        }
        System.out.printf(Locale.ROOT, "amax = %.4f, E8M0 byte = %d, scale = 2^%d = %s%n", amax, e, e - 127, d);
        System.out.printf(Locale.ROOT, "largest representable = 6 * scale = %s%n", 6 * d);
        System.out.println("codes = " + codes);
        System.out.printf(Locale.ROOT, "mean abs error = %.4f%n", err / 32);
        System.out.println("storage: 1 scale byte + 16 bytes of nibbles = 17 bytes per 32 values = " + 17 * 8 / 32.0 + " bits per value");
    }
}
Output
amax = 3.8544, E8M0 byte = 126, scale = 2^-1 = 0.5
largest representable = 6 * scale = 3.0
codes = 19c30cdeab03402d00bc9c59d9e07acc
mean abs error = 0.1147
storage: 1 scale byte + 16 bytes of nibbles = 17 bytes per 32 values = 4.25 bits per value

The largest value is 3.8544. floor(log2(3.8544)) = 1, so the exponent is 1 - 2 = -1, and the scale is 2^-1 = 0.5. The E8M0 byte stores -1 + 127 = 126. The block's grid is the E2M1 values times 0.5: 0, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, and their negatives.

The largest value, 3.85, is more than 6 times the scale (3.0), so it is clipped to 3.0. This is the weakness of a power-of-two scale. The block's largest value can land anywhere from 4 to 8 times the scale, and everything above 6 times the scale is clipped.

The strength is cost. The scale takes one byte, and multiplying by a power of two only adds to the exponent. Storage is 1 scale byte plus 16 bytes of nibbles (32 values at 4 bits), which is 17 bytes per 32 values, or 4.25 bits per value.

llama.cpp's MXFP4 tensor type has this layout (block_mxfp4: one e byte and 16 bytes of qs). The gpt-oss models ship their expert weights in it.

Example 15: NVFP4, 16-value blocks with an FP8 scale

NVFP4 is NVIDIA's 4-bit format for Blackwell GPUs. It fixes the two weaknesses of MXFP4 described above. As NVIDIA defines it:

  • Each block has 16 values instead of 32. A smaller block means one large value affects fewer neighbors.
  • Each block's scale is an FP8 E4M3 number instead of a power of two, so the scale can fall between powers of two and fit the block's largest value more closely.
  • The whole tensor has one more FP32 scale. The per-tensor scale moves the block scales into E4M3's range: tensorScale = tensorAmax / (448 * 6).

The factor 448 * 6 comes from the two ranges: the largest E4M3 scale (448) times the largest E2M1 value (6).

The program quantizes the same 32 values as Example 14, split into two blocks of 16. It reuses the E4M3 encoder from Example 8 and the E2M1 search from Example 14. For each block it prints the largest value, the chosen scale code, and the effective scale (the E4M3 scale times the tensor scale).

Java
import java.util.Locale;
import java.util.Random;

public class Nvfp4Blocks {
    static final float[] E2M1 = {0, 0.5f, 1, 1.5f, 2, 3, 4, 6, -0f, -0.5f, -1, -1.5f, -2, -3, -4, -6};
    static float e4m3(int c) {
        int s = (c >> 7) & 1, e = (c >> 3) & 0xF, m = c & 7;
        if (e == 15 && m == 7) return Float.NaN;
        float v = (e == 0) ? m / 8f * (float) Math.pow(2, -6) : (1 + m / 8f) * (float) Math.pow(2, e - 7);
        return s == 1 ? -v : v;
    }
    static int toE4m3(float x) {
        int best = 0; float bestErr = Float.MAX_VALUE;
        for (int c = 0; c < 128; c++) {                       // positive codes only: scales are positive
            float v = e4m3(c);
            if (Float.isNaN(v)) continue;
            float err = Math.abs(v - x);
            if (err < bestErr || (err == bestErr && (c & 1) == 0)) { best = c; bestErr = err; }
        }
        return best;
    }
    static float nearestE2m1(float x) {
        float best = 0;
        for (float v : E2M1) if (Math.abs(v - x) < Math.abs(best - x)) best = v;
        return best;
    }

    public static void main(String[] args) {
        Random rnd = new Random(3);
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) rnd.nextGaussian();
        float tensorAmax = 0;
        for (float v : x) tensorAmax = Math.max(tensorAmax, Math.abs(v));
        float tensorScale = tensorAmax / (448f * 6f);
        System.out.printf(Locale.ROOT, "tensor scale (FP32) = %.6e%n", tensorScale);

        double err = 0;
        for (int b = 0; b < 2; b++) {
            float amax = 0;
            for (int i = 0; i < 16; i++) amax = Math.max(amax, Math.abs(x[b * 16 + i]));
            int code = toE4m3(amax / 6f / tensorScale);
            float blockScale = e4m3(code) * tensorScale;
            for (int i = 0; i < 16; i++) {
                float q = nearestE2m1(x[b * 16 + i] / blockScale) * blockScale;
                err += Math.abs(q - x[b * 16 + i]);
            }
            System.out.printf(Locale.ROOT, "block %d: amax %.4f, E4M3 code 0x%02X = %s, effective scale %.4f%n",
                    b, amax, code, e4m3(code), blockScale);
        }
        System.out.printf(Locale.ROOT, "mean abs error = %.4f%n", err / 32);
        System.out.println("storage: 16 nibbles (8 bytes) + 1 scale byte per 16 values = " + 9 * 8 / 16.0 + " bits per value");
    }
}
Output
tensor scale (FP32) = 1.433914e-03
block 0: amax 2.1022, E4M3 code 0x77 = 240.0, effective scale 0.3441
block 1: amax 3.8544, E4M3 code 0x7E = 448.0, effective scale 0.6424
mean abs error = 0.0751
storage: 16 nibbles (8 bytes) + 1 scale byte per 16 values = 4.5 bits per value

The same 32 values now have a mean error of 0.075 instead of 0.115. Follow block 1, which holds the tensor's largest value, 3.85. The program computes 3.8544 / 6 / tensorScale = 448, so the block's scale code is the largest E4M3 value, 448. The effective scale is 448 * 0.001434 = 0.642. Then 3.85 maps to exactly 6 times that scale, so it is stored without clipping.

Block 0 has a smaller largest value, 2.10. Its scale is 2.1022 / 6 / 0.001434 = 244, and the nearest E4M3 value is 240 (E4M3 has steps of 16 between 128 and 256). That gives an effective scale of 0.344, which is not a power of two. The largest value of block 0 also lands at about 6 times its scale, and its small values get a finer grid than they had with the shared scale of Example 14.

The cost is 4.5 bits per value instead of 4.25: 8 bytes of nibbles plus 1 scale byte for 16 values is 9 * 8 / 16 = 4.5.

ggml groups NVFP4 differently. block_nvfp4 in ggml-common.h holds 64 values as four 16-value sub-blocks, each with an unsigned E4M3 scale byte (ggml_fp32_to_ue4m3 in ggml-impl.h rounds half up rather than to even). That is 36 bytes per 64 values. The per-tensor scale is a separate tensor.

Example 16: bytes per parameter and model size

All of the formats above lead to one number that matters in practice: bytes per parameter. It sets the size of a model file. Through the roofline model from lesson 17 (the rule that decode speed is limited by how fast weights can be read from memory), it also sets how fast the model can generate text.

Block formats add scale bytes to every block, so their cost per weight is higher than the bit width of their codes. The cost in bits per weight is the block's total bytes times 8, divided by the number of values in the block. For example, NVFP4 has 9 bytes for 16 values: 9 * 8 / 16 = 4.5.

The program lists seven formats with their bits per weight, bytes per parameter, and the size of an 8-billion-parameter model (8.03e9 parameters, the size of Llama 3.1 8B).

Java
import java.util.Locale;

public class BytesPerParameter {
    public static void main(String[] args) {
        String[] names = {"FP32", "FP16 / BF16", "FP8 (per-tensor scale)", "Q8_0 (INT8, 32-value blocks)",
                          "NVFP4 (16-value blocks)", "Q4_0 (INT4, 32-value blocks)", "MXFP4 (32-value blocks)"};
        double[] bits = {32, 16, 8, 34 * 8 / 32.0, 9 * 8 / 16.0, 18 * 8 / 32.0, 17 * 8 / 32.0};
        double params = 8.03e9;
        System.out.println("format                          bits/weight  bytes/param  8B model GB");
        for (int i = 0; i < names.length; i++) {
            System.out.printf(Locale.ROOT, "%-31s %11.3f %12.4f %12.2f%n",
                    names[i], bits[i], bits[i] / 8, params * bits[i] / 8 / 1e9);
        }
    }
}
Output
format                          bits/weight  bytes/param  8B model GB
FP32                                 32.000       4.0000        32.12
FP16 / BF16                          16.000       2.0000        16.06
FP8 (per-tensor scale)                8.000       1.0000         8.03
Q8_0 (INT8, 32-value blocks)          8.500       1.0625         8.53
NVFP4 (16-value blocks)               4.500       0.5625         4.52
Q4_0 (INT4, 32-value blocks)          4.500       0.5625         4.52
MXFP4 (32-value blocks)               4.250       0.5313         4.27

Read the last column: the same model is 32 GB in FP32, 16 GB in FP16 or BF16, 8 GB in FP8, and about 4.3 to 4.5 GB in the 4-bit block formats. Each halving of the bits halves the file.

Q8_0 stores 32 INT8 values and one FP16 scale per block. That is 34 bytes per 32 values, so the cost is 8.5 bits per weight. Q4_0 stores 32 nibbles and one FP16 scale, which is 18 bytes, or 4.5 bits per weight. These two ggml formats are built byte for byte in lesson 21.

Real files mix formats per tensor. Norm weights, and often the embedding table, stay at higher precision. A "4-bit" file is therefore somewhat larger than the 4-bit row in this table.

Interview points

  • FP32 uses 1 sign bit, 8 exponent bits, and 23 mantissa bits with bias 127. FP16 uses 1, 5, and 10 bits with bias 15 and a maximum of 65504. BF16 uses 1, 8, and 7 bits. It is the top half of FP32 and has FP32's range.
  • FP16 overflows at 65520, which rounds to infinity. Below 6.1e-5 it switches to subnormals with fewer significant bits, and values under about 3e-8 become zero. BF16 gives up 3 bits of precision to keep FP32's range.
  • Adding up squares or dot products in FP16 can overflow when the data has outliers. Inference engines accumulate in FP32 to avoid this.
  • FP8 E4M3 has bias 7, no infinity, and a maximum of 448, with 3 mantissa bits. It is used for weights and activations. FP8 E5M2 has bias 15, follows IEEE rules, and has a maximum of 57344. It is used for gradients, where range matters more.
  • Low-precision formats need scales to use their range. FP8 uses one scale per tensor. MXFP4 uses one E8M0 scale per 32 values. NVFP4 uses one E4M3 scale per 16 values plus an FP32 scale per tensor. Q8_0 and Q4_0 use one FP16 scale per 32 values.
  • An INT8 dot product multiplies and adds integers into an int accumulator, then multiplies once by the two scales. VNNI on x86 and sdot on ARM do this in hardware.

Summary

  • A normal FP32 value is (-1)^s * (1 + m / 2^23) * 2^(e - 127). The leading 1 is implied and not stored.
  • Float.floatToFloat16 and Float.float16ToFloat (Java 20) convert with round-to-nearest-even, and values that are too large become infinity.
  • BF16 keeps about 2 to 3 significant decimal digits and FP32's range of about 1e-38 to 3e38.
  • One outlier of 300 in a 4,096-value vector makes an FP16 sum of squares overflow, because 300 squared is 90,000, above 65504.
  • E4M3 has 8 steps per power of two. Its grid near 1 is 0.125 apart, and it saturates at 448 in the conversion used together with scales.
  • A per-tensor scale of amax / 448 moved small weights out of E4M3's subnormal range and cut the mean error by more than 13 times.
  • ggml's kvalues_fp4 table stores the E2M1 values doubled: 0, 1, 2, 3, 4, 6, 8, 12 and their negatives.
  • MXFP4's scale exponent is floor(log2(amax)) - 2, and one block is 17 bytes for 32 values.
  • NVFP4 as defined by NVIDIA uses 16-value blocks with E4M3 scales and an FP32 tensor scale. ggml stores it as 64-value blocks with four UE4M3 scales.
  • An 8B model is 16 GB in FP16, 8.5 GB in Q8_0, and about 4.5 GB in Q4_0 or NVFP4.