codeloopback
Guide contents

Lesson 21 of 73

Building Q8_0 and Q4_0 Like ggml

The previous lesson, Quantization Fundamentals, showed the ideas: store small integer codes plus a scale, and give each small block of values its own scale. This lesson turns those ideas into the real thing. GGUF files, the model files used by llama.cpp, store most weights in formats defined by ggml, the tensor library under llama.cpp. Here you rebuild the four simplest ones, Q8_0, Q4_0, Q4_1, and Q5_0, in Java so that the bytes are identical to what llama.cpp writes.

Why match the bytes exactly? A quantizer that is only approximately like ggml cannot produce files that other tools trust, and a reader that misunderstands one nibble (half a byte) returns garbage weights. Working at the byte level also shows where the "4.5 bits per weight" of Q4_0 really comes from.

After this lesson you can read and write these blocks by hand, explain what each byte means, describe how the CPU multiplies quantized weights without turning them back into floats, and measure what each format costs in accuracy.

Every format below uses blocks of 32 values. The table is a map for the lesson. "FP16" is a 16-bit floating point number, and a "nibble" is 4 bits, half a byte.

Format Block Bytes Stored per block Bits per weight Example
Q8_0 32 34 FP16 scale d, 32 int8 8.5 1 to 4, 10
Q4_0 32 18 FP16 scale d, 16 bytes of nibbles 4.5 5, 6, 11
Q4_1 32 20 FP16 scale d, FP16 minimum m, 16 bytes of nibbles 5.0 7
Q5_0 32 22 FP16 scale d, 4 bytes of fifth bits qh, 16 bytes of nibbles 5.5 8

The ground truth is the pinned llama.cpp source. The block structs (C data layouts) are in ggml/src/ggml-common.h. The reference quantizers quantize_row_q8_0_ref, quantize_row_q4_0_ref, quantize_row_q4_1_ref, and quantize_row_q5_0_ref are in ggml/src/ggml-quants.c. The dot products are in ggml/src/ggml-cpu/quants.c.

The "reference" quantizers are the ones ggml uses when it writes model files, so their output is deterministic across machines. gguf-py 0.19.0, the Python package in the llama.cpp repository, reimplements them in NumPy in gguf/quants.py, and Example 14 checks the Java output against it.

This lesson builds on symmetric and asymmetric scales, block-wise scales, and integer dot products from Quantization Fundamentals, and on FP16 conversion with Float.floatToFloat16 from Number Formats.

Example 1: the Q8_0 block layout

Start with the biggest and simplest format. Q8_0 keeps 8 bits per value. A block is the unit that shares one scale: 32 values are stored together as one small package.

In ggml-common.h, a Q8_0 block is a C struct with two fields. The first is an FP16 scale named d (for delta, the step size between neighboring codes). The second is 32 signed bytes named qs (the quants, meaning the integer codes). A static_assert in the C code checks that the struct is exactly 34 bytes, so the compiler refuses to build if the layout ever changes.

The program below describes that layout as a Java record and prints the byte offsets. Look at where each field starts and how the total size works out.

Java
public class Q8_0Layout {
    static final int QK8_0 = 32;                 // values per block
    record Field(String name, String type, int offset, int bytes) {}

    public static void main(String[] args) {
        Field[] fields = {
            new Field("d", "ggml_half (FP16)", 0, 2),
            new Field("qs", "int8_t[32]", 2, QK8_0),
        };
        int size = 0;
        for (Field f : fields) {
            System.out.printf("%-3s %-17s bytes %2d..%2d%n", f.name(), f.type(), f.offset(), f.offset() + f.bytes() - 1);
            size += f.bytes();
        }
        System.out.println("block size = " + size + " bytes for " + QK8_0 + " values");
        System.out.println("bits per weight = " + size * 8.0 / QK8_0);
        System.out.println("blocks in a 4096-value row = " + 4096 / QK8_0 + ", row bytes = " + 4096 / QK8_0 * size);
    }
}
Output
d   ggml_half (FP16)  bytes  0.. 1
qs  int8_t[32]        bytes  2..33
block size = 34 bytes for 32 values
bits per weight = 8.5
blocks in a 4096-value row = 128, row bytes = 4352

The first field d takes bytes 0 and 1 and qs takes bytes 2 to 33, so a block is 2 + 32 = 34 bytes for 32 values. That is 34 * 8 / 32 = 8.5 bits per weight: 8 bits for the code and half a bit for the shared scale. A row of 4,096 values is 128 blocks, which is 4,352 bytes.

A tensor row is a sequence of blocks with nothing between them. So the row length must be a multiple of 32 (gguf-py's quant_shape_to_byte_shape raises an error otherwise). The FP16 scale is stored in the machine's byte order, which is little-endian on every platform llama.cpp targets in practice, and GGUF files are little-endian by default. Little-endian means the least significant byte of a number comes first in memory. To get a value back, multiply the code by the scale: qs[j] * d.

Example 2: quantize one Q8_0 block

Now fill a block with real numbers. quantize_row_q8_0_ref does four things for each block:

  1. Find amax, the largest absolute value in the block.
  2. Compute the scale d = amax / 127 and its inverse id = 1 / d (or 0 when d is 0, which happens when the whole block is zero).
  3. Store d as FP16.
  4. Store roundf(x * id) for each value, which is the value divided by the scale and rounded.

There is one trap. C's roundf rounds halves away from zero (2.5 becomes 3, -2.5 becomes -3). Java's Math.round rounds halves toward positive infinity (-2.5 becomes -2). To get identical bytes, the program defines its own roundf helper. Also notice that the codes are computed with the full-precision id, not with the FP16-rounded scale. Only the stored scale is rounded.

The input is 32 numbers computed by a fixed formula. The program prints the first few inputs, the scale in float and FP16 form, all 32 codes, and the rounding comparison.

Java
import java.util.Arrays;
import java.util.Locale;

public class QuantizeQ8_0 {
    // C roundf: halves away from zero. |v| + 0.5 is exact in double for any float v.
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    public static void main(String[] args) {
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) (((i * 7919) % 1001) - 500) / 337f;

        float amax = 0;
        for (float v : x) amax = Math.max(amax, Math.abs(v));
        float d = amax / 127f;
        float id = d != 0 ? 1.0f / d : 0f;
        short dHalf = Float.floatToFloat16(d);
        byte[] qs = new byte[32];
        for (int j = 0; j < 32; j++) qs[j] = (byte) roundf(x[j] * id);

        System.out.printf(Locale.ROOT, "x[0..5] = %.5f %.5f %.5f %.5f %.5f %.5f%n", x[0], x[1], x[2], x[3], x[4], x[5]);
        System.out.printf(Locale.ROOT, "amax = %.6f, d = %.9f, id = %.6f%n", amax, d, id);
        System.out.printf(Locale.ROOT, "d as FP16 = 0x%04X = %.9f%n", dHalf & 0xFFFF, Float.float16ToFloat(dHalf));
        System.out.println("qs = " + Arrays.toString(qs));
        System.out.println("roundf(2.5) = " + roundf(2.5f) + ", roundf(-2.5) = " + roundf(-2.5f)
                + ", Math.round(-2.5f) = " + Math.round(-2.5f));
    }
}
Output
x[0..5] = -1.48368 1.22255 0.95846 0.69436 0.43027 0.16617
amax = 1.483680, d = 0.011682516, id = 85.598000
d as FP16 = 0x21FB = 0.011680603
qs = [-127, 105, 82, 59, 37, 14, -8, -31, -54, -76, -99, -121, 110, 88, 65, 42, 20, -3, -25, -48, -71, -93, -116, 116, 93, 71, 48, 25, 3, -20, -42, -65]
roundf(2.5) = 3, roundf(-2.5) = -3, Math.round(-2.5f) = -2

The value with the largest magnitude, -1.48368, maps to -127, and every other value lands in between (for example 1.22255 times 85.598 is 104.65, which rounds to 105). The FP16 scale (0.011680603) differs from the float scale (0.011682516) in the fourth significant digit, because FP16 keeps only 11 bits of precision.

The last line shows why the helper is needed. For -2.5, C's roundf gives -3 and Java's Math.round gives -2. With real data such exact halves are rare, but a quantizer meant to match ggml byte for byte must handle them the same way.

Example 3: dequantize and measure the error

To use a block, the loader multiplies each code by the scale: qs[j] * d, where d is read back from FP16. This means the stored result has two sources of error. The first is rounding each value to a code. The second is rounding the scale itself to FP16.

The program quantizes the block from Example 2, dequantizes it, and reports the largest error twice: once as if the scale were stored exactly, and once with the real FP16 scale.

Java
import java.util.Locale;

public class DequantQ8_0 {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    public static void main(String[] args) {
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) (((i * 7919) % 1001) - 500) / 337f;
        float amax = 0;
        for (float v : x) amax = Math.max(amax, Math.abs(v));
        float d = amax / 127f, id = d != 0 ? 1.0f / d : 0f;
        float dStored = Float.float16ToFloat(Float.floatToFloat16(d));

        double maxErr = 0, sumSq = 0, power = 0, maxErrExactScale = 0;
        for (float v : x) {
            int q = roundf(v * id);
            float back = q * dStored;
            maxErr = Math.max(maxErr, Math.abs(back - v));
            maxErrExactScale = Math.max(maxErrExactScale, Math.abs(q * d - v));
            sumSq += (back - v) * (back - v);
            power += v * v;
        }
        System.out.printf(Locale.ROOT, "step d = %.6f, half step = %.6f%n", d, d / 2);
        System.out.printf(Locale.ROOT, "max |error| with float d = %.6f%n", maxErrExactScale);
        System.out.printf(Locale.ROOT, "max |error| with FP16 d  = %.6f%n", maxErr);
        System.out.printf(Locale.ROOT, "relative RMSE = %.4f%%%n", 100 * Math.sqrt(sumSq / power));
    }
}
Output
step d = 0.011683, half step = 0.005841
max |error| with float d = 0.005094
max |error| with FP16 d  = 0.005206
relative RMSE = 0.3832%

The step d is 0.011683, so the largest possible rounding error is half a step, 0.005841. With the exact float scale the largest error is 0.005094, inside that bound, as lesson 20 predicted. With the FP16 scale it grows to 0.005206. The reason is that a scale off by a relative 0.02% shifts the largest value 127 * d by 0.02% of amax, a small extra error at the block's extremes.

The relative RMSE of Q8_0 is 0.38%, a fraction of a percent. Lesson 24 shows how to measure what such errors mean for a whole model, with perplexity and KL divergence against the F16 file.

Example 4: pack into little-endian bytes and read back

So far the block lived in Java variables. On disk it is 34 bytes: 2 bytes of FP16 scale (least significant byte first), then the 32 codes as signed bytes. Java's ByteBuffer with ByteOrder.LITTLE_ENDIAN writes and reads exactly this layout.

The program packs the block from Example 2, prints it in hex (two hex digits per byte), reads it back, and checks that the decoded values match. The last check quantizes the decoded values again and compares the bytes.

Java
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.HexFormat;

public class PackQ8_0 {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    static byte[] quantize(float[] x) {
        ByteBuffer b = ByteBuffer.allocate(34).order(ByteOrder.LITTLE_ENDIAN);
        float amax = 0;
        for (float v : x) amax = Math.max(amax, Math.abs(v));
        float d = amax / 127f, id = d != 0 ? 1.0f / d : 0f;
        b.putShort(Float.floatToFloat16(d));
        for (int j = 0; j < 32; j++) b.put((byte) roundf(x[j] * id));
        return b.array();
    }

    static float[] dequantize(byte[] block) {
        ByteBuffer b = ByteBuffer.wrap(block).order(ByteOrder.LITTLE_ENDIAN);
        float d = Float.float16ToFloat(b.getShort());
        float[] y = new float[32];
        for (int j = 0; j < 32; j++) y[j] = b.get() * d;
        return y;
    }

    public static void main(String[] args) {
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) (((i * 7919) % 1001) - 500) / 337f;
        byte[] block = quantize(x);
        String hex = HexFormat.of().formatHex(block);
        System.out.println("bytes = " + block.length);
        System.out.println("d  = " + hex.substring(0, 4) + "  (little-endian: low byte first)");
        System.out.println("qs = " + hex.substring(4));
        float[] y = dequantize(block);
        System.out.println("y[0] = " + y[0] + ", x[0] = " + x[0]);
        System.out.println("round trip of bytes stable: " + java.util.Arrays.equals(block, quantize(y)));
    }
}
Output
bytes = 34
d  = fb21  (little-endian: low byte first)
qs = 8169523b250ef8e1cab49d876e58412a14fde7d0b9a38c745d47301903ecd6bf
y[0] = -1.4834366, x[0] = -1.4836795
round trip of bytes stable: true

The first two bytes are the scale, low byte first: the FP16 value 0x21FB from Example 2 is stored as fb 21. The 32 codes follow as two's complement bytes, which is how computers store negative numbers in a byte. So 81 is -127 (the byte value 129 minus 256) and 7f would be +127. The decoded y[0] is -1.4834366 against the original -1.4836795, a difference of about 0.0002.

Quantizing the dequantized values again gives the same bytes. The grid points of the quantizer are fixed points: a value that is already on the grid maps back to itself. This is a useful test for any implementation.

Example 5: the Q4_0 layout, the signed maximum, and nibble order

Q4_0 stores 32 four-bit codes in 16 bytes, plus an FP16 scale, so 18 bytes per block. Four bits give 16 possible codes, so the layout needs three details that Q8_0 did not have. They are the main source of confusion when people read ggml code.

  1. The scale comes from the signed maximum. The scale is computed from the value max with the largest magnitude, keeping its sign: d = max / -8. If that value is negative, d is positive. If it is positive, d is negative.
  2. How a code is computed. The code is min(15, (int8_t)(x * id + 8.5f)). Adding 8.5 and truncating rounds to the nearest code from 0 to 15. The stored nibble means code - 8. The extreme value max becomes max * id = -8, which is code 0, so the extreme value always uses the code for -8, the one extra negative code in a 4-bit signed range (-8 to 7).
  3. Nibble order. Byte j holds value j in its low 4 bits and value j + 16 in its high 4 bits. The first half of the block goes into the low nibbles and the second half into the high nibbles. Neighbors are not packed into one byte.

A picture of the packing for a block of 32 values:

Text
value index:   0   1   2  ...  15 | 16  17  18  ...  31
               |   |   |       |    |   |   |        |
byte index:    0   1   2  ...  15 |  0   1   2  ...  15
nibble:       low low low     low | high high high  high

The program shows the scale sign and the nibble placement for two small blocks. In block a, the largest magnitude is positive. In block b, all signs are flipped.

Java
import java.util.Locale;

public class Q4_0Layout {
    static void show(String label, float[] x) {
        float amax = 0, max = 0;
        for (float v : x) if (amax < Math.abs(v)) { amax = Math.abs(v); max = v; }
        float d = max / -8f, id = d != 0 ? 1.0f / d : 0f;
        int[] code = new int[32];
        for (int j = 0; j < 32; j++) code[j] = Math.min(15, (int) (x[j] * id + 8.5f));
        System.out.printf(Locale.ROOT, "%s: max = %.2f, d = %.4f%n", label, max, d);
        for (int j = 0; j < 2; j++) {
            int b = code[j] | (code[j + 16] << 4);
            System.out.printf(Locale.ROOT, "  byte %d = 0x%02X: low = value %2d (x=%5.2f, code %2d), high = value %2d (x=%5.2f, code %2d)%n",
                    j, b, j, x[j], code[j], j + 16, x[j + 16], code[j + 16]);
        }
    }

    public static void main(String[] args) {
        float[] a = new float[32], b = new float[32];
        for (int i = 0; i < 32; i++) { a[i] = (i - 12) / 4f; b[i] = -a[i]; }
        show("largest magnitude positive", a);       // a[31] = 4.75
        show("largest magnitude negative", b);       // b[31] = -4.75
        System.out.println("block bytes = 2 + 32 / 2 = " + (2 + 32 / 2) + ", bits per weight = " + 18 * 8 / 32.0);
    }
}
Output
largest magnitude positive: max = 4.75, d = -0.5938
  byte 0 = 0x6D: low = value  0 (x=-3.00, code 13), high = value 16 (x= 1.00, code  6)
  byte 1 = 0x6D: low = value  1 (x=-2.75, code 13), high = value 17 (x= 1.25, code  6)
largest magnitude negative: max = -4.75, d = 0.5938
  byte 0 = 0x6D: low = value  0 (x= 3.00, code 13), high = value 16 (x=-1.00, code  6)
  byte 1 = 0x6D: low = value  1 (x= 2.75, code 13), high = value 17 (x=-1.25, code  6)
block bytes = 2 + 32 / 2 = 18, bits per weight = 4.5

For block a, the largest magnitude is +4.75, so d is negative (-0.59375) and 4.75 becomes code 0. For block b, the signs flip, d is positive, and -4.75 becomes code 0. In both cases the extreme value maps to code 0, which means -8, and because -8 * d equals max exactly, the largest value is stored without error (up to the FP16 rounding of d).

The other side of the block can only reach code 15, which means +7. So a block whose values are symmetric around zero loses a little accuracy at its opposite end.

Byte 0 holds values 0 and 16 (shown as 0x6D: low nibble 13, high nibble 6), byte 1 holds values 1 and 17, and so on. A kernel that unpacks a byte into two values gets elements j and j + 16. That is why the dot product in Example 11 multiplies the low nibble with activation j and the high nibble with activation j + 16.

Example 6: quantize and dequantize one Q4_0 block, error against Q8_0

Decoding a Q4_0 value reverses the steps of Example 5. In dequantize_row_q4_0, value j is ((qs[j] & 0x0F) - 8) * d and value j + 16 is ((qs[j] >> 4) - 8) * d. The expression & 0x0F keeps only the low 4 bits and >> 4 moves the high 4 bits down.

The program quantizes the 32-value block from Example 2 both ways, with Q4_0 and with Q8_0, and compares the first values and the errors.

Java
import java.util.Locale;

public class Q4_0Block {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
    static float half(float v) { return Float.float16ToFloat(Float.floatToFloat16(v)); }

    static float[] q4_0(float[] x) {
        float amax = 0, max = 0;
        for (float v : x) if (amax < Math.abs(v)) { amax = Math.abs(v); max = v; }
        float d = max / -8f, id = d != 0 ? 1.0f / d : 0f, dq = half(d);
        byte[] qs = new byte[16];
        for (int j = 0; j < 16; j++) {
            int lo = Math.min(15, (int) (x[j] * id + 8.5f)), hi = Math.min(15, (int) (x[j + 16] * id + 8.5f));
            qs[j] = (byte) (lo | (hi << 4));
        }
        float[] y = new float[32];
        for (int j = 0; j < 16; j++) {
            y[j] = ((qs[j] & 0x0F) - 8) * dq;
            y[j + 16] = (((qs[j] & 0xFF) >> 4) - 8) * dq;
        }
        return y;
    }

    static float[] q8_0(float[] x) {
        float amax = 0;
        for (float v : x) amax = Math.max(amax, Math.abs(v));
        float d = amax / 127f, id = d != 0 ? 1.0f / d : 0f, dq = half(d);
        float[] y = new float[32];
        for (int j = 0; j < 32; j++) y[j] = roundf(x[j] * id) * dq;
        return y;
    }

    static String stats(float[] x, float[] y) {
        double e = 0, p = 0, m = 0;
        for (int i = 0; i < x.length; i++) { e += Math.pow(y[i] - x[i], 2); p += x[i] * x[i]; m = Math.max(m, Math.abs(y[i] - x[i])); }
        return String.format(Locale.ROOT, "max |error| %.4f, relative RMSE %6.3f%%", m, 100 * Math.sqrt(e / p));
    }

    public static void main(String[] args) {
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) (((i * 7919) % 1001) - 500) / 337f;
        float[] y4 = q4_0(x), y8 = q8_0(x);
        System.out.printf(Locale.ROOT, "x[0..3]    = %8.4f %8.4f %8.4f %8.4f%n", x[0], x[1], x[2], x[3]);
        System.out.printf(Locale.ROOT, "Q4_0[0..3] = %8.4f %8.4f %8.4f %8.4f%n", y4[0], y4[1], y4[2], y4[3]);
        System.out.printf(Locale.ROOT, "Q8_0[0..3] = %8.4f %8.4f %8.4f %8.4f%n", y8[0], y8[1], y8[2], y8[3]);
        System.out.println("Q4_0: " + stats(x, y4));
        System.out.println("Q8_0: " + stats(x, y8));
    }
}
Output
x[0..3]    =  -1.4837   1.2226   0.9585   0.6944
Q4_0[0..3] =  -1.4834   1.2980   0.9271   0.7417
Q8_0[0..3] =  -1.4834   1.2265   0.9578   0.6892
Q4_0: max |error| 0.0888, relative RMSE  6.206%
Q8_0: max |error| 0.0052, relative RMSE  0.383%

Q4_0 values land on a grid of 16 points spaced |d| apart, about 0.19 for this block, so each value can move by up to about 0.09 (the largest measured error is 0.0888). For example 1.2226 becomes 1.2980 in Q4_0 but 1.2265 in Q8_0. Q8_0's grid is 16 times finer.

The relative RMSE differs by roughly that factor of 16 (6.206% against 0.383%). The 6 dB-per-bit rule from lesson 20 predicts this: 4 extra bits is 24 dB, and 24 dB is a factor of about 16 in amplitude.

One Java detail: the & 0xFF before >> 4 converts the signed Java byte to its unsigned value. Without it, a byte such as 0xF0 would shift in sign bits and give wrong high nibbles.

Example 7: Q4_1, a scale and a minimum

Q4_0's grid is centered on zero. Some blocks are not centered on zero, and for those a grid that starts at the block's minimum fits better. Q4_1 is the asymmetric version of Q4_0 (the zero-point idea from lesson 20).

quantize_row_q4_1_ref computes the block's min and max and sets d = (max - min) / 15, so the 16 codes span exactly the block's range. It stores both d and m = min as FP16, and stores each code as min(15, (int8_t)((x - min) * id + 0.5f)). Dequantization is code * d + m. The block is 20 bytes: d, m, then 16 bytes of nibbles in the same order as Q4_0.

The program tries two blocks. One is centered on zero, as weights usually are. The other has all values in a narrow band away from zero (the first block shifted by 2).

Java
import java.util.Locale;

public class Q4_1Block {
    static float half(float v) { return Float.float16ToFloat(Float.floatToFloat16(v)); }

    static float[] q4_1(float[] x) {
        float min = Float.MAX_VALUE, max = -Float.MAX_VALUE;
        for (float v : x) { min = Math.min(min, v); max = Math.max(max, v); }
        float d = (max - min) / 15f, id = d != 0 ? 1.0f / d : 0f;
        float dq = half(d), mq = half(min);
        float[] y = new float[32];
        for (int j = 0; j < 32; j++) {
            int code = Math.min(15, (int) ((x[j] - min) * id + 0.5f));
            y[j] = code * dq + mq;
        }
        return y;
    }

    static double relRmse(float[] x, float[] y) {
        double e = 0, p = 0;
        for (int i = 0; i < x.length; i++) { e += Math.pow(y[i] - x[i], 2); p += x[i] * x[i]; }
        return 100 * Math.sqrt(e / p);
    }

    public static void main(String[] args) {
        float[] centered = new float[32], shifted = new float[32];
        for (int i = 0; i < 32; i++) {
            centered[i] = (float) (((i * 7919) % 1001) - 500) / 337f;
            shifted[i] = 2.0f + Math.abs(centered[i]);           // all positive, like activations after a ReLU
        }
        for (float[] x : new float[][] {centered, shifted}) {
            float min = Float.MAX_VALUE, max = -Float.MAX_VALUE;
            for (float v : x) { min = Math.min(min, v); max = Math.max(max, v); }
            System.out.printf(Locale.ROOT, "range [%.3f, %.3f]: Q4_1 d = %.4f, m = %.4f, relative RMSE %.3f%%%n",
                    min, max, (max - min) / 15, min, relRmse(x, q4_1(x)));
        }
        System.out.println("block bytes = 2 + 2 + 16 = 20, bits per weight = " + 20 * 8 / 32.0);
    }
}
Output
range [-1.484, 1.353]: Q4_1 d = 0.1891, m = -1.4837, relative RMSE 6.350%
range [2.033, 3.484]: Q4_1 d = 0.0967, m = 2.0326, relative RMSE 0.996%
block bytes = 2 + 2 + 16 = 20, bits per weight = 5.0

For the block centered on zero (range -1.484 to 1.353), Q4_1's step (max - min) / 15 = 0.1891 is about the same as Q4_0's amax / 8, and its relative RMSE is 6.350%, no better than Q4_0's 6.206% from Example 6. So the extra 2 bytes buy nothing here.

For the block whose values lie in a narrow band away from zero (2.033 to 3.484), the step shrinks to the width of that band divided by 15, which is 0.0967, and the error falls to 0.996%.

Weight blocks are mostly centered on zero, which is one reason Q4_0 and later the k-quants are used more than Q4_1. The k-quants (Q4_K and Q5_K) reuse the scale-and-minimum idea with cheaper, quantized scales (lesson 22).

Example 8: Q5_0, the fifth bit stored separately

Five bits do not fit neatly into bytes. Q5_0 solves this by splitting each 5-bit code into two parts, so that no space is wasted on alignment.

The code range is 0 to 31, meaning -16 to 15. The low 4 bits of each code go into nibbles exactly as in Q4_0. The fifth bits (the top bit) of all 32 codes are collected into one 32-bit word called qh. Bit j of qh holds the fifth bit of value j, and bit j + 16 holds the fifth bit of value j + 16.

quantize_row_q5_0_ref uses d = max / -16 and the code min(31, (int8_t)(x * id + 16.5f)). The block is d (2 bytes), qh (4 bytes), and qs (16 bytes): 22 bytes in total.

The program builds the block, prints the qh bits and the hex bytes, then decodes the block the way dequantize_row_q5_0 does and prints the relative error.

Java
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.HexFormat;
import java.util.Locale;

public class Q5_0Block {
    public static void main(String[] args) {
        float[] x = new float[32];
        for (int i = 0; i < 32; i++) x[i] = (float) (((i * 7919) % 1001) - 500) / 337f;
        float amax = 0, max = 0;
        for (float v : x) if (amax < Math.abs(v)) { amax = Math.abs(v); max = v; }
        float d = max / -16f, id = d != 0 ? 1.0f / d : 0f;

        int qh = 0;
        byte[] qs = new byte[16];
        for (int j = 0; j < 16; j++) {
            int lo = Math.min(31, (int) (x[j] * id + 16.5f)), hi = Math.min(31, (int) (x[j + 16] * id + 16.5f));
            qs[j] = (byte) ((lo & 0x0F) | ((hi & 0x0F) << 4));
            qh |= ((lo & 0x10) >> 4) << j;
            qh |= ((hi & 0x10) >> 4) << (j + 16);
        }
        ByteBuffer b = ByteBuffer.allocate(22).order(ByteOrder.LITTLE_ENDIAN);
        b.putShort(Float.floatToFloat16(d)).putInt(qh).put(qs);
        System.out.printf(Locale.ROOT, "max = %.4f, d = %.5f%n", max, d);
        System.out.println("qh bits (value 31 .. 0) = " + String.format("%32s", Integer.toBinaryString(qh)).replace(' ', '0'));
        System.out.println("block = " + HexFormat.of().formatHex(b.array()));

        // dequantize as dequantize_row_q5_0 does
        float dq = Float.float16ToFloat(Float.floatToFloat16(d));
        double e = 0, p = 0;
        for (int j = 0; j < 16; j++) {
            int xh0 = ((qh >>> j) << 4) & 0x10, xh1 = (qh >>> (j + 12)) & 0x10;
            int x0 = ((qs[j] & 0x0F) | xh0) - 16, x1 = (((qs[j] & 0xFF) >> 4) | xh1) - 16;
            e += Math.pow(x0 * dq - x[j], 2) + Math.pow(x1 * dq - x[j + 16], 2);
        }
        for (float v : x) p += v * v;
        System.out.printf(Locale.ROOT, "relative RMSE = %.3f%%%n", 100 * Math.sqrt(e / p));
    }
}
Output
max = -1.4837, d = 0.09273
qh bits (value 31 .. 0) = 00011111100000111111000000111110
block = ef2d3ef0831f200ddaa775421ffcc99664310eebb885
relative RMSE = 3.211%

In qh, each 1 bit marks a value whose code is 16 or more, that is, a value on the side opposite to max. In the printed bit string, bit 31 is on the left and bit 0 on the right.

The decoder rebuilds the fifth bit with two shifts. ((qh >> j) << 4) & 0x10 moves bit j to position 4. (qh >> (j + 12)) & 0x10 moves bit j + 16 to position 4. Those two lines are copied from dequantize_row_q5_0.

One extra bit per value halves the step, so the relative RMSE (3.211%) is about half of Q4_0's (6.206%) for the same block, at 5.5 bits per weight instead of 4.5.

Example 9: quantize a whole tensor row by row

A weight tensor in GGUF is quantized row by row, and each row is a sequence of blocks. Nothing separates blocks or rows, so the total size follows from simple arithmetic: rows * cols / 32 * bytesPerBlock.

The program quantizes a seeded 64 by 4,096 tensor into Q8_0 bytes and Q4_0 bytes. It checks that each buffer was filled exactly to its computed size and computes the bits per weight and the saving compared with 32-bit floats (F32).

Java
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Locale;
import java.util.Random;

public class QuantizeTensor {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    static void q8_0(float[] x, int off, ByteBuffer out) {
        float amax = 0;
        for (int j = 0; j < 32; j++) amax = Math.max(amax, Math.abs(x[off + j]));
        float d = amax / 127f, id = d != 0 ? 1.0f / d : 0f;
        out.putShort(Float.floatToFloat16(d));
        for (int j = 0; j < 32; j++) out.put((byte) roundf(x[off + j] * id));
    }
    static void q4_0(float[] x, int off, ByteBuffer out) {
        float amax = 0, max = 0;
        for (int j = 0; j < 32; j++) { float v = x[off + j]; if (amax < Math.abs(v)) { amax = Math.abs(v); max = v; } }
        float d = max / -8f, id = d != 0 ? 1.0f / d : 0f;
        out.putShort(Float.floatToFloat16(d));
        for (int j = 0; j < 16; j++)
            out.put((byte) (Math.min(15, (int) (x[off + j] * id + 8.5f)) | (Math.min(15, (int) (x[off + j + 16] * id + 8.5f)) << 4)));
    }

    public static void main(String[] args) {
        int rows = 64, cols = 4096;
        if (cols % 32 != 0) throw new IllegalArgumentException("row length must be a multiple of 32");
        Random rnd = new Random(21);
        float[] w = new float[rows * cols];
        for (int i = 0; i < w.length; i++) w[i] = (float) rnd.nextGaussian() * 0.02f;

        ByteBuffer b8 = ByteBuffer.allocate(rows * cols / 32 * 34).order(ByteOrder.LITTLE_ENDIAN);
        ByteBuffer b4 = ByteBuffer.allocate(rows * cols / 32 * 18).order(ByteOrder.LITTLE_ENDIAN);
        for (int r = 0; r < rows; r++)
            for (int blk = 0; blk < cols / 32; blk++) {
                q8_0(w, r * cols + blk * 32, b8);
                q4_0(w, r * cols + blk * 32, b4);
            }
        long f32 = (long) rows * cols * 4;
        System.out.println("F32 bytes  = " + f32);
        System.out.printf(Locale.ROOT, "Q8_0 bytes = %d (written %d), %.2f bits/weight, %.1fx smaller%n",
                b8.capacity(), b8.position(), b8.capacity() * 8.0 / (rows * cols), (double) f32 / b8.capacity());
        System.out.printf(Locale.ROOT, "Q4_0 bytes = %d (written %d), %.2f bits/weight, %.1fx smaller%n",
                b4.capacity(), b4.position(), b4.capacity() * 8.0 / (rows * cols), (double) f32 / b4.capacity());
        System.out.println("row stride: Q8_0 " + cols / 32 * 34 + " bytes, Q4_0 " + cols / 32 * 18 + " bytes");
    }
}
Output
F32 bytes  = 1048576
Q8_0 bytes = 278528 (written 278528), 8.50 bits/weight, 3.8x smaller
Q4_0 bytes = 147456 (written 147456), 4.50 bits/weight, 7.1x smaller
row stride: Q8_0 4352 bytes, Q4_0 2304 bytes

The F32 tensor takes 64 * 4096 * 4 = 1,048,576 bytes. Q8_0 takes 64 * 128 * 34 = 278,528 bytes, which is 3.8 times smaller. Q4_0 takes 64 * 128 * 18 = 147,456 bytes, which is 7.1 times smaller.

Every ByteBuffer was filled exactly to its capacity (the "written" value equals the size), so the layout has no padding.

The row stride is the number of bytes from the start of one row to the start of the next: 4,352 for Q8_0 and 2,304 for Q4_0 here. It is what a matrix-vector kernel uses to find row r. ggml stores it in the tensor's nb[1] field. Section 7 reads it from a real file in the lesson on loading GGUF with the FFM API, later in this course.

Example 10: the Q8_0 by Q8_0 dot product

A model spends most of its time multiplying a weight row with an activation vector. For Q8_0 this multiplication works on the codes directly. ggml_vec_dot_q8_0_q8_0_generic in ggml-cpu/quants.c is the portable version of the kernel (a kernel is a small, heavily used function). For each block it does two things:

  1. Compute an integer sum of the 32 products of int8 codes.
  2. Add sumi * (dx * dy) to a running float total, with both scales converted from FP16.

The SIMD versions compute the same quantity using vector instructions, which process many values per instruction. The program builds two Q8_0 vectors of 4,096 values (one is a weight-like vector, one an activation-like vector) and computes their dot product block by block in the same order of operations. It compares the result with the exact float answer.

Java
import java.util.Locale;
import java.util.Random;

public class DotQ8_0 {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    static short[] quantize(float[] x, byte[] qs) {
        short[] d = new short[x.length / 32];
        for (int b = 0; b < d.length; b++) {
            float amax = 0;
            for (int j = 0; j < 32; j++) amax = Math.max(amax, Math.abs(x[b * 32 + j]));
            float s = amax / 127f, id = s != 0 ? 1.0f / s : 0f;
            d[b] = Float.floatToFloat16(s);
            for (int j = 0; j < 32; j++) qs[b * 32 + j] = (byte) roundf(x[b * 32 + j] * id);
        }
        return d;
    }

    public static void main(String[] args) {
        int n = 4096;
        Random rnd = new Random(22);
        float[] a = new float[n], b = new float[n];
        for (int i = 0; i < n; i++) { a[i] = (float) rnd.nextGaussian() * 0.02f; b[i] = (float) rnd.nextGaussian(); }
        byte[] qa = new byte[n], qb = new byte[n];
        short[] da = quantize(a, qa), db = quantize(b, qb);

        float sumf = 0;
        for (int ib = 0; ib < n / 32; ib++) {
            int sumi = 0;
            for (int j = 0; j < 32; j++) sumi += qa[ib * 32 + j] * qb[ib * 32 + j];
            sumf += sumi * (Float.float16ToFloat(da[ib]) * Float.float16ToFloat(db[ib]));
        }
        double exact = 0;
        for (int i = 0; i < n; i++) exact += (double) a[i] * b[i];
        System.out.printf(Locale.ROOT, "Q8_0 x Q8_0 = %.6f%n", sumf);
        System.out.printf(Locale.ROOT, "exact       = %.6f%n", exact);
        double sumAbs = 0;
        for (int i = 0; i < n; i++) sumAbs += Math.abs(a[i] * b[i]);
        System.out.printf(Locale.ROOT, "error       = %.6f (%.3f%% of the sum of |a*b|)%n", sumf - exact, 100 * Math.abs(sumf - exact) / sumAbs);
        System.out.println("blocks = " + n / 32 + ", int8 products = " + n + ", float multiply-adds = " + 2 * n / 32);
    }
}
Output
Q8_0 x Q8_0 = -1.570552
exact       = -1.566664
error       = -0.003888 (0.008% of the sum of |a*b|)
blocks = 128, int8 products = 4096, float multiply-adds = 256

The Q8_0 result is -1.570552 and the exact value is -1.566664. The error is reported relative to the sum of absolute products, which gives 0.008%. A plain relative error would be misleading, because a dot product of random vectors can land close to zero.

The kernel did 4,096 integer products and only 256 float operations (two per block of 128 blocks). In llama.cpp, the matrix-vector product for a Q8_0 weight tensor runs this function once per output row, with b being the activation vector quantized to Q8_0 once per token.

Example 11: Q4_0 weights by Q8_0 activations, as the CPU backend computes it

Weights and activations do not need the same format. For Q4_0 weights, the CPU backend's type table in ggml-cpu.c sets vec_dot_type = GGML_TYPE_Q8_0. This means that before a matrix-vector product, the activation vector is quantized to Q8_0, and ggml_vec_dot_q4_0_q8_0 multiplies each Q4_0 block with the matching Q8_0 block.

For one block, the kernel keeps two integer sums. One is for the low nibbles (values j times activations j). The other is for the high nibbles (values j + 16 times activations j + 16). Then it adds sumi * dx * dy to the total. This is why the nibble order of Example 5 matters.

The program quantizes a weight vector to Q4_0 and an activation vector to Q8_0, runs this kernel, and compares the result with two references: the exact answer, and the answer with Q4_0 weights but float activations.

Java
import java.util.Locale;
import java.util.Random;

public class DotQ4_0Q8_0 {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    public static void main(String[] args) {
        int n = 4096, nb = n / 32;
        Random rnd = new Random(23);
        float[] w = new float[n], x = new float[n];
        for (int i = 0; i < n; i++) { w[i] = (float) rnd.nextGaussian() * 0.02f; x[i] = (float) rnd.nextGaussian(); }

        short[] dw = new short[nb], dx = new short[nb];
        byte[] qw = new byte[n / 2], qx = new byte[n];
        float[] wDeq = new float[n];
        for (int b = 0; b < nb; b++) {
            float amax = 0, max = 0;                                         // Q4_0 weights
            for (int j = 0; j < 32; j++) { float v = w[b * 32 + j]; if (amax < Math.abs(v)) { amax = Math.abs(v); max = v; } }
            float d = max / -8f, id = d != 0 ? 1.0f / d : 0f;
            dw[b] = Float.floatToFloat16(d);
            for (int j = 0; j < 16; j++) {
                int lo = Math.min(15, (int) (w[b * 32 + j] * id + 8.5f)), hi = Math.min(15, (int) (w[b * 32 + j + 16] * id + 8.5f));
                qw[b * 16 + j] = (byte) (lo | (hi << 4));
                wDeq[b * 32 + j] = (lo - 8) * Float.float16ToFloat(dw[b]);
                wDeq[b * 32 + j + 16] = (hi - 8) * Float.float16ToFloat(dw[b]);
            }
            float xa = 0;                                                    // Q8_0 activations
            for (int j = 0; j < 32; j++) xa = Math.max(xa, Math.abs(x[b * 32 + j]));
            float s = xa / 127f, is = s != 0 ? 1.0f / s : 0f;
            dx[b] = Float.floatToFloat16(s);
            for (int j = 0; j < 32; j++) qx[b * 32 + j] = (byte) roundf(x[b * 32 + j] * is);
        }

        float sumf = 0;                                                      // ggml_vec_dot_q4_0_q8_0
        for (int ib = 0; ib < nb; ib++) {
            int sumi0 = 0, sumi1 = 0;
            for (int j = 0; j < 16; j++) {
                int v0 = (qw[ib * 16 + j] & 0x0F) - 8, v1 = ((qw[ib * 16 + j] & 0xFF) >> 4) - 8;
                sumi0 += v0 * qx[ib * 32 + j];
                sumi1 += v1 * qx[ib * 32 + j + 16];
            }
            sumf += (sumi0 + sumi1) * Float.float16ToFloat(dw[ib]) * Float.float16ToFloat(dx[ib]);
        }
        double exact = 0, weightOnly = 0, sumAbs = 0;
        for (int i = 0; i < n; i++) { exact += (double) w[i] * x[i]; weightOnly += (double) wDeq[i] * x[i]; sumAbs += Math.abs(w[i] * x[i]); }
        System.out.printf(Locale.ROOT, "exact (FP32 weights, FP32 x)      = %.6f%n", exact);
        System.out.printf(Locale.ROOT, "Q4_0 weights, FP32 x              = %.6f (error %.3f%%)%n", weightOnly, 100 * Math.abs(weightOnly - exact) / sumAbs);
        System.out.printf(Locale.ROOT, "Q4_0 weights, Q8_0 x (CPU kernel) = %.6f (error %.3f%%)%n", sumf, 100 * Math.abs(sumf - exact) / sumAbs);
    }
}
Output
exact (FP32 weights, FP32 x)      = -0.231642
Q4_0 weights, FP32 x              = -0.338090 (error 0.203%)
Q4_0 weights, Q8_0 x (CPU kernel) = -0.343866 (error 0.214%)

The exact value is -0.231642. With Q4_0 weights and float activations the result is -0.338090 (error 0.203% of the sum of absolute products). With Q8_0 activations too, which is what the CPU kernel does, it is -0.343866 (error 0.214%).

Most of the error comes from the 4-bit weights. Quantizing the activations to Q8_0 adds a much smaller amount, because 8 bits per activation with one scale per 32 values is fine-grained. That is the reason for this design: the integer kernel is much faster than converting every weight to a float, and the extra activation error is small.

The weight row is never expanded to floats. The nibbles go straight into integer multiply-adds. On x86, the AVX2 version of this kernel unpacks 32 nibbles into bytes with a shift and a mask, subtracts 8, and uses byte multiply-add instructions. Section 7 writes the same kernel with the Java Vector API, in the lesson on quantized kernels later in this course.

One detail about the activation quantizer on x86 (quantize_row_q8_0 in ggml-cpu/arch/x86/quants.c): it computes id as 127 / amax and rounds with the SIMD instruction's round-to-nearest-even mode. In rare tie cases its codes can differ by one from the reference function used here. Weight files are always written by the reference functions, so this affects only activations at run time.

Example 12: an error table for F16, Q8_0, Q5_0, Q4_1, and Q4_0

With all formats available, they can be compared on the same data. The program quantizes 64K (65,536) seeded Gaussian values with a few outliers (one value in 500 is multiplied by 10). It uses each format's exact ggml rounding and reports the relative RMSE, the signal-to-noise ratio, and the bits per weight. F16 is included as the baseline, because most model files are converted from F16.

Java
import java.util.Locale;
import java.util.Random;

public class ErrorTable {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
    static float h(float v) { return Float.float16ToFloat(Float.floatToFloat16(v)); }
    static float maxSigned(float[] x, int o) { float a = 0, m = 0; for (int j = 0; j < 32; j++) if (a < Math.abs(x[o + j])) { a = Math.abs(x[o + j]); m = x[o + j]; } return m; }

    static float[] fake(String type, float[] x) {                 // quantize, then dequantize
        float[] y = new float[x.length];
        for (int o = 0; o < x.length; o += 32) {
            switch (type) {
                case "F16" -> { for (int j = 0; j < 32; j++) y[o + j] = h(x[o + j]); }
                case "Q8_0" -> { float a = 0; for (int j = 0; j < 32; j++) a = Math.max(a, Math.abs(x[o + j]));
                    float d = a / 127f, id = d != 0 ? 1f / d : 0f;
                    for (int j = 0; j < 32; j++) y[o + j] = roundf(x[o + j] * id) * h(d); }
                case "Q5_0" -> { float d = maxSigned(x, o) / -16f, id = d != 0 ? 1f / d : 0f;
                    for (int j = 0; j < 32; j++) y[o + j] = (Math.min(31, (int) (x[o + j] * id + 16.5f)) - 16) * h(d); }
                case "Q4_0" -> { float d = maxSigned(x, o) / -8f, id = d != 0 ? 1f / d : 0f;
                    for (int j = 0; j < 32; j++) y[o + j] = (Math.min(15, (int) (x[o + j] * id + 8.5f)) - 8) * h(d); }
                case "Q4_1" -> { float mn = Float.MAX_VALUE, mx = -Float.MAX_VALUE;
                    for (int j = 0; j < 32; j++) { mn = Math.min(mn, x[o + j]); mx = Math.max(mx, x[o + j]); }
                    float d = (mx - mn) / 15f, id = d != 0 ? 1f / d : 0f;
                    for (int j = 0; j < 32; j++) y[o + j] = Math.min(15, (int) ((x[o + j] - mn) * id + 0.5f)) * h(d) + h(mn); }
                default -> throw new IllegalArgumentException(type);
            }
        }
        return y;
    }

    public static void main(String[] args) {
        Random rnd = new Random(24);
        float[] x = new float[65536];
        for (int i = 0; i < x.length; i++) x[i] = (float) rnd.nextGaussian() * 0.02f * (rnd.nextInt(500) == 0 ? 10 : 1);
        String[] types = {"F16", "Q8_0", "Q5_0", "Q4_1", "Q4_0"};
        double[] bpw = {16, 8.5, 5.5, 5.0, 4.5};
        System.out.println("type   bits/weight   relative RMSE   SNR dB");
        for (int t = 0; t < types.length; t++) {
            float[] y = fake(types[t], x);
            double e = 0, p = 0;
            for (int i = 0; i < x.length; i++) { e += Math.pow(y[i] - x[i], 2); p += x[i] * x[i]; }
            System.out.printf(Locale.ROOT, "%-5s %12.1f %14.4f%% %8.2f%n", types[t], bpw[t], 100 * Math.sqrt(e / p), 10 * Math.log10(p / e));
        }
    }
}
Output
type   bits/weight   relative RMSE   SNR dB
F16           16.0         0.0213%    73.44
Q8_0           8.5         0.7205%    42.85
Q5_0           5.5         5.7053%    24.87
Q4_1           5.0         8.6991%    21.21
Q4_0           4.5        11.1656%    19.04

The table orders the formats as their bit counts suggest. F16 has almost no error (0.02%). Q8_0's error is under 1%. Q5_0 and Q4_1 sit between Q8_0 and Q4_0, and Q4_0 has the largest error at 11.17%.

Q4_1 spends its extra half bit on a minimum, which helps less than Q5_0's extra bit per value on data centered at zero (Example 7), so Q4_1 is worse than Q5_0 despite using fewer bits (5.0 against 5.5).

These are weight errors on synthetic data. Real models have structure: some tensors and some layers are more sensitive than others. The k-quant mixes in lesson 22 exploit this by giving sensitive tensors more bits.

Example 13: matrix-vector output error against FP32

What a layer passes on is y = W x, not the weights themselves. The program builds a 256 by 4,096 weight matrix like the data in Example 12 and quantizes it to F16, Q8_0, and Q4_0. It runs the matrix-vector product with Q8_0 activations for the quantized formats (as the CPU backend does) and with FP32 activations for F16. Then it reports the relative output error against the exact FP32 result.

Java
import java.util.Locale;
import java.util.Random;

public class OutputErrorByFormat {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
    static float h(float v) { return Float.float16ToFloat(Float.floatToFloat16(v)); }

    static float[] fake(String t, float[] x) {               // quantize and dequantize, rules as in Example 12
        float[] y = new float[x.length];
        for (int o = 0; o < x.length; o += 32) {
            float a = 0, m = 0;
            for (int j = 0; j < 32; j++) if (a < Math.abs(x[o + j])) { a = Math.abs(x[o + j]); m = x[o + j]; }
            float d = t.equals("Q8_0") ? a / 127f : m / -8f, id = d != 0 ? 1f / d : 0f;
            for (int j = 0; j < 32; j++) y[o + j] = switch (t) {
                case "F16" -> h(x[o + j]);
                case "Q8_0" -> roundf(x[o + j] * id) * h(d);
                default -> (Math.min(15, (int) (x[o + j] * id + 8.5f)) - 8) * h(d);
            };
        }
        return y;
    }

    public static void main(String[] args) {
        int rows = 256, cols = 4096;
        Random rnd = new Random(25);
        float[] w = new float[rows * cols], x = new float[cols];
        for (int i = 0; i < w.length; i++) w[i] = (float) rnd.nextGaussian() * 0.02f * (rnd.nextInt(500) == 0 ? 10 : 1);
        for (int i = 0; i < cols; i++) x[i] = (float) rnd.nextGaussian();
        float[] xq = fake("Q8_0", x);
        double[] ref = new double[rows];
        for (int r = 0; r < rows; r++) for (int c = 0; c < cols; c++) ref[r] += (double) w[r * cols + c] * x[c];

        System.out.println("weights  activations  relative output error");
        for (String t : new String[] {"F16", "Q8_0", "Q4_0"}) {
            float[] wq = fake(t, w);
            float[] act = t.equals("F16") ? x : xq;
            double e = 0, p = 0;
            for (int r = 0; r < rows; r++) {
                double y = 0;
                for (int c = 0; c < cols; c++) y += (double) wq[r * cols + c] * act[c];
                e += (y - ref[r]) * (y - ref[r]); p += ref[r] * ref[r];
            }
            System.out.printf(Locale.ROOT, "%-8s %-12s %10.4f%%%n", t, t.equals("F16") ? "FP32" : "Q8_0", 100 * Math.sqrt(e / p));
        }
    }
}
Output
weights  activations  relative output error
F16      FP32             0.0207%
Q8_0     Q8_0             0.9605%
Q4_0     Q8_0            11.6399%

The output errors follow the weight errors of Example 12 closely. Random rounding errors in 4,096 products add like noise, so the relative output error is about the relative weight error. The Q8_0 row (0.96%) includes the Q8_0 activation error too and is still under 1%. Q4_0 is at 11.64%.

For a whole model, these per-layer errors compound through dozens of layers and through the residual stream (the running sum that carries information from layer to layer). Lesson 24 measures the end result with perplexity and KL divergence on a real model (Making, Quantizing, and Testing Your Own GGUF).

Example 14: a byte-exact cross-check with gguf-py

A quantizer that writes GGUF files must produce exactly the bytes llama.cpp produces. One way to test that is to compare against a second implementation. gguf-py's quants.py reimplements the reference quantizers in NumPy, and its source documents it as bit-exact for Q8_0.

The Java program prints the hex bytes of all four formats for two blocks of fixed input. The first block is the one used throughout this lesson. The second contains an outlier of 9.5 at index 40, which changes the scales and the codes.

Java
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.HexFormat;

public class HexCrossCheck {
    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
    static float maxSigned(float[] x, int o) { float a = 0, m = 0; for (int j = 0; j < 32; j++) if (a < Math.abs(x[o + j])) { a = Math.abs(x[o + j]); m = x[o + j]; } return m; }

    static byte[] quantize(String type, float[] x, int o) {
        ByteBuffer b = ByteBuffer.allocate(switch (type) { case "Q8_0" -> 34; case "Q4_0" -> 18; case "Q4_1" -> 20; default -> 22; })
                .order(ByteOrder.LITTLE_ENDIAN);
        switch (type) {
            case "Q8_0" -> { float a = 0; for (int j = 0; j < 32; j++) a = Math.max(a, Math.abs(x[o + j]));
                float d = a / 127f, id = d != 0 ? 1f / d : 0f; b.putShort(Float.floatToFloat16(d));
                for (int j = 0; j < 32; j++) b.put((byte) roundf(x[o + j] * id)); }
            case "Q4_0" -> { float d = maxSigned(x, o) / -8f, id = d != 0 ? 1f / d : 0f; b.putShort(Float.floatToFloat16(d));
                for (int j = 0; j < 16; j++) b.put((byte) (Math.min(15, (int) (x[o + j] * id + 8.5f)) | Math.min(15, (int) (x[o + j + 16] * id + 8.5f)) << 4)); }
            case "Q4_1" -> { float mn = Float.MAX_VALUE, mx = -Float.MAX_VALUE;
                for (int j = 0; j < 32; j++) { mn = Math.min(mn, x[o + j]); mx = Math.max(mx, x[o + j]); }
                float d = (mx - mn) / 15f, id = d != 0 ? 1f / d : 0f;
                b.putShort(Float.floatToFloat16(d)).putShort(Float.floatToFloat16(mn));
                for (int j = 0; j < 16; j++) b.put((byte) (Math.min(15, (int) ((x[o + j] - mn) * id + 0.5f)) | Math.min(15, (int) ((x[o + j + 16] - mn) * id + 0.5f)) << 4)); }
            default -> { float d = maxSigned(x, o) / -16f, id = d != 0 ? 1f / d : 0f; int qh = 0; byte[] qs = new byte[16];
                for (int j = 0; j < 16; j++) {
                    int lo = Math.min(31, (int) (x[o + j] * id + 16.5f)), hi = Math.min(31, (int) (x[o + j + 16] * id + 16.5f));
                    qs[j] = (byte) ((lo & 0x0F) | ((hi & 0x0F) << 4));
                    qh |= ((lo & 0x10) >> 4) << j | ((hi & 0x10) >> 4) << (j + 16);
                }
                b.putShort(Float.floatToFloat16(d)).putInt(qh).put(qs); }
        }
        return b.array();
    }

    public static void main(String[] args) {
        float[] x = new float[64];
        for (int i = 0; i < 64; i++) x[i] = (float) (((i * 7919) % 1001) - 500) / 337f;
        x[40] = 9.5f;
        for (int blk = 0; blk < 2; blk++)
            for (String t : new String[] {"Q8_0", "Q4_0", "Q4_1", "Q5_0"})
                System.out.println(t + " " + HexFormat.of().formatHex(quantize(t, x, blk * 32)));
    }
}
Output
Q8_0 fb218169523b250ef8e1cab49d876e58412a14fde7d0b9a38c745d47301903ecd6bf
Q4_0 ef31908f6d5c4a2917f6e5c3b2a08f7e5c4b
Q4_1 0d32efbd908e6d5c3a2917f6e5c3b2908f7d5c4a
Q5_0 ef2d3ef0831f200ddaa775421ffcc99664310eebb885
Q8_0 ca2cf2ef130f0c0805017ffaf7f3f014100d090602fffbf8f4f1ed110e0a070300fc
Q4_0 c0bc79898787879798989078797989878787
Q4_1 d339afbd31302424131312020f41313130242423
Q5_0 c0b8831efce1f2f20e0e1f1f2f2020e1e1f2f20e0e1e

The same input built in NumPy, quantized with gguf-py's reference quantizers:

Terminal
cat > cross.py <<'EOF'
import numpy as np
from gguf.quants import quantize
from gguf.constants import GGMLQuantizationType as T
i = np.arange(64)
x = (((i * 7919) % 1001) - 500).astype(np.float32) / np.float32(337)
x[40] = np.float32(9.5)
for blk in range(2):
    b = x[blk*32:(blk+1)*32].reshape(1, 32)
    for name in ["Q8_0", "Q4_0", "Q4_1", "Q5_0"]:
        print(name, quantize(b, getattr(T, name)).tobytes().hex())
EOF
PYTHONPATH=llama.cpp/gguf-py python3 cross.py

Output (from gguf-py 0.19.0 with NumPy; deterministic):

Output
Q8_0 fb218169523b250ef8e1cab49d876e58412a14fde7d0b9a38c745d47301903ecd6bf
Q4_0 ef31908f6d5c4a2917f6e5c3b2a08f7e5c4b
Q4_1 0d32efbd908e6d5c3a2917f6e5c3b2908f7d5c4a
Q5_0 ef2d3ef0831f200ddaa775421ffcc99664310eebb885
Q8_0 ca2cf2ef130f0c0805017ffaf7f3f014100d090602fffbf8f4f1ed110e0a070300fc
Q4_0 c0bc79898787879798989078797989878787
Q4_1 d339afbd31302424131312020f41313130242423
Q5_0 c0b8831efce1f2f20e0e1f1f2f2020e1e1f2f20e0e1e

All eight lines are identical to the Java output. Both sides compute the inputs with IEEE float32 division, which is correctly rounded in Java and in NumPy, so the inputs are bit-identical too.

The same comparison over 20,000 seeded blocks also matched for all four formats. Those blocks held Gaussian values, many exact quarter-steps that create rounding ties, and one large outlier per 97 values.

To see how to read a line, take the second Q4_0 line. c0bc is the FP16 scale 0xBCC0 (little-endian, low byte first), which is about -1.19 and equals 9.5 / -8. The first data byte 79 holds code 9 for value 32 in its low nibble and code 7 for value 48 in its high nibble.

Example 15: Project 5, a quantizer with an error report

Project 5 combines the lesson into one tool. Quantizer converts a float tensor (rows by columns) to Q8_0 or Q4_0 bytes, dequantizes the bytes back, and reports size, bits per weight, weight error, and matrix-vector output error. It is built in three stages, each a part of the program:

  1. Formats as an enum. GgmlType holds the block size and bytes per block, and has quantizeBlock and dequantizeBlock methods with ggml's exact rules (Examples 2, 4, 5, 6).
  2. Tensor conversion. quantize walks rows and blocks into one little-endian ByteBuffer and checks the row length (Example 9).
  3. Report. The report compares sizes and errors with F32 and checks that re-quantizing the dequantized tensor reproduces the same bytes (the fixed-point test from Example 4).

The main method quantizes a 512 by 2048 tensor with both formats, prints one table row each, and then tries a row length of 100 to show the error check.

Java
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Arrays;
import java.util.Locale;
import java.util.Random;

public class Quantizer {
    enum GgmlType {
        Q8_0(32, 34), Q4_0(32, 18);
        final int block, bytes;
        GgmlType(int block, int bytes) { this.block = block; this.bytes = bytes; }

        void quantizeBlock(float[] x, int o, ByteBuffer out) {
            if (this == Q8_0) {
                float a = 0; for (int j = 0; j < 32; j++) a = Math.max(a, Math.abs(x[o + j]));
                float d = a / 127f, id = d != 0 ? 1f / d : 0f;
                out.putShort(Float.floatToFloat16(d));
                for (int j = 0; j < 32; j++) out.put((byte) roundf(x[o + j] * id));
            } else {
                float a = 0, m = 0; for (int j = 0; j < 32; j++) if (a < Math.abs(x[o + j])) { a = Math.abs(x[o + j]); m = x[o + j]; }
                float d = m / -8f, id = d != 0 ? 1f / d : 0f;
                out.putShort(Float.floatToFloat16(d));
                for (int j = 0; j < 16; j++)
                    out.put((byte) (Math.min(15, (int) (x[o + j] * id + 8.5f)) | Math.min(15, (int) (x[o + j + 16] * id + 8.5f)) << 4));
            }
        }

        void dequantizeBlock(ByteBuffer in, float[] y, int o) {
            float d = Float.float16ToFloat(in.getShort());
            if (this == Q8_0) { for (int j = 0; j < 32; j++) y[o + j] = in.get() * d; }
            else for (int j = 0; j < 16; j++) { int b = in.get() & 0xFF; y[o + j] = ((b & 0x0F) - 8) * d; y[o + j + 16] = ((b >> 4) - 8) * d; }
        }
    }

    static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }

    static ByteBuffer quantize(GgmlType t, float[] w, int rows, int cols) {
        if (cols % t.block != 0) throw new IllegalArgumentException("row length " + cols + " is not a multiple of " + t.block);
        ByteBuffer out = ByteBuffer.allocate(rows * (cols / t.block) * t.bytes).order(ByteOrder.LITTLE_ENDIAN);
        for (int o = 0; o < w.length; o += t.block) t.quantizeBlock(w, o, out);
        return out.flip();
    }

    static float[] dequantize(GgmlType t, ByteBuffer data, int n) {
        float[] y = new float[n];
        ByteBuffer in = data.duplicate().order(ByteOrder.LITTLE_ENDIAN);
        for (int o = 0; o < n; o += t.block) t.dequantizeBlock(in, y, o);
        return y;
    }

    static double outputError(float[] w, float[] wq, float[] x, int rows, int cols) {
        double e = 0, p = 0;
        for (int r = 0; r < rows; r++) {
            double y = 0, yq = 0;
            for (int c = 0; c < cols; c++) { y += (double) w[r * cols + c] * x[c]; yq += (double) wq[r * cols + c] * x[c]; }
            e += (yq - y) * (yq - y); p += y * y;
        }
        return Math.sqrt(e / p);
    }

    public static void main(String[] args) {
        int rows = 512, cols = 2048;
        Random rnd = new Random(26);
        float[] w = new float[rows * cols], x = new float[cols];
        for (int i = 0; i < w.length; i++) w[i] = (float) rnd.nextGaussian() * 0.02f * (rnd.nextInt(500) == 0 ? 10 : 1);
        for (int i = 0; i < cols; i++) x[i] = (float) rnd.nextGaussian();

        System.out.printf(Locale.ROOT, "tensor %d x %d, F32 %d bytes%n", rows, cols, (long) w.length * 4);
        System.out.println("type  bytes    bits/weight  ratio  weight RMSE  output error  stable");
        for (GgmlType t : GgmlType.values()) {
            ByteBuffer q = quantize(t, w, rows, cols);
            float[] wq = dequantize(t, q, w.length);
            double e = 0, p = 0;
            for (int i = 0; i < w.length; i++) { e += Math.pow(wq[i] - w[i], 2); p += w[i] * w[i]; }
            boolean stable = quantize(t, wq, rows, cols).equals(q);
            System.out.printf(Locale.ROOT, "%-5s %-8d %11.2f %6.2f %11.3f%% %12.3f%%  %s%n", t, q.remaining(),
                    q.remaining() * 8.0 / w.length, (double) w.length * 4 / q.remaining(),
                    100 * Math.sqrt(e / p), 100 * outputError(w, wq, x, rows, cols), stable);
        }
        try { quantize(GgmlType.Q4_0, new float[100], 1, 100); }
        catch (IllegalArgumentException ex) { System.out.println("error: " + ex.getMessage()); }
    }
}
Output
tensor 512 x 2048, F32 4194304 bytes
type  bytes    bits/weight  ratio  weight RMSE  output error  stable
Q8_0  1114112         8.50   3.76       0.701%        0.690%  true
Q4_0  589824          4.50   7.11      11.022%       10.397%  true
error: row length 100 is not a multiple of 32

The report shows what lessons 18 to 21 predicted. Q8_0 at 8.5 bits per weight is 3.8 times smaller than F32 (3.76 in the table) with under 1% error. Q4_0 at 4.5 bits is 7.1 times smaller with an error about 15 times larger (11.0% against 0.7%). Both re-quantize to identical bytes (stable is true). The output error is close to the weight error in both cases, as in Example 13.

A row length that is not a multiple of 32 is rejected, as gguf-py does. llama-quantize does not stop in that case. It prints a warning and stores such a tensor in a fallback type (F16 for a 32-value format such as Q4_0).

Section 3 writes these bytes into a real GGUF file (The GGUF Format). Section 7 loads such a file and runs the dot products from Examples 10 and 11 with the Vector API.

Interview points

  • Q8_0 takes 34 bytes per 32 values: an FP16 scale d = amax / 127 and 32 int8 codes roundf(x / d). That is 8.5 bits per weight.
  • Q4_0 takes 18 bytes per 32 values. The scale is d = max / -8, taken from the signed value with the largest magnitude. Codes are min(15, (int)(x / d + 8.5)) and are stored as code - 8. That is 4.5 bits per weight.
  • Q4_0 nibble order: byte j holds value j in the low nibble and value j + 16 in the high nibble.
  • Q4_1 adds an FP16 minimum and decodes as code * d + m. Q5_0 stores the fifth bits of its 32 codes in a 32-bit word qh.
  • Codes are computed with the float inverse scale. Only the stored scale is rounded to FP16, which adds a small error at the block extremes.
  • C's roundf rounds halves away from zero and Java's Math.round does not, so a byte-exact port needs its own rounding helper.
  • The CPU backend quantizes activations to Q8_0 for Q4_0 and Q8_0 weights. It computes integer sums per block and uses two float multiplies per block.
  • Weight error and matrix-vector output error are close for random inputs. On Gaussian weights with outliers, Q8_0 is under 1% and Q4_0 is about 11%.
  • A ported quantizer is verified by comparing its bytes with gguf-py's quants.py on fixed inputs, including inputs that create rounding ties.

Summary

  • Block structs and sizes are defined in ggml-common.h with static_assert. gguf-py lists them in GGML_QUANT_SIZES: Q8_0 is 34, Q4_0 is 18, Q4_1 is 20, and Q5_0 is 22 bytes per 32 values.
  • The reference quantizers in ggml-quants.c are deterministic, and model files are written with them.
  • The FP16 scale is stored little-endian as the first two bytes of each block.
  • A Q4_0 block's extreme value maps to code 0 (meaning -8) and is stored exactly, up to the FP16 rounding of the scale.
  • Re-quantizing dequantized values reproduces the same bytes.
  • A 64 by 4,096 tensor is 278,528 bytes in Q8_0 and 147,456 bytes in Q4_0, with no padding between blocks or rows.
  • ggml_vec_dot_q8_0_q8_0 and ggml_vec_dot_q4_0_q8_0 sum integer products per block and then multiply by both FP16 scales.
  • The x86 activation quantizer rounds ties to even and computes 127 / amax, so it can differ from the reference in rare ties.
  • On Gaussian weights with outliers, error decreases in the order Q4_0, Q4_1, Q5_0, Q8_0, F16.
  • Java and gguf-py produced identical bytes for all four formats on the lesson's blocks and on 20,000 seeded blocks.
  • Quantizer (Project 5) reports bytes, bits per weight, weight RMSE, output error, and the fixed-point check for Q8_0 and Q4_0.