GPTQ, AWQ, EXL3, NF4, and FP8 Serving

k-quants, i-quants, and the Importance Matrix covered the formats that llama.cpp reads from GGUF files. Those formats are built so that they run well on a CPU or on a mix of CPU and GPU. Programs that run models on a GPU, such as vLLM, ExLlama, and Hugging Face Transformers (called "engines" in this lesson), use a different family of formats. They are stored in safetensors files, a plain format for saving tensors (tables of numbers), together with the extra tensors they need, such as scales.

The most important idea in this lesson is that the GPU methods ask a different question from the one used so far. Earlier lessons asked how close each rounded weight is to the original weight. The GPU methods ask how close the output of the layer is, because the output is what the rest of the model sees. Once that idea is clear, GPTQ and AWQ are two different ways to reach a better answer to the same question. The remaining methods (NF4, EXL2, EXL3, FP8, MXFP4, NVFP4) change the grid, the bit budget, or the number format.

After the lesson, you can look at a model name such as ...-GPTQ-Int4, ...-AWQ, ...-FP8, or ...-exl3-4.0bpw and say what is in the file and what hardware runs it fast.

The ideas of Quantization Fundamentals carry over: a quantizer picks a grid (the list of values it can store) and a scale (the number that stretches the grid to the weights' size), and its quality is the error it leaves. Two terms are used often. bpw means bits per weight, the file size in bits divided by the number of weights, scales included. An activation is a number the model computes while it runs, as opposed to a weight, which is stored in the file. The difference between the methods is which error they try to minimize.

Example 1: the layer output error as the real objective

A layer in a neural network computes y = W x: it multiplies a matrix of weights W by an input vector x. The problem is deciding which error a quantizer should make small. The obvious choice is the weight error, the distance between each original weight and its rounded copy. This example shows that the weight error can be the same for two choices while the error that matters is very different.

The program uses a layer with 2 weights whose two inputs are almost equal (input 1 is input 0 plus a little noise). It compares two quantized versions. Both move each weight by 0.01, so their weight errors are identical. Look at the "output error" column, which measures how much the layer's result changes on 1,000 sample inputs.

import java.util.Locale;
import java.util.Random;

public class OutputObjective {
    public static void main(String[] args) {
        Random rnd = new Random(1);
        int tokens = 1000;
        double[][] x = new double[tokens][2];
        for (double[] t : x) {
            t[0] = rnd.nextGaussian();
            t[1] = t[0] + 0.05 * rnd.nextGaussian();   // input 1 almost equals input 0
        }
        double[] w = {0.40, -0.25};
        double[][] candidates = {{0.41, -0.24}, {0.41, -0.26}};   // same size errors, different signs
        for (double[] q : candidates) {
            double weightErr = Math.pow(q[0] - w[0], 2) + Math.pow(q[1] - w[1], 2);
            double outErr = 0;
            for (double[] t : x) {
                double diff = (w[0] - q[0]) * t[0] + (w[1] - q[1]) * t[1];
                outErr += diff * diff / tokens;
            }
            System.out.println(String.format(Locale.ROOT, "q = [%.2f, %.2f]  weight error %.6f  output error %.6f",
                    q[0], q[1], weightErr, outErr));
        }
    }
}

q = [0.41, -0.24]  weight error 0.000200  output error 0.000410
q = [0.41, -0.26]  weight error 0.000200  output error 0.000000

A layer computes y = W x. What later layers see is y, not W. The quantity that matters is the output error ||W X - Ŵ X||^2 over realistic inputs X, where Ŵ is the quantized matrix. (The double bars mean the size of the difference, and X is a table of many input vectors.)

The output shows why the two errors differ. Both candidates have weight error 0.000200. In the first, both weights move up by 0.01. Because input 1 almost equals input 0, the two errors add up in the output: 0.000410. In the second, one weight moves up and the other down, so the errors cancel in the output: 0.000000 at six decimals. A small calculation shows it. With inputs x0 = x1 = 1, the first candidate changes the output by 0.01 * 1 + 0.01 * 1 = 0.02, and the second by 0.01 * 1 - 0.01 * 1 = 0.

Round-to-nearest (RTN) is the plain method: it rounds every weight to its closest grid value and treats each weight alone, so it ignores this effect. GPTQ and AWQ both use calibration inputs, a small sample of realistic inputs, to minimize the output error instead. They differ in how they do it. An objective measured on outputs needs inputs, which is why both methods ask for a calibration dataset (a few hundred samples of text) when they quantize a model.

Example 2: GPTQ: one column at a time with inverse-Hessian compensation

GPTQ (Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", 2022) builds directly on Example 1. It quantizes the columns of a weight matrix in order. After rounding each column, it adjusts the columns that are not rounded yet, so that they absorb the rounding error. The adjustment is chosen so that the layer's output stays as close as possible to the original.

The program implements this update on a layer with 16 inputs that are correlated (they share a common part, so they tend to move together) and compares the output error with RTN at 4 and 3 bits. A 4-bit grid has 16 levels and a 3-bit grid has 8, so the 3-bit case rounds much more coarsely. The error is measured on test inputs that were not used for calibration.

import java.util.Locale;
import java.util.Random;

public class GptqToy {
    static final int N = 16, ROWS = 8;

    // Symmetric grid with levels -(qmax+1) .. qmax: qmax 7 is 4-bit, qmax 3 is 3-bit.
    static double quant(double v, double s, int qmax) { return s * Math.max(-qmax - 1, Math.min(qmax, Math.rint(v / s))); }

    static double[][] inverse(double[][] a) {                    // Gauss-Jordan elimination
        int n = a.length; double[][] m = new double[n][2 * n];
        for (int i = 0; i < n; i++) { System.arraycopy(a[i], 0, m[i], 0, n); m[i][n + i] = 1; }
        for (int c = 0; c < n; c++) {
            double p = m[c][c];
            for (int k = 0; k < 2 * n; k++) m[c][k] /= p;
            for (int r = 0; r < n; r++) if (r != c) { double f = m[r][c]; for (int k = 0; k < 2 * n; k++) m[r][k] -= f * m[c][k]; }
        }
        double[][] inv = new double[n][n];
        for (int i = 0; i < n; i++) System.arraycopy(m[i], n, inv[i], 0, n);
        return inv;
    }

    static double[] gptqRow(double[] row, double[][] h, double s, int qmax) {
        double[] w = row.clone(), q = new double[N];
        double[][] hinv = inverse(h);
        for (int i = 0; i < N; i++) {
            q[i] = quant(w[i], s, qmax);
            double err = (w[i] - q[i]) / hinv[i][i];
            for (int j = i + 1; j < N; j++) w[j] -= err * hinv[i][j];      // compensate the columns not yet quantized
            double[][] next = new double[N][N];                             // remove column i from the inverse
            for (int a = 0; a < N; a++) for (int b = 0; b < N; b++) next[a][b] = hinv[a][b] - hinv[a][i] * hinv[i][b] / hinv[i][i];
            hinv = next;
        }
        return q;
    }

    static double[][] inputs(Random rnd, int tokens) {           // correlated input channels
        double[][] x = new double[tokens][N];
        for (double[] t : x) { double shared = rnd.nextGaussian(); for (int j = 0; j < N; j++) t[j] = shared + 0.5 * rnd.nextGaussian(); }
        return x;
    }

    public static void main(String[] args) {
        Random rnd = new Random(5);
        double[][] calib = inputs(rnd, 512), test = inputs(rnd, 512), w = new double[ROWS][N];
        double[][] h = new double[N][N];                              // H = 2 X^T X / tokens
        for (double[] t : calib) for (int a = 0; a < N; a++) for (int b = 0; b < N; b++) h[a][b] += 2 * t[a] * t[b] / calib.length;
        for (double[] row : w) for (int j = 0; j < N; j++) row[j] = rnd.nextGaussian() * 0.02;
        for (int qmax : new int[] {7, 3}) {
            double rtnErr = 0, gptqErr = 0;
            for (double[] row : w) {
                double amax = 0;
                for (double v : row) amax = Math.max(amax, Math.abs(v));
                double s = amax / qmax;
                double[] rtn = new double[N];
                for (int j = 0; j < N; j++) rtn[j] = quant(row[j], s, qmax);
                double[] g = gptqRow(row, h, s, qmax);
                for (double[] t : test) {
                    double y = 0, yr = 0, yg = 0;
                    for (int j = 0; j < N; j++) { y += row[j] * t[j]; yr += rtn[j] * t[j]; yg += g[j] * t[j]; }
                    rtnErr += (y - yr) * (y - yr); gptqErr += (y - yg) * (y - yg);
                }
            }
            System.out.println(String.format(Locale.ROOT, "%d-bit: round-to-nearest output MSE %.3e, GPTQ output MSE %.3e",
                    qmax == 7 ? 4 : 3, rtnErr / (ROWS * test.length), gptqErr / (ROWS * test.length)));
        }
    }
}

4-bit: round-to-nearest output MSE 1.870e-05, GPTQ output MSE 1.730e-05
3-bit: round-to-nearest output MSE 1.983e-04, GPTQ output MSE 7.524e-05

The update needs a matrix that describes how errors in different columns interact. For calibration inputs X, this matrix is H = 2 X^T X / tokens. It is the Hessian of the squared output error with respect to the weights. A Hessian is the matrix of second derivatives, which says how sharply the error grows when each weight (or pair of weights) is changed. GPTQ uses its inverse. The loop in gptqRow has four steps:

1. Quantize column i: q_i = quant(w_i).
2. Compute the scaled error err = (w_i - q_i) / Hinv[i][i].
3. Update every later column: w_j -= err * Hinv[i][j]. Columns correlated with column i move in the direction that cancels its error in the output, as in Example 1.
4. Remove column i from the inverse Hessian (the rank-one update in gptqRow) and continue with the next column.

The output shows that the effect grows as the grid gets coarser. At 4 bits, GPTQ lowers the output error by 7 percent (1.870e-05 to 1.730e-05). At 3 bits the RTN error is about 10 times larger (1.983e-04), and GPTQ cuts it by 62 percent (to 7.524e-05). The GPTQ paper reports its main results at 3 and 4 bits.

The paper adds three engineering changes that do not change the math:

• It computes the needed rows of the inverse with a Cholesky decomposition, a standard way to factor a matrix, for numerical stability.
• It applies updates in blocks of 128 columns to use the GPU efficiently ("lazy batch updates").
• It adds a small damping term (1 percent of the mean diagonal) to H, which keeps the inverse stable.

With these changes, the paper reports quantizing 175-billion-parameter models in about four GPU hours. All rows of a matrix share the same H, so the inverse is computed once per layer.

Example 3: GPTQ group size, act-order, and storage

A GPTQ checkpoint stores more than the codes. A group is a run of consecutive input columns that share one scale and one zero point per output row. (A zero point is a number that shifts the grid so that it covers a range not centered on zero.) Small groups follow the local range of the weights better but store more scales. This example puts numbers on that cost and also explains the option called act-order.

The program prints bits per weight for common group sizes, using the formula bits + (16 + bits) / group (a 16-bit scale and a zero point of bits bits per group). It then builds an act-order column permutation, that is, a new order for the columns, for a toy list of eight numbers.

import java.util.Arrays;
import java.util.Comparator;
import java.util.Locale;

public class GptqLayout {
    public static void main(String[] args) {
        int in = 4096, out = 4096;
        System.out.println("bits  group   bpw (weights + FP16 scale + packed zero)");
        for (int bits : new int[] {4, 3}) {
            for (int group : new int[] {32, 64, 128, in}) {
                double bpw = bits + (16.0 + bits) / group;
                System.out.println(String.format(Locale.ROOT, "%4d  %5s   %.4f", bits, group == in ? "row" : String.valueOf(group), bpw));
            }
        }
        System.out.println(String.format(Locale.ROOT, "act-order g_idx: 32 bits per input column = %.4f bpw", 32.0 / out));

        double[] hDiag = {0.8, 3.1, 0.2, 5.6, 1.0, 0.9, 2.4, 0.1};    // diagonal of H: mean x_j^2 per column
        Integer[] order = {0, 1, 2, 3, 4, 5, 6, 7};
        Arrays.sort(order, Comparator.comparingDouble((Integer j) -> -hDiag[j]));
        System.out.println("act-order column order: " + Arrays.toString(order));
        int[] gIdx = new int[8];
        for (int pos = 0; pos < 8; pos++) gIdx[order[pos]] = pos / 4;  // group size 4 in quantization order
        System.out.println("g_idx (group of each column): " + Arrays.toString(gIdx));
    }
}

bits  group   bpw (weights + FP16 scale + packed zero)
   4     32   4.6250
   4     64   4.3125
   4    128   4.1563
   4    row   4.0049
   3     32   3.5938
   3     64   3.2969
   3    128   3.1484
   3    row   3.0046
act-order g_idx: 32 bits per input column = 0.0078 bpw
act-order column order: [3, 1, 6, 4, 5, 0, 2, 7]
g_idx (group of each column): [1, 0, 1, 0, 0, 1, 0, 1]

Smaller groups cost more:

• Group 128 at 4 bits is 4.156 bpw: 4 bits of code, plus a 16-bit scale and a 4-bit zero point shared by 128 weights (20 / 128 = 0.156).
• Group 32 is 4.625 bpw, between Q4_0 (4.5) and Q4_1 (5.0).
• One group per row (written -1 in configuration files) is almost exactly 4.0 bpw.

The common checkpoint layout from the AutoGPTQ project stores each linear layer as several tensors:

A quantize_config.json file, or the quantization_config block in config.json, records bits, group_size, sym, and desc_act.

Act-order (desc_act: true) quantizes columns in order of decreasing H[j][j]. That number is twice the mean squared activation of column j, the same quantity (up to the factor 2) as the importance matrix in k-quants, i-quants, and the Importance Matrix. The columns with the largest inputs are quantized first, while many unquantized columns remain to absorb their error. In the output, column 3 (5.6) goes first and column 7 (0.1) goes last.

Groups are formed in quantization order, so the columns of a group are no longer next to each other in memory. The array g_idx records which group each column belongs to: [1, 0, 1, 0, 0, 1, 0, 1]. Columns 3, 1, 6, and 4 (the first four in the order) form group 0, which is why positions 1, 3, 4, and 6 hold 0. The extra index costs 32 bits per input column (0.0078 bpw for a 4096-wide layer). The larger cost is speed: kernels (GPU programs for one operation) must look up scales through it, and some kernels handle that more slowly than plain groups.

Example 4: AWQ: scale the salient input channels before quantizing

AWQ (Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", 2023) starts from an observation. A small fraction of input channels carry much larger activations than the rest, and the weights in those channels matter most, because every error in them is multiplied by a large input. The problem is how to protect them without storing them in higher precision, which would complicate the file.

The program gives 2 of 32 input channels activations 8 times larger than the others. It then tries scaling exponents alpha from 0 to 1. At alpha = 0 nothing is scaled, which is plain round-to-nearest. Compare the output error for each alpha.

import java.util.Locale;
import java.util.Random;

public class AwqToy {
    static final int N = 32, ROWS = 32;

    // 4-bit symmetric, one scale per row (a group of 32 weights).
    static double[] quantRow(double[] w) {
        double amax = 0;
        for (double v : w) amax = Math.max(amax, Math.abs(v));
        double s = amax / 7; double[] q = new double[w.length];
        for (int j = 0; j < w.length; j++) q[j] = s * Math.max(-8, Math.min(7, Math.rint(w[j] / s)));
        return q;
    }

    static double[][] inputs(Random rnd, int tokens) {
        double[][] x = new double[tokens][N];
        for (double[] t : x) for (int j = 0; j < N; j++) t[j] = rnd.nextGaussian() * (j < 2 ? 8.0 : 1.0);  // 2 salient channels
        return x;
    }

    public static void main(String[] args) {
        Random rnd = new Random(11);
        double[][] calib = inputs(rnd, 512), test = inputs(rnd, 512);
        double[] meanAbs = new double[N];
        for (double[] t : calib) for (int j = 0; j < N; j++) meanAbs[j] += Math.abs(t[j]) / calib.length;
        double[][] w = new double[ROWS][N];
        for (double[] row : w) for (int j = 0; j < N; j++) row[j] = rnd.nextGaussian() * 0.02;

        for (int a = 0; a <= 10; a += 2) {
            double alpha = a / 10.0, err = 0;
            double[] s = new double[N];
            for (int j = 0; j < N; j++) s[j] = Math.pow(meanAbs[j], alpha);
            for (double[] row : w) {
                double[] scaled = new double[N];
                for (int j = 0; j < N; j++) scaled[j] = row[j] * s[j];        // W * diag(s)
                double[] q = quantRow(scaled);
                for (double[] t : test) {
                    double y = 0, yq = 0;
                    for (int j = 0; j < N; j++) { y += row[j] * t[j]; yq += q[j] * (t[j] / s[j]); }  // X / s
                    err += (y - yq) * (y - yq);
                }
            }
            System.out.println(String.format(Locale.ROOT, "alpha %.1f  output MSE %.3e%s", alpha, err / (ROWS * test.length),
                    a == 0 ? "  (plain round-to-nearest)" : ""));
        }
    }
}

alpha 0.0  output MSE 5.149e-04  (plain round-to-nearest)
alpha 0.2  output MSE 3.132e-04
alpha 0.4  output MSE 3.349e-04
alpha 0.6  output MSE 5.585e-04
alpha 0.8  output MSE 1.179e-03
alpha 1.0  output MSE 2.489e-03

The trick rests on a fact about multiplication: W x does not change if column j of W is multiplied by s_j and input x_j is divided by s_j. For example, 0.02 * 8 is 0.16, and x / 8 undoes it exactly. AWQ uses that freedom in four steps:

1. Measure mean|x_j| (the average absolute value) per input channel on calibration data.
2. Set s_j = mean|x_j|^alpha.
3. Quantize W * diag(s) with plain round-to-nearest. (diag(s) is the matrix with the scales on its diagonal, so the notation means "multiply each column by its scale".)
4. At inference, divide the input by s. In practice the division is folded into the previous operation, such as a norm weight, so it costs nothing at run time.

Scaling a salient column up makes its weights larger relative to the group's step, so their relative rounding error shrinks. The cost is that the group's absmax (largest absolute value) may grow, which coarsens the grid for the other columns. The output shows both effects. alpha = 0 is plain RTN (5.149e-04). alpha = 0.2 cuts the output error by 39 percent (3.132e-04). Larger values protect the salient channels too much and enlarge the step for the rest, so the error rises again, to 2.489e-03 at alpha = 1.

AWQ picks alpha per layer by this kind of grid search on calibration data, which is the loop in the program. It does no reconstruction and no weight updates beyond the scaling, so it needs less calibration data than GPTQ and, according to the paper, depends less on which data is used. The paper reports that keeping about 1 percent of salient channels in FP16 already recovers most of the quality, and that scaling achieves the same without mixed precision.

Example 5: GPTQ and AWQ side by side

Examples 2 and 4 used different toy layers, so their numbers cannot be compared with each other. This example puts both methods on one layer. The program builds a layer whose 16 input channels are correlated (as in Example 2) and whose channels 0 and 1 are 8 times larger (as in Example 4). It quantizes the 16 rows with round-to-nearest, with GPTQ, and with AWQ at the best alpha found on the calibration inputs. The test error is measured on inputs that neither method saw.

import java.util.Arrays;
import java.util.Locale;
import java.util.Random;

public class GptqVsAwq {
    static final int N = 16, ROWS = 16;

    static double quant(double v, double s, int qmax) { return s * Math.max(-qmax - 1, Math.min(qmax, Math.rint(v / s))); }
    static double absmax(double[] r) { double m = 0; for (double v : r) m = Math.max(m, Math.abs(v)); return m; }

    static double[][] inverse(double[][] a) {                       // Gauss-Jordan, as in Example 2
        double[][] m = new double[N][2 * N];
        for (int i = 0; i < N; i++) { System.arraycopy(a[i], 0, m[i], 0, N); m[i][N + i] = 1; }
        for (int c = 0; c < N; c++) {
            double p = m[c][c];
            for (int k = 0; k < 2 * N; k++) m[c][k] /= p;
            for (int r = 0; r < N; r++) if (r != c) { double f = m[r][c]; for (int k = 0; k < 2 * N; k++) m[r][k] -= f * m[c][k]; }
        }
        double[][] inv = new double[N][N];
        for (int i = 0; i < N; i++) System.arraycopy(m[i], N, inv[i], 0, N);
        return inv;
    }

    static double[] gptq(double[] row, double[][] hinv, int qmax, boolean compensate) {   // Example 2
        double s = absmax(row) / qmax; double[] w = row.clone(), q = new double[N];
        for (int i = 0; i < N; i++) {
            q[i] = quant(w[i], s, qmax);
            if (!compensate) continue;                                // plain round-to-nearest
            double err = (w[i] - q[i]) / hinv[i][i];
            for (int j = i + 1; j < N; j++) w[j] -= err * hinv[i][j];
            double[][] next = new double[N][N];
            for (int a = 0; a < N; a++) for (int b = 0; b < N; b++) next[a][b] = hinv[a][b] - hinv[a][i] * hinv[i][b] / hinv[i][i];
            hinv = next;
        }
        return q;
    }

    static double[][] inputs(Random rnd, int tokens) {              // correlated channels; channels 0 and 1 salient
        double[][] x = new double[tokens][N];
        for (double[] t : x) { double sh = rnd.nextGaussian(); for (int j = 0; j < N; j++) t[j] = (sh + 0.5 * rnd.nextGaussian()) * (j < 2 ? 8 : 1); }
        return x;
    }

    // Quantizes every row (AWQ scaling s, then GPTQ or RTN) and returns the output MSE on inputs x.
    static double run(double[][] w, double[][] hinv, double[] s, int qmax, boolean gptq, double[][] x) {
        double e = 0;
        for (double[] row : w) {
            double[] scaled = new double[N];
            for (int j = 0; j < N; j++) scaled[j] = row[j] * s[j];
            double[] q = gptq(scaled, hinv, qmax, gptq);
            for (double[] t : x) {
                double y = 0, yq = 0;
                for (int j = 0; j < N; j++) { y += row[j] * t[j]; yq += q[j] * t[j] / s[j]; }
                e += (y - yq) * (y - yq);
            }
        }
        return e / (w.length * x.length);
    }

    public static void main(String[] args) {
        Random rnd = new Random(5);
        double[][] calib = inputs(rnd, 512), test = inputs(rnd, 512), w = new double[ROWS][N], h = new double[N][N];
        for (double[] row : w) for (int j = 0; j < N; j++) row[j] = rnd.nextGaussian() * 0.02;
        double[] meanAbs = new double[N], ones = new double[N];
        Arrays.fill(ones, 1.0);
        for (double[] t : calib) for (int a = 0; a < N; a++) {
            meanAbs[a] += Math.abs(t[a]) / calib.length;
            for (int b = 0; b < N; b++) h[a][b] += 2 * t[a] * t[b] / calib.length;
        }
        double[][] hinv = inverse(h);
        for (int qmax : new int[] {7, 3}) {
            double[] best = ones; double bestErr = Double.MAX_VALUE, bestAlpha = 0;
            for (int a = 0; a <= 10; a++) {                             // AWQ: grid search alpha on calibration data
                double[] s = new double[N];
                for (int j = 0; j < N; j++) s[j] = Math.pow(meanAbs[j], a / 10.0);
                double err = run(w, hinv, s, qmax, false, calib);
                if (err < bestErr) { bestErr = err; best = s; bestAlpha = a / 10.0; }
            }
            System.out.println(String.format(Locale.ROOT, "%d-bit test MSE: RTN %.3e  GPTQ %.3e  AWQ (alpha %.1f) %.3e",
                    qmax == 7 ? 4 : 3, run(w, hinv, ones, qmax, false, test), run(w, hinv, ones, qmax, true, test),
                    bestAlpha, run(w, hinv, best, qmax, false, test)));
        }
    }
}

4-bit test MSE: RTN 5.443e-04  GPTQ 1.264e-04  AWQ (alpha 0.4) 1.702e-04
3-bit test MSE: RTN 2.247e-03  GPTQ 6.565e-04  AWQ (alpha 0.2) 1.404e-03

The helper gptq is the column loop from Example 2. With compensate set to false it is plain round-to-nearest. AWQ is the same rounding applied to W * diag(s), with s = mean|x|^alpha as in Example 4.

The output shows two things:

• Both methods beat round-to-nearest at both bit widths. At 4 bits GPTQ cuts the error by 77 percent (5.443e-04 to 1.264e-04) and AWQ by 69 percent (to 1.702e-04).
• At 3 bits the gap between them widens: GPTQ keeps a 71 percent reduction (6.565e-04), while AWQ reaches only 38 percent (1.404e-03). AWQ can only rescale whole channels, so it cannot cancel errors between correlated channels. GPTQ's column updates do exactly that.

The toy has one layer and synthetic inputs. Published comparisons on real models depend on the model, the bit width, and the calibration data, so the result here shows the mechanism, not a ranking. The table compares the methods by design.

Both are weight-only methods. The activations stay in FP16 or BF16 (16-bit float formats), and the kernel converts the weights back to 16 bits on the fly, which is called dequantizing. That saves memory and memory bandwidth, which is what limits single-user decode (the step that produces one token at a time; see the roofline model). It does not use 4-bit arithmetic. Methods that also quantize activations (FP8 W8A8, Example 9) speed up compute-bound work, meaning work limited by arithmetic speed rather than memory, such as large batches and long prompts.

Example 6: bitsandbytes NF4 and double quantization

The grids so far are evenly spaced. Example 10 of k-quants, i-quants, and the Importance Matrix showed that placing levels closer together near zero lowers the error, because trained weights cluster near zero. NF4 ("4-bit NormalFloat", from the QLoRA paper by Dettmers et al., 2023) takes this idea to its logical end. It places its 16 levels at quantiles of a normal distribution (a quantile is a cut point that leaves a given share of the data below it), so that each level covers about the same share of normally distributed weights. Each block of weights is scaled by its absmax, then rounded to the nearest level.

The program rebuilds the table with the same construction as create_normal_map in the bitsandbytes source. It then compares NF4 with uniform INT4 (evenly spaced 4-bit integers) on Gaussian blocks of 64 weights, and prints the cost of the scales.

import java.util.Locale;
import java.util.Random;

public class Nf4Table {
    static double cdf(double z) {                                 // standard normal CDF by Simpson integration
        int n = 20000; double a = 0, h = z / n, sum = 0;
        for (int i = 0; i <= n; i++) {
            double x = a + i * h, f = Math.exp(-x * x / 2);
            sum += (i == 0 || i == n) ? f : (i % 2 == 1 ? 4 * f : 2 * f);
        }
        return 0.5 + sum * h / 3 / Math.sqrt(2 * Math.PI);
    }

    static double ppf(double p) {                                 // inverse CDF by bisection
        double lo = -10, hi = 10;
        for (int i = 0; i < 60; i++) { double mid = (lo + hi) / 2; if (cdf(mid) < p) lo = mid; else hi = mid; }
        return (lo + hi) / 2;
    }

    public static void main(String[] args) {
        double offset = 0.9677083;
        double[] v = new double[16];
        for (int i = 0; i < 8; i++) v[8 + i] = ppf(offset + (0.5 - offset) * i / 8.0);   // 8 positive
        for (int i = 0; i < 7; i++) v[i] = -ppf(offset + (0.5 - offset) * i / 7.0);      // 7 negative
        java.util.Arrays.sort(v);
        double max = v[15];
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < 16; i++) { v[i] /= max; sb.append(String.format(Locale.ROOT, "%.4f ", v[i])); }
        System.out.println("NF4: " + sb.toString().trim());

        Random rnd = new Random(42);
        double errNf4 = 0, errInt4 = 0; int n = 64 * 256;
        for (int b = 0; b < 256; b++) {
            double[] x = new double[64]; double amax = 0;
            for (int j = 0; j < 64; j++) { x[j] = rnd.nextGaussian(); amax = Math.max(amax, Math.abs(x[j])); }
            for (double xj : x) {
                double best = 0;
                for (double c : v) if (Math.abs(xj / amax - c) < Math.abs(xj / amax - best)) best = c;
                errNf4 += Math.pow(xj - best * amax, 2);
                double s = amax / 7;
                errInt4 += Math.pow(xj - s * Math.max(-8, Math.min(7, Math.rint(xj / s))), 2);
            }
        }
        System.out.println(String.format(Locale.ROOT, "block 64, absmax scale: NF4 RMSE %.4f, INT4 RMSE %.4f",
                Math.sqrt(errNf4 / n), Math.sqrt(errInt4 / n)));
        System.out.println(String.format(Locale.ROOT, "scale bits per weight: FP32 per 64 = %.3f, double-quantized = %.3f",
                32.0 / 64, 8.0 / 64 + 32.0 / (64 * 256)));
    }
}

NF4: -1.0000 -0.6962 -0.5251 -0.3949 -0.2844 -0.1848 -0.0910 0.0000 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.7230 1.0000
block 64, absmax scale: NF4 RMSE 0.0927, INT4 RMSE 0.1070
scale bits per weight: FP32 per 64 = 0.500, double-quantized = 0.127

The construction in bitsandbytes is:

• 8 positive values: the normal inverse CDF (the function that turns a probability into a position on the bell curve) at 8 evenly spaced probabilities from 0.9677083 down to 0.5. The last point, 0.5, gives the value 0 and is dropped.
• 7 negative values: the same with 7 points, negated.
• One exact zero.
• All values divided by the largest, so the table spans -1 to 1.

The 16 values in the output match the table hard-coded in bitsandbytes functional.py to four decimals (-1.0, -0.6962, ..., 0.7230, 1.0). The offset 0.9677083 controls how far into the tails the outermost levels reach. The bitsandbytes docstring says it covers about 1.845 standard deviations. Each level then covers a region of about equal probability for normal data, so levels are dense near zero, as in IQ4_NL. On Gaussian blocks NF4 gives RMSE (the typical size of the rounding error) of 0.0927 against 0.1070 for uniform INT4 with the same absmax scale.

Scales have a cost too. bitsandbytes stores one FP32 absmax per 64 weights (the default blocksize), which costs 32 / 64 = 0.5 bpw. Double quantization (compress_statistics=True in quantize_4bit) reduces this by quantizing the scales themselves. It subtracts the mean of the absmax values and stores them as 8-bit numbers in blocks of 256, with their own FP32 scales. The cost drops to 8/64 + 32/(64*256) = 0.127 bpw, the figure given in the QLoRA paper. This is the same "quantize the scales" idea as the k-quant super-blocks.

NF4 is usually applied when the model is loaded, not shipped as a separate file. Hugging Face Transformers takes a BitsAndBytesConfig with load_in_4bit=True, bnb_4bit_quant_type="nf4", and bnb_4bit_use_double_quant=True, and quantizes the BF16 weights on the GPU. Its kernels dequantize to the compute dtype (the number format used for arithmetic) for each multiplication. NF4 is widely used for fine-tuning: QLoRA trains small adapter matrices on top of frozen NF4 weights. It is less often used for fast serving.

Example 7: EXL2: per-layer bit widths to hit an average

Different layers react differently to rounding, as the _M mixes in llama.cpp showed. The ExLlamaV2 project's EXL2 format builds on this: one model can use several bit widths (2, 3, 4, 5, 6, and 8 bits according to its documentation), so the average can be any value, such as 4.0 or 4.65 bpw. The problem is deciding which layers get more bits.

The program allocates bit widths to the 7 matrices of one transformer block with a greedy rule under a 4.0 bpw budget. "Greedy" means that at every step it takes the single best-looking upgrade and never reconsiders. Each layer has a sensitivity number, a stand-in for how much that layer's error hurts. Look at which layers end with high bit widths and which ones give bits up.

import java.util.Locale;

public class BitAllocation {
    public static void main(String[] args) {
        String[] layers = {"attn_q", "attn_k", "attn_v", "attn_o", "ffn_gate", "ffn_up", "ffn_down"};
        long[] params = {16, 4, 4, 16, 56, 56, 56};               // millions of weights per layer of a 7B-class block
        double[] sens = {1.0, 1.0, 8.0, 2.0, 0.5, 0.5, 6.0};       // error at 0 bits, from a calibration pass
        double[] options = {2, 2.5, 3, 3.5, 4, 5, 6, 8};
        int[] pick = new int[layers.length];                      // index into options, start at 2 bits
        long total = 0; for (long p : params) total += p;
        double target = 4.0;
        while (true) {
            double bits = 0;
            for (int i = 0; i < layers.length; i++) bits += params[i] * options[pick[i]];
            int best = -1; double bestGain = 0;
            for (int i = 0; i < layers.length; i++) {
                if (pick[i] + 1 >= options.length) continue;
                double extra = params[i] * (options[pick[i] + 1] - options[pick[i]]);
                if ((bits + extra) / total > target) continue;
                double gain = params[i] * sens[i] * (Math.pow(4, -options[pick[i]]) - Math.pow(4, -options[pick[i] + 1])) / extra;
                if (gain > bestGain) { bestGain = gain; best = i; }
            }
            if (best < 0) break;
            pick[best]++;
        }
        double bits = 0, err = 0, uniformErr = 0;
        for (int i = 0; i < layers.length; i++) {
            bits += params[i] * options[pick[i]];
            err += params[i] * sens[i] * Math.pow(4, -options[pick[i]]);
            uniformErr += params[i] * sens[i] * Math.pow(4, -target);
            System.out.println(String.format(Locale.ROOT, "%-9s %3d M weights  %.1f bits", layers[i], params[i], options[pick[i]]));
        }
        System.out.println(String.format(Locale.ROOT, "average %.3f bpw, error %.4f (uniform 4-bit error %.4f)",
                bits / total, err / total, uniformErr / total));
    }
}

attn_q     16 M weights  4.0 bits
attn_k      4 M weights  5.0 bits
attn_v      4 M weights  6.0 bits
attn_o     16 M weights  5.0 bits
ffn_gate   56 M weights  3.5 bits
ffn_up     56 M weights  3.0 bits
ffn_down   56 M weights  5.0 bits
average 4.000 bpw, error 0.0052 (uniform 4-bit error 0.0089)

The model behind the toy is that each layer's error is sensitivity * 4^-bits, so one more bit divides the squared error by 4. (k-quants, i-quants, and the Importance Matrix, Example 13, measured a factor of about 2 in RMSE per bit, which is 4 in squared error.) The sensitivities stand in for what a calibration pass measures. EXL2's converter runs such a measurement pass over the model before it chooses settings.

The greedy rule starts every layer at 2 bits. It then repeatedly upgrades the layer with the largest error reduction per extra bit, as long as the upgrade fits in the budget, until nothing more fits. The output shows the result:

• attn_v (sensitivity 8, only 4 M weights) gets 6 bits, because each extra bit there is cheap.
• ffn_down (sensitivity 6, 56 M weights) gets 5 bits.
• ffn_up (sensitivity 0.5) drops to 3 bits to pay for the rest.

The average is exactly 4.000 bpw, and the total error is 0.0052 against 0.0089 for a uniform 4-bit model, a 42 percent reduction at the same size. This is the same idea as llama.cpp's _M mixes, chosen per model from measurements instead of fixed rules. ExLlamaV2 quantizes each layer with a GPTQ-style method once its bit width is chosen.

Example 8: EXL3 and trellis-coded quantization

EXL3 is the format of ExLlamaV3. Its README describes it as a streamlined variant of QTIP (Tseng et al., "QTIP: Quantization with Trellises and Incoherence Processing", 2024). A trellis quantizer does not fit in a short program, so this example runs only the first of QTIP's two ideas in Java and describes the second in prose. Both ideas are explained here, because EXL3 files are built from them.

Idea 1, incoherence processing. Before quantizing, the weight matrix is multiplied on both sides by random orthogonal matrices built from Hadamard transforms. An orthogonal matrix rotates a vector without changing its length, so the layer's output stays the same as long as the inputs are rotated in the same way. After the rotation, the entries of the matrix look like samples from a Gaussian (bell-shaped) distribution, without the outliers of the original. An outlier is one value far larger than the rest. Gaussian-looking weights are easier to quantize with a fixed codebook, because no single large value forces a coarse scale on everything else.

Idea 2, trellis-coded quantization (TCQ). Vector quantization over long groups gives lower error than scalar quantization. (k-quants, i-quants, and the Importance Matrix, Example 12, showed it for groups of 8.) The difficulty is size: a codebook for a group of 256 values at 2 bits per value would need 2^512 entries, which cannot be stored. TCQ makes the codes of neighboring weights overlap. Each weight's code is read from a sliding window over a bit stream, and a dynamic-programming search (the Viterbi algorithm, which finds the best path through a graph step by step) picks the bit stream that minimizes the total error. The decoder only reads a window of bits and turns it into a value.

The program shows idea 1, on the input side only. It uses 200 weight rows of 64 values, each with one outlier weight (0.3 among values with standard deviation 0.02). It multiplies each row by random signs and then applies a Walsh-Hadamard transform. That transform is a 64 by 64 matrix of +1 and -1 entries, scaled by 1/sqrt(64), that mixes every value into every output. The same map is applied to the input, and the rotated row is quantized with 4-bit round-to-nearest. Compare the peak-to-RMS ratio (the largest value divided by the typical value) and the error before and after.

import java.util.Locale;
import java.util.Random;

public class Incoherence {
    static final int N = 64;

    // Random signs, then the fast Walsh-Hadamard transform scaled by 1/sqrt(N): an orthogonal map.
    static double[] rotate(double[] v, int[] signs) {
        double[] a = new double[N];
        for (int i = 0; i < N; i++) a[i] = v[i] * signs[i];
        for (int len = 1; len < N; len <<= 1)
            for (int i = 0; i < N; i += 2 * len)
                for (int j = i; j < i + len; j++) { double x = a[j], y = a[j + len]; a[j] = x + y; a[j + len] = x - y; }
        for (int i = 0; i < N; i++) a[i] /= Math.sqrt(N);
        return a;
    }

    static double[] rtn4(double[] w) {                               // 4-bit absmax, one scale per row
        double amax = 0;
        for (double v : w) amax = Math.max(amax, Math.abs(v));
        double s = amax / 7; double[] q = new double[N];
        for (int j = 0; j < N; j++) q[j] = s * Math.max(-8, Math.min(7, Math.rint(w[j] / s)));
        return q;
    }

    static double dot(double[] a, double[] b) { double s = 0; for (int i = 0; i < N; i++) s += a[i] * b[i]; return s; }
    static double peakToRms(double[] v) { double m = 0; for (double x : v) m = Math.max(m, Math.abs(x)); return m / Math.sqrt(dot(v, v) / N); }

    public static void main(String[] args) {
        Random rnd = new Random(8);
        int[] signs = new int[N];
        for (int i = 0; i < N; i++) signs[i] = rnd.nextBoolean() ? 1 : -1;
        double errPlain = 0, errRot = 0, peakPlain = 0, peakRot = 0, maxDotDiff = 0;
        int rows = 200;
        for (int r = 0; r < rows; r++) {
            double[] w = new double[N], x = new double[N];
            for (int j = 0; j < N; j++) { w[j] = rnd.nextGaussian() * 0.02; x[j] = rnd.nextGaussian(); }
            w[rnd.nextInt(N)] = 0.3;                                    // one outlier weight per row
            double[] wr = rotate(w, signs), xr = rotate(x, signs);
            maxDotDiff = Math.max(maxDotDiff, Math.abs(dot(w, x) - dot(wr, xr)));
            peakPlain += peakToRms(w) / rows; peakRot += peakToRms(wr) / rows;
            errPlain += Math.pow(dot(w, x) - dot(rtn4(w), x), 2) / rows;
            errRot += Math.pow(dot(w, x) - dot(rtn4(wr), xr), 2) / rows;
        }
        System.out.println("w.x equals (Hw).(Hx) within 1e-12: " + (maxDotDiff < 1e-12));
        System.out.println(String.format(Locale.ROOT, "peak / rms of a row: original %.2f, rotated %.2f", peakPlain, peakRot));
        System.out.println(String.format(Locale.ROOT, "4-bit output MSE:   original %.3e, rotated %.3e", errPlain, errRot));
    }
}

w.x equals (Hw).(Hx) within 1e-12: true
peak / rms of a row: original 7.06, rotated 1.99
4-bit output MSE:   original 8.392e-03, rotated 6.534e-04

The first line shows that the rotation does not change the layer's result. The map is orthogonal, so the dot product of the rotated row with the rotated input equals the original one. The second line shows what it does change. In the original rows the largest value is 7.06 times the root mean square, because of the outlier. After rotation the outlier's size is spread over all 64 positions, and the ratio drops to 1.99. With a smaller peak, the absmax scale is smaller, so the 63 normal weights no longer round to zero. The 4-bit output error falls by a factor of about 13 (8.392e-03 to 6.534e-04). The engine pays for this by rotating the inputs at run time. A fast Walsh-Hadamard transform costs n log2 n additions for n values.

QTIP's "bitshift trellis" makes decoding cheap. The next state of the trellis is the current state shifted by k bits with k new bits appended. The value for a state can be computed from the state's bits with a few arithmetic operations, instead of a large lookup table. The paper reports better quality than earlier vector-quantization methods at 2 to 4 bits.

The statements in this example come from the QTIP paper and the ExLlamaV3 README. The details of the EXL3 file layout are not covered here.

Example 9: FP8 serving: weight scales and dynamic activation scales

FP8 (Number Formats) is 8-bit floating point. The E4M3 variant has 4 exponent bits and 3 mantissa bits (the digits after the leading one), and its largest value is 448. A floating-point number keeps the same relative precision at every size, while an integer keeps the same absolute step. GPU engines use FP8 to quantize both weights and activations, written W8A8 (8-bit weights, 8-bit activations), so the matrix multiply itself runs on FP8 tensor cores, the GPU units that do small matrix multiplications at high speed.

The problem is how the granularity of the scale matters for FP8 and for INT8. The program quantizes a layer whose first 2 rows are 250 times larger than the rest. It tries per-tensor weight scales (one scale for the whole matrix) and per-channel weight scales (one per row), and uses a dynamic per-token scale for the activations in every case. Compare the four error numbers.

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Random;

public class Fp8W8A8 {
    static final List<Double> E4M3 = new ArrayList<>();
    static {
        for (int e = 0; e < 16; e++)
            for (int m = 0; m < 8; m++) {
                if (e == 15 && m == 7) continue;                      // S.1111.111 is NaN
                double v = e == 0 ? m / 8.0 * Math.pow(2, -6) : (1 + m / 8.0) * Math.pow(2, e - 7);
                E4M3.add(v); E4M3.add(-v);
            }
    }
    static double fp8(double x) {                                     // nearest E4M3 value, max 448
        double best = 0;
        for (double v : E4M3) if (Math.abs(x - v) < Math.abs(x - best)) best = v;
        return best;
    }
    static double int8(double x) { return Math.max(-127, Math.min(127, Math.rint(x))); }
    static double code(boolean fp, double x) { return fp ? fp8(x) : int8(x); }

    public static void main(String[] args) {
        Random rnd = new Random(9);
        int out = 16, in = 64, tokens = 32;
        double[][] w = new double[out][in];
        for (int r = 0; r < out; r++) for (int j = 0; j < in; j++) w[r][j] = rnd.nextGaussian() * (r < 2 ? 0.5 : 0.002);
        double[][] x = new double[tokens][in];
        for (int t = 0; t < tokens; t++) for (int j = 0; j < in; j++) x[t][j] = rnd.nextGaussian() * (1 + t);
        double tensorMax = 0;
        for (double[] row : w) for (double v : row) tensorMax = Math.max(tensorMax, Math.abs(v));

        for (boolean fp : new boolean[] {false, true})
            for (boolean perChannel : new boolean[] {false, true}) {
                double qmax = fp ? 448 : 127, relSum = 0;
                for (int r = 0; r < out; r++) {
                    double rowMax = 0;
                    for (double v : w[r]) rowMax = Math.max(rowMax, Math.abs(v));
                    double s = (perChannel ? rowMax : tensorMax) / qmax;      // static weight scale
                    double num = 0, den = 0;
                    for (double[] xt : x) {
                        double amax = 0;
                        for (double v : xt) amax = Math.max(amax, Math.abs(v));
                        double sa = amax / qmax;                              // dynamic per-token activation scale
                        double y = 0, yq = 0;
                        for (int j = 0; j < in; j++) { y += w[r][j] * xt[j]; yq += code(fp, w[r][j] / s) * s * code(fp, xt[j] / sa) * sa; }
                        num += (y - yq) * (y - yq); den += y * y;
                    }
                    relSum += Math.sqrt(num / den);
                }
                System.out.println(String.format(Locale.ROOT, "%-4s %-11s weight scales: mean relative row error %.4f",
                        fp ? "FP8" : "INT8", perChannel ? "per-channel" : "per-tensor", relSum / out));
            }
    }
}

INT8 per-tensor  weight scales: mean relative row error 0.8675
INT8 per-channel weight scales: mean relative row error 0.0090
FP8  per-tensor  weight scales: mean relative row error 0.0384
FP8  per-channel weight scales: mean relative row error 0.0386

Three kinds of scale appear in W8A8:

• A static weight scale is computed once when the checkpoint is made and stored next to the weights: one number per tensor, or one per output channel (row).
• A dynamic activation scale is computed at run time from each token's activation vector (amax / 448), so every token uses the full FP8 range.
• A static activation scale is the alternative: one calibrated number per tensor. It saves the amax computation but risks clipping inputs that are larger than anything in the calibration data.

The output shows why FP8 checkpoints often use per-tensor weight scales while INT8 needs per-channel scales. The numbers are the mean relative row error, the typical size of the error divided by the size of the correct answer.

INT8 has a fixed step. With one scale for the whole tensor, the scale is set by the two huge rows. The small rows fall into only a few steps around zero, and their relative error is 87 percent. FP8 has a fixed relative precision across its range (about 3 mantissa bits, so roughly 6 percent relative error per value at worst). A 250 times smaller row is therefore still represented with the same relative precision, as long as it stays above the subnormal range (the very small values where FP8 loses precision). FP8 with any scale is less precise than INT8 with a good per-channel scale (0.0386 against 0.0090). That is the trade for not having to choose scales carefully.

vLLM can load FP8 checkpoints, and its documentation also describes --quantization fp8, which converts BF16 weights to FP8 with per-tensor scales when the model loads. FP8 tensor cores are available on NVIDIA GPUs from the Ada and Hopper generations onward, according to NVIDIA's documentation.

Example 10: MXFP4 and NVFP4: 4-bit floats with block scales

FP4 E2M1 (Number Formats) is a 4-bit floating-point format with 2 exponent bits and 1 mantissa bit. It has only 16 codes: 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6. Such a small set needs a scale shared by a block of values, and the two current block formats built on it differ in how that scale is made. The program implements the scale rules of ggml's quantize_row_mxfp4_ref and quantize_row_nvfp4_ref on Gaussian data. Compare the bits per weight and the error of the two.

import java.util.Locale;
import java.util.Random;

public class Fp4Blocks {
    static final double[] E2M1 = {0, 0.5, 1, 1.5, 2, 3, 4, 6};

    static double nearest(double x, double d) {
        double best = 0;
        for (double m : E2M1) for (double v : new double[] {m, -m}) if (Math.abs(x - v * d) < Math.abs(x - best)) best = v * d;
        return best;
    }

    static double e4m3(double x) {                                // nearest positive E4M3 value (Example 9)
        double best = 0;
        for (int e = 0; e < 16; e++) for (int m = 0; m < 8; m++) {
            if (e == 15 && m == 7) continue;
            double v = e == 0 ? m / 8.0 * Math.pow(2, -6) : (1 + m / 8.0) * Math.pow(2, e - 7);
            if (Math.abs(x - v) < Math.abs(x - best)) best = v;
        }
        return best;
    }

    public static void main(String[] args) {
        Random rnd = new Random(42);
        double[] x = new double[4096];
        for (int i = 0; i < x.length; i++) x[i] = rnd.nextGaussian();
        double errMx = 0, errNv = 0;
        for (int b = 0; b < x.length; b += 32) {                    // MXFP4: 32 values, power-of-two scale
            double amax = 0;
            for (int j = 0; j < 32; j++) amax = Math.max(amax, Math.abs(x[b + j]));
            double d = Math.pow(2, Math.floor(Math.log(amax) / Math.log(2)) - 2);
            for (int j = 0; j < 32; j++) errMx += Math.pow(x[b + j] - nearest(x[b + j], d), 2);
        }
        for (int b = 0; b < x.length; b += 16) {                    // NVFP4: 16 values, E4M3 scale
            double amax = 0;
            for (int j = 0; j < 16; j++) amax = Math.max(amax, Math.abs(x[b + j]));
            double d = e4m3(amax / 6);
            for (int j = 0; j < 16; j++) errNv += Math.pow(x[b + j] - nearest(x[b + j], d), 2);
        }
        System.out.println(String.format(Locale.ROOT, "MXFP4 (32 x 4 bits + 8-bit scale) %.2f bpw  RMSE %.4f", (32 * 4 + 8) / 32.0, Math.sqrt(errMx / x.length)));
        System.out.println(String.format(Locale.ROOT, "NVFP4 (16 x 4 bits + 8-bit scale) %.2f bpw  RMSE %.4f", (16 * 4 + 8) / 16.0, Math.sqrt(errNv / x.length)));
    }
}

MXFP4 (32 x 4 bits + 8-bit scale) 4.25 bpw  RMSE 0.1145
NVFP4 (16 x 4 bits + 8-bit scale) 4.50 bpw  RMSE 0.0949

The two formats differ in block size and scale type:

The bpw values come from the block layout: MXFP4 is 32 * 4 + 8 = 136 bits per 32 values, which is 4.25. NVFP4 is 16 * 4 + 8 = 72 bits per 16 values, which is 4.5.

MXFP4 comes from the Open Compute Project's Microscaling (MX) specification. ggml picks the exponent as floor(log2(amax)) - 2, so the block's largest value lands near the top of the E2M1 range. A power-of-two scale cannot fit the block's range exactly, because it can only double or halve. That costs accuracy: RMSE 0.1145. NVFP4 is NVIDIA's format. Its smaller blocks and finer FP8 scale (amax / 6, so the largest value maps to 6) give RMSE 0.0949 for 0.25 bpw more. NVIDIA also describes a second, per-tensor FP32 scale for NVFP4. ggml's block stores only the sub-block scales.

Both formats appear in released models and hardware. OpenAI's gpt-oss models ship their mixture-of-experts weights in MXFP4. (A mixture-of-experts model has many parallel feed-forward "experts", and each token uses only a few.) llama-quantize has a matching mix, MXFP4_MOE, which in llama-quant.cpp puts expert tensors in MXFP4 and other tensors in Q8_0. NVIDIA lists FP4 tensor core support, including NVFP4, for its Blackwell GPUs. On GPUs without FP4 tensor cores, engines dequantize these formats to 16 bits inside the kernel, the same way they handle GPTQ.

Example 11: KV cache quantization in vLLM and llama.cpp

Weights are not the only large tensor in memory. While a model generates text, it keeps the KV cache (key-value cache), a record of the "key" and "value" vectors of every token already processed, so it does not need to recompute them. The cache grows with the length of the conversation and can become as large as the weights. It can be quantized too. The problem is how much that saves and what it costs in accuracy.

The program first computes KV cache sizes for an 8B model with grouped-query attention at several cache types. It then measures the error of attention scores q · k (the dot product of a query vector and a key vector, which tells the model how much to attend to a token) when keys are stored as q8_0 or q4_0.

import java.util.Locale;
import java.util.Random;

public class KvCacheQuant {
    static double f16(double x) { return Float.float16ToFloat(Float.floatToFloat16((float) x)); }

    public static void main(String[] args) {
        int layers = 32, kvHeads = 8, headDim = 128;
        long perToken = 2L * layers * kvHeads * headDim;            // values per token (K and V)
        String[] names = {"f16", "q8_0", "q4_0", "fp8"};
        double[] bits = {16, 8.5, 4.5, 8};
        for (int i = 0; i < names.length; i++)
            System.out.println(String.format(Locale.ROOT, "%-5s %6.1f KiB per token, %5.2f GiB at 32768 tokens",
                    names[i], perToken * bits[i] / 8 / 1024, perToken * bits[i] / 8 * 32768 / (1L << 30)));

        Random rnd = new Random(4);
        double[] q = new double[headDim], k = new double[headDim];
        double errQ8 = 0, errQ4 = 0, dot = 0;
        for (int trial = 0; trial < 200; trial++) {
            for (int j = 0; j < headDim; j++) { q[j] = rnd.nextGaussian(); k[j] = rnd.nextGaussian() * (j % 32 == 0 ? 5 : 1); }
            double exact = 0, s8 = 0, s4 = 0;
            for (int b = 0; b < headDim; b += 32) {
                double amax = 0, max = 0;
                for (int j = 0; j < 32; j++) if (Math.abs(k[b + j]) > amax) { amax = Math.abs(k[b + j]); max = k[b + j]; }
                double d8 = f16(amax / 127), d4 = f16(max / -8);
                for (int j = 0; j < 32; j++) {
                    exact += q[b + j] * k[b + j];
                    s8 += q[b + j] * d8 * Math.rint(k[b + j] / d8);
                    s4 += q[b + j] * d4 * (Math.min(15, (int) (k[b + j] / (max / -8) + 8.5)) - 8);
                }
            }
            errQ8 += Math.abs(exact - s8); errQ4 += Math.abs(exact - s4); dot += Math.abs(exact);
        }
        System.out.println(String.format(Locale.ROOT, "mean |q.k| %.3f, error q8_0 %.4f, error q4_0 %.4f", dot / 200, errQ8 / 200, errQ4 / 200));
    }
}

f16    128.0 KiB per token,  4.00 GiB at 32768 tokens
q8_0    68.0 KiB per token,  2.13 GiB at 32768 tokens
q4_0    36.0 KiB per token,  1.13 GiB at 32768 tokens
fp8     64.0 KiB per token,  2.00 GiB at 32768 tokens
mean |q.k| 10.305, error q8_0 0.0872, error q4_0 1.5009

The sizes use the formula from Memory Math: 2 * layers * kv_heads * head_dim values per token (the 2 is for keys and values). At 32K tokens the cache drops from 4.00 GiB at F16 to 2.13 GiB at q8_0 and 1.13 GiB at q4_0.

The accuracy part generates key vectors with one large channel in every 32 (a pattern seen in real keys) and compares q · k with the exact value. q8_0 changes the score by 0.087 on average, while the scores average 10.3 in magnitude. q4_0 changes it by 1.50, which is 17 times more. The size of the error matters because attention scores pass through softmax (Self-Attention Step by Step), the step that turns scores into weights using the exponential function. An error of 1.5 in a score multiplies that token's weight by up to e^1.5, about 4.5.

The two engines expose this differently:

K and V can use different types in llama.cpp. Keys feed the softmax through q · k, so their errors are amplified as shown above. Values are multiplied by the attention weights and summed, so a value error enters the output scaled by its token's weight.

Example 12: engine and format support

Two questions remain: which engine runs which format, and how large the same model is in each one. The program answers the size part first. It computes the file size of a model with the Llama 3 8B shapes (from k-quants, i-quants, and the Importance Matrix, Example 6) in each GPU format, using the bits per weight derived in Examples 3, 6, 9, and 10. The columns of Format are the bits per weight for the decoder blocks and for the embeddings and output head. Compare every row with the GGUF Q4_K_M line printed at the end.

import java.util.Locale;

public class FormatSizes {
    record Format(String name, double blockBpw, double edgeBpw) {}   // bits per weight: decoder blocks, embeddings + head

    public static void main(String[] args) {
        long d = 4096, ff = 14336, kv = 1024, vocab = 128256; int layers = 32;   // Llama 3 8B shapes (lesson 19)
        long blocks = layers * (2 * d * d + 2 * d * kv + 3 * d * ff);            // the linear layers inside the blocks
        long edges = 2 * vocab * d;                                              // token_embd and output
        long norms = (2L * layers + 1) * d;                                      // kept at 16 bits by every format here
        Format[] formats = {
            new Format("BF16", 16, 16),
            new Format("FP8 W8A8, per-channel FP32 scales", 8 + 32.0 / d, 16),
            new Format("GPTQ 4-bit, group 128, act-order", 4 + 20.0 / 128 + 32.0 / 4096, 16),
            new Format("AWQ 4-bit, group 128", 4 + 20.0 / 128, 16),
            new Format("NF4 with double quantization", 4 + 8.0 / 64 + 32.0 / (64 * 256), 16),
            new Format("NVFP4 (16 values + E4M3 scale)", 4.5, 16),
            new Format("MXFP4 (32 values + E8M0 scale)", 4.25, 16)};
        System.out.println("format                              GiB");
        for (Format f : formats) {
            double bits = blocks * f.blockBpw() + edges * f.edgeBpw() + norms * 16.0;
            System.out.println(String.format(Locale.ROOT, "%-35s %5.2f", f.name(), bits / 8 / (1L << 30)));
        }
        System.out.println("GGUF Q4_K_M (lesson 19, Example 6)   4.58");
        System.out.println(String.format(Locale.ROOT, "share of weights in the blocks: %.1f%%", 100.0 * blocks / (blocks + edges + norms)));
    }
}

format                              GiB
BF16                                14.96
FP8 W8A8, per-channel FP32 scales    8.46
GPTQ 4-bit, group 128, act-order     5.34
AWQ 4-bit, group 128                 5.33
NF4 with double quantization         5.31
NVFP4 (16 values + E4M3 scale)       5.61
MXFP4 (32 values + E8M0 scale)       5.41
GGUF Q4_K_M (lesson 19, Example 6)   4.58
share of weights in the blocks: 86.9%

The calculator assumes that every GPU format quantizes only the linear layers inside the transformer blocks, and keeps the token embeddings and the output head at 16 bits. That is the default of the common GPTQ, AWQ, and bitsandbytes tools. A given checkpoint may differ, and its quantization_config says which modules were skipped. The two 16-bit tensors are 13 percent of the weights, which is why every 4-bit GPU file here is 5.3 to 5.6 GiB. Q4_K_M quantizes the embeddings too (and puts the output head at Q6_K), so it is 4.58 GiB. The FP8 row uses one FP32 scale per 4,096 weights as an approximation of one scale per output row.

Support for each format is a property of the engine, and it changes between releases. The table states what each project's source or documentation shows. The llama.cpp column is checked against the pinned source. The vLLM column is checked against the source of the vLLM 0.30.0 package on PyPI (vLLM is pinned before section 5, and its support list changes between releases). The ExLlama and Transformers columns follow those projects' READMEs and documentation.

The ExLlama and Transformers columns are the part of this lesson most likely to change. Each project's own documentation is the reference for other versions.

The criteria for choosing a format depend on the hardware and the job:

Interview points

• GPTQ and AWQ minimize the layer output error ||WX - ŴX|| on calibration inputs, not the weight error. Example 1 shows two quantizations with equal weight error and very different output error.
• GPTQ quantizes columns in order and updates the remaining columns with w_j -= (w_i - q_i) / Hinv[i][i] * Hinv[i][j], where H = 2 X^T X. The gain over round-to-nearest grows at lower bit widths.
• GPTQ checkpoints store qweight, scales, qzeros, and g_idx. Group size 128 at 4 bits is about 4.16 bpw. Act-order quantizes high-activation columns first and needs g_idx.
• AWQ multiplies salient input channels by s = mean|x|^alpha before rounding and divides the input by s. The value of alpha is grid-searched per layer.
• Both GPTQ and AWQ are weight-only: they save memory and bandwidth for decode but do not speed up compute-bound prefill (the step that reads the whole prompt at once).
• NF4 has 16 levels at normal quantiles (8 positive, 7 negative, and zero) with an absmax per 64 weights. Double quantization lowers the scale cost from 0.5 to 0.127 bpw.
• EXL2 mixes bit widths per layer to reach an average. EXL3 uses QTIP's trellis-coded quantization with Hadamard incoherence processing.
• FP8 W8A8 uses static weight scales and usually dynamic per-token activation scales. FP8's relative precision makes per-tensor weight scales workable where INT8 needs per-channel scales.
• MXFP4 is 32 E2M1 values with a power-of-two E8M0 scale (4.25 bpw). NVFP4 is 16 values with an FP8 E4M3 scale (4.5 bpw).
• In llama.cpp a quantized V cache requires flash attention. Key errors are amplified by softmax more than value errors.

Summary

• The output error, not the weight error, measures a quantizer's effect on a layer.
• In the toy, GPTQ reduced output error by 7 percent at 4 bits and by 62 percent at 3 bits compared with round-to-nearest.
• The inverse Hessian of the squared output error tells GPTQ how to move unquantized weights to cancel each column's error.
• Smaller GPTQ groups cost more: 4.625 bpw at group 32, 4.156 at group 128, and about 4.0 per row.
• AWQ with alpha = 0.2 cut the toy's output error by 39 percent. Too large an alpha makes the error worse than no scaling.
• On one layer with correlated and salient channels, GPTQ cut 3-bit output error by 71 percent and AWQ by 38 percent. AWQ rescales channels but cannot cancel errors between them.
• The NF4 table is built from normal quantiles with offset 0.9677083 and normalized to [-1, 1].
• bitsandbytes uses 64-value blocks for weights and 256-value blocks for double-quantized scales.
• A greedy per-layer bit allocation at 4.0 bpw lowered the toy model's error by 42 percent against uniform 4 bits.
• A random-sign Hadamard rotation keeps w · x unchanged and lowered a row's peak-to-RMS ratio from 7.06 to 1.99, which cut the 4-bit error by a factor of about 13 in the toy.
• QTIP decodes trellis codes by computing values from a sliding window of bits instead of looking them up in a large codebook.
• Per-tensor INT8 scales destroyed the small rows of the toy layer (87 percent error). Per-channel INT8 gave 0.9 percent, and FP8 gave about 3.8 percent with either.
• GPU formats usually keep the embeddings and output head at 16 bits, so a 4-bit GPTQ or AWQ file of an 8B Llama-shaped model is about 5.3 GiB against 4.58 GiB for Q4_K_M.
• MXFP4 and NVFP4 are ggml types 39 and 40. MXFP4_MOE puts expert tensors in MXFP4 and the rest in Q8_0.
• q8_0 halves the KV cache and changed attention scores by under 1 percent in the toy. q4_0 quarters it with a 17 times larger score error.