k-quants, i-quants, and the Importance Matrix

A model file is a very long list of numbers called weights. Quantization stores each weight with fewer bits so the file is smaller and fits in memory. The price is a small rounding error in every weight. Building Q8_0 and Q4_0 Like ggml showed the simplest scheme: cut the weights into blocks of 32 and give every block its own scale, which is the number that converts small integer codes back into real weights.

This lesson covers the formats that most downloaded GGUF files use today. Names such as Q4_K_M and IQ4_XS look like codes, but each part has a meaning. The lesson explains how those formats shrink the cost of scales, how one file can use different formats for different parts of the model, how the quantizer can be told which weights matter most, and how the newer i-quants squeeze weights into 2 or 3 bits.

After the lesson, you can read a quant name, predict the file size for a given model, explain why two files that are both called "4-bit" differ in quality, and pick a file for a given amount of memory.

Layouts, block sizes, and constants come from ggml/src/ggml-common.h, ggml/src/ggml-quants.c, and src/llama-quant.cpp at the pinned llama.cpp commit, and from gguf-py 0.19.0 constants.py. ggml is the tensor library inside llama.cpp, and a tensor is a table of numbers. Every Java program uses seeded random weights (the random generator always starts from the same number), so the output is the same on every run.

Two measuring terms appear throughout. Bits per weight (bpw) is the file size in bits divided by the number of weights, so it includes the cost of scales. RMSE (root mean squared error) is the typical size of the difference between the original weight and the weight after rounding. A lower RMSE means a more faithful copy.

Example 1: two-level scales and what they save

Every 4-bit format stores each weight as a code from 0 to 15, which takes half a byte. The code alone is not enough, because the reader also needs to know what real number each code stands for. That information is the scale (and sometimes a minimum), and storing it costs extra bits. The problem in this example is how to keep 4-bit codes but pay less for the scales and minimums.

The program counts the bytes for four layouts that all store 4-bit codes. Look at the last two numbers on each line: the total bits per weight and the part of that total spent on scales.

import java.util.Locale;

public class SuperBlockStorage {
    static void row(String name, int bytes, int weights, int scaleBytes) {
        System.out.println(String.format(Locale.ROOT, "%-34s %4d bytes / %3d weights = %.4f bpw (scales %.4f)",
                name, bytes, weights, bytes * 8.0 / weights, scaleBytes * 8.0 / weights));
    }

    public static void main(String[] args) {
        // 4-bit codes take 0.5 byte per weight in every layout below.
        row("Q4_0: FP16 scale per 32", 2 + 16, 32, 2);
        row("Q4_1: FP16 scale + min per 32", 2 + 2 + 16, 32, 4);
        row("flat: FP16 scale + min per 32, x8", 8 * (2 + 2) + 128, 256, 32);
        row("Q4_K: 6-bit scale + min per 32", 2 + 2 + 12 + 128, 256, 16);
        System.out.println("6-bit pairs: 8 sub-blocks x 12 bits = " + (8 * 12) + " bits = " + (8 * 12 / 8) + " bytes");
    }
}

Q4_0: FP16 scale per 32              18 bytes /  32 weights = 4.5000 bpw (scales 0.5000)
Q4_1: FP16 scale + min per 32        20 bytes /  32 weights = 5.0000 bpw (scales 1.0000)
flat: FP16 scale + min per 32, x8   160 bytes / 256 weights = 5.0000 bpw (scales 1.0000)
Q4_K: 6-bit scale + min per 32      144 bytes / 256 weights = 4.5000 bpw (scales 0.5000)
6-bit pairs: 8 sub-blocks x 12 bits = 96 bits = 12 bytes

All four layouts spend 0.5 byte per weight on the 4-bit codes. They differ only in what the scales cost. Here is how to read each line.

• Q4_0 stores one FP16 scale (2 bytes, which is 16 bits) for every 32 weights. The overhead is 16 bits / 32 weights = 0.5 bits per weight. The total is 4 + 0.5 = 4.5 bpw.
• Q4_1 adds an FP16 minimum. A minimum lets the 16 levels cover a range that does not sit centered on zero, for example from -0.1 to 0.4. The overhead doubles to 1 bpw.
• The "flat" row is eight Q4_1 blocks side by side. Putting them in a bigger group saves nothing, because each block still has its own two FP16 numbers.
• Q4_K keeps a scale and a minimum for each 32-value sub-block, but stores them as 6-bit integers instead of FP16. Two FP16 numbers per 256 weights, called d and dmin, turn those integers back into real values: sub-block scale = d * ls and sub-block minimum = dmin * lm, where ls and lm are the stored 6-bit integers.

A small case shows why this saves space. Suppose a sub-block needs the scale 0.0123. Instead of storing 0.0123 as a 16-bit number, Q4_K stores ls = 40 in 6 bits. If d is 0.0003075, then d * ls = 0.0123. Eight sub-blocks share the same d, so the cost of the FP16 number is spread over 256 weights instead of 32.

A super-block is the group of 256 weights (QK_K = 256 in ggml-common.h) that shares d and dmin. This is quantization applied twice: first the weights are quantized with per-sub-block scales, then the scales themselves are quantized with a per-super-block scale. The result has the flexibility of Q4_1 (a scale and a minimum for every 32 weights) at the cost of Q4_0 (4.5 bpw).

super-block of 256 weights
  d, dmin             2 FP16 numbers, shared by all 8 sub-blocks
  sub-block 0   32 weights   ls0, lm0 (6 bits each)
  sub-block 1   32 weights   ls1, lm1
  ...
  sub-block 7   32 weights   ls7, lm7

Example 2: the Q4_K block layout and its 12 scale bytes

The previous example said that a Q4_K block stores eight 6-bit scales and eight 6-bit minimums. That is sixteen 6-bit numbers, and the question is how they fit into a clean number of bytes. block_q4_K in ggml-common.h is a structure with four parts: d (FP16), dmin (FP16), scales[12] (12 bytes), and qs[128] (128 bytes of 4-bit codes).

Sixteen numbers of 6 bits is 96 bits, which is exactly 12 bytes. The packing is not a plain stream of bits, though, because ggml wants the unpacking to be fast. The program packs the numbers exactly as quantize_row_q4_K_ref does and unpacks them with a copy of get_scale_min_k4, the ggml function that reads them back. Watch that every pair comes back unchanged.

public class Q4KScales {
    // Packs 8 six-bit scales and 8 six-bit mins into 12 bytes, as quantize_row_q4_K_ref does.
    static byte[] pack(int[] ls, int[] lm) {
        int[] s = new int[12];
        for (int j = 0; j < 8; j++) {
            if (j < 4) {
                s[j] = ls[j];
                s[j + 4] = lm[j];
            } else {
                s[j + 4] = (ls[j] & 0xF) | ((lm[j] & 0xF) << 4);
                s[j - 4] |= (ls[j] >> 4) << 6;
                s[j] |= (lm[j] >> 4) << 6;
            }
        }
        byte[] out = new byte[12];
        for (int i = 0; i < 12; i++) out[i] = (byte) s[i];
        return out;
    }

    // get_scale_min_k4 from ggml-quants.c
    static int[] unpack(int j, byte[] b) {
        int[] q = new int[12];
        for (int i = 0; i < 12; i++) q[i] = b[i] & 0xFF;
        if (j < 4) return new int[] {q[j] & 63, q[j + 4] & 63};
        return new int[] {(q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4), (q[j + 4] >> 4) | ((q[j] >> 6) << 4)};
    }

    public static void main(String[] args) {
        int[] ls = {63, 40, 12, 5, 33, 50, 17, 1};
        int[] lm = {0, 7, 20, 63, 31, 2, 45, 9};
        byte[] packed = pack(ls, lm);
        StringBuilder hex = new StringBuilder();
        for (byte v : packed) hex.append(String.format("%02x ", v & 0xFF));
        System.out.println("12 scale bytes: " + hex.toString().trim());
        for (int j = 0; j < 8; j++) {
            int[] sm = unpack(j, packed);
            System.out.println("sub-block " + j + ": scale " + sm[0] + ", min " + sm[1]);
        }
        System.out.println("offsets: d=0 dmin=2 scales=4 qs=16 end=" + (16 + 128));
    }
}

12 scale bytes: bf e8 4c 05 40 07 94 3f f1 22 d1 91
sub-block 0: scale 63, min 0
sub-block 1: scale 40, min 7
sub-block 2: scale 12, min 20
sub-block 3: scale 5, min 63
sub-block 4: scale 33, min 31
sub-block 5: scale 50, min 2
sub-block 6: scale 17, min 45
sub-block 7: scale 1, min 9
offsets: d=0 dmin=2 scales=4 qs=16 end=144

Sixteen 6-bit numbers are 96 bits, which is exactly 12 bytes, so nothing is wasted. The layout works like this:

• Sub-blocks 0 to 3 put their scale in the low 6 bits of bytes 0 to 3 and their minimum in the low 6 bits of bytes 4 to 7. Each of those bytes still has 2 unused bits at the top.
• Sub-blocks 4 to 7 put the low 4 bits of the scale and the low 4 bits of the minimum together in bytes 8 to 11. Their top 2 bits go into the unused top 2 bits of bytes 0 to 7.

Byte 0 in the output is bf, which is 1011 1111 in binary. The low 6 bits, 111111, are 63, the scale of sub-block 0. The top 2 bits, 10, are the high bits of sub-block 4's scale. The number 33 is 100001 in 6 bits, so its low 4 bits are 0001 and its high 2 bits are 10. Byte 8 is f1: the low half 1 is the low 4 bits of the scale 33, and the high half f (1111) is the low 4 bits of the minimum 31. The unpacker reverses all of this, and the eight printed pairs match the inputs.

The offsets line describes the whole block. d is at byte 0, dmin at byte 2, the scales at byte 4, and the 128 bytes of 4-bit codes at byte 16, for 144 bytes in total. The 4-bit codes are also arranged differently from Q4_0. Each group of 32 bytes holds 64 weights, with weight j in the low nibble (the low 4 bits of a byte) and weight j + 32 in the high nibble. Anyone reading the raw bytes must follow that order.

Example 3: a simplified Q4_K quantizer against Q4_0

The extra flexibility of Q4_K should help, but the claim needs a measurement. The program quantizes 4,096 random weights with a bell-shaped (Gaussian) distribution and a standard deviation of 0.02, which is a typical size for the weights of a trained model. It uses the Q4_0 method from Building Q8_0 and Q4_0 Like ggml and a simplified Q4_K method. Both formats cost 4.5 bpw, so any difference in error comes from the design alone.

import java.util.Locale;
import java.util.Random;

public class SimpleQ4K {
    static float f16(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }

    static float[] q4_0(float[] x) {               // lesson 18
        float[] out = new float[x.length];
        for (int b = 0; b < x.length; b += 32) {
            float amax = 0, max = 0;
            for (int j = 0; j < 32; j++) if (Math.abs(x[b + j]) > amax) { amax = Math.abs(x[b + j]); max = x[b + j]; }
            float d = max / -8f, id = d != 0 ? 1f / d : 0f, dq = f16(d);
            for (int j = 0; j < 32; j++) out[b + j] = (Math.min(15, (int) (x[b + j] * id + 8.5f)) - 8) * dq;
        }
        return out;
    }

    static float[] q4_K(float[] x) {               // simplified: no scale search
        float[] out = new float[x.length];
        for (int sb = 0; sb < x.length; sb += 256) {
            float[] scale = new float[8], min = new float[8];
            float maxScale = 0, maxMin = 0;
            for (int s = 0; s < 8; s++) {
                float lo = x[sb + 32 * s], hi = lo;
                for (int j = 0; j < 32; j++) { lo = Math.min(lo, x[sb + 32 * s + j]); hi = Math.max(hi, x[sb + 32 * s + j]); }
                lo = Math.min(lo, 0f);
                scale[s] = (hi - lo) / 15f;
                min[s] = -lo;
                maxScale = Math.max(maxScale, scale[s]);
                maxMin = Math.max(maxMin, min[s]);
            }
            float d = f16(maxScale / 63f), dmin = f16(maxMin / 63f);
            for (int s = 0; s < 8; s++) {
                int ls = Math.min(63, (int) Math.rint(scale[s] * 63f / maxScale));
                int lm = maxMin > 0 ? Math.min(63, (int) Math.rint(min[s] * 63f / maxMin)) : 0;
                float ds = d * ls, ms = dmin * lm;
                for (int j = 0; j < 32; j++) {
                    int i = sb + 32 * s + j;
                    int q = ds == 0 ? 0 : Math.max(0, Math.min(15, (int) Math.rint((x[i] + ms) / ds)));
                    out[i] = ds * q - ms;
                }
            }
        }
        return out;
    }

    static double rmse(float[] a, float[] b) {
        double s = 0;
        for (int i = 0; i < a.length; i++) s += (a[i] - b[i]) * (a[i] - b[i]);
        return Math.sqrt(s / a.length);
    }

    public static void main(String[] args) {
        Random rnd = new Random(42);
        float[] w = new float[4096];
        for (int i = 0; i < w.length; i++) w[i] = (float) (rnd.nextGaussian() * 0.02);
        System.out.println(String.format(Locale.ROOT, "Q4_0 (4.5 bpw)            RMSE %.6f", rmse(w, q4_0(w))));
        System.out.println(String.format(Locale.ROOT, "Q4_K simplified (4.5 bpw) RMSE %.6f", rmse(w, q4_K(w))));
    }
}

Q4_0 (4.5 bpw)            RMSE 0.001662
Q4_K simplified (4.5 bpw) RMSE 0.001543

The simplified Q4_K method follows the structure of quantize_row_q4_K_ref:

1. For each 32-value sub-block, take the minimum (never above zero) and the maximum. The sub-block scale is (max - min) / 15, so codes 0 to 15 span the whole range.
2. Find the largest sub-block scale and the largest minimum in the super-block. Then d = maxScale / 63 and dmin = maxMin / 63, rounded to FP16.
3. Store each sub-block scale as the 6-bit integer round(scale * 63 / maxScale), and each minimum the same way.
4. Quantize each weight with the quantized scale and minimum, so the codes match what a reader will reconstruct later.

The output shows a lower error for Q4_K (0.001543 against 0.001662, about 7 percent less). The reason is that each sub-block uses all 16 levels between its own minimum and maximum. Q4_0 centers its grid on zero and derives the step from the largest magnitude, so when a block leans to one side, part of the grid goes unused.

The real quantize_row_q4_K_ref adds a search on top of this. Its helper make_qkx2_quants tries 21 candidate scales around the first guess (rmin = -1, rdelta = 0.1, nstep = 20). For each candidate it refits the scale and minimum with weighted least squares (a standard way to find the numbers that give the smallest weighted error), and it keeps the candidate with the lowest weighted error. The weights are av_x + |x|, where av_x is the root mean square of the sub-block, so large weights count more. Example 8 shows the idea of a weighted scale search on 8 values.

Example 4: the k-quant family: sizes from gguf-py block sizes

Q4_K is one member of a family: Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, and Q8_K. The number is the main bit width and the K marks the super-block design. This example answers how many bits per weight each member really costs.

The program computes the bytes in one block with the same formulas as GGML_QUANT_SIZES in gguf-py constants.py, then prints bits per weight. Compare the bpw column with the nominal number in the type name.

import java.util.List;
import java.util.Locale;

public class KQuantSizes {
    record QType(String name, int id, int blockSize, int typeSize, String layout) {
        double bpw() { return typeSize * 8.0 / blockSize; }
    }

    public static void main(String[] args) {
        int qk = 256;
        List<QType> types = List.of(
            new QType("Q2_K", 10, qk, 2 + 2 + qk / 16 + qk / 4, "16 x 16, 4-bit scale+min"),
            new QType("Q3_K", 11, qk, 2 + qk / 4 + qk / 8 + 12, "16 x 16, 6-bit scale"),
            new QType("Q4_K", 12, qk, 2 + 2 + qk / 2 + 12, "8 x 32, 6-bit scale+min"),
            new QType("Q5_K", 13, qk, 2 + 2 + qk / 2 + qk / 8 + 12, "8 x 32, 6-bit scale+min"),
            new QType("Q6_K", 14, qk, 2 + qk / 2 + qk / 4 + qk / 16, "16 x 16, 8-bit scale"),
            new QType("Q8_K", 15, qk, 4 + qk + qk / 8, "activations only"));
        System.out.println("type  id  block  bytes   bpw     sub-blocks");
        for (QType t : types)
            System.out.println(String.format(Locale.ROOT, "%-5s %3d  %5d  %5d  %6.4f  %s",
                    t.name(), t.id(), t.blockSize(), t.typeSize(), t.bpw(), t.layout()));
    }
}

type  id  block  bytes   bpw     sub-blocks
Q2_K   10    256     84  2.6250  16 x 16, 4-bit scale+min
Q3_K   11    256    110  3.4375  16 x 16, 6-bit scale
Q4_K   12    256    144  4.5000  8 x 32, 6-bit scale+min
Q5_K   13    256    176  5.5000  8 x 32, 6-bit scale+min
Q6_K   14    256    210  6.5625  16 x 16, 8-bit scale
Q8_K   15    256    292  9.1250  activations only

The type ids are the numbers stored in a GGUF tensor info record to say which format a tensor uses (The GGUF Format reads them). Each type has a block size of 256 and a different byte count. The bpw column is the byte count times 8, divided by 256. In every row it is higher than the number in the name, because scales and minimums cost bits too. The table lists what one block holds.

Q3_K and Q6_K have no minimum: their codes are symmetric around zero, like Q4_0. Q2_K, Q4_K, and Q5_K have minimums. The comments in ggml-common.h give the same numbers as the output: "Effectively 2.625 bits per weight" for Q2_K, 3.4375 for Q3_K, 4.5 for Q4_K, 5.5 for Q5_K, and 6.5625 for Q6_K.

Q8_K is not a storage format for weights. During inference (running the model to produce text), the CPU backend converts the activations to Q8_K before it multiplies them with k-quant weights. An activation is a number computed while the model runs, as opposed to a weight, which is stored in the file. The stored sums of each 16 codes (bsums) let the dot product apply the weights' sub-block minimums and offsets with one multiplication per 16 activations instead of one per value.

Example 5: quant mixes: what _S, _M, and _L change per tensor

A model is made of many tensors. A tensor here is one table of weights, such as the matrix that produces the attention values in layer 5. Not all tensors are equally sensitive to rounding. A file called Q4_K_M is therefore not all Q4_K: the letters S, M, and L (small, medium, large) tell llama-quantize to give some tensors more bits than the base type.

The program copies a subset of llama_tensor_get_type_impl from src/llama-quant.cpp, the function that picks a type for each tensor. It does this for a Llama-style model with 32 layers and prints one digit per layer: 4 for Q4_K, 6 for Q6_K, and so on. Read each row as a strip of 32 layers from first to last.

The tensor names matter here. attn_v is the attention tensor that produces values, ffn_down is the last matrix of the feed-forward part of a layer, attn_q produces attention queries, and output is the final matrix that turns the model's internal state into scores for every token.

public class QuantMixRules {
    // A subset of llama_tensor_get_type_impl for a Llama-style model:
    // grouped-query attention with 4 query heads per KV head, no experts, rows divisible by 256.
    static boolean useMoreBits(int i, int n) { return i < n / 8 || i >= 7 * n / 8 || (i - n / 8) % 3 == 2; }

    static String type(String ftype, String tensor, int i, int n) {
        String base = ftype.substring(0, 4);                   // Q3_K, Q4_K, Q5_K, Q6_K
        if (tensor.equals("output")) return "Q6_K";
        switch (tensor) {
            case "attn_v" -> {
                if (ftype.equals("Q3_K_M")) return i < 2 ? "Q5_K" : "Q4_K";
                if (ftype.equals("Q3_K_L")) return "Q5_K";
                if ((ftype.equals("Q4_K_M") || ftype.equals("Q5_K_M")) && useMoreBits(i, n)) return "Q6_K";
                if (ftype.equals("Q4_K_S") && i < 4) return "Q5_K";
            }
            case "ffn_down" -> {
                if (ftype.equals("Q3_K_M")) return i < n / 16 ? "Q5_K" : "Q4_K";
                if (ftype.equals("Q3_K_L")) return "Q5_K";
                if ((ftype.equals("Q4_K_M") || ftype.equals("Q5_K_M")) && useMoreBits(i, n)) return "Q6_K";
                if (ftype.equals("Q4_K_S") && i < n / 8) return "Q5_K";
            }
            case "attn_output" -> {
                if (ftype.equals("Q3_K_M")) return "Q4_K";
                if (ftype.equals("Q3_K_L")) return "Q5_K";
            }
            default -> { }
        }
        return base;
    }

    public static void main(String[] args) {
        int n = 32;
        for (String ftype : new String[] {"Q3_K_M", "Q4_K_S", "Q4_K_M", "Q5_K_M"}) {
            for (String tensor : new String[] {"attn_v", "ffn_down", "attn_q"}) {
                StringBuilder line = new StringBuilder();
                for (int i = 0; i < n; i++) line.append(type(ftype, tensor, i, n).charAt(1));
                System.out.printf("%-6s %-8s %s%n", ftype, tensor, line);
            }
        }
        System.out.println("output tensor: " + type("Q4_K_M", "output", 0, n));
    }
}

Q3_K_M attn_v   55444444444444444444444444444444
Q3_K_M ffn_down 55444444444444444444444444444444
Q3_K_M attn_q   33333333333333333333333333333333
Q4_K_S attn_v   55554444444444444444444444444444
Q4_K_S ffn_down 55554444444444444444444444444444
Q4_K_S attn_q   44444444444444444444444444444444
Q4_K_M attn_v   66664464464464464464464464466666
Q4_K_M ffn_down 66664464464464464464464464466666
Q4_K_M attn_q   44444444444444444444444444444444
Q5_K_M attn_v   66665565565565565565565565566666
Q5_K_M ffn_down 66665565565565565565565565566666
Q5_K_M attn_q   55555555555555555555555555555555
output tensor: Q6_K

The name before the suffix sets the default type (llama_ftype_get_default_type): Q3_K_S, Q3_K_M, and Q3_K_L all start from Q3_K, and Q4_K_S and Q4_K_M start from Q4_K. The suffix then raises the type of selected tensors. The attn_q rows show an unchanged default: every layer keeps the base type. The attn_v and ffn_down rows show the raised types.

use_more_bits(i, n) decides whether layer i of n layers gets the larger type. It is true for the first eighth of the layers, the last eighth, and every third layer in between: i < n/8 || i >= 7*n/8 || (i - n/8) % 3 == 2. For 32 layers, n/8 is 4, so layers 0 to 3 and 28 to 31 qualify (8 layers), and the middle rule adds layers 6, 9, 12, 15, 18, 21, 24, and 27 (8 more). That is 16 of 32 layers, which matches the 666644644... pattern in the output. Tensors that are not listed in the table (attn_q, attn_k, ffn_gate, ffn_up, token_embd) keep the default type, as the attn_q rows show.

The real function has more branches than this copy. Some depend on the architecture (Falcon is one family), on mixture-of-experts models with 8 experts, on grouped-query attention (n_gqa() >= 4 moves attn_v to Q4_K for Q2_K), and on whether an importance matrix is present. Two rules apply to every type:

• Tensors with fewer than 2 dimensions (norm weights and biases) are never quantized.
• A tensor whose row length is not a multiple of the type's block size falls back to a type with a smaller block (Q4_K to Q5_0, Q6_K to Q8_0).

Making, Quantizing, and Testing Your Own GGUF shows that fallback in a real llama-quantize log.

Example 6: the average bits per weight of a mixed model

Because a mix uses several types, the real size of a file is not the size of its base type. This example computes the size of an 8B model (8 billion weights) at every mix. The program applies the Example 5 rules to each tensor of a model with the Llama 3 8B shapes, adds up the bits, and divides by the number of weights.

The shapes are: hidden size 4096 (the width of the model's internal vectors), FFN size 14336 (the width inside the feed-forward part), 32 layers, 8 KV heads of 128 dimensions (the groups used by grouped-query attention), a vocabulary of 128256 tokens, and separate input and output embeddings (the tables that turn a token into a vector and back). The norm tensors stay in F32 because they are never quantized.

import java.util.LinkedHashMap;
import java.util.Locale;
import java.util.Map;

public class MixedModelSize {
    static boolean useMoreBits(int i, int n) { return i < n / 8 || i >= 7 * n / 8 || (i - n / 8) % 3 == 2; }

    static String type(String ftype, String tensor, int i, int n) {   // same rules as Example 5
        if (ftype.equals("Q8_0")) return "Q8_0";
        if (tensor.equals("output")) return "Q6_K";
        boolean m = ftype.equals("Q4_K_M") || ftype.equals("Q5_K_M");
        if (tensor.equals("attn_v")) {
            if (ftype.equals("Q3_K_M")) return i < 2 ? "Q5_K" : "Q4_K";
            if (ftype.equals("Q3_K_L")) return "Q5_K";
            if (m && useMoreBits(i, n)) return "Q6_K";
            if (ftype.equals("Q4_K_S") && i < 4) return "Q5_K";
        } else if (tensor.equals("ffn_down")) {
            if (ftype.equals("Q3_K_M")) return i < n / 16 ? "Q5_K" : "Q4_K";
            if (ftype.equals("Q3_K_L")) return "Q5_K";
            if (m && useMoreBits(i, n)) return "Q6_K";
            if (ftype.equals("Q4_K_S") && i < n / 8) return "Q5_K";
        } else if (tensor.equals("attn_output")) {
            if (ftype.equals("Q3_K_M")) return "Q4_K";
            if (ftype.equals("Q3_K_L")) return "Q5_K";
        }
        return ftype.substring(0, 4);
    }

    static final Map<String, Double> BPW = Map.of("Q3_K", 3.4375, "Q4_K", 4.5, "Q5_K", 5.5, "Q6_K", 6.5625, "Q8_0", 8.5);

    public static void main(String[] args) {
        // Llama 3 8B shapes: hidden 4096, FFN 14336, 32 layers, 8 KV heads of 128, vocabulary 128256.
        long d = 4096, ff = 14336, kv = 1024, vocab = 128256; int n = 32;
        Map<String, Long> perLayer = new LinkedHashMap<>();
        perLayer.put("attn_q", d * d); perLayer.put("attn_k", d * kv); perLayer.put("attn_v", d * kv);
        perLayer.put("attn_output", d * d); perLayer.put("ffn_gate", d * ff); perLayer.put("ffn_up", d * ff);
        perLayer.put("ffn_down", d * ff);
        long norms = (2L * n + 1) * d;                         // F32, never quantized
        for (String ftype : new String[] {"Q3_K_S", "Q3_K_M", "Q3_K_L", "Q4_K_S", "Q4_K_M", "Q5_K_S", "Q5_K_M", "Q6_K", "Q8_0"}) {
            double bits = norms * 32.0;
            long params = norms;
            bits += vocab * d * BPW.get(type(ftype, "token_embd", 0, n)) + vocab * d * BPW.get(type(ftype, "output", 0, n));
            params += 2 * vocab * d;
            for (int i = 0; i < n; i++)
                for (var e : perLayer.entrySet()) {
                    bits += e.getValue() * BPW.get(type(ftype, e.getKey(), i, n));
                    params += e.getValue();
                }
            System.out.println(String.format(Locale.ROOT, "%-7s %.3f bpw  %.2f GiB  (%d params)",
                    ftype, bits / params, bits / 8 / (1L << 30), params));
        }
    }
}

Q3_K_S  3.643 bpw  3.41 GiB  (8030261248 params)
Q3_K_M  3.996 bpw  3.74 GiB  (8030261248 params)
Q3_K_L  4.298 bpw  4.02 GiB  (8030261248 params)
Q4_K_S  4.667 bpw  4.36 GiB  (8030261248 params)
Q4_K_M  4.894 bpw  4.58 GiB  (8030261248 params)
Q5_K_S  5.570 bpw  5.21 GiB  (8030261248 params)
Q5_K_M  5.704 bpw  5.33 GiB  (8030261248 params)
Q6_K    6.563 bpw  6.14 GiB  (8030261248 params)
Q8_0    8.501 bpw  7.95 GiB  (8030261248 params)

The parameter count, 8,030,261,248, is the one published for Llama 3 8B, which confirms the shapes. The sizes match the table that llama-quantize --help prints for Llama-3-8B at the pinned commit to within 0.01 GiB: 3.41G for Q3_K_S, 3.74G for Q3_K_M, 4.58G for Q4_K_M, 5.33G for Q5_K_M, 6.14G for Q6_K, and 7.96G for Q8_0. The small remaining differences come from metadata such as the tokenizer, which this calculation leaves out. (A GiB is 2^30 bytes, about 1.07 billion bytes.)

Check one row by hand. Q4_K_M averages 4.894 bpw over 8,030,261,248 weights, which is 39.30 billion bits, or 4.91 billion bytes, or 4.58 GiB. Two facts follow from the numbers.

• A "4-bit" file is 4.67 to 4.89 bpw, not 4.0. The 0.5 bpw of scales is part of every Q4_K block, and the mix adds more on top. Memory estimates in Memory Math must use the mix's average, not the nominal bit width.
• Q4_K_M costs 0.22 GiB more than Q4_K_S (5 percent). All of that goes to half of the attn_v and ffn_down tensors.

Example 7: sensitive tensors and the cost of extra bits

The source gives tensors such as output, attn_v, and ffn_down more bits. Extra bits are only a good deal when the tensor is small. This example measures what each tensor group costs.

The program prints each tensor group's share of the 8B model's weights, its size at Q4_K in MiB (2^20 bytes), and how many MiB it would add to move that group from Q4_K to Q6_K. The last column is the price of upgrading.

import java.util.Locale;

public class SensitiveTensorCost {
    public static void main(String[] args) {
        long d = 4096, ff = 14336, kv = 1024, vocab = 128256; int n = 32;
        String[] names = {"token_embd", "output", "attn_q (all layers)", "attn_k (all layers)", "attn_v (all layers)",
                "attn_output (all layers)", "ffn_gate (all layers)", "ffn_up (all layers)", "ffn_down (all layers)"};
        long[] params = {vocab * d, vocab * d, n * d * d, n * d * kv, n * d * kv, n * d * d, n * d * ff, n * d * ff, n * d * ff};
        long total = 0;
        for (long p : params) total += p;
        System.out.println("tensor group               share   Q4_K MiB  +MiB for Q6_K");
        for (int i = 0; i < names.length; i++) {
            double q4 = params[i] * 4.5 / 8 / (1 << 20);
            double q6 = params[i] * 6.5625 / 8 / (1 << 20);
            System.out.println(String.format(Locale.ROOT, "%-26s %5.1f%%  %8.1f  %8.1f",
                    names[i], 100.0 * params[i] / total, q4, q6 - q4));
        }
    }
}

tensor group               share   Q4_K MiB  +MiB for Q6_K
token_embd                   6.5%     281.8     129.2
output                       6.5%     281.8     129.2
attn_q (all layers)          6.7%     288.0     132.0
attn_k (all layers)          1.7%      72.0      33.0
attn_v (all layers)          1.7%      72.0      33.0
attn_output (all layers)     6.7%     288.0     132.0
ffn_gate (all layers)       23.4%    1008.0     462.0
ffn_up (all layers)         23.4%    1008.0     462.0
ffn_down (all layers)       23.4%    1008.0     462.0

The table explains the choice of tensors in terms of cost:

• attn_v is 1.7 percent of the weights. With grouped-query attention it has 1024 rows against 4096 for attn_q, so moving all of it to Q6_K adds only 33 MiB. A comment in llama-quant.cpp makes the same argument for the 70B model: attn_v is 8 times smaller than attn_q, so more bits give "a nice boost in quantization accuracy with nearly negligible increase in model size".
• output is 6.5 percent. It maps the final internal state to the logits, the raw scores for every possible next token, at every step. An error there changes the scores the sampler reads directly, without later layers in between to smooth it out. Every k-quant mix stores it as Q6_K (129 MiB more than Q4_K here).
• The ffn_* tensors hold 70 percent of the weights. Raising all of ffn_down costs 462 MiB, so Q4_K_M raises only half of its layers, with the first and last layers always included.

The source does not give a single measured reason for each rule. The comments describe the choices as found by trying them ("Guard against craziness in the first few ffn_down layers"). The facts that can be stated are the cost (this table) and the measured effect. The effect is measured with perplexity (how surprised the model is by a test text, where lower is better) and KL divergence (a number for how far the quantized model's predictions drift from the original's). Making, Quantizing, and Testing Your Own GGUF measures both with llama-perplexity.

Example 8: importance-weighted error and the best scale

A quantizer must pick one scale per block. A larger scale covers bigger values but rounds small ones more coarsely, so the choice is a trade. The problem is that some weights in the block matter more to the model's output than others. This example shows how importance changes the best scale.

The program quantizes 8 values to 4 bits with three scales: the plain absmax scale (the largest absolute value divided by the largest code), the scale with the lowest plain squared error, and the scale with the lowest importance-weighted squared error. The array w holds the importance of each weight. Weights 1, 3, and 6 are marked as important (values 9.0, 8.0, and 7.5).

import java.util.Locale;

public class WeightedScale {
    static double error(float[] x, float[] w, float scale, boolean weighted) {
        double e = 0;
        for (int i = 0; i < x.length; i++) {
            int q = Math.max(-8, Math.min(7, (int) Math.rint(x[i] / scale)));
            double diff = x[i] - q * scale;
            e += (weighted ? w[i] : 1.0) * diff * diff;
        }
        return e;
    }

    public static void main(String[] args) {
        float[] x = {0.90f, -0.35f, 0.12f, 0.51f, -0.77f, 0.05f, 0.33f, -0.18f};
        float[] w = {0.1f, 9.0f, 0.2f, 8.0f, 0.1f, 0.3f, 7.5f, 0.2f};   // importance of each weight
        float absmaxScale = 0.90f / 7;
        float bestPlain = absmaxScale, bestWeighted = absmaxScale;
        for (int k = 0; k <= 60; k++) {
            float s = absmaxScale * (0.70f + 0.01f * k);
            if (error(x, w, s, false) < error(x, w, bestPlain, false)) bestPlain = s;
            if (error(x, w, s, true) < error(x, w, bestWeighted, true)) bestWeighted = s;
        }
        for (float s : new float[] {absmaxScale, bestPlain, bestWeighted}) {
            System.out.println(String.format(Locale.ROOT, "scale %.5f  plain error %.5f  weighted error %.5f",
                    s, error(x, w, s, false), error(x, w, s, true)));
        }
    }
}

scale 0.12857  plain error 0.00962  weighted error 0.03620
scale 0.12600  plain error 0.00910  weighted error 0.02602
scale 0.16714  plain error 0.01368  weighted error 0.00502

Importance-weighted error is sum_i w_i * (x_i - s * q_i)^2, where x_i is the weight, s is the scale, q_i is its integer code, and w_i is how much weight i matters. With every w_i = 1 it is the ordinary squared error. For one weight, for example, a rounding error of 0.1 with importance 9 costs 9 * 0.01 = 0.09, while the same error with importance 0.1 costs only 0.001.

• The absmax scale (0.12857) puts 0.90 exactly on code 7, but it is not the best even by plain error.
• The best plain scale (0.12600) lowers the plain error from 0.00962 to 0.00910.
• The best weighted scale (0.16714) is very different. This scale places -0.35, 0.51, and 0.33 (the important weights) close to grid points. Its plain error is higher (0.01368), but its weighted error is 7 times lower than the plain choice's (0.00502 against 0.03620).

So a quantizer that knows which weights matter makes a different trade: it lets unimportant weights carry more error so that important ones carry less. All ggml quantizers that accept quant_weights (quantize_q4_K, quantize_iq4_xs, and the others) work this way: their searches minimize a weighted error. The next example shows where the weights come from.

Example 9: a toy importance matrix from activation statistics

The previous example assumed that someone hands over the importance of each weight. This example shows how to measure it. The idea: a weight matters when the input it multiplies is large. A weight that always meets an input near zero contributes almost nothing, so rounding it badly costs almost nothing.

The program builds an importance matrix from 512 "calibration" activation vectors (sample inputs, here random numbers). In a few columns, called "hot" channels, the inputs are much larger. The program then quantizes 64 weight rows with and without the importance values, and measures the error of the layer output on 512 different "test" activations that were not used to build the matrix. Look at the three imatrix numbers first, then at the two error lines.

import java.util.Locale;
import java.util.Random;

public class ToyImatrix {
    static float[] quantize(float[] w, double[] imp) {           // 4-bit symmetric, one block, best scale by search
        float amax = 0;
        for (float v : w) amax = Math.max(amax, Math.abs(v));
        float best = amax / 7; double bestErr = Double.MAX_VALUE;
        for (int k = 0; k <= 80; k++) {
            float s = amax / 7 * (0.6f + 0.01f * k);
            double e = 0;
            for (int i = 0; i < w.length; i++) {
                double diff = w[i] - s * Math.max(-8, Math.min(7, Math.rint(w[i] / s)));
                e += imp[i] * diff * diff;
            }
            if (e < bestErr) { bestErr = e; best = s; }
        }
        float[] out = new float[w.length];
        for (int i = 0; i < w.length; i++) out[i] = best * (float) Math.max(-8, Math.min(7, Math.rint(w[i] / best)));
        return out;
    }

    static double[][] activations(Random rnd, int tokens, int cols) {
        double[][] x = new double[tokens][cols];
        for (double[] row : x)
            for (int j = 0; j < cols; j++) row[j] = rnd.nextGaussian() * (j % 8 == 3 ? 6.0 : 1.0); // hot channels
        return x;
    }

    static double outputError(float[] w, float[] q, double[][] x) {
        double s = 0;
        for (double[] row : x) {
            double y = 0, yq = 0;
            for (int j = 0; j < w.length; j++) { y += w[j] * row[j]; yq += q[j] * row[j]; }
            s += (y - yq) * (y - yq);
        }
        return s / x.length;
    }

    public static void main(String[] args) {
        Random rnd = new Random(7);
        int cols = 32;
        double[][] calib = activations(rnd, 512, cols), test = activations(rnd, 512, cols);
        double[] imatrix = new double[cols], ones = new double[cols];
        for (double[] row : calib) for (int j = 0; j < cols; j++) imatrix[j] += row[j] * row[j] / calib.length;
        java.util.Arrays.fill(ones, 1.0);
        System.out.println(String.format(Locale.ROOT, "imatrix: col0 %.2f  col3 %.2f  col11 %.2f", imatrix[0], imatrix[3], imatrix[11]));
        double plain = 0, weighted = 0;
        for (int r = 0; r < 64; r++) {                            // 64 rows of a weight matrix
            float[] w = new float[cols];
            for (int j = 0; j < cols; j++) w[j] = (float) (rnd.nextGaussian() * 0.02);
            plain += outputError(w, quantize(w, ones), test);
            weighted += outputError(w, quantize(w, imatrix), test);
        }
        System.out.println(String.format(Locale.ROOT, "output MSE without imatrix: %.3e", plain / 64));
        System.out.println(String.format(Locale.ROOT, "output MSE with imatrix:    %.3e", weighted / 64));
    }
}

imatrix: col0 0.96  col3 36.73  col11 34.65
output MSE without imatrix: 4.453e-04
output MSE with imatrix:    2.864e-04

For a weight row w and input x, the layer output is y = sum_j w_j * x_j. An error e_j in weight j changes y by e_j * x_j. Over many inputs, the expected squared change is e_j^2 * mean(x_j^2). So mean(x_j^2), the average of the squared input for column j, is the natural importance of every weight in input column j.

The importance matrix (imatrix) is that number for each input column of each weight matrix in the model. llama-imatrix computes it by running the model on calibration text and adding x[j] * x[j] for every token in every matrix multiplication (acc[j] += x[j] * x[j] in tools/imatrix/imatrix.cpp), then dividing by the token count.

In the toy, every eighth column is a hot channel with standard deviation 6, so its imatrix value is about 36 (36.73 and 34.65 in the output) against about 1 for normal columns (0.96 for column 0). The number 36 is 6 squared, as the formula predicts. Quantizing with those weights cuts the output error from 4.453e-04 to 2.864e-04, a 36 percent reduction, with the same 4 bits and the same storage. The test activations were not used to build the imatrix, which shows the statistics carry over to new inputs from the same distribution.

ggml does not use the raw imatrix value as the weight. In quantize_row_q4_K_impl and quantize_row_iq4_nl_impl the weight of value x[j] is qw[j] * sqrt(sigma2 + x[j]^2), where qw is the imatrix entry and sigma2 is twice the mean square of the super-block. Large weights in hot columns therefore get the most protection. The imatrix changes only the search for scales and codes. The file format is the same with or without it, so any program that reads GGUF can read an imatrix-quantized file.

Example 10: IQ4_NL: a non-linear 16-value grid

A 4-bit code can name 16 values. The formats so far space those 16 values evenly, so the gap between neighboring values is the same everywhere. But trained weights are not spread evenly: most are close to zero and few are large. The problem is whether evenly spaced levels are the best use of 16 codes.

IQ4_NL replaces the grid -8 .. 7 with the 16 values of kvalues_iq4nl from ggml-common.h, which are packed closer together near zero. The formats whose names start with IQ are called i-quants. The program implements quantize_row_iq4_nl_impl for 32-value blocks, as quantize_iq4_nl calls it, and compares the error with Q4_0. The last line prints the gaps between neighboring table values.

import java.util.Locale;
import java.util.Random;

public class Iq4Nl {
    static final int[] VALUES = {-127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113};
    static float f16(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }

    static int bestIndex(float x) {                               // best_index_int8
        if (x <= VALUES[0]) return 0;
        if (x >= VALUES[15]) return 15;
        int lo = 0, hi = 15;
        while (hi - lo > 1) { int mid = (lo + hi) / 2; if (x < VALUES[mid]) hi = mid; else lo = mid; }
        return x - VALUES[hi - 1] < VALUES[hi] - x ? hi - 1 : hi;
    }

    static float[] iq4nl(float[] x) {                            // quantize_row_iq4_nl_impl, block 32, ntry 7
        float[] out = new float[x.length];
        for (int b = 0; b < x.length; b += 32) {
            float amax = 0, max = 0;
            for (int j = 0; j < 32; j++) if (Math.abs(x[b + j]) > amax) { amax = Math.abs(x[b + j]); max = x[b + j]; }
            float bestD = 0, best = 0;
            for (int itry = -7; itry <= 7; itry++) {
                float id = (itry + VALUES[0]) / max, sumqx = 0, sumq2 = 0;
                for (int j = 0; j < 32; j++) {
                    float w = x[b + j] * x[b + j], q = VALUES[bestIndex(id * x[b + j])];
                    sumqx += w * q * x[b + j];
                    sumq2 += w * q * q;
                }
                if (sumq2 > 0 && sumqx * sumqx > best * sumq2) { bestD = sumqx / sumq2; best = bestD * sumqx; }
            }
            float d = f16(bestD), id = d != 0 ? 1f / d : 0f;
            for (int j = 0; j < 32; j++) out[b + j] = d * VALUES[bestIndex(id * x[b + j])];
        }
        return out;
    }

    static float[] q4_0(float[] x) {                              // lesson 18
        float[] out = new float[x.length];
        for (int b = 0; b < x.length; b += 32) {
            float amax = 0, max = 0;
            for (int j = 0; j < 32; j++) if (Math.abs(x[b + j]) > amax) { amax = Math.abs(x[b + j]); max = x[b + j]; }
            float d = max / -8f, id = d != 0 ? 1f / d : 0f, dq = f16(d);
            for (int j = 0; j < 32; j++) out[b + j] = (Math.min(15, (int) (x[b + j] * id + 8.5f)) - 8) * dq;
        }
        return out;
    }

    static double rmse(float[] a, float[] b) {
        double s = 0;
        for (int i = 0; i < a.length; i++) s += (a[i] - b[i]) * (a[i] - b[i]);
        return Math.sqrt(s / a.length);
    }

    public static void main(String[] args) {
        Random rnd = new Random(42);
        float[] w = new float[4096];
        for (int i = 0; i < w.length; i++) w[i] = (float) (rnd.nextGaussian() * 0.02);
        System.out.println(String.format(Locale.ROOT, "Q4_0   RMSE %.6f", rmse(w, q4_0(w))));
        System.out.println(String.format(Locale.ROOT, "IQ4_NL RMSE %.6f", rmse(w, iq4nl(w))));
        StringBuilder gaps = new StringBuilder("gaps between table values:");
        for (int i = 1; i < 16; i++) gaps.append(' ').append(VALUES[i] - VALUES[i - 1]);
        System.out.println(gaps);
    }
}

Q4_0   RMSE 0.001662
IQ4_NL RMSE 0.001487
gaps between table values: 23 21 18 16 14 13 12 11 12 12 13 15 16 20 24

IQ4_NL stores a block exactly like Q4_0: an FP16 scale and 16 bytes of 4-bit indices, which is 18 bytes per 32 weights (4.5 bpw). The difference is the meaning of a code. In Q4_0, code q means d * (q - 8). In IQ4_NL, code q means d * kvalues_iq4nl[q], and the table is -127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113.

The gaps line shows the shape: 11 to 13 between the values near zero, up to 23 and 24 at the ends. Weights in a trained model have a bell-shaped distribution, with many values near zero and few large ones. Placing levels closer together where values are dense lowers the average error, and the output confirms it: 0.001487 for IQ4_NL against 0.001662 for Q4_0, about 11 percent less, at the same size.

The quantizer also searches for a good scale. After a first guess d = max / 127, it tries 15 inverse scales (itry + values[0]) / max for itry from -7 to 7. For each one it maps every value to its nearest table entry with a binary search (best_index_int8), refits the scale by weighted least squares (d = sum(w*q*x) / sum(w*q*q)), and keeps the best. Without an imatrix the weights are x^2. The "NL" stands for non-linear: the mapping from code to value is a table instead of a straight line.

Example 11: IQ4_XS: IQ4_NL values in super-blocks

IQ4_NL pays 0.5 bpw for its FP16 scales, like Q4_0. Example 1 showed that super-blocks cut that cost. This example applies the same trick to the non-linear grid and measures what it saves.

The program implements the super-block path of quantize_row_iq4_nl_impl that quantize_iq4_xs uses. Each of the 8 sub-blocks gets a 6-bit signed scale, and one FP16 value d per 256 weights converts them. The program prints the error and the block size.

import java.util.Locale;
import java.util.Random;

public class Iq4Xs {
    static final int[] VALUES = {-127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113};
    static float f16(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }

    static int bestIndex(float x) {
        if (x <= VALUES[0]) return 0;
        if (x >= VALUES[15]) return 15;
        int lo = 0, hi = 15;
        while (hi - lo > 1) { int mid = (lo + hi) / 2; if (x < VALUES[mid]) hi = mid; else lo = mid; }
        return x - VALUES[hi - 1] < VALUES[hi] - x ? hi - 1 : hi;
    }

    static float blockScale(float[] x, int b) {                  // same search as IQ4_NL (Example 10)
        float amax = 0, max = 0;
        for (int j = 0; j < 32; j++) if (Math.abs(x[b + j]) > amax) { amax = Math.abs(x[b + j]); max = x[b + j]; }
        float bestD = 0, best = 0;
        for (int itry = -7; itry <= 7; itry++) {
            float id = (itry + VALUES[0]) / max, sumqx = 0, sumq2 = 0;
            for (int j = 0; j < 32; j++) {
                float w = x[b + j] * x[b + j], q = VALUES[bestIndex(id * x[b + j])];
                sumqx += w * q * x[b + j];
                sumq2 += w * q * q;
            }
            if (sumq2 > 0 && sumqx * sumqx > best * sumq2) { bestD = sumqx / sumq2; best = bestD * sumqx; }
        }
        return bestD;
    }

    static float[] iq4xs(float[] x) {
        float[] out = new float[x.length];
        for (int sb = 0; sb < x.length; sb += 256) {
            float[] scales = new float[8];
            float maxScale = 0, amaxScale = 0;
            for (int ib = 0; ib < 8; ib++) {
                scales[ib] = blockScale(x, sb + 32 * ib);
                if (Math.abs(scales[ib]) > amaxScale) { amaxScale = Math.abs(scales[ib]); maxScale = scales[ib]; }
            }
            float d = -maxScale / 32, dq = f16(d), id = d != 0 ? 1 / d : 0;
            for (int ib = 0; ib < 8; ib++) {
                int l = Math.max(-32, Math.min(31, (int) Math.rint(id * scales[ib])));   // 6-bit sub-block scale
                float dl = dq * l, idl = dl != 0 ? 1 / dl : 0;
                for (int j = 0; j < 32; j++) {
                    int i = sb + 32 * ib + j;
                    out[i] = dl * VALUES[bestIndex(idl * x[i])];
                }
            }
        }
        return out;
    }

    public static void main(String[] args) {
        Random rnd = new Random(42);
        float[] w = new float[4096];
        for (int i = 0; i < w.length; i++) w[i] = (float) (rnd.nextGaussian() * 0.02);
        double s = 0;
        float[] q = iq4xs(w);
        for (int i = 0; i < w.length; i++) s += (w[i] - q[i]) * (w[i] - q[i]);
        System.out.println(String.format(Locale.ROOT, "IQ4_XS RMSE %.6f", Math.sqrt(s / w.length)));
        int bytes = 2 + 2 + 256 / 64 + 256 / 2;
        System.out.println(String.format(Locale.ROOT, "block: %d bytes per 256 weights = %.4f bpw", bytes, bytes * 8.0 / 256));
    }
}

IQ4_XS RMSE 0.001501
block: 136 bytes per 256 weights = 4.2500 bpw

block_iq4_xs holds an FP16 d, a 16-bit scales_h, 4 bytes of scales_l, and 128 bytes of 4-bit indices: 136 bytes per 256 weights, which is 4.25 bpw. Each of the 8 sub-blocks has a 6-bit signed scale l from -32 to 31, stored as l + 32: the low 4 bits in scales_l (two per byte) and the high 2 bits in scales_h. The super-block scale is d = -maxScale / 32, so the sub-block with the largest scale gets l = -32.

The error (0.001501) is almost the same as IQ4_NL (0.001487) at 0.25 bpw less, and lower than Q4_0 at 0.25 bpw less. IQ4_XS needs rows whose length is a multiple of 256. For other rows llama-quantize falls back to IQ4_NL. With these numbers it is the smallest 4-bit format in the table, and the llama-quantize help lists it as "4.25 bpw non-linear quantization".

Example 12: codebooks for IQ2 and IQ3: a toy vector quantizer

Below 3 bits, a scalar grid (one independent grid of levels per weight) has too few levels. A 2-bit code names only 4 values, so each weight can only be one of 4 numbers. The problem is how IQ2_XXS reaches 2.06 bpw with acceptable error.

The idea is to stop rounding weights one at a time. Instead, take a group of 8 weights, look for the most similar entry in a table of ready-made 8-value patterns, and store only the position of that entry. A table of 256 patterns needs an 8-bit position. This is called vector quantization, because a group of values is treated as one vector. The table is called a codebook.

The program builds a toy codebook: 256 vectors of 8 magnitudes, trained with k-means on |w|. K-means is a procedure that repeatedly assigns every group to its nearest table entry and then moves each entry to the average of its group. Each group of 8 weights stores an 8-bit index into the table and 7 sign bits. The program compares the error with a 2-bit scalar quantizer.

import java.util.Locale;
import java.util.Random;

public class ToyCodebook {
    public static void main(String[] args) {
        Random rnd = new Random(3);
        int groups = 4096, dim = 8, k = 256;
        float[][] w = new float[groups][dim];
        for (float[] g : w) for (int j = 0; j < dim; j++) g[j] = (float) rnd.nextGaussian();

        // Codebook of magnitude vectors, trained with a few rounds of k-means on |w|.
        float[][] book = new float[k][];
        for (int c = 0; c < k; c++) { book[c] = new float[dim]; for (int j = 0; j < dim; j++) book[c][j] = Math.abs(w[c * 7][j]); }
        int[] assign = new int[groups];
        for (int iter = 0; iter < 8; iter++) {
            float[][] sum = new float[k][dim]; int[] count = new int[k];
            for (int g = 0; g < groups; g++) {
                assign[g] = nearest(book, w[g]);
                count[assign[g]]++;
                for (int j = 0; j < dim; j++) sum[assign[g]][j] += Math.abs(w[g][j]);
            }
            for (int c = 0; c < k; c++) if (count[c] > 0) for (int j = 0; j < dim; j++) book[c][j] = sum[c][j] / count[c];
        }

        double errVq = 0, err2bit = 0;
        for (int g = 0; g < groups; g++) {
            float[] code = book[nearest(book, w[g])];
            int negatives = 0, smallest = 0;
            for (int j = 0; j < dim; j++) {
                if (w[g][j] < 0) negatives++;
                if (Math.abs(w[g][j]) < Math.abs(w[g][smallest])) smallest = j;
            }
            for (int j = 0; j < dim; j++) {
                boolean neg = w[g][j] < 0;
                if (negatives % 2 == 1 && j == smallest) neg = !neg;      // 7 sign bits: force even parity
                double v = neg ? -code[j] : code[j];
                errVq += (w[g][j] - v) * (w[g][j] - v);
                double q = Math.max(-2, Math.min(1, Math.floor(w[g][j] / 0.996)));   // 2-bit grid {-2,-1,0,1}+0.5
                double v2 = (q + 0.5) * 0.996;
                err2bit += (w[g][j] - v2) * (w[g][j] - v2);
            }
        }
        int n = groups * dim;
        System.out.println(String.format(Locale.ROOT, "codebook: 8-bit index + 7 sign bits = %.3f bits per weight", 15.0 / 8));
        System.out.println(String.format(Locale.ROOT, "codebook RMSE    %.4f", Math.sqrt(errVq / n)));
        System.out.println(String.format(Locale.ROOT, "2-bit scalar RMSE %.4f (2 bits per weight)", Math.sqrt(err2bit / n)));
    }

    static int nearest(float[][] book, float[] g) {
        int best = 0; double bestD = Double.MAX_VALUE;
        for (int c = 0; c < book.length; c++) {
            double dist = 0;
            for (int j = 0; j < g.length; j++) { double diff = Math.abs(g[j]) - book[c][j]; dist += diff * diff; }
            if (dist < bestD) { bestD = dist; best = c; }
        }
        return best;
    }
}

codebook: 8-bit index + 7 sign bits = 1.875 bits per weight
codebook RMSE    0.3106
2-bit scalar RMSE 0.3450 (2 bits per weight)

A table of 256 entries needs an 8-bit index, so 8 weights cost 8 bits for the magnitudes, or 1 bit each.

The signs use a parity trick from ggml. Eight signs need 8 bits, but the quantizer forces the number of negative signs in a group to be even. If the count is odd, it flips the sign of the weight with the smallest magnitude, where the flip costs the least error. With even parity the eighth sign follows from the other seven, so 7 bits are enough. The toy then costs (8 + 7) / 8 = 1.875 bits per weight before scales.

The codebook error (0.3106) is lower than a 2-bit scalar quantizer with an optimal fixed step (0.3450), even though the codebook uses fewer bits (1.875 against 2). A scalar quantizer treats every weight alone. The codebook spends its 256 entries on the 8-value patterns that actually occur. The toy trains and tests on the same data, so the real gap is smaller, but the direction holds.

ggml's tables are fixed in the source, not trained per model. The table below compares the two codebook types in ggml.

In dequantize_row_iq2_xxs (the function that turns stored bits back into numbers) each 32-weight group reads two 32-bit words. The first holds four 8-bit grid indices. The second holds four 7-bit sign indices (looked up in ksigns_iq2xs, whose entries all have an even number of set bits) and a 4-bit scale in the top bits. One FP16 d per 256 weights completes the block: 8 groups of 8 bytes plus 2 bytes is 66 bytes, or 2.0625 bpw. IQ2_XS, IQ2_S, IQ3_S, IQ1_S, and IQ1_M use other tables and scale layouts.

These very low-bit formats have a rule. llama-quantize refuses to produce IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, and IQ3_XXS tensors without an importance matrix (tensor_requires_imatrix in llama-quant.cpp). It stops with the message "this quantization requires an importance matrix!". The same rule covers Q2_K tensors in a Q2_K_S file. The token embedding and output tensors are exempt.

Example 13: error against bits per weight

Every format trades size for error. This example measures the curve. The program quantizes 32,768 Gaussian weights with symmetric absmax quantizers from 8 down to 2 bits (32-value blocks, FP16 scale), and with the IQ4_NL grid. It prints the RMSE divided by the weights' standard deviation, so the number is the error as a fraction of a typical weight. A value of 0.05 means a typical rounding error is 5 percent of a typical weight.

import java.util.Locale;
import java.util.Random;

public class ErrorVsBits {
    static final int[] NL = {-127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113};
    static float f16(float x) { return Float.float16ToFloat(Float.floatToFloat16(x)); }

    // Symmetric b-bit, 32-value blocks, FP16 scale from absmax: levels -(2^(b-1)-1) .. 2^(b-1)-1.
    static double symmetric(float[] x, int bits) {
        int qmax = (1 << (bits - 1)) - 1; double err = 0;
        for (int b = 0; b < x.length; b += 32) {
            float amax = 0;
            for (int j = 0; j < 32; j++) amax = Math.max(amax, Math.abs(x[b + j]));
            float d = f16(amax / qmax);
            for (int j = 0; j < 32; j++) {
                double v = d * Math.max(-qmax, Math.min(qmax, Math.rint(x[b + j] / d)));
                err += (x[b + j] - v) * (x[b + j] - v);
            }
        }
        return Math.sqrt(err / x.length);
    }

    // Non-linear 4-bit table with the plain absmax scale (no search), to isolate the effect of the grid.
    static double nonLinear(float[] x) {
        double err = 0;
        for (int b = 0; b < x.length; b += 32) {
            float amax = 0, max = 0;
            for (int j = 0; j < 32; j++) if (Math.abs(x[b + j]) > amax) { amax = Math.abs(x[b + j]); max = x[b + j]; }
            float d = f16(max / -127f);
            for (int j = 0; j < 32; j++) {
                double bestV = 0, bestE = Double.MAX_VALUE;
                for (int v : NL) { double e = Math.abs(x[b + j] - d * v); if (e < bestE) { bestE = e; bestV = d * v; } }
                err += (x[b + j] - bestV) * (x[b + j] - bestV);
            }
        }
        return Math.sqrt(err / x.length);
    }

    public static void main(String[] args) {
        Random rnd = new Random(42);
        float[] w = new float[32768];
        for (int i = 0; i < w.length; i++) w[i] = (float) (rnd.nextGaussian() * 0.02);
        System.out.println("format              bpw    RMSE / sigma");
        for (int bits = 8; bits >= 2; bits--)
            System.out.println(String.format(Locale.ROOT, "symmetric %d-bit    %5.2f   %.4f", bits, bits + 0.5, symmetric(w, bits) / 0.02));
        System.out.println(String.format(Locale.ROOT, "non-linear 4-bit    %5.2f   %.4f", 4.5, nonLinear(w) / 0.02));
    }
}

format              bpw    RMSE / sigma
symmetric 8-bit     8.50   0.0053
symmetric 7-bit     7.50   0.0107
symmetric 6-bit     6.50   0.0217
symmetric 5-bit     5.50   0.0447
symmetric 4-bit     4.50   0.0962
symmetric 3-bit     3.50   0.2256
symmetric 2-bit     2.50   0.6494
non-linear 4-bit     4.50   0.0793

Each bit removed roughly doubles the error from 8 down to 4 bits (0.0053, 0.0107, 0.0217, 0.0447, 0.0962), because halving the number of levels doubles the step between them. Below 4 bits the error grows faster: 0.2256 at 3 bits and 0.6494 at 2 bits. The symmetric 2-bit quantizer here has only three levels (-1, 0, 1) so that zero is exact, and most weights are rounded to zero. That is why low-bit formats spend more effort on grids, minimums, codebooks, and importance weights than 8-bit formats do.

The non-linear 4-bit row uses the IQ4_NL table with a plain absmax scale and no search. It sits between 4 and 5 bits (0.0793, compared with 0.0962 at symmetric 4 bits and 0.0447 at 5 bits), which isolates the gain of the grid alone. The search in Example 10 adds the rest.

Combining Examples 3, 10, 11, and 13 for the same seed-42 weights gives one comparison of the 4-bit formats:

Example 14: choosing a GGUF quant by memory budget

All of this leads to a practical decision: which file to download for a given machine. The file must fit in memory together with two other things. The KV cache (key-value cache) is the memory where the model keeps data about every token it has already read, so it grows with the context length (the number of tokens in the conversation). The runtime also needs working buffers.

The program takes the 8B file sizes from Example 6, subtracts the KV cache and a 1 GiB runtime allowance from the memory budget, and picks the largest quant that fits. It tries budgets of 6, 8, and 12 GiB with contexts of 8,192 and 32,768 tokens.

import java.util.Locale;

public class PickQuant {
    record Quant(String name, double gib) {}

    public static void main(String[] args) {
        // File sizes for an 8B Llama-shaped model, from Example 6.
        Quant[] quants = {new Quant("Q8_0", 7.95), new Quant("Q6_K", 6.14), new Quant("Q5_K_M", 5.33),
                new Quant("Q5_K_S", 5.21), new Quant("Q4_K_M", 4.58), new Quant("Q4_K_S", 4.36),
                new Quant("IQ4_XS", 4.17), new Quant("Q3_K_M", 3.74), new Quant("Q3_K_S", 3.41)};
        double kvGibPer1k = 0.125;            // 8B with GQA at F16: 128 KiB per token (lesson 16)
        double overheadGib = 1.0;             // compute buffers and runtime
        for (double budget : new double[] {6, 8, 12}) {
            for (int ctx : new int[] {8192, 32768}) {
                double free = budget - overheadGib - kvGibPer1k * ctx / 1024;
                String pick = "none fits";
                for (Quant q : quants) if (q.gib() <= free) { pick = q.name(); break; }
                System.out.println(String.format(Locale.ROOT, "budget %4.1f GiB, context %5d: %.2f GiB for weights -> %s",
                        budget, ctx, free, pick));
            }
        }
    }
}

budget  6.0 GiB, context  8192: 4.00 GiB for weights -> Q3_K_M
budget  6.0 GiB, context 32768: 1.00 GiB for weights -> none fits
budget  8.0 GiB, context  8192: 6.00 GiB for weights -> Q5_K_M
budget  8.0 GiB, context 32768: 3.00 GiB for weights -> none fits
budget 12.0 GiB, context  8192: 10.00 GiB for weights -> Q8_0
budget 12.0 GiB, context 32768: 7.00 GiB for weights -> Q6_K

The KV cache size comes from Memory Math: Llama 3 8B stores 2 * 32 layers * 8 KV heads * 128 dims * 2 bytes = 128 KiB per token at F16, which is 0.125 GiB per 1,024 tokens. The IQ4_XS size (4.17 GiB) uses the same method as Example 6 with the IQ4_XS rules: attn_v at Q5_K because of grouped-query attention, output at Q6_K, and the first 4 ffn_down tensors at Q5_K when no imatrix is given.

Work through the first line by hand: 6 GiB budget, minus 1 GiB for the runtime, minus 8 * 0.125 = 1 GiB for an 8K cache, leaves 4 GiB for the weights. The largest file that is at most 4.00 GiB is Q3_K_M (3.74 GiB), because Q4_K_S (4.36) is too big.

The output shows that context length moves the answer as much as the budget does. At 8 GiB, an 8K context leaves room for Q5_K_M, while a 32K context leaves 3 GiB, and none of these files fits. The options there are a smaller model, a quantized KV cache (Memory Math), or offloading some layers to system RAM.

The criteria for choosing within what fits are in the table.

Interview points

• k-quants use 256-weight super-blocks. Each 16- or 32-weight sub-block has a small integer scale (and, for Q2_K, Q4_K, and Q5_K, a minimum), and one or two FP16 values per super-block turn those integers into real scales. This saves space because the FP16 numbers are shared by many weights.
• Q4_K is 144 bytes per 256 weights (4.5 bpw): FP16 d and dmin, 12 bytes of 6-bit scales and minimums, and 128 bytes of 4-bit codes. It has a per-32 minimum like Q4_1 at the cost of Q4_0.
• The bpw of the k-quants are Q2_K 2.625, Q3_K 3.4375, Q4_K 4.5, Q5_K 5.5, and Q6_K 6.5625. Q8_K is used for activations in the CPU dot product, not for stored weights.
• _S, _M, and _L are per-tensor mixes chosen in llama_tensor_get_type_impl. Q4_K_M puts output and half of the attn_v and ffn_down layers (the layers selected by use_more_bits) at Q6_K, and the rest at Q4_K.
• An 8B Llama-shaped model at Q4_K_M averages about 4.89 bpw (4.58 GiB), not 4 bits, because scales and the higher-bit tensors add to the nominal width.
• attn_v is cheap to upgrade under grouped-query attention (1.7 percent of an 8B model), and output feeds the logits directly. Both get more bits in most mixes.
• The importance matrix is mean(x_j^2) per input column, collected by llama-imatrix on calibration text. The quantizer minimizes sum w_j * err_j^2 with weights derived from it. The file format does not change.
• IQ4_NL uses a fixed 16-value non-linear table with 4.5 bpw. IQ4_XS puts the same table in 256-value super-blocks with 6-bit sub-block scales for 4.25 bpw.
• IQ2_XXS and IQ3_XXS are codebook formats: 8-bit indices into fixed tables of 8- or 4-value magnitude patterns, plus 7 sign bits per 8 weights with even parity.
• Rows whose length is not a multiple of 256 cannot use 256-block types. llama-quantize falls back to a 32-block type (Q4_K to Q5_0, IQ4_XS to IQ4_NL).

Summary

• QK_K = 256 is the super-block size of every k-quant and of IQ2_*, IQ3_*, IQ1_*, and IQ4_XS.
• Quantizing the scales themselves (a 6-bit scale and minimum per 32 weights) gives Q4_K the flexibility of Q4_1 at 4.5 bpw.
• The 12 Q4_K scale bytes pack sixteen 6-bit values, and get_scale_min_k4 unpacks them.
• A per-sub-block minimum lets 16 levels cover an asymmetric range, which lowers error at the same size (0.001543 against 0.001662 in Example 3).
• llama-quantize mixes types per tensor. Q4_K_M for an 8B model is 4.58 GiB and Q4_K_S is 4.36 GiB.
• use_more_bits selects the first and last eighth of the layers and every third layer between them.
• Tensors with fewer than 2 dimensions are never quantized, so norm weights stay F32.
• The imatrix weights each column by its mean squared activation. In the toy it cut layer-output error by 36 percent at the same storage.
• ggml's weighted searches use qw[j] * sqrt(sigma2 + x[j]^2) as the weight when an imatrix is present.
• The IQ4_NL table is -127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113, denser near zero.
• IQ4_XS is 136 bytes per 256 weights (4.25 bpw).
• IQ2_XXS is 66 bytes per 256 weights (2.0625 bpw), built from a 256-entry grid of 8-value patterns with magnitudes 8, 25, and 43.
• From 8 to 4 bits, each removed bit roughly doubles the error, and below 4 bits it grows faster.
• The right quant is the largest one whose file, KV cache, and buffers fit in memory, adjusted for the decode speed the task needs.