Lesson 20 of 73
Quantization Fundamentals
A language model is mostly a very large pile of numbers called weights. In the usual format each weight takes 4 bytes (a 32-bit float), so a model with 7 billion weights needs about 28 GB to store, and the computer must read all of that memory for every token it generates. Quantization shrinks the pile: each weight is replaced by a small integer, called a code, plus a shared number called a scale that says how big one step of the code is. A weight that took 32 bits can then take 8, 4, or even 2 bits.
Shrinking numbers loses information, so the real question is how to do it with as little damage as possible. Number Formats showed the formats that hold the numbers. This lesson shows how to choose the scale and the codes, how to measure the error that remains, and why the GGUF formats (the file formats used by llama.cpp) store values in blocks of 32 with one scale per block.
After this lesson, you can quantize a list of numbers by hand, predict how large the error will be, explain why one extreme value can ruin a whole tensor, and describe how small numeric errors turn into a measurable loss of model quality. The next lesson, Building Q8_0 and Q4_0 Like ggml, uses these ideas to build the real file formats.
| Idea | Question it answers | Example |
|---|---|---|
| Symmetric and asymmetric scales | How do real values map to integer codes? | 1 to 3 |
| Bit width | How fast does the error grow as bits shrink? | 4, 11 |
| Granularity: tensor, row, block | How much damage can one extreme value do? | 5 to 8 |
| Rounding and scale search | How should codes and scales be picked? | 9, 10 |
| Output error and integer math | What does the model actually see, and how do kernels compute with codes? | 12 to 14 |
| Perplexity and KL divergence | How much do the model's predictions change? | 15 |
Every program uses seeded java.util.Random data, so the numbers are the same on every run. Real model weights follow a bell-shaped distribution: most values sit close to zero and a few are large. Seeded Gaussian values (random numbers that follow a bell curve), sometimes with large values added on purpose, stand in for them.
Example 1: symmetric absmax quantization to INT8
Suppose you have 8 numbers between -1.27 and 1.27 and you want to store each in one byte. A byte can hold an integer from -128 to 127, so the plan is to stretch or shrink the numbers until the largest one lands on 127. That stretch factor is the scale.
Symmetric quantization maps the range [-amax, amax] onto the integer codes [-127, 127], where amax is the largest absolute value in the data ("absmax" means "absolute maximum"). The scale is amax / 127, the size of one step. Each value becomes q = round(x / scale). Zero maps exactly to code 0, which is why the scheme is called symmetric: the positive and negative sides are treated the same.
A small case by hand: with amax = 1.27, the scale is 1.27 / 127 = 0.01. The value 0.50 becomes round(0.50 / 0.01) = 50, and the value -1.27 becomes round(-1.27 / 0.01) = -127.
The program below does this for 8 values with one shared scale. Look at how the printed codes relate to the original values.
import java.util.Arrays;
import java.util.Locale;
public class AbsmaxInt8 {
public static void main(String[] args) {
float[] x = {0.12f, -0.50f, 0.03f, 0.98f, -1.27f, 0.40f, -0.07f, 0.66f};
float amax = 0;
for (float v : x) amax = Math.max(amax, Math.abs(v));
float scale = amax / 127f;
int[] q = new int[x.length];
for (int i = 0; i < x.length; i++) q[i] = Math.round(x[i] / scale);
System.out.printf(Locale.ROOT, "amax = %.2f, scale = %.6f%n", amax, scale);
System.out.println("codes = " + Arrays.toString(q));
System.out.println("bytes: float " + x.length * 4 + ", int8 " + x.length + " + 4 for the scale");
}
}amax = 1.27, scale = 0.010000
codes = [12, -50, 3, 98, -127, 40, -7, 66]
bytes: float 32, int8 8 + 4 for the scaleThe values were chosen so that amax / 127 is exactly 0.01, so the codes can be read directly: each code is the value in hundredths (0.12 becomes 12, -0.50 becomes -50). The largest value, -1.27, maps to -127, the end of the range, so no code is wasted at the high end. Storage drops from 4 bytes per value (32 bytes for 8 values) to 1 byte per value plus one shared 4-byte scale (12 bytes). With more values, the shared scale becomes negligible and the saving approaches 4 times.
Example 2: dequantize and measure the error
A stored code is only useful if you can turn it back into an approximate number. Dequantization reverses the mapping: x' = q * scale. The difference x' - x is the quantization error. It exists because rounding throws away everything between two neighboring codes.
With rounding to nearest, a value is never more than half a step from its code, so each error is at most scale / 2. In the next program the values do not fall exactly on the grid, so errors appear. The program prints each original value, its code, the value recovered from the code, and the error.
import java.util.Locale;
public class DequantError {
public static void main(String[] args) {
float[] x = {0.123f, -0.501f, 0.0349f, 0.987f, -1.27f, 0.4012f, -0.0666f, 0.6649f};
float amax = 1.27f, scale = amax / 127f;
double sumSq = 0, maxErr = 0;
for (float v : x) {
int q = Math.round(v / scale);
float back = q * scale;
float err = back - v;
sumSq += err * err;
maxErr = Math.max(maxErr, Math.abs(err));
System.out.printf(Locale.ROOT, "%8.4f -> %4d -> %8.4f error %+.4f%n", v, q, back, err);
}
System.out.printf(Locale.ROOT, "max |error| = %.4f (bound scale/2 = %.4f), RMSE = %.4f%n",
maxErr, scale / 2, Math.sqrt(sumSq / x.length));
}
} 0.1230 -> 12 -> 0.1200 error -0.0030
-0.5010 -> -50 -> -0.5000 error +0.0010
0.0349 -> 3 -> 0.0300 error -0.0049
0.9870 -> 99 -> 0.9900 error +0.0030
-1.2700 -> -127 -> -1.2700 error +0.0000
0.4012 -> 40 -> 0.4000 error -0.0012
-0.0666 -> -7 -> -0.0700 error -0.0034
0.6649 -> 66 -> 0.6600 error -0.0049
max |error| = 0.0049 (bound scale/2 = 0.0050), RMSE = 0.0032Every error is below 0.005, which is half of the 0.01 step. Look at the value 0.0349: it becomes code 3, which is 0.03, an error of -0.0049, almost the largest possible. The value -1.27 has error 0 because it is exactly on the grid.
The last line reports the RMSE (root mean square error): square every error, average the squares, then take the square root. It summarizes the error of a whole tensor in one number. If errors are spread evenly across a step, the RMSE is about scale / sqrt(12), which is 0.0029 for this scale. The measured 0.0032 from only 8 values is close. Later examples compare formats by RMSE, often as a fraction of the RMS of the values themselves, which is called the relative RMSE.
Example 3: asymmetric quantization with a zero point
Symmetric quantization has a weakness. If all the values are positive, for example the outputs of a ReLU (a function that turns every negative number into 0), then the negative half of the codes is never used. Half of the available precision is wasted.
Asymmetric quantization fixes this by mapping [min, max], the actual range of the data, onto all codes [0, 255]. The scale is (max - min) / 255. Because the range no longer has zero in the middle, the scheme stores a zero point zp = round(-min / scale), which is the code that represents the real value 0. A value becomes q = round(x / scale) + zp, and it is recovered as (q - zp) * scale.
A small case: for data between 0.2 and 2.2, scale = 2.0 / 255 = 0.00784. The symmetric scale for the same data would be 2.2 / 127 = 0.0173, more than twice as large.
The program compares symmetric and asymmetric 8-bit quantization on 1,000 positive values. Compare the scales and the RMSE on the last two lines.
import java.util.Locale;
import java.util.Random;
public class ZeroPoint {
public static void main(String[] args) {
Random rnd = new Random(5);
float[] x = new float[1000];
for (int i = 0; i < x.length; i++) x[i] = 0.2f + 2.0f * rnd.nextFloat(); // values in [0.2, 2.2)
float min = Float.MAX_VALUE, max = -Float.MAX_VALUE;
for (float v : x) { min = Math.min(min, v); max = Math.max(max, v); }
float sSym = Math.max(Math.abs(min), Math.abs(max)) / 127f;
float sAsym = (max - min) / 255f;
int zp = Math.round(-min / sAsym);
double eSym = 0, eAsym = 0; int lo = 255, hi = 0;
for (float v : x) {
int qs = Math.round(v / sSym);
int qa = Math.round(v / sAsym) + zp;
lo = Math.min(lo, qa); hi = Math.max(hi, qa);
eSym += Math.pow(qs * sSym - v, 2);
eAsym += Math.pow((qa - zp) * sAsym - v, 2);
}
System.out.printf(Locale.ROOT, "min %.4f max %.4f%n", min, max);
System.out.printf(Locale.ROOT, "symmetric: scale %.6f, codes used %d..%d, RMSE %.6f%n",
sSym, Math.round(min / sSym), Math.round(max / sSym), Math.sqrt(eSym / x.length));
System.out.printf(Locale.ROOT, "asymmetric: scale %.6f, zero point %d, codes used %d..%d, RMSE %.6f%n",
sAsym, zp, lo, hi, Math.sqrt(eAsym / x.length));
}
}min 0.2006 max 2.1988
symmetric: scale 0.017313, codes used 12..127, RMSE 0.005102
asymmetric: scale 0.007836, zero point -26, codes used 0..255, RMSE 0.002314Symmetric quantization reserves codes -127 to -1 for negative values that never occur, so only the positive half of the codes is used and each step is twice as large. Asymmetric quantization uses all 256 codes for the actual interval, so the step and the RMSE drop to less than half (0.0023 against 0.0051).
Here 0 lies below the data's minimum, so the zero point is -26, which is outside the code range 0 to 255. That is fine: the code is never stored, but it still makes the formula (q - zp) * scale give the right answer for the codes that are stored.
The cost of asymmetric quantization is a second stored number (the zero point, or equivalently the minimum) and an extra term in every dot product. ggml's Q4_1 stores a scale and a minimum per block for this reason (lesson 21).
Example 4: error at 8, 6, 4, 3, and 2 bits
With b bits, symmetric quantization has codes from -(2^(b-1) - 1) to 2^(b-1) - 1. For 8 bits that is -127 to 127, for 4 bits it is -7 to 7, and for 2 bits it is -1 to 1. Each bit removed halves the number of codes, so the step between codes doubles.
To compare bit widths, the program uses two measures. The first is the relative RMSE, the error as a percentage of the data's own size. The second is the signal-to-noise ratio (SNR) in decibels, 10 * log10(signal power / error power). A higher SNR means the error is smaller compared with the data. Every 10 dB is a factor of 10 in power, so 20 dB means the error power is 100 times smaller than the signal power.
The program quantizes 4,096 seeded Gaussian values at five bit widths with a single scale.
import java.util.Locale;
import java.util.Random;
public class BitWidths {
public static void main(String[] args) {
Random rnd = new Random(11);
float[] x = new float[4096];
double power = 0;
for (int i = 0; i < x.length; i++) { x[i] = (float) rnd.nextGaussian(); power += x[i] * x[i]; }
float amax = 0;
for (float v : x) amax = Math.max(amax, Math.abs(v));
System.out.println("bits qmax relative RMSE SNR dB");
for (int bits : new int[] {8, 6, 4, 3, 2}) {
int qmax = (1 << (bits - 1)) - 1;
float scale = amax / qmax;
double err = 0;
for (float v : x) {
float back = Math.round(v / scale) * scale;
err += (back - v) * (back - v);
}
System.out.printf(Locale.ROOT, "%4d %4d %12.4f%% %7.2f%n",
bits, qmax, 100 * Math.sqrt(err / power), 10 * Math.log10(power / err));
}
}
}bits qmax relative RMSE SNR dB
8 127 0.8299% 41.62
6 31 3.3851% 29.41
4 7 15.1461% 16.39
3 3 34.8079% 9.17
2 1 89.8797% 0.93From 8 to 4 bits, each bit removed costs about 6 dB of SNR. The reason is that halving the number of codes doubles the step, which quadruples the error power (error power grows with the square of the step), and 10 * log10(4) is 6.02. At 8 bits the error is under 1% of the signal, and at 4 bits it is about 15%.
At 3 and 2 bits the grid is so coarse that the 6 dB rule stops working. With 2 bits the codes are only -1, 0, and 1, and the error is 90% of the signal, which means the quantized tensor is almost useless. These numbers use one scale for all 4,096 values. The next examples show that smaller groups of values with their own scales do much better at the same bit width.
Example 5: one outlier ruins a per-tensor scale
A scale chosen from the largest value has a hidden problem: the whole tensor pays for its single most extreme value. Trained transformers do contain a few weights and activations far larger than the rest. Such a value is called an outlier.
A concrete picture: if almost all values are between -2 and 2, but one value is 40, then amax = 40. With 4 bits (qmax = 7) the scale is 40 / 7 = 5.7. A value of 1.5 divides to 0.26 and rounds to code 0. Every ordinary value falls to zero.
The program takes the Gaussian tensor from Example 4, first quantizes it to 4 bits with one scale, then sets one value to 40 and does it again. Watch the line that counts values mapped to code 0.
import java.util.Locale;
import java.util.Random;
public class OutlierScale {
static double relRmse(float[] x, int bits) {
int qmax = (1 << (bits - 1)) - 1;
float amax = 0;
for (float v : x) amax = Math.max(amax, Math.abs(v));
float scale = amax / qmax;
double err = 0, power = 0; int zeros = 0;
for (float v : x) {
int q = Math.round(v / scale);
if (q == 0) zeros++;
err += Math.pow(q * scale - v, 2); power += v * v;
}
System.out.printf(Locale.ROOT, " amax %.2f, scale %.3f, values mapped to code 0: %d of %d%n",
amax, scale, zeros, x.length);
return Math.sqrt(err / power);
}
public static void main(String[] args) {
Random rnd = new Random(11);
float[] x = new float[4096];
for (int i = 0; i < x.length; i++) x[i] = (float) rnd.nextGaussian();
System.out.println("without outlier:");
System.out.printf(Locale.ROOT, " relative RMSE %.2f%%%n", 100 * relRmse(x, 4));
x[100] = 40f;
System.out.println("with one value of 40:");
System.out.printf(Locale.ROOT, " relative RMSE %.2f%%%n", 100 * relRmse(x, 4));
}
}without outlier:
amax 3.62, scale 0.518, values mapped to code 0: 810 of 4096
relative RMSE 15.15%
with one value of 40:
amax 40.00, scale 5.714, values mapped to code 0: 4082 of 4096
relative RMSE 84.34%Without the outlier, the scale is 0.518 and 810 of 4,096 values map to code 0. With the outlier, the scale grows by about 10 times, and 4,082 of the 4,096 values map to code 0. The 4-bit tensor has turned into "zero, plus one large value".
The relative RMSE (84%) understates the damage. It divides by the power of the data, and the outlier's own large power is part of that total. For the ordinary values, almost all the information is gone. The fix is to avoid sharing one scale across values that differ this much, which is what the next two examples do.
Example 6: per-row scales
A weight matrix is a table of numbers with rows and columns. In a neural network each row produces one output value, so a row is also called an output channel. Different rows often have very different sizes. One scale for the whole matrix is then set by the largest row and is far too coarse for the small rows.
Per-row quantization (also called per-channel) gives each row its own scale, computed from that row's largest value. The program builds a 64 by 256 matrix in which the row magnitudes vary by a factor of 50, then compares one scale for the whole tensor with one scale per row at 4 bits.
import java.util.Locale;
import java.util.Random;
public class PerRow {
static double quantErr(float[] w, int from, int to, float scale, int qmax) {
double e = 0;
for (int i = from; i < to; i++) {
int q = Math.max(-qmax, Math.min(qmax, Math.round(w[i] / scale)));
e += Math.pow(q * scale - w[i], 2);
}
return e;
}
public static void main(String[] args) {
int rows = 64, cols = 256, qmax = 7;
Random rnd = new Random(2);
float[] w = new float[rows * cols];
double power = 0;
for (int r = 0; r < rows; r++) {
float rowScale = 0.02f + r * 0.0155f; // 0.02 .. about 1.0
for (int c = 0; c < cols; c++) { w[r * cols + c] = (float) rnd.nextGaussian() * rowScale; power += Math.pow(w[r * cols + c], 2); }
}
float amax = 0;
for (float v : w) amax = Math.max(amax, Math.abs(v));
double eTensor = quantErr(w, 0, w.length, amax / qmax, qmax);
double eRow = 0;
for (int r = 0; r < rows; r++) {
float rmax = 0;
for (int c = 0; c < cols; c++) rmax = Math.max(rmax, Math.abs(w[r * cols + c]));
eRow += quantErr(w, r * cols, (r + 1) * cols, rmax / qmax, qmax);
}
System.out.printf(Locale.ROOT, "per-tensor: 1 scale, relative RMSE %.2f%%%n", 100 * Math.sqrt(eTensor / power));
System.out.printf(Locale.ROOT, "per-row: %d scales, relative RMSE %.2f%%%n", rows, 100 * Math.sqrt(eRow / power));
System.out.printf(Locale.ROOT, "scale overhead per row: %.3f bits per weight (one FP16 scale per %d weights)%n", 16.0 / cols, cols);
}
}per-tensor: 1 scale, relative RMSE 23.20%
per-row: 64 scales, relative RMSE 12.75%
scale overhead per row: 0.063 bits per weight (one FP16 scale per 256 weights)Per-row scales fit each row's own range, so small rows are no longer rounded to zero by a scale set in a large row. The relative RMSE falls from 23.20% to 12.75%.
The price is one FP16 (16-bit float) scale per row. Spread over 256 weights, that is 16 / 256 = 0.0625 bits per weight, which is the 0.063 on the last line. Per-row scales are common in GPU formats (FP8 per-channel scales, GPTQ and AWQ group scales in lesson 23).
A per-row scale is still shared by a whole row, so one outlier inside a row (as in Example 5) still damages that row. Smaller groups are needed to confine it.
Example 7: block-wise scales, error against storage overhead
The idea behind per-row scales can go further: cut each row into short pieces and give every piece its own scale. Block-wise quantization splits each row into blocks of B consecutive values with one scale per block. An outlier then affects only the B values in its own block.
There is a trade-off. Smaller blocks fit the data better, but every block needs its own stored scale, so more of the file is spent on scales instead of codes. The program measures 4-bit error for block sizes from 16 to 4,096 on Gaussian data with 8 outliers, and prints the scale cost per weight (16 / B bits for an FP16 scale) next to the error.
import java.util.Locale;
import java.util.Random;
public class BlockSizes {
public static void main(String[] args) {
Random rnd = new Random(4);
float[] x = new float[4096];
double power = 0;
for (int i = 0; i < x.length; i++) x[i] = (float) rnd.nextGaussian();
for (int k = 0; k < 8; k++) x[rnd.nextInt(x.length)] *= 8; // a few outliers
for (float v : x) power += v * v;
int qmax = 7;
System.out.println("block scale bits/weight total bits/weight relative RMSE");
for (int b : new int[] {16, 32, 64, 128, 256, 4096}) {
double err = 0;
for (int start = 0; start < x.length; start += b) {
float amax = 0;
for (int i = start; i < start + b; i++) amax = Math.max(amax, Math.abs(x[i]));
float scale = amax / qmax;
for (int i = start; i < start + b; i++) err += Math.pow(Math.round(x[i] / scale) * scale - x[i], 2);
}
System.out.printf(Locale.ROOT, "%5d %17.4f %17.4f %12.2f%%%n",
b, 16.0 / b, 4 + 16.0 / b, 100 * Math.sqrt(err / power));
}
}
}block scale bits/weight total bits/weight relative RMSE
16 1.0000 5.0000 8.90%
32 0.5000 4.5000 10.52%
64 0.2500 4.2500 12.33%
128 0.1250 4.1250 14.36%
256 0.0625 4.0625 17.34%
4096 0.0039 4.0039 38.33%Going from one scale for all 4,096 values to blocks of 32 cuts the relative error from 38.33% to 10.52%. The 8 outliers now affect only 8 blocks of 32 values instead of all 4,096 values. Going further down to 16 values lowers the error only to 8.90% while doubling the scale cost from 0.5 to 1.0 bits per weight, so the gain per extra bit shrinks.
Blocks of 32 with an FP16 scale cost half a bit per weight. That is where ggml's Q4_0 (4.5 bits per weight) and Q8_0 (8.5 bits per weight) numbers come from: 4 or 8 bits for the code, plus 0.5 bits for the shared scale. The k-quants of lesson 22 keep small blocks but store the block scales themselves in 6 or 8 bits, which brings the overhead down further.
Example 8: bits per weight including scales
Marketing names such as "4-bit" hide the real cost. The true storage cost is the bits per weight: the code bits plus the share of the scale and any other stored values.
bits per weight = codeBits + (scaleBits + otherBits) / blockSize
Q4_0: 4 + (16 + 0) / 32 = 4.5
Q4_1: 4 + (16 + 16) / 32 = 5.0The first line is the formula. The two lines below compute it by hand for Q4_0 (an FP16 scale per block of 32) and Q4_1 (an FP16 scale plus an FP16 minimum per block of 32).
The program applies the formula to formats that differ in code width, block size, and extra stored values. For four of them it also checks the result against the block byte sizes in gguf-py's GGML_QUANT_SIZES (gguf-py is the Python package that ships with llama.cpp and describes the GGUF formats).
import java.util.Locale;
public class BitsPerWeight {
record Format(String name, int codeBits, int block, int scaleBits, int extraBits, int ggufBytes) {
double bpw() { return codeBits + (double) (scaleBits + extraBits) / block; }
}
public static void main(String[] args) {
Format[] formats = {
new Format("Q8_0: int8, FP16 scale", 8, 32, 16, 0, 34),
new Format("Q4_0: 4-bit, FP16 scale", 4, 32, 16, 0, 18),
new Format("Q4_1: 4-bit, FP16 scale + FP16 min", 4, 32, 16, 16, 20),
new Format("Q5_0: 5-bit, FP16 scale", 5, 32, 16, 0, 22),
new Format("4-bit, FP16 scale per 128", 4, 128, 16, 0, 0),
new Format("4-bit, FP32 scale per 4096", 4, 4096, 32, 0, 0),
};
System.out.println("format bits/weight bytes/block gguf-py bytes");
for (Format f : formats) {
double bytes = f.bpw() * f.block() / 8;
System.out.printf(Locale.ROOT, "%-37s %11.4f %12.1f %s%n", f.name(), f.bpw(), bytes,
f.ggufBytes() == 0 ? "-" : String.valueOf(f.ggufBytes()));
}
}
}format bits/weight bytes/block gguf-py bytes
Q8_0: int8, FP16 scale 8.5000 34.0 34
Q4_0: 4-bit, FP16 scale 4.5000 18.0 18
Q4_1: 4-bit, FP16 scale + FP16 min 5.0000 20.0 20
Q5_0: 5-bit, FP16 scale 5.5000 22.0 22
4-bit, FP16 scale per 128 4.1250 66.0 -
4-bit, FP32 scale per 4096 4.0078 2052.0 -The computed bytes per block agree with gguf-py for the four ggml formats: Q8_0 is 34 bytes per 32 values, Q4_0 is 18, Q4_1 is 20 (it adds a minimum, as in Example 3), and Q5_0 is 22 (its fifth bits take 4 more bytes).
A 4-bit format with one scale per 128 values costs only 4.125 bits per weight, but Example 7 showed that its error is higher than with blocks of 32. Quantization design is this trade between bits spent on scales and bits spent on codes.
Example 9: round to nearest against stochastic rounding
Up to now every value was rounded to the nearest code. That is not the only choice. Round to nearest (RTN) always picks the closer code. Stochastic rounding picks the upper code with a probability equal to the fractional distance. A value 30% of the way from code 2 to code 3 (that is, 2.3) becomes 3 thirty percent of the time and 2 the other seventy percent. Here "stochastic" means "chosen at random with set probabilities".
RTN has the smaller error for each single value. Stochastic rounding is unbiased: its average over many roundings equals the true value. The program first rounds the value 2.3 ten thousand times both ways, then adds up ten thousand values of 0.3.
import java.util.Locale;
import java.util.Random;
public class StochasticRounding {
static int stochastic(float v, Random rnd) {
int lo = (int) Math.floor(v);
return rnd.nextFloat() < v - lo ? lo + 1 : lo;
}
public static void main(String[] args) {
Random rnd = new Random(9);
float v = 2.3f; // in units of the step
long rtnSum = 0, stoSum = 0;
double rtnSq = 0, stoSq = 0;
int n = 10_000;
for (int i = 0; i < n; i++) {
int a = Math.round(v), b = stochastic(v, rnd);
rtnSum += a; stoSum += b;
rtnSq += (a - v) * (a - v); stoSq += (b - v) * (b - v);
}
System.out.printf(Locale.ROOT, "value 2.3: RTN mean %.4f, stochastic mean %.4f%n", (double) rtnSum / n, (double) stoSum / n);
System.out.printf(Locale.ROOT, "RMSE per value: RTN %.4f, stochastic %.4f%n", Math.sqrt(rtnSq / n), Math.sqrt(stoSq / n));
double exact = 0; long rtn = 0, sto = 0;
for (int i = 0; i < n; i++) { float u = 0.3f; exact += u; rtn += Math.round(u); sto += stochastic(u, rnd); }
System.out.printf(Locale.ROOT, "sum of 10000 values of 0.3: exact %.1f, RTN %d, stochastic %d%n", exact, rtn, sto);
}
}value 2.3: RTN mean 2.0000, stochastic mean 2.3002
RMSE per value: RTN 0.3000, stochastic 0.4583
sum of 10000 values of 0.3: exact 3000.0, RTN 0, stochastic 2959RTN turns 2.3 into 2 every time. That is a fixed error of -0.3, and it is a bias: the error always points the same direction, so it adds up when many such values are summed. In the last line, 10,000 values of 0.3 should sum to 3,000, but RTN rounds each one to 0 and gets a sum of 0.
Stochastic rounding has a larger error for each value (RMSE 0.4583 against 0.3000), but its average (2.3002) is close to the true value, and the sum of 0.3 values comes out at 2,959, close to 3,000. This matters when many small updates accumulate, as in training with low-precision weights.
For storing finished weights, every GGUF quantizer uses deterministic rounding. Each weight is used exactly as stored, nothing accumulates, and a lower per-value error is what reduces the output error.
Example 10: searching for a better scale than absmax
Absmax guarantees that no value is clipped, meaning no value is forced to the end code because it is too large. At low bit widths that guarantee is not always the best choice. A slightly smaller scale gives every ordinary value a finer grid. The cost is that the few largest values no longer fit, and they are clipped to the end code.
This suggests a search: try several candidate scales, measure the squared error for each, and keep the best. The program tries scales from 60% to 100% of absmax on 4-bit Gaussian blocks of 32 values and keeps the one with the smallest squared error in each block.
import java.util.Locale;
import java.util.Random;
public class ScaleSearch {
static double err(float[] x, int from, int n, float scale, int qmax) {
double e = 0;
for (int i = from; i < from + n; i++) {
int q = Math.max(-qmax, Math.min(qmax, Math.round(x[i] / scale)));
e += Math.pow(q * scale - x[i], 2);
}
return e;
}
public static void main(String[] args) {
Random rnd = new Random(6);
float[] x = new float[4096];
double power = 0;
for (int i = 0; i < x.length; i++) { x[i] = (float) rnd.nextGaussian(); power += x[i] * x[i]; }
int qmax = 7, block = 32;
double eAbsmax = 0, eSearch = 0, factorSum = 0;
for (int s = 0; s < x.length; s += block) {
float amax = 0;
for (int i = s; i < s + block; i++) amax = Math.max(amax, Math.abs(x[i]));
eAbsmax += err(x, s, block, amax / qmax, qmax);
double best = Double.MAX_VALUE, bestF = 1;
for (int k = 60; k <= 100; k++) {
double e = err(x, s, block, amax * k / 100f / qmax, qmax);
if (e < best) { best = e; bestF = k / 100.0; }
}
eSearch += best; factorSum += bestF;
}
int blocks = x.length / block;
System.out.printf(Locale.ROOT, "absmax scale: relative RMSE %.2f%%%n", 100 * Math.sqrt(eAbsmax / power));
System.out.printf(Locale.ROOT, "searched scale: relative RMSE %.2f%%, average factor %.3f of absmax%n",
100 * Math.sqrt(eSearch / power), factorSum / blocks);
}
}absmax scale: relative RMSE 9.77%
searched scale: relative RMSE 9.15%, average factor 0.943 of absmaxThe search lowers the relative RMSE from 9.77% to 9.15%. The average best factor is 0.943, so on average the best scale is about 6% smaller than absmax. The format does not change: the stored scale is a different number, so there is no extra storage cost.
This is how ggml's k-quant quantizers work: they search over candidate scales and minimums for each block. With an importance matrix, the search minimizes a weighted error instead (lesson 22). Q4_0 and Q8_0, the formats rebuilt in the next lesson, compute their scale directly from the block's largest value, with no search.
Example 11: Gaussian weights and the quantization grid
The codes of a quantizer are evenly spaced, but real weights are not evenly spread. Most sit near zero, and large values are rare. This means the codes near zero are busy and the codes at the ends are mostly idle.
The program quantizes Gaussian blocks of 32 values to 4 bits (codes -7 to 7) with absmax scales and counts how often each code is used. Each # stands for 200 uses.
import java.util.Random;
public class CodeHistogram {
public static void main(String[] args) {
Random rnd = new Random(8);
int[] counts = new int[15]; // codes -7..7
int block = 32, n = 32 * 1024;
float[] x = new float[block];
for (int b = 0; b < n / block; b++) {
float amax = 0;
for (int i = 0; i < block; i++) { x[i] = (float) rnd.nextGaussian(); amax = Math.max(amax, Math.abs(x[i])); }
float scale = amax / 7;
for (int i = 0; i < block; i++) counts[Math.round(x[i] / scale) + 7]++;
}
for (int c = -7; c <= 7; c++) {
int k = counts[c + 7];
System.out.printf("%3d %6d %s%n", c, k, "#".repeat(k / 200));
}
}
} -7 715 ###
-6 627 ###
-5 1082 #####
-4 1663 ########
-3 2479 ############
-2 3431 #################
-1 4141 ####################
0 4354 #####################
1 4083 ####################
2 3390 ################
3 2521 ############
4 1748 ########
5 1126 #####
6 689 ###
7 719 ###The codes near zero are used most often: code 0 appears 4,354 times and code -7 only 715 times. The end codes, -7 and 7, are used slightly more than ±6 only because each block's largest value lands there by construction (there are 1,024 blocks, and each block's largest value is positive or negative about half the time).
In information terms, the codes are not used equally, so 4 bits carry less than 4 bits of information about the values. This motivates formats with non-uniform grids that place more codes near zero. IQ4_NL uses a fixed table of 16 values spaced like a bell curve, and NF4 places its codes at quantiles of a normal distribution (the points that split the distribution into equally likely parts). See lesson 22 and lesson 23.
Example 12: error in layer outputs, not only in weights
So far the examples measured the error in the weights. What a layer passes to the next layer is its output y = W x, where W is the weight matrix and x is the input vector. Each output value is a sum of many products w * x. The rounding error in each weight is small and random, so in a sum the errors partly cancel each other.
The program quantizes a 256 by 1024 weight matrix to 4 bits three ways: one scale for the whole tensor, one per row, and one per block of 32 values. For each, it prints the error in the weights and the error in the output.
import java.util.Locale;
import java.util.Random;
public class OutputError {
static float[] quantize(float[] w, int group, int qmax) {
float[] out = new float[w.length];
for (int s = 0; s < w.length; s += group) {
float amax = 0;
for (int i = s; i < s + group; i++) amax = Math.max(amax, Math.abs(w[i]));
float scale = amax / qmax;
for (int i = s; i < s + group; i++) out[i] = Math.round(w[i] / scale) * scale;
}
return out;
}
static float[] matVec(float[] w, float[] x, int rows, int cols) {
float[] y = new float[rows];
for (int r = 0; r < rows; r++) for (int c = 0; c < cols; c++) y[r] += w[r * cols + c] * x[c];
return y;
}
static double rel(float[] a, float[] b) {
double e = 0, p = 0;
for (int i = 0; i < a.length; i++) { e += Math.pow(a[i] - b[i], 2); p += b[i] * b[i]; }
return Math.sqrt(e / p);
}
public static void main(String[] args) {
int rows = 256, cols = 1024;
Random rnd = new Random(12);
float[] w = new float[rows * cols], x = new float[cols];
for (int i = 0; i < w.length; i++) w[i] = (float) rnd.nextGaussian() * 0.02f;
for (int k = 0; k < 64; k++) w[rnd.nextInt(w.length)] *= 10; // outlier weights
for (int i = 0; i < cols; i++) x[i] = (float) rnd.nextGaussian();
float[] y = matVec(w, x, rows, cols);
String[] names = {"per-tensor", "per-row", "block 32"};
int[] groups = {w.length, cols, 32};
for (int g = 0; g < 3; g++) {
float[] wq = quantize(w, groups[g], 7);
System.out.printf(Locale.ROOT, "%-10s weight error %6.2f%% output error %6.2f%%%n",
names[g], 100 * rel(wq, w), 100 * rel(matVec(wq, x, rows, cols), y));
}
}
}per-tensor weight error 78.46% output error 81.05%
per-row weight error 21.66% output error 24.83%
block 32 weight error 10.07% output error 11.12%In this test the output error follows the weight error closely (78% against 81%, 22% against 25%, 10% against 11%). The input vector has no special structure, so the random weight errors add up as random noise.
Two consequences follow. First, ranking formats by weight error usually gives the same order as ranking them by output error, which is why weight RMSE is a useful first test. Second, the cases where the two differ come from the inputs. If a few input channels are much larger than the others (these are activation outliers), the errors in the weights that multiply those channels dominate the output error. Methods such as AWQ and the importance matrix use this fact by measuring activations on sample text (lesson 22, lesson 23).
Example 13: a quantized dot product with integer sums per block
A dot product multiplies two vectors element by element and adds the results. It is the core operation of every matrix multiplication in a model. With quantized data there is a faster way than converting every code back to a float.
When both vectors are quantized in blocks of 32, each block of one vector has a scale sA and each block of the other has a scale sB. Within a block, the real product of two values is (qA * sA) * (qB * sB) = qA * qB * sA * sB. The scales are constant inside the block, so the kernel adds up the integer products qA * qB first and multiplies by sA * sB once:
dot = sum over blocks of ( scaleA * scaleB * (sum of qA * qB inside the block) )The inner sum uses only int8 integers, which processors handle very quickly. The program computes a 1,024-value dot product this way and counts the integer and float operations.
import java.util.Locale;
import java.util.Random;
public class BlockDot {
public static void main(String[] args) {
int n = 1024, block = 32;
Random rnd = new Random(13);
float[] a = new float[n], b = new float[n];
for (int i = 0; i < n; i++) { a[i] = (float) rnd.nextGaussian(); b[i] = (float) rnd.nextGaussian(); }
float[] sa = new float[n / block], sb = new float[n / block];
byte[] qa = new byte[n], qb = new byte[n];
for (int k = 0; k < n / block; k++) {
float ma = 0, mb = 0;
for (int i = k * block; i < (k + 1) * block; i++) { ma = Math.max(ma, Math.abs(a[i])); mb = Math.max(mb, Math.abs(b[i])); }
sa[k] = ma / 127; sb[k] = mb / 127;
for (int i = k * block; i < (k + 1) * block; i++) { qa[i] = (byte) Math.round(a[i] / sa[k]); qb[i] = (byte) Math.round(b[i] / sb[k]); }
}
float result = 0; long intOps = 0, floatMuls = 0;
for (int k = 0; k < n / block; k++) {
int sum = 0;
for (int i = k * block; i < (k + 1) * block; i++) { sum += qa[i] * qb[i]; intOps++; }
result += sa[k] * sb[k] * sum; floatMuls += 2;
}
float exact = 0;
for (int i = 0; i < n; i++) exact += a[i] * b[i];
System.out.printf(Locale.ROOT, "block int8 dot = %.4f, float dot = %.4f%n", result, exact);
System.out.println("integer multiply-adds = " + intOps + ", float multiplies = " + floatMuls + " (float dot: " + n + ")");
}
}block int8 dot = -7.8576, float dot = -7.6905
integer multiply-adds = 1024, float multiplies = 64 (float dot: 1024)The quantized result (-7.8576) is within about 2% of the float result (-7.6905). The kernel did 1,024 integer multiply-adds but only 64 float multiplies, two per block for 32 blocks.
Integer multiply-adds on 8-bit values are what SIMD dot-product instructions (SIMD means one instruction works on many values at once; VNNI on x86 and sdot on ARM are examples) and GPU integer tensor cores do fastest. This is the structure of ggml's ggml_vec_dot_q8_0_q8_0. The next lesson rebuilds it with the exact Q8_0 layout, and section 7 vectorizes it with the Vector API.
Example 14: weight-only quantization against W8A8
A model computation has two kinds of numbers. Weights are fixed after training. Activations are the intermediate values computed from the input text, and they change with every token. This leads to two choices.
Weight-only quantization stores the weights in few bits and keeps the activations in floating point. The kernel converts weights back to floats (or to 8-bit integers) as it reads them. W8A8 (8-bit weights, 8-bit activations) quantizes both so that the whole product can run in integer arithmetic. Because activations change with every token, their scale cannot be computed in advance. It is computed at run time for each token, which is called a dynamic per-token scale.
The program compares the two methods on a vector in which one activation channel is very large (60, while the others are around 1). It runs once without the large channel and once with it.
import java.util.Locale;
import java.util.Random;
public class W8A8 {
static byte[] q8(float[] v, float scale) {
byte[] q = new byte[v.length];
for (int i = 0; i < v.length; i++) q[i] = (byte) Math.round(v[i] / scale);
return q;
}
static float absmax(float[] v) { float m = 0; for (float x : v) m = Math.max(m, Math.abs(x)); return m; }
public static void main(String[] args) {
int n = 4096;
Random rnd = new Random(14);
float[] w = new float[n], x = new float[n];
for (int i = 0; i < n; i++) { w[i] = (float) rnd.nextGaussian() * 0.02f; x[i] = (float) rnd.nextGaussian(); }
float exact = 0;
for (int i = 0; i < n; i++) exact += w[i] * x[i];
for (float outlier : new float[] {0f, 60f}) {
float[] xs = x.clone();
if (outlier > 0) xs[7] = outlier;
float ref = exact - w[7] * x[7] + w[7] * xs[7];
float sw = absmax(w) / 127; byte[] qw = q8(w, sw);
float weightOnly = 0;
for (int i = 0; i < n; i++) weightOnly += qw[i] * sw * xs[i]; // activations stay float
float sx = absmax(xs) / 127; byte[] qx = q8(xs, sx); // dynamic per-token scale
int acc = 0;
for (int i = 0; i < n; i++) acc += qw[i] * qx[i];
float w8a8 = acc * sw * sx;
System.out.printf(Locale.ROOT, "outlier %4.0f: exact %8.4f weight-only %8.4f (err %.4f) W8A8 %8.4f (err %.4f)%n",
outlier, ref, weightOnly, Math.abs(weightOnly - ref), w8a8, Math.abs(w8a8 - ref));
}
}
}outlier 0: exact 1.3690 weight-only 1.3496 (err 0.0194) W8A8 1.3522 (err 0.0168)
outlier 60: exact 1.5735 weight-only 1.5710 (err 0.0025) W8A8 1.6008 (err 0.0273)Without the outlier, both methods are close to the exact result (errors of about 0.02). With one activation of 60, the per-token activation scale grows about 15 times, every ordinary activation is rounded much more coarsely, and the W8A8 error grows to 0.0273. The weight-only result is unaffected (error 0.0025) because its activations stay in floating point.
This is the main difficulty of activation quantization. It is the reason methods such as SmoothQuant move scale from activations into weights, and why FP8 activations (which have a wider dynamic range) are popular on GPUs that support them.
llama.cpp's CPU backend uses a middle path. The weights are in a GGUF type, and the activations are quantized on the fly into 8-bit blocks. The blocks are Q8_0 or Q8_1 blocks of 32 values for the formats of the next lesson, and Q8_K blocks of 256 values for the k-quants of lesson 22. Each block of activations gets its own scale, so one outlier affects only its own block.
Example 15: model-level quality, perplexity and KL divergence
Weight error and output error are proxies. What users care about is how much the model's predictions change. Two standard measures answer that.
Perplexity is exp(average negative log probability) of the correct next tokens on a text (lesson 5). It can be read as "how many equally likely choices the model is torn between at each step". A perplexity of 3.3 means the model is about as uncertain as if it picked among 3.3 equally likely tokens. Lower is better.
KL divergence (Kullback-Leibler divergence), KL(P || Q) = sum P(i) * ln(P(i) / Q(i)), compares two probability distributions. Here P is the original model's distribution over the next token and Q is the quantized model's. At every position, KL measures how far Q is from P. It is 0 when they are identical and grows as they differ.
A small case: if P = (0.5, 0.5) and Q = (0.5, 0.5), every term is 0.5 * ln(1) = 0, so KL is 0. If Q = (0.6, 0.4), the sum is 0.5 * ln(0.5/0.6) + 0.5 * ln(0.5/0.4), about 0.0204, a small positive number.
The program builds a toy model with a 6-token vocabulary. Its logits (the raw scores before turning them into probabilities) are a seeded 6 by 16 matrix times a hidden vector. It quantizes the matrix at several bit widths and measures perplexity and mean KL over 200 positions of a text sampled from the original model.
import java.util.Locale;
import java.util.Random;
public class ToyQuality {
static float[] softmax(float[] z) {
float m = Float.NEGATIVE_INFINITY; for (float v : z) m = Math.max(m, v);
float s = 0; float[] p = new float[z.length];
for (int i = 0; i < z.length; i++) { p[i] = (float) Math.exp(z[i] - m); s += p[i]; }
for (int i = 0; i < z.length; i++) p[i] /= s;
return p;
}
static float[] logits(float[] w, float[] h) {
float[] z = new float[6];
for (int t = 0; t < 6; t++) for (int j = 0; j < 16; j++) z[t] += w[t * 16 + j] * h[j];
return z;
}
static float[] quantize(float[] w, int bits) { // per-row absmax
int qmax = (1 << (bits - 1)) - 1; float[] out = new float[w.length];
for (int r = 0; r < 6; r++) {
float amax = 0; for (int j = 0; j < 16; j++) amax = Math.max(amax, Math.abs(w[r * 16 + j]));
for (int j = 0; j < 16; j++) out[r * 16 + j] = Math.round(w[r * 16 + j] / (amax / qmax)) * (amax / qmax);
}
return out;
}
public static void main(String[] args) {
Random rnd = new Random(15);
float[] w = new float[96];
for (int i = 0; i < w.length; i++) w[i] = (float) rnd.nextGaussian();
int positions = 200;
float[][] hidden = new float[positions][16]; int[] next = new int[positions];
for (int p = 0; p < positions; p++) {
for (int j = 0; j < 16; j++) hidden[p][j] = (float) rnd.nextGaussian() * 0.5f;
float[] probs = softmax(logits(w, hidden[p]));
float u = rnd.nextFloat(), c = 0; next[p] = 5;
for (int t = 0; t < 6; t++) { c += probs[t]; if (u < c) { next[p] = t; break; } } // text sampled from the model
}
for (int bits : new int[] {32, 8, 4, 3, 2}) {
float[] wq = bits == 32 ? w : quantize(w, bits);
double nll = 0, kl = 0;
for (int p = 0; p < positions; p++) {
float[] P = softmax(logits(w, hidden[p])), Q = softmax(logits(wq, hidden[p]));
nll -= Math.log(Q[next[p]]);
for (int t = 0; t < 6; t++) kl += P[t] * Math.log(P[t] / Q[t]);
}
System.out.printf(Locale.ROOT, "%2d bits: perplexity %.4f mean KL %.6f%n", bits, Math.exp(nll / positions), kl / positions);
}
}
}32 bits: perplexity 3.3009 mean KL 0.000000
8 bits: perplexity 3.3033 mean KL 0.000021
4 bits: perplexity 3.3390 mean KL 0.006735
3 bits: perplexity 3.5704 mean KL 0.043762
2 bits: perplexity 5.1742 mean KL 0.378855The 32-bit row is the original model, so its KL is 0. At 8 bits, perplexity and KL barely move (3.3033 and 0.000021). At 4 bits, KL is still small (0.0067). At 3 and 2 bits, both grow quickly, and at 2 bits perplexity has risen from 3.30 to 5.17.
Perplexity summarizes how well the model predicts one particular text, so small changes can hide in noise and can even go down by chance. KL divergence compares the full distributions at every position against the original, so it detects changes that perplexity misses. This is why llama.cpp's llama-perplexity tool has a --kl-divergence mode that compares a quantized model with saved logits from the full-precision model (Making, Quantizing, and Testing Your Own GGUF).
The lesson on evaluating models and agents later in this course adds task-level evaluation, which matters for coding: a small KL can still change which tool call a model makes.
Interview points
- Symmetric absmax quantization uses
scale = amax / qmaxandq = round(x / scale). The maximum error per value is half a step, because rounding to nearest never moves a value further than that. - Asymmetric quantization adds a zero point (or a minimum) and uses all codes for
[min, max]. For one-sided data this halves the step, at the cost of one more stored value per group. - Each bit removed doubles the step and costs about 6 dB of signal-to-noise ratio, as long as the grid is not already very coarse.
- One outlier inflates a shared absmax scale and rounds ordinary values to zero. Smaller groups (rows, blocks of 32) confine the damage to the group that contains the outlier.
- Bits per weight is code bits plus (scale and minimum bits) divided by block size. This gives Q8_0 8.5, Q4_0 4.5, Q4_1 5.0, and Q5_0 5.5.
- Round to nearest minimizes the error of each value. Stochastic rounding is unbiased and matters when many small updates accumulate, not when finished weights are stored.
- A scale search below absmax trades clipping of a few values for a finer grid on all the others. It lowers block error at no storage cost.
- A quantized dot product sums the integer products inside each block and applies the two scales once per block, which keeps most of the work in fast integer arithmetic.
- Weight-only quantization keeps activations in float. W8A8 needs a dynamic per-token activation scale and suffers when activations contain outliers.
- KL divergence against the original model detects quality loss that perplexity can hide.
Summary
- Quantization maps values to integer codes with a scale (and optionally a zero point). Dequantization multiplies the codes back.
- The RMSE of uniform rounding error is about
scale / sqrt(12). - Symmetric quantization of all-positive data wastes half the codes. Asymmetric quantization recovers them.
- At one scale per 4,096 Gaussian values, 8 bits gave under 1% relative error and 4 bits gave about 15%.
- A single value of 40 in a 4-bit tensor of Gaussian values sent most other values to code 0.
- Per-row scales cost 16 bits per row. Blocks of 32 with an FP16 scale cost 0.5 bits per weight.
- Q8_0, Q4_0, Q4_1, and Q5_0 take 34, 18, 20, and 22 bytes per 32 values in gguf-py.
- 10,000 values of 0.3 sum to 0 under round to nearest and to about 3,000 under stochastic rounding.
- On Gaussian blocks, most 4-bit codes used are near zero. Non-uniform grids such as IQ4_NL and NF4 respond to that.
- A block-quantized int8 dot product of 1,024 values needs 64 float multiplies.
- One activation of 60 in a 4,096-value vector raised the W8A8 error, while weight-only quantization was unaffected.
- On the toy model, perplexity and KL changed little at 8 and 4 bits and grew quickly at 3 and 2 bits.