Quantization

Someone wants to run an AI model on their own laptop instead of using a service on the internet. They open a download page and find the same model offered as several files. One is very large. The others are much smaller and carry labels such as "quantized", "8-bit", or "4-bit".

The smaller files exist because of quantization. The term comes up as soon as the topic is inference, which is the stage where a finished model is run to answer requests. Running a model needs memory, and quantization is the most common way to make a model need less of it.

This lesson explains what quantization changes inside a model, why it saves so much memory, what is lost in exchange, and why it decides whether a model can run on an ordinary computer. After this lesson you can read a model download page and explain the difference between the full file and the quantized ones.

1. Quantization changes the numbers, not the design

A model has a design and it has numbers. The design is the layout: how many layers there are and how they connect. The numbers are the weights that training filled in.

Quantization does not touch the design. It keeps the same layers and the same connections. It also keeps the same weights in the sense that each weight still stands for the same connection and has nearly the same value. The only change is how each weight is written down: with fewer digits.

That one change is enough to alter what it takes to run the model.

2. A weight is the number on one connection

A neural network is made of many small units joined by connections, as the lesson on parameters and weights described. Each connection carries one number, its weight. The weight says how strongly one unit influences the next.

flowchart LR
    A["Unit"] -- "weight 0.31827491" --> D["Unit"]
    B["Unit"] --> D
    C["Unit"] --> D
    A --> E["Unit"]
    B --> E
    C --> E

The diagram shows a very small piece of a network: three units on the left, two on the right, and a connection between every pair. One connection is labelled with its weight, 0.31827491. That value is an illustration of what a weight looks like. Every other connection has a number of its own.

3. Weights are long numbers

Weights are not round numbers such as 1 or 5. They are decimal numbers with many digits after the point. Here is what a small slice of one layer's weights could look like:

 0.31827491  -0.09461203   0.77310582
-0.42095718   0.18604937  -0.65228104
 0.05913862   0.94270351  -0.23748096
 0.86051274  -0.37192640   0.11408593

These values are made up to show the shape. Now take one of them apart:

0.31827491
  ^^^^^^^^
  eight places after the point,
  every one of them written down

Each of those eight digits has to be stored somewhere.

A computer does not store decimal digits directly. It stores bits. A bit is the smallest unit of storage, a single 0 or 1. The more bits are spent on a number, the more exactly the number can be recorded. A number stored with decimals in this way is called a floating point number. Model weights are very often stored as 16-bit or 32-bit floating point numbers. That is a lot of precision for each weight.

One detail about the example: a 16-bit floating point number holds only about three reliable decimal digits, and a 32-bit one holds about seven. The eight-digit example above is written long to make the idea visible. The point is the same at any size: a weight written with more bits is more exact and takes more space.

4. All those numbers take up memory

One weight takes almost no space. The difficulty is the count. A model has billions of weights, and to run the model, the machine must hold every one of them in memory at the same time.

A worked case shows the scale. Eight bits make one byte, and about a billion bytes make one gigabyte (GB), the unit in which a computer's memory is usually stated. Suppose a model has 7 billion weights, each stored as a 16-bit number.

7,000,000,000 weights x 16 bits = 112,000,000,000 bits
112,000,000,000 bits / 8        =  14,000,000,000 bytes
                                =  about 14 GB

So this model needs about 14 GB of memory only to hold its weights, before it has read a single word. The model size is an example. Many laptops have 8 or 16 GB of memory in total, and the operating system and other programs need part of that.

Each number is small. The number of numbers is not.

5. Cutting off the end of each number

Quantization starts from a plain observation: the last digits of each weight contribute the least. Keep the front of the number and stop storing the rest.

The shortened weight, 0.318, is very close to the original, 0.31827491. The difference is less than one part in a thousand. Yet the shortened version needs far fewer bits to store.

Repeat the earlier calculation with each weight stored in 4 bits instead of 16:

7,000,000,000 weights x 4 bits = 28,000,000,000 bits
28,000,000,000 bits / 8        =  3,500,000,000 bytes
                               =  about 3.5 GB

The same model drops from about 14 GB to about 3.5 GB. It has the same number of weights and the same design. Only the storage size of each weight changed.

Chopping digits off the end is the simplest way to picture the idea. Real quantization methods are more careful. They usually round each weight to the nearest value from a small fixed set of allowed values, and they choose that set to fit the weights of each layer. With 4 bits there are only 16 allowed values, and with 8 bits there are 256. The effect is the one described here: each weight is stored less exactly and takes less room.

The word "quantization" comes from this step. To quantize something is to restrict it to a limited set of values.

6. The trade: a little quality for a lot of memory

The idea in one sentence: shorten the numbers and keep the model. Some precision is given up and a lot of memory is returned.

The two sides of the trade are not equal in size.

The loss in quality is small because of how a network produces its output. No single weight decides an answer. Each output comes from adding up the effects of a very large number of weights. A tiny error in each one mostly averages out, in the same way that rounding every price on a long receipt to the nearest cent barely changes the total.

The loss is not zero, and it grows as more precision is removed. A model quantized to 8 bits usually behaves almost like the original. At 4 bits the difference is usually still small. Below that, quality tends to fall off quickly, and the model makes more mistakes. The exact effect depends on the model and the method, so the figures on a download page or a test on your own tasks are the reliable guide.

A quantized model often also runs faster, because the machine has less data to move for every token it generates.

7. Why this matters for running a model locally

Running a model locally means running it on your own machine instead of sending requests to a provider's servers. People do this for privacy, to work offline, or to avoid paying per token.

Three facts meet here.

A data center can add more hardware. A laptop cannot. So for local use, memory is the hard limit, and a model that does not fit will not load at all.

8. Two files for one model

This is why model download pages offer more than one file for the same model.

Both files hold the same model. One writes the weights out in full, and the other writes them shorter. The quantized file gives up a little quality, and that is its known cost. In return it needs far less memory, which is the reason the second file exists.

At the time of writing, quantized files are often labelled with the number of bits per weight, with tags such as "8-bit", "4-bit", or "Q4". A lower number means a smaller file and a larger loss of quality.

9. Quantization decides whether the model fits

For local use, the question is often not "which version is better" but "which version loads".

memory this machine has:        ================ 16 GB

model at full precision:        ============================ 28 GB   does not fit, will not load
the same model, quantized:      ======= 7 GB                         fits, so it runs

The numbers are made up: a model that needs 28 GB with 16-bit weights needs about a quarter of that, 7 GB, with 4-bit weights. The machine's memory is a ceiling. The full-precision model is above the ceiling, so it cannot be loaded. The quantized model is below it, so it runs.

This is what makes local models approachable. Quantization does not make the model better. It makes the model small enough to fit on hardware that people already own.

The criteria for choosing a version follow from this. If the full-precision model fits in memory and speed is acceptable, it gives the best quality. If it does not fit, choose the quantized version with the most bits per weight that still fits. A small loss of quality from a model that runs is the only option when the full version does not run.

Interview points

• Quantization stores a model's weights with fewer bits each. The architecture and the number of weights stay the same.
• Weights are commonly stored as 16-bit or 32-bit floating point numbers. Quantized versions commonly use 8 or 4 bits per weight.
• Memory for weights is roughly the number of weights times the bits per weight. Going from 16 bits to 4 bits cuts the memory to a quarter.
• The whole set of weights must be in memory during inference, so memory is the limit that decides whether a model can run on a given machine.
• Real quantization rounds each weight to the nearest value in a small set of allowed values. Cutting digits off the end is a simplified picture of it.
• The quality loss is usually small because outputs depend on sums over very many weights, so small errors in individual weights mostly cancel out. The loss grows as the bit count drops.
• Quantization is the main reason large models can run on laptops and phones. A quantized model is often faster as well as smaller.
• Choose the highest precision that fits in the available memory and meets the speed requirement, and test quality on the real task.

Summary

• Quantization changes how the weights are written, not the model's design.
• A weight is a long decimal number, and a model has billions of them.
• All weights must sit in memory at once while the model runs.
• Storing each weight with less precision saves a large amount of memory.
• The cost is a small loss in answer quality, which grows with stronger quantization.
• Download pages offer a full file and smaller quantized files of the same model.
• A quantized model can fit on a machine where the full model will not load.