Quantization: Number Formats and Approaches

A trained model is a very large collection of numbers. Each number takes some number of bits in memory, and the GPU must read those bits every time it generates a token. Quantization means storing and computing with the same numbers in fewer bits, for example 8 bits instead of 16. Fewer bits mean less data to read and faster arithmetic. The price is that every number becomes a little less exact, and when that goes wrong the answers of the model get worse.

This lesson starts the techniques chapter of the course. It explains the number formats that GPUs support, why two formats with the same number of bits can behave very differently, which parts of a model can be quantized safely, and how to check afterwards that quality did not drop. Afterwards you can pick a precision for a deployment, estimate the memory it saves, and design a test that shows whether the quantized model is as good as the original.

First, the setting for the whole chapter. Inference engineering is unusual because research papers reach production within months or even weeks. Some of the most visible work in the field is bridging that gap. Four ideas frame the techniques in the next five lessons:

• More constraints give better performance. The more you can fix about your system (one engine only does prefill, one model only runs one precision), the faster you can make it. Disaggregation (see Prefill-Decode Disaggregation) is an example of adding a constraint on purpose.
• More traffic allows more optimizations. Large model parallelism, routing by cache contents, and dynamic disaggregation only pay off when many GPUs, often several nodes, serve the same model. With low traffic they cost more than they save.
• Tuning never ends. Real traffic does not follow tidy rules. Parameters of inference engines, speculation algorithms, and model servers can be adjusted by redeploying or by changing them at run time. As an example, one Baseten inference engineer working on a code autocomplete model tried 77 configurations with a handwritten script during an internal hackathon. The search ended with a non-obvious setup that doubled tokens per second (TPS) for a customer's model.
• Techniques interact. Some help each other and some conflict. Quantizing the KV cache (the stored attention keys and values, explained in KV Caching) removes a bottleneck in disaggregation. Raising the batch size removes the spare compute that speculation needs. The goal is a balanced set of optimizations that delivers more than the sum of its parts.

The chapter covers five categories: quantization, speculation, caching, parallelism, and disaggregation. For each one, look for the conditions where it is recommended and the bottleneck or trade-off it introduces.

Terms used here come from earlier lessons. Prefill is the first phase of a request, where the model reads the whole prompt at once. It is limited by arithmetic speed (compute-bound). Decode is the second phase, where tokens are produced one by one. It is limited by how fast the weights can be read from memory (memory-bound). TTFT (time to first token) is the wait before the first output token, and TPS is tokens per second. See Finding the Bottleneck and Measuring Latency and Throughput for the full explanations.

Example 1: why fewer bits help both phases

Quantization improves latency (both TTFT and TPS), raises system throughput, and leaves headroom for other techniques such as disaggregation, speculation, and prefix caching to work better. It improves both phases for different reasons.

Models are trained with weights, activations, and other parts stored in a native number format. This is usually BF16 or FP16 (16 bits each). Training in 8-bit and 4-bit native precision is becoming more common. Post-training quantization takes finished weights and converts them from the native format to a lower-precision format. Halving the precision helps like this:

Working with quantized data adds overhead, so going from 16 to 8 bits is not exactly twice as fast. In practice, quantizing down one level of precision gives about 30 to 50 percent better performance for LLMs.

The table below shows the memory side with real sizes. Formula: GB = billions of parameters x bits / 8. For the 70B model at 16 bits: 70 x 16 / 8 = 140 GB. It covers three models at 16, 8, and 4 bits. A parameter at 16 bits takes 2 bytes, so a billion parameters take 2 GB. At 8 bits, a billion parameters take about 1 GB, a figure the parallelism lesson uses again. Weights only are counted here. The KV cache and other buffers come on top.

Weight memory in GB (billions of parameters x bits / 8)
model          16-bit    8-bit    4-bit
8B model         16.0      8.0      4.0
70B model       140.0     70.0     35.0
671B model     1342.0    671.0    335.5

Each column is half of the one before it, because each step halves the bits per parameter. The 70B model drops from 140 GB (more than one 80 GB GPU can hold) to 70 GB. Smaller weights also leave more VRAM (the GPU's own memory) free for the KV cache. This is why quantization "opens up headroom" for other techniques.

Example 2: why precision errors matter

The catch is that every quantized number is slightly wrong, and errors do not stay small. They are multiplied, added, and passed from layer to layer, and a model repeats this for every token. A quick demonstration uses the number Pi stored with fewer digits. The result squares and cubes three versions of Pi.

Pi         squared      cubed       
3.14159    9.869588     31.006198   
3.14       9.859600     30.959144   
3.0        9.000000     27.000000   

Pi cubed with full double precision: 31.006277
Error of the 3.0 version: 12.9%

The input changed by about 4.5 percent (3.0 against 3.14159), but the cube is off by almost 13 percent. The error roughly triples because the number is used three times. A model uses each weight in many multiplications across many layers and tokens, so most of the work of quantization is about two things: preventing precision errors and limiting their effect on the final output.

Example 3: number formats

A number format says how a pattern of bits is read as a value. These are the common ones, with the GPU architecture that first supported them:

The architecture names are GPU generations (see GPU Generations). Which formats matter for inference:

• FP64, "double precision", is used only for high-precision scientific computing, not for AI training or inference.
• FP32 is sometimes used in training and almost never in inference.
• FP6 is experimental at the time of writing, although AMD GPUs are adopting it quickly.
• That leaves 16, 8, and 4 bits as the main inference precisions.

Every format has three attributes:

• Precision: how many bits express one value. FP16 uses 16.
• Type: whether the bits mean an integer (no decimal part) or a floating-point number (has a decimal point that can move).
• Scale factor: a multiplier that maps values from the low-precision format back to the higher-precision one. Example 4 shows one at work.

These attributes decide two properties that determine how well a format can hold the numbers of a model:

• Dynamic range: the distance between the smallest and the largest value the format can represent.
• Granularity: how many parameters or other values share one scale factor.

Dynamic range is the key to low-precision inference without quality loss. Counting alone shows the limit: 16 bits can express 65,536 distinct values, and 8 bits only 256. The dynamic range describes how those values are spread between the smallest and largest one.

A floating-point number has three parts:

• Sign: one bit, positive or negative.
• Exponent: a group of bits that give a power of two.
• Mantissa: a group of bits that give the base value, which is then multiplied by two to the power of the exponent.

FP8 in the E4M3 layout means 4 exponent bits, 3 mantissa bits, and the one remaining bit for the sign. An integer format has only a sign and value bits.

graph LR
    A["FP8 E4M3: 8 bits"] --> S["1 bit: sign"]
    A --> E["4 bits: exponent"]
    A --> M["3 bits: mantissa"]
    B["INT8: 8 bits"] --> S2["1 bit: sign"]
    B --> V["7 bits: value"]

The exponent is the reason floating-point formats beat integer formats for inference. An integer format spaces its values evenly, so it cannot cover both very small and very large numbers. The exponent lets a floating-point format cover both. This matters because outliers (rare values far from the rest) carry a lot of meaning in a model, and a format must still represent them after quantization.

The next table gives the range of some formats, worked out from their bit layout. For a floating-point format with an exponent bias, the largest value is (1 + largest mantissa fraction) times two to the largest exponent, and the smallest normal value is two to the power (1 minus bias). Formats vary in how they reserve bit patterns for infinity and not-a-number. IEEE-style formats (FP16, BF16, E5M2) give the top exponent to those special values. E4M3 as used on GPUs keeps the top exponent for ordinary numbers and reserves only one pattern for not-a-number, which is why its maximum is 448 and not 240.

FP16       16 bits  largest 65504.0      smallest normal 6.10352e-05 
BF16       16 bits  largest 3.38953e+38  smallest normal 1.17549e-38 
FP8 E5M2    8 bits  largest 57344.0      smallest normal 6.10352e-05 
FP8 E4M3    8 bits  largest 448.000      smallest normal 0.0156250   
INT8        8 bits  range -128 to 127 (evenly spaced, step 1)
INT4        4 bits  range -8 to 7 (evenly spaced, step 1)
Distinct bit patterns: 16 bits = 65536, 8 bits = 256

Read the lines in pairs. FP16 and BF16 both use 16 bits, yet BF16 covers numbers up to about 3.4e38 because it spends 8 bits on the exponent instead of 5. It pays with a shorter mantissa (7 bits instead of 10), so its steps between neighbors are coarser. The two 8-bit floating-point layouts show the same trade: E5M2 reaches 57,344, E4M3 reaches 448 but is finer inside its range. The 8-bit integer format covers only -128 to 127 in steps of one, which is why it needs a scale factor for almost every real tensor.

Example 4: scale factors and granularity

A scale factor is how a small format stands in for a large range. The idea with a small case: take INT8, which holds integers from -127 to 127. A group of weights has a largest absolute value of 2.0. Choose the scale 2.0 / 127 = 0.01575. Each weight is divided by the scale and rounded, which gives an integer, and multiplied by the scale again to get an approximation back. A weight of 1.0 becomes round(1.0 / 0.01575) = 63, and returns as 63 x 0.01575 = 0.992. The error is 0.008.

Now the problem. One huge outlier makes the scale large, and then all small values collapse to a few integers. This depends on how many values share one scale factor, which is the granularity. Quantization can use scale factors at three levels:

• Tensor level: one scale factor for the whole QKV tensor (the combined query, key, and value tensor of attention).
• Channel level: one scale factor for each feature vector inside the tensor.
• Block level: each feature vector is cut into blocks of N values, and each block gets its own scale factor.

The experiment quantizes 64 values to INT8 (sample values 0.5 x sin(0.7 x i)). Most values are small, and one is an outlier of 50. It compares one scale for all 64 values with one scale per block of 16 and one per block of 4.

one scale per 64 values: mean error 0.08471, value[11] 0.4941 -> 0.3937
one scale per 16 values: mean error 0.02105, value[11] 0.4941 -> 0.3937
one scale per  4 values: mean error 0.00362, value[11] 0.4941 -> 0.3937

Reading this: the mean error shrinks as the blocks shrink, from 0.08471 with one scale for all 64 values to 0.00362 with a scale every 4 values. The outlier of 50 forces a scale of about 0.39 in its own block, so every value in that block is rounded to a multiple of 0.39. Value 11 sits in the same block as the outlier in all three runs (indices 8 to 11 are one block even at size 4), so it is badly rounded each time: 0.4941 becomes 0.3937. The difference is how many other values suffer. With one scale for everything, all 64 values are rounded that coarsely. With blocks of 4, only 4 are. This is the point of the rule: more granular quantization has a lower chance of smoothing over outliers, which preserves quality. The cost is overhead for storing and applying more scale factors.

Microscaling formats

The formats added with Blackwell use block scale factors built into the format:

• MXFP8 and MXFP4 are "microscaling" formats. They compute a scale factor for each block of 32 parameters, which reduces the damage from the lower dynamic range of the 8-bit and 4-bit types.
• NVFP4 is a 4-bit format by NVIDIA with finer granularity than the MX formats. Its block size is 16, and it adds a second, 32-bit global scale factor for the whole tensor to fight the quality loss of 4-bit values.

The trade-off: block scale factors must be stored in memory, which slightly reduces the gain from quantization. Both the tensor scale and the block scales must be applied, which adds a little compute. Blackwell GPUs offset this overhead by applying scale factors inside the Tensor Cores.

Local and edge inference

This course focuses on inference in data centers, but quantization is also central to local and edge inference, especially for large models. GGUF, a binary file format for storing models, is the most popular way to distribute heavily quantized models on Hugging Face. Individual researchers and companies use it to squeeze huge models such as DeepSeek onto consumer hardware like Apple computers. These schemes use dynamic quantization: some layers or components stay at their original precision while others are quantized to integers with as few as one bit. The reported precision is the average, which is why a name such as the 1.58-bit quantization from the fine-tuning lab Unsloth appears.

These schemes suit local use. Production systems should stick to floating-point formats. Integer formats are not suitable for quality-sensitive workloads because of their small dynamic range. Among the floating-point formats, the 8-bit ones (FP8, MXFP8) are generally the sweet spot: better performance without losing quality. FP4, especially NVFP4 with its finer granularity, is promising, but FP8 and MXFP8 are the most flexible, particularly when quantizing the KV cache.

Example 5: how to quantize, and when

The more parameters a model has, the less it reacts to quantization, because each single parameter matters less. Even so, large models still need careful quantization. There are two moments at which quantization can happen:

Some labs release models made with QAT, for example GPT-OSS in MXFP4 and Kimi K2 Thinking in INT4. Inference engineers who use open models only have finished weights, so they can only do post-training quantization.

A leading tool for PTQ is NVIDIA TensorRT Model Optimizer (ModelOpt). It is an open-source library that also supports pruning, distillation, and sparsity. Its outputs work with all of the main inference engines: vLLM, SGLang, and TensorRT-LLM (see Inference Engines).

Before running PTQ, two decisions remain after picking a precision:

1. Which parts of the model to quantize: weights, activations, KV cache, or attention.
2. Which number format has the dynamic range and granularity the workload needs.

These choices turn quantization from a yes-or-no switch into a range of trade-offs between speed and quality.

Sensitivity of each component

Parts of a model react differently to quantization. Reducing precision in a more sensitive part carries a higher risk of quality loss. From least to most sensitive:

graph LR
    W["Weights and activations: low risk"] --> K["KV cache: moderate risk"] --> A["Attention: high risk"]

You can be more selective inside each component. Even in linear layers and activations, the early and late layers (the input and output layers of the network) may stay in their original precision because they are more sensitive.

Quantizing weights and activations helps speed directly. Quantizing the KV cache gives an extra boost to prefix caching and disaggregation, because the KV cache is a valuable resource: a smaller one lets the engine keep more of it in memory and read it faster. The risk is compounding. The cached values of each token are used by every later token, so a precision error introduced by quantization can grow from token to token.

Compounding is also why attention is the riskiest layer to quantize. Attention is very sensitive to dynamic range, and each attention calculation depends on the results of earlier ones. Over thousands of tokens, the errors add up fast. All but the most aggressive schemes keep functions such as softmax in their original precision.

A moderate recipe for low-precision inference follows from this. Use a format with high dynamic range, such as FP8 (preferably a microscaling format such as MXFP8). Carefully quantize selected linear layers, the activations, and often the KV cache values. Components of the attention layer are rarely quantized even then.

Example 6: measuring the quality impact

The standard for production quantization is zero perceptible quality loss. After quantizing, the output must be tested against the original precision. There are three methods:

1. Perplexity: compute the perplexity score of the quantized model and compare it with the original.
2. Intelligence benchmarks: run a standard benchmark such as MMLU (a broad knowledge test) or SWE-bench (a software engineering test) and compare the scores.
3. Custom evals: run an evaluation suite built for your product and compare it with the original weights.

In each case, look for a score difference that cannot be told apart from noise. LLMs are non-deterministic, so scores vary slightly between runs.

Perplexity is the simplest check. The model is not asked to write anything. Instead, it is given sequences that are known to be good, and the check measures how likely the model thought each real next token was. A high perplexity means the model is more "surprised" by the real text, which is the opposite of what a next-token predictor should be. After quantization, look for a minimal increase.

The formula is: perplexity = exp(-(1/N) x sum of ln p), where p is the probability the model gave to each of the N real tokens. Small case by hand with two tokens where the model gave probabilities 0.5 and 0.25: ln 0.5 = -0.693 and ln 0.25 = -1.386, the average is -1.040, and exp(1.040) = 2.83. Read in words: the model was about as uncertain as if it chose among 2.83 equally likely tokens at each step. The result compares an original model (probabilities 0.50, 0.40, 0.25, 0.80), a slightly worse quantized one (0.49, 0.39, 0.24, 0.79), and a damaged one (0.30, 0.20, 0.10, 0.50) on four real tokens.

by-hand case        : 2.83
original            : 2.236
quantized (small)   : 2.292
quantized (damaged) : 4.273

Reading the numbers: the lightly quantized model gives each real token a slightly smaller probability, so perplexity rises a little. The damaged model gives much smaller probabilities, so perplexity rises a lot. A small increase is the target. Perplexity says nothing about task quality directly, which is why the other two methods exist.

A more complete check uses a public benchmark or, better, a domain-specific eval that matches real usage. On evals, look for a minimal drop in the score. Running all three kinds of check and comparing each against the original weights (an apples-to-apples comparison) gives the full picture.

Quantization is a scale and not a switch. Quantizing to FP8 instead of FP4, or quantizing fewer components (for example weights only), still gives a performance gain with less risk. For a highly sensitive domain where quality cannot be risked, every other technique in this chapter is lossless, so the rest of the chapter still applies.

Interview points

• Quantization lowers the bits per value. Prefill gains because lower-precision Tensor Cores have twice the FLOPS. Decode gains because each value is half the size, so effective memory bandwidth doubles. The real gain for one precision step is about 30 to 50 percent, not 100 percent, because of overhead.
• Dynamic range, not only the bit count, decides quality. Floating-point formats have an exponent, so they represent outliers better than integer formats of the same width.
• Granularity is the number of values sharing one scale factor (tensor, channel, or block). Finer granularity protects outliers and costs extra storage and compute. MXFP8 and MXFP4 use blocks of 32 values. NVFP4 uses blocks of 16 plus a 32-bit global scale.
• Component sensitivity from least to most: weights, activations, KV cache, attention. Softmax usually stays in the original precision.
• Open-weight users can only do post-training quantization. Quantization-aware training needs the training run. ModelOpt is the tool named for PTQ.
• Check quality with perplexity, a public benchmark (MMLU, SWE-bench), and a custom eval. A difference that cannot be told apart from noise passes.
• Integer formats suit local deployments (GGUF, dynamic quantization such as the 1.58-bit Unsloth builds), while FP8 or MXFP8 is the usual production choice.

Summary

• Quantization stores a model's numbers in fewer bits to cut memory traffic and raise arithmetic throughput.
• Common formats: FP64, FP32, FP16, BF16, FP8, MXFP8, INT8, FP6, FP4, MXFP4, NVFP4, INT4. For inference the main ones are 16, 8, and 4 bits.
• A format has precision, type, and scale factor, which together set its dynamic range and granularity.
• Floating-point formats have sign, exponent, and mantissa bits. E4M3 is 1 sign, 4 exponent, and 3 mantissa bits.
• Scale factors can be applied per tensor, per channel, or per block. Finer granularity keeps quality better and costs more overhead.
• Quantizing weights is the safest. Activations come next, then the KV cache, and attention is the riskiest because errors compound over tokens.
• Perplexity, benchmarks, and custom evals together show whether a quantized model keeps its quality.
• Every other technique in this chapter is lossless in quality.