Making, Quantizing, and Testing Your Own GGUF

A model that you download from Hugging Face is usually a folder of files in the safetensors format. Safetensors is a file format that stores a model's weights, which are the millions of numbers the model learned during training. llama.cpp, the program this course uses to run models on your own machine, does not read that folder. It reads a single file in its own format, called GGUF. This lesson shows how to make that file yourself, shrink it, and check how much quality the shrinking cost.

Making your own GGUF matters for three reasons. A new model often appears on Hugging Face days before anyone publishes a GGUF of it. Published GGUF files come in a fixed set of sizes, and you may need a different one. And when you make the file yourself, you can measure it and report real numbers instead of trusting someone else's claim. Quantization is the step that stores each weight with fewer bits so that the file is smaller and faster to read. The two previous lessons explained what is inside the Q4_K_M and IQ4_XS blocks. This lesson runs the tools that produce them. See k-quants, i-quants, and the Importance Matrix for the block layouts.

The whole pipeline looks like this:

Hugging Face folder (safetensors + config + tokenizer)
        |  convert_hf_to_gguf.py
        v
16-bit GGUF file  ---->  llama-imatrix (optional)  ---->  imatrix.gguf
        |                                                     |
        |  llama-quantize  <----------------------------------+
        v
Q8_0, Q4_K_M, IQ4_XS ... .gguf files
        |  llama-perplexity (compare each one to the 16-bit file)
        v
numbers: perplexity, KL divergence

Each tool in the pipeline has one job:

Every flag in this lesson is checked against the pinned llama.cpp commit 0c1e57098. That means each flag was read from the --help output of the built binaries or from the argparse definitions (the code that declares command-line options) of the Python scripts.

Examples 1 to 4 show the commands for a real model. Examples 5 to 11 were run for real on a tiny model with random weights. This model has the Llama architecture, 4 layers, 256 dimensions, the Qwen2 tokenizer with 151,936 tokens, and 80.6 million parameters stored as F32 (32-bit floating point numbers). Its output formats, file sizes, and tensor types are real. Its perplexity values do not measure language quality, because random weights predict text no better than chance. The tiny model is useful for learning the tools because it is small enough to convert and quantize in seconds.

Example 1: get the llama.cpp source and the Python requirements

The conversion scripts live in the llama.cpp repository, and they need Python packages that the compiled binaries do not. This example installs both the program and those packages.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt

The build commands are the ones in docs/build.md. They put llama-quantize, llama-imatrix, llama-perplexity, llama-gguf-split, and the other tools in build/bin/. A prebuilt release archive contains the same binaries but not the Python scripts, so you need the source checkout for conversion either way.

The python3 -m venv .venv line creates a virtual environment, which is a private folder of Python packages. It keeps the converter's packages separate from the rest of your system.

requirements/requirements-convert_hf_to_gguf.txt pulls in the legacy converter's requirements (NumPy, sentencepiece, transformers, protobuf, and the gguf package). It also pins torch==2.11.0 and adds the CPU wheel index https://download.pytorch.org/whl/cpu as an extra package index. The converter uses PyTorch only to read tensors from the files. It never runs the model, so no GPU is needed. The top-level requirements.txt includes the requirement files of every script, but the single file above is enough for conversion.

The scripts and the binaries must come from the same checkout. The converter writes metadata keys and tensor names that the C++ loader of the same version expects. A model architecture that was added recently exists only in recent versions of both.

Example 2: download a safetensors model with the hf CLI

The converter needs only a few files from the model repository, so this example downloads only those. The hf command comes with the huggingface_hub Python package. The Hugging Face Hub covers it in detail.

pip install huggingface_hub
hf download Qwen/Qwen2.5-Coder-0.5B-Instruct \
  --include "*.json" --include "*.safetensors" --include "merges.txt" \
  --local-dir ./Qwen2.5-Coder-0.5B-Instruct

hf download REPO_ID takes these options:

• --include and --exclude take glob patterns (file name patterns with * as a wildcard) that select files.
• --revision takes a branch, tag, or commit.
• --local-dir writes plain files into a directory instead of the shared cache.

All four options appear in hf download --help for huggingface_hub 2.1.1. The files the converter reads are:

• config.json for the architecture name and the hyperparameters (the sizes and settings that define the model's shape),
• *.safetensors for the tensors, plus model.safetensors.index.json when the weights are split across several files,
• tokenizer.json and tokenizer_config.json for the vocabulary and the chat template, and for BPE (byte-pair encoding) models also vocab.json and merges.txt.

Some repositories also contain PyTorch .bin files, which store the same weights a second time. Downloading only *.safetensors halves the download. The safetensors Format explains why the converter prefers safetensors.

Example 3: convert to a 16-bit GGUF with --outtype

convert_hf_to_gguf.py writes one GGUF file that holds all the tensors, the tokenizer, and the chat template. The choice you make is the number format of the tensors, set with --outtype. The 16-bit file you create here is the source for every quantization that follows, so it should lose as little as possible.

python3 convert_hf_to_gguf.py ./Qwen2.5-Coder-0.5B-Instruct \
  --outtype bf16 \
  --outfile Qwen2.5-Coder-0.5B-Instruct-BF16.gguf

The --outtype choices at the pinned commit are f32, f16, bf16, q8_0, tq1_0, tq2_0, and auto, with auto as the default. With auto, the script looks at the first tensor that has 2 or more dimensions. If that tensor is BF16, the output is BF16 (the script prints "heuristics detected bfloat16 tensor dtype, setting --outtype bf16"). If it is FP16, the output is FP16. Otherwise it falls back to FP16.

The log below is a reconstructed excerpt, not captured from a run. Each line is the format string of a logger.info call in the pinned convert_hf_to_gguf.py or conversion/base.py, printed in Python's default LEVEL:logger:message format with the logger name hf-to-gguf. The ... marks values that depend on the model, and the script prints many more lines between these. Look at the first lines to confirm the architecture name, and at the last line to confirm the export finished.

INFO:hf-to-gguf:Loading model: Qwen2.5-Coder-0.5B-Instruct
INFO:hf-to-gguf:Model architecture: Qwen2ForCausalLM
INFO:hf-to-gguf:gguf: context length = ...
INFO:hf-to-gguf:gguf: embedding length = ...
INFO:hf-to-gguf:gguf: head count = ...
INFO:hf-to-gguf:gguf: key-value head count = ...
INFO:hf-to-gguf:gguf: file type = ...
INFO:hf-to-gguf:Model successfully exported to Qwen2.5-Coder-0.5B-Instruct-BF16.gguf

Each line confirms one thing. Model architecture shows which converter class the script chose, based on config.json. The gguf: lines show hyperparameters that are about to be stored in the file. The last line names the finished file.

Choose the output type with this table:

BF16 and FP16 are two ways to store a number in 16 bits. BF16 covers a much wider range of values, and FP16 keeps more fine detail. The overflow warning for f16 exists because FP16 cannot store anything larger than 65504.

You always quantize from the 16-bit file, not from an already quantized one. llama-quantize can requantize an already quantized file only with --allow-requantize. Its help text warns that this "can severely reduce quality compared to quantizing from 16bit or 32bit". Rounding twice adds the errors together.

Without --outfile, the script builds a file name from the model's metadata with naming_convention in gguf-py. The name joins the base name, size label, fine-tune, version, and encoding with hyphens. Example 13 returns to the naming scheme.

Example 4: inspect the result with gguf-dump

After a conversion you want to check that the metadata and tensors are what you expect. gguf-dump prints the contents of a GGUF file. It is a script in gguf-py (gguf/scripts/gguf_dump.py), installed as the gguf-dump command with pip install gguf.

gguf-dump tiny-Q4_K_M.gguf
# without installing: PYTHONPATH=gguf-py python3 gguf-py/gguf/scripts/gguf_dump.py tiny-Q4_K_M.gguf

The output below is for the tiny model after Example 5's Q4_K_M quantization. It comes from the pinned gguf-py, and the tensor list is shortened. The file has two parts: a list of key-value pairs (metadata) and a list of tensors. Look first at the keys that start with llama. and tokenizer., then at the tensor types in the second column from the right.

INFO:gguf-dump:* Loading: tiny-Q4_K_M.gguf
* File is LITTLE endian, script is running on a LITTLE endian host.
* Dumping 25 key/value pair(s)
      1: UINT32     |        1 | GGUF.version = 3
      2: UINT64     |        1 | GGUF.tensor_count = 39
      3: UINT64     |        1 | GGUF.kv_count = 22
      4: STRING     |        1 | general.architecture = 'llama'
      5: STRING     |        1 | general.name = 'tiny-random-llama'
      6: UINT32     |        1 | llama.context_length = 4096
      7: UINT32     |        1 | llama.embedding_length = 256
      8: UINT32     |        1 | llama.block_count = 4
      9: UINT32     |        1 | llama.feed_forward_length = 704
     10: UINT32     |        1 | llama.attention.head_count = 8
     11: UINT32     |        1 | llama.attention.head_count_kv = 2
     12: UINT32     |        1 | llama.rope.dimension_count = 32
     13: FLOAT32    |        1 | llama.attention.layer_norm_rms_epsilon = 9.999999747378752e-06
     14: FLOAT32    |        1 | llama.rope.freq_base = 500000.0
     15: STRING     |        1 | tokenizer.ggml.model = 'gpt2'
     16: STRING     |        1 | tokenizer.ggml.pre = 'qwen2'
     17: [STRING]   |   151936 | tokenizer.ggml.tokens = ['!', '"', '#', '$', '%', '&', ...]
     18: [INT32]    |   151936 | tokenizer.ggml.token_type = [1, 1, 1, 1, 1, 1, ...]
     19: [STRING]   |   151387 | tokenizer.ggml.merges = ['Ġ Ġ', 'ĠĠ ĠĠ', 'i n', 'Ġ t', 'ĠĠĠĠ ĠĠĠĠ', 'e r', ...]
     20: UINT32     |        1 | tokenizer.ggml.eos_token_id = 151643
     21: UINT32     |        1 | tokenizer.ggml.padding_token_id = 151643
     22: UINT32     |        1 | tokenizer.ggml.bos_token_id = 151643
     23: STRING     |        1 | tokenizer.chat_template = '{% for message in messages %}{% if loop.first and message...'
     24: UINT32     |        1 | general.quantization_version = 2
     25: UINT32     |        1 | general.file_type = 15
* Dumping 39 tensor(s)
      1:   38895616 |   256, 151936,     1,     1 | Q6_K    | output.weight
      2:        256 |   256,     1,     1,     1 | F32     | output_norm.weight
      3:   38895616 |   256, 151936,     1,     1 | Q4_K    | token_embd.weight
      4:      16384 |   256,    64,     1,     1 | Q4_K    | blk.0.attn_k.weight
      5:        256 |   256,     1,     1,     1 | F32     | blk.0.attn_norm.weight
      6:      65536 |   256,   256,     1,     1 | Q4_K    | blk.0.attn_output.weight
      7:      65536 |   256,   256,     1,     1 | Q4_K    | blk.0.attn_q.weight
      8:      16384 |   256,    64,     1,     1 | Q4_K    | blk.0.attn_v.weight
      9:     180224 |   704,   256,     1,     1 | Q5_0    | blk.0.ffn_down.weight
     10:     180224 |   256,   704,     1,     1 | Q4_K    | blk.0.ffn_gate.weight

Here is what to check in the dump:

• The first three rows are header fields (version 3, 39 tensors, 22 key-value pairs). The dump lists them as pseudo-keys, which is why it says 25 pairs: 22 real ones plus 3 header fields.
• general.architecture names the loader that llama.cpp uses. The llama.* keys hold the hyperparameters copied from config.json. A wrong head count or RoPE base (the base frequency of rotary position encoding) here produces garbage text later.
• tokenizer.ggml.* holds the whole vocabulary, the merges, and the special token ids. tokenizer.chat_template holds the Jinja template (a text template language) from tokenizer_config.json.
• general.file_type = 15 is MOSTLY_Q4_K_M in gguf-py's LlamaFileType. general.quantization_version = 2 is written by both the converter and llama-quantize.
• Each tensor row gives the element count, the dimensions with the innermost first, the ggml type, and the name. For example 256, 151936 is a matrix that PyTorch calls [151936, 256]. The GGUF Format reads all of these fields byte by byte.

In the tensor list you can already see the mix of types: output.weight is Q6_K, most matrices are Q4_K, blk.0.ffn_down.weight is Q5_0, and the norm weights stay F32. Example 5 explains why.

gguf-dump --json prints the same data as JSON, and --no-tensors stops after the metadata.

Example 5: llama-quantize to Q8_0 and Q4_K_M

llama-quantize reads a 16-bit (or 32-bit) GGUF and writes a new GGUF in which most tensors use a smaller type. The command takes an input file, an output file, and a type name or number from the list in --help. Here the input is tiny-f32.gguf, the tiny model stored as 32-bit floats.

./build/bin/llama-quantize --help
./build/bin/llama-quantize tiny-f32.gguf tiny-Q8_0.gguf Q8_0
./build/bin/llama-quantize tiny-f32.gguf tiny-Q4_K_M.gguf Q4_K_M

The first block below is part of the type list from --help at the pinned commit. The size and perplexity figures are the tool's own reference numbers for Llama-3-8B, so they show the trade-off for one real model. Read each line as "type, file size, how much worse the perplexity gets compared with the 16-bit model". For example, Q4_K_M is 4.58 GB and adds 0.1754 to the perplexity, while Q8_0 is 7.96 GB and adds only 0.0026. A smaller file costs more quality.

   2  or  Q4_0    :  4.34G, +0.4685 ppl @ Llama-3-8B
  30  or  IQ4_XS  :  4.25 bpw non-linear quantization
  14  or  Q4_K_S  :  4.37G, +0.2689 ppl @ Llama-3-8B
  15  or  Q4_K_M  :  4.58G, +0.1754 ppl @ Llama-3-8B
  16  or  Q5_K_S  :  5.21G, +0.1049 ppl @ Llama-3-8B
  17  or  Q5_K_M  :  5.33G, +0.0569 ppl @ Llama-3-8B
  18  or  Q6_K    :  6.14G, +0.0217 ppl @ Llama-3-8B
   7  or  Q8_0    :  7.96G, +0.0026 ppl @ Llama-3-8B
   1  or  F16     : 14.00G, +0.0020 ppl @ Mistral-7B
  32  or  BF16    : 14.00G, -0.0050 ppl @ Mistral-7B
          COPY    : only copy tensors, no quantizing

Next is the output of the Q4_K_M run on the tiny random-weight model (paths shortened, middle tensors omitted). Each numbered line is one tensor, with the type it had, the type it was converted to, and its size before and after. The warning lines come first because the tool plans the types before it starts converting.

llama_model_loader: - type  f32:   39 tensors
warning: blk.0.ffn_down.weight                - ncols    704 not divisible by 256 (required for type    q4_K) -> falling back to    q5_0
warning: blk.1.ffn_down.weight                - ncols    704 not divisible by 256 (required for type    q4_K) -> falling back to    q5_0
warning: blk.2.ffn_down.weight                - ncols    704 not divisible by 256 (required for type    q6_K) -> falling back to    q8_0
warning: blk.3.ffn_down.weight                - ncols    704 not divisible by 256 (required for type    q6_K) -> falling back to    q8_0
[   1/  39] output.weight                        - [   256, 151936,      1,      1], type =    f32, converting to q6_K .. size =   148.38 MiB ->    30.43 MiB
[   2/  39] output_norm.weight                   - [   256,      1,      1,      1], type =    f32, size =    0.001 MiB
[   3/  39] token_embd.weight                    - [   256, 151936,      1,      1], type =    f32, converting to q4_K .. size =   148.38 MiB ->    20.87 MiB
[   8/  39] blk.0.attn_v.weight                  - [   256,     64,      1,      1], type =    f32, converting to q4_K .. size =     0.06 MiB ->     0.01 MiB
[  26/  39] blk.2.attn_v.weight                  - [   256,     64,      1,      1], type =    f32, converting to q6_K .. size =     0.06 MiB ->     0.01 MiB
[  35/  39] blk.3.attn_v.weight                  - [   256,     64,      1,      1], type =    f32, converting to q6_K .. size =     0.06 MiB ->     0.01 MiB
llama_model_quantize_impl: model size  =   307.51 MiB (32.00 BPW)
llama_model_quantize_impl: quant size  =    53.04 MiB (5.52 BPW)
llama_model_quantize_impl: WARNING: 4 of 39 tensor(s) required fallback quantization

Every line matches a rule from k-quants, i-quants, and the Importance Matrix. The tool does not use one type for every tensor. It applies a per-tensor mix, which spends more bits on the tensors that hurt quality most when rounded.

• output.weight becomes Q6_K, and the norm weights stay F32. One-dimensional tensors are never quantized, because they are tiny and the quantization blocks work on rows of a matrix.
• attn_v is Q4_K in layers 0 and 1 and Q6_K in layers 2 and 3. A function called use_more_bits(i, 4) decides this. For 4 layers it is true for i >= 7*4/8 = 3 and for (i - 0) % 3 == 2, which gives layers 2 and 3.
• ffn_down has rows of 704 values. The Q4_K and Q6_K types store values in blocks of 256, and 704 is not a multiple of 256 (704 / 256 = 2.75). So the planned Q4_K falls back to Q5_0 and the planned Q6_K to Q8_0, which use blocks of 32 (704 / 32 = 22). The warnings come from tensor_type_fallback.
• The average is 5.52 bits per weight (bpw), well above 4.5. The reason is that the tiny model's two 151,936-row tensors (the embedding and the output matrix) hold most of its weights, and one of them is Q6_K. For the 8B model in the k-quants lesson the same mix averages 4.89 bpw.

The sizes can be checked by hand. The token_embd.weight tensor has 256 x 151,936 = 38,895,616 values. As F32 that is 38,895,616 x 4 bytes = 148.38 MiB. Q4_K costs 4.5 bits per value, so 38,895,616 x 4.5 / 8 = 21,878,784 bytes, which is 20.87 MiB. That matches the log.

The Q8_0 run converts every 2-dimensional tensor to q8_0. The output tensor stays Q8_0 because the mix keeps it when the target is Q8_0. The run ends with quant size = 81.69 MiB (8.50 BPW), which is exactly 34 bytes per 32 weights: 32 one-byte values plus a 2-byte scale.

You can change the mix with several options:

• --pure turns the mix off and uses one type for every tensor.
• --tensor-type attn_v=q8_0, --output-tensor-type, and --token-embedding-type override single tensors.
• --dry-run prints the final size without writing a file.

Example 6: llama-imatrix on calibration text

Plain quantization treats every weight as equally important. Some weights matter more, because they multiply inputs that are often large. An importance matrix (imatrix) records which input columns matter most, so that the quantizer can protect them. llama-imatrix makes it by running the model over a text file (the calibration text) and recording the mean squared input of every matrix multiplication. The k-quants lesson describes the idea.

./build/bin/llama-imatrix -m tiny-f32.gguf -f calibration.txt -o imatrix.gguf -c 256 --chunks 20 -t 4
./build/bin/llama-imatrix -m tiny-f32.gguf --in-file imatrix.gguf --show-statistics

Below is representative output of the first command. It comes from the pinned build on the random-weight model, and the timings differ by machine. Each [k] value is the running perplexity after k chunks of text.

compute_imatrix: tokenizing the input ..
compute_imatrix: tokenization took 34.642 ms
compute_imatrix: computing over 20 chunks, n_ctx=256, batch_size=2048, n_seq=8
compute_imatrix: 4.27 seconds per pass - ETA 0.17 minutes
[1]161046.5631,[2]167349.3654,[3]163314.6611,[4]163902.4102,[5]164473.2431,[6]163260.9848,[7]162472.6773,[8]163215.9556,
[9]163123.2284,[10]163479.9074,[11]163650.6003,[12]164682.5388,[13]164078.9846,[14]163598.9664,[15]163051.2454,[16]163135.2124,
[17]162616.6170,[18]162168.9294,[19]162555.6955,[20]162229.4668,
Final estimate: PPL = 162229.4668 +/- 1006.89997

The tool cuts the text into chunks of -c tokens (256 here), and --chunks 20 stops after 20 of them. While it collects statistics, it also computes perplexity (--no-ppl turns that off). Perplexity is explained in Example 8. For now, read it as "how surprised the model is by the text".

The random model's perplexity of about 162,000 is close to the vocabulary size of 151,936. A model that gives roughly equal probability to every token is as uncertain as picking among all 151,936 tokens, so its perplexity equals the number of tokens. A trained model on ordinary text scores in the single or low double digits.

The output file is imatrix.gguf by default. --output-format dat writes the older binary format instead. At the pinned commit the result covered 28 tensors: 7 matrices in each of 4 layers. The embedding is a lookup, not a matrix multiplication, so it has no entry. The output tensor is included only with --process-output.

The second command prints statistics from the finished file. Here is part of the --show-statistics table. Each row is one matrix, and the columns describe the spread of the recorded values across the matrix's input columns.

 Layer	       Tensor	          Σ(Act²)	  Min	            Max	           μ	   σ	 % Active	N	   Entropy	E (norm)	ZD	  CosSim
    3	attn_k              	    255.78	  0.4140	     2.0335	  1.00	  0.30	  100.00%	   256	    7.9355	 99.19%	     15.62%	  0.9450
    0	attn_output         	      0.89	  0.0018	     0.0062	  0.00	  0.00	  100.00%	   256	    7.9633	 99.54%	     12.11%	  0.0000
    1	ffn_down            	      2.04	  0.0007	     0.0193	  0.00	  0.00	  100.00%	   704	    9.2341	 97.62%	     10.80%	  0.7965

Σ(Act²) is the sum of the per-column values. N is the number of input columns (256, or 704 for ffn_down). Min, Max, and μ (the mean) describe the spread across columns. A column whose value is much larger than the others is a column whose weights the quantizer will protect.

The calibration text matters. Text close to the model's real use (code for a coding model, several languages for a multilingual one) gives statistics close to what the model sees in use.

Example 7: quantize with --imatrix to IQ4_XS

The --imatrix option passes the importance matrix to llama-quantize. The quantizer then uses it to choose better rounding for the weights in the important columns. The file format does not change. Only the stored numbers do.

./build/bin/llama-quantize --imatrix imatrix.gguf tiny-f32.gguf tiny-IQ4_XS.gguf IQ4_XS

Below is representative output from the pinned build on the random-weight model, with the middle tensors omitted. Look for the lines that mention the importance matrix, and for the types chosen for each tensor.

llama_model_quantize_impl: have importance matrix data with 28 entries
warning: blk.0.ffn_down.weight                - ncols    704 not divisible by 256 (required for type  iq4_xs) -> falling back to  iq4_nl
[   1/  39] output.weight                        - [   256, 151936,      1,      1], type =    f32,
====== llama_model_quantize_impl: did not find weights for output.weight
converting to q6_K .. load_imatrix: imatrix datasets=['calibration.txt']
load_imatrix: loaded 28 importance matrix entries from imatrix.gguf computed on 20 chunks
size =   148.38 MiB ->    30.43 MiB
[   3/  39] token_embd.weight                    - [   256, 151936,      1,      1], type =    f32,
====== llama_model_quantize_impl: did not find weights for token_embd.weight
converting to iq4_xs .. size =   148.38 MiB ->    19.71 MiB
[   8/  39] blk.0.attn_v.weight                  - [   256,     64,      1,      1], type =    f32, converting to q5_K .. size =     0.06 MiB ->     0.01 MiB
[   9/  39] blk.0.ffn_down.weight                - [   704,    256,      1,      1], type =    f32, converting to iq4_nl .. size =     0.69 MiB ->     0.10 MiB
llama_model_quantize_impl: quant size  =    51.60 MiB (5.37 BPW)

Here is how to read the log:

• The imatrix file records the calibration file name (imatrix datasets=[...]) and the chunk count, so you can trace where a GGUF's statistics came from.
• "did not find weights" for output.weight and token_embd.weight is expected. Example 6 collected no statistics for them, so they are quantized without importance weights.
• attn_v became Q5_K in every layer. The model has 8 query heads and 2 KV heads, so n_gqa() = 8 / 2 = 4 (grouped-query attention: 4 query heads share one KV head). The IQ4_XS rule moves attn_v to Q5_K when n_gqa() >= 4.
• ffn_down falls back from IQ4_XS (256-value blocks) to IQ4_NL (32-value blocks), the fallback listed in tensor_type_fallback. The reason is the same as in Example 5: rows of 704 values are not a multiple of 256.

Some types cannot work without an imatrix: IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, and IQ3_XXS. Running them without --imatrix stops with "this quantization requires an importance matrix!". These types use so few bits that unguided rounding would damage the model badly. --include-weights and --exclude-weights limit the imatrix to named tensors.

Example 8: llama-perplexity on a text file

To compare quantizations you need one number for a model's quality. Perplexity (What a Language Model Computes) is that number. The tool shows the model a test text, and at each position it checks how much probability the model gave to the token that actually came next. Perplexity is exp of the mean negative log probability of those tokens. Lower is better. A perplexity of 10 means the model is on average as unsure as if it were choosing among 10 equally likely tokens.

./build/bin/llama-perplexity -m tiny-Q8_0.gguf -f test.txt -c 256 --chunks 8 -t 4

Below is representative output from the pinned build on the random-weight model. The timings differ by machine.

perplexity: tokenizing the input ..
perplexity: tokenization took 63.078 ms
perplexity: calculating perplexity over 8 chunks, n_ctx=256, batch_size=2048, n_seq=8
perplexity: 3.77 seconds per pass - ETA 0.05 minutes
[1]152680.5277,[2]156981.6239,[3]159137.2574,[4]159867.9773,[5]160959.0366,[6]160299.1451,[7]163786.4034,[8]164783.4533,
Final estimate: PPL = 164783.4533 +/- 1650.42870

The tool splits the text into chunks of -c tokens. It evaluates each chunk from an empty context and scores only the second half, so every scored token has at least half a context of history. [k] is the running perplexity after k chunks, and the final line adds a standard error (the +/- value, which shows how much the estimate could move).

Perplexity numbers are comparable only when the test text, the context size, and the tokenizer are the same. The perplexity README at the pinned commit names the WikiText-2 test set as the convention among contributors, and scripts/get-wikitext-2.sh downloads it.

A perplexity difference smaller than the printed error says little on its own. The same is true for a difference of a few hundredths between two quantizations. This is the reason for the next example.

Example 9: KL divergence against the 16-bit model

Perplexity only looks at the probability of the correct token. Two models can have the same perplexity and still disagree on every other token. KL divergence (Quantization Fundamentals) measures how different two probability distributions are, and it is 0 when they are identical. Here the two distributions are the base model's prediction for the next token and the quantized model's prediction at the same position. llama-perplexity computes it in two runs. The first run saves the base model's logits (its raw scores for every token). The second run compares a quantized model against the saved scores.

./build/bin/llama-perplexity -m tiny-f32.gguf -f test.txt -c 256 --chunks 4 --kl-divergence-base logits-f32.bin
./build/bin/llama-perplexity -m tiny-Q4_K_M.gguf --kl-divergence-base logits-f32.bin --kl-divergence

The second run needs no -f option, because the saved file contains the tokens as well. Below is representative output of the second run from the pinned build on the random-weight model, with the token-probability statistics omitted. The top table shows running values per chunk. The two summary sections after it are what you report.

chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1    153930.0933 ± 4551.1601       0.00761 ±    0.00726       0.00210 ±    0.00001     0.000 ±  0.000 %    62.205 ±  4.320 %
   2    156321.2077 ± 3093.0076      -0.00363 ±    0.00485       0.00212 ±    0.00001     0.000 ±  0.000 %    66.142 ±  2.975 %
   3    159150.1136 ± 2495.8130       0.00052 ±    0.00371       0.00206 ±    0.00001     0.000 ±  0.000 %    62.730 ±  2.480 %
   4    159412.2309 ± 2175.0924      -0.00235 ±    0.00315       0.00206 ±    0.00001     0.000 ±  0.000 %    62.205 ±  2.153 %

====== Perplexity statistics ======
Mean PPL(Q)                   : 159412.230921 ± 2175.092368
Mean PPL(base)                : 159787.462361 ± 2172.201240
Cor(ln(PPL(Q)), ln(PPL(base))):  97.32%
Mean ln(PPL(Q)/PPL(base))     :  -0.002351 ±   0.003153
Mean PPL(Q)/PPL(base)         :   0.997652 ±   0.003145
Mean PPL(Q)-PPL(base)         : -375.231439 ± 503.102733

====== KL divergence statistics ======
Mean    KLD:   0.002062 ±   0.000007
Maximum KLD:   0.002490
99.9%   KLD:   0.002482
99.0%   KLD:   0.002419
95.0%   KLD:   0.002330
90.0%   KLD:   0.002287
Median  KLD:   0.002051
10.0%   KLD:   0.001879
 5.0%   KLD:   0.001834
 1.0%   KLD:   0.001747
 0.1%   KLD:   0.001675
Minimum KLD:   0.001651

Here is how to read the summary:

• Mean KLD is the average, over all scored tokens, of the KL divergence between the base model's next-token distribution and the quantized model's. It uses the whole distribution, not only the probability of the correct token. So it detects changes that perplexity averages away.
• The high percentiles (99.0%, 99.9%, Maximum) show the worst tokens. A quantized model with a low mean and a high maximum is usually right and occasionally very wrong. That matters for code, where one wrong token can break a program.
• Same top p is the percentage of tokens where both models rank the same token first. For greedy decoding (always taking the most likely token) it is the fraction of steps where the two models produce the same token.
• Δp is the change in the probability of the correct token. Here it is 0.000 because the random model gives every token a probability of about 1/151,936.
• Mean PPL(Q)/PPL(base) compares perplexities directly. It is 0.9977, below 1 but within its error of 1. With random weights the quantized model is not "better", only differently random. This is why KL divergence, which cannot be negative, is the more reliable comparison.

The logits file stores the base model's log probabilities for every scored token over the whole vocabulary, so it is large. Here it is 154 MB for 4 chunks of 128 scored tokens with a 151,936-token vocabulary. According to the perplexity README, it is 11 GiB for LLaMA 2 or 37 GiB for LLaMA 3 on the WikiText-2 test set.

Example 10: a comparison table across quants

With perplexity and KL divergence available, you can compare the files from Examples 5 and 7 side by side. The commands from Example 9 were run for each file. The llama-bench tool, shown below, measured speed.

./build/bin/llama-bench -m tiny-f32.gguf -m tiny-Q8_0.gguf -m tiny-Q4_K_M.gguf -m tiny-IQ4_XS.gguf -p 128 -n 32 -t 4

The table below shows representative results from the pinned build on the random-weight model. The speeds come from one 4-thread CPU and differ by machine. In the speed columns, pp128 is prompt processing (reading 128 tokens) and tg32 is token generation (writing 32 tokens), both in tokens per second.

Keep the random weights in mind when reading the table:

• Q8_0 is almost identical to the base. Its mean KLD is about 100 times lower than the 4-bit files.
• The two 4-bit files have similar mean KLD. IQ4_XS is 1.4 MiB smaller and slightly lower on the mean, but higher at the 99th percentile. With a trained model and a real test text, the same kind of table is how you compare two quantizations.
• Generation speed (tg32) follows file size for the F32 and Q8_0 rows. Fewer bytes per token means more tokens per second (the roofline model explains why). The 4-bit rows are slower than Q8_0 on this machine even though they are smaller. This model is so small that its weights fit in the CPU caches, so the work of unpacking 4-bit and 6-bit blocks dominates. On a 7B model, where weights must stream from RAM, the order follows size.
• The "Same top p" values for the 4-bit files are low because the random model's distributions are almost flat. A tiny change in a score changes which token is first. A trained model has peaked distributions, and its same-top-p values for 4-bit quantizations are much higher.

Example 11: llama-gguf-split split and merge

Some hosts and file systems limit the size of one file, and a large model can exceed it. llama-gguf-split cuts a GGUF into shards that llama.cpp loads as one model, and it can merge them back. The first command below splits the tiny Q8_0 file with at most 16 tensors per shard. The second merges the shards.

./build/bin/llama-gguf-split --split --split-max-tensors 16 tiny-Q8_0.gguf tiny-Q8_0
./build/bin/llama-gguf-split --merge tiny-Q8_0-00001-of-00003.gguf tiny-Q8_0-merged.gguf

Below is representative output from the pinned build. The split lists how many tensors and how many megabytes go into each shard. The first shard is large because it holds the 39 MiB output tensor and the 39 MiB embedding.

n_split: 3
split 00001: n_tensors = 16, total_size = 89M
split 00002: n_tensors = 16, total_size = 1M
split 00003: n_tensors = 7, total_size = 0M
Writing file tiny-Q8_0-00001-of-00003.gguf ... done
Writing file tiny-Q8_0-00002-of-00003.gguf ... done
Writing file tiny-Q8_0-00003-of-00003.gguf ... done
gguf_split: 3 gguf split written with a total of 39 tensors.
gguf_merge: tiny-Q8_0-00001-of-00003.gguf -> tiny-Q8_0-merged.gguf
gguf_merge: reading metadata tiny-Q8_0-00001-of-00003.gguf ...done
gguf_merge: tiny-Q8_0-merged.gguf merged from 3 split with 39 tensors.

Splitting works on whole tensors, never inside one. You choose the limit with --split-max-tensors N (default 128) or --split-max-size N(M|G) such as --split-max-size 2G. A shard cannot be smaller than its largest tensor. Asking for 40M here fails with "one of splits have 0 tensors", because the 39 MiB output tensor and the 39 MiB embedding cannot share a 40 MB shard. The output names follow the pattern PREFIX-00001-of-00003.gguf.

Every shard is a valid GGUF file. The first holds all the metadata, and the others hold only three keys. gguf-dump on the second shard shows them:

      4: UINT16     |        1 | split.no = 1
      5: INT32      |        1 | split.tensors.count = 39
      6: UINT16     |        1 | split.count = 3

These keys record the shard's position (split.no), the total tensor count, and the number of shards.

To load a split model, pass the first shard, for example llama-cli -m tiny-Q8_0-00001-of-00003.gguf. llama.cpp opens the others by name. convert_hf_to_gguf.py takes the same --split-max-tensors and --split-max-size options, and llama-quantize --keep-split keeps the input's shard layout. The merged file is not byte-identical to the original, because it keeps the split.* keys (with split.no = 0 and split.count = 0), but it holds the same tensors.

Example 12: LoRA adapters with convert_lora_to_gguf.py and --lora

Fine-tuning a whole model is expensive. A LoRA adapter (low-rank adaptation) is a cheaper alternative. For each adapted layer it is a pair of small matrices whose product is added to the frozen base weight (How Models Are Trained). The adapter file is small, and one base model can serve many adapters. This example shows how to use an adapter with llama.cpp without merging it into the base model.

python3 convert_lora_to_gguf.py ./my-lora-adapter \
  --base ./Qwen2.5-Coder-0.5B-Instruct \
  --outtype f16 --outfile my-lora-F16.gguf

./build/bin/llama-cli -m Qwen2.5-Coder-0.5B-Instruct-Q4_K_M.gguf --lora my-lora-F16.gguf
./build/bin/llama-cli -m Qwen2.5-Coder-0.5B-Instruct-Q4_K_M.gguf --lora-scaled my-lora-F16.gguf:0.5

convert_lora_to_gguf.py takes the directory of a PEFT adapter (the format of Hugging Face's parameter-efficient fine-tuning library), which holds adapter_config.json and adapter_model.safetensors. The --base option points to a directory with the base model's config.json. The base weights themselves are not needed. Without --base, the script reads the base model id from the adapter config and fetches its config from the Hub, and --base-model-id names a different one. Its --outtype default is f32, unlike the main converter's auto.

At run time, --lora FNAME applies an adapter at scale 1.0, and --lora-scaled FNAME:SCALE applies it at a scale you choose, such as 0.5 for half strength. Both accept a comma-separated list to stack several adapters. The adapter applies on top of a quantized base model, so one Q4_K_M base can serve several adapters. llama-server adds --lora-init-without-apply and a POST /lora-adapters endpoint to change scales while it runs.

Example 13: naming and uploading a GGUF repository

To share your files so that tools and people can find them, two conventions matter: the file names inside the repository and the repository's model card (its description page).

hf auth login
hf repos create my-user/Qwen2.5-Coder-0.5B-Instruct-GGUF
hf upload my-user/Qwen2.5-Coder-0.5B-Instruct-GGUF . . --include "*.gguf" --include "README.md"

File names follow the scheme implemented by naming_convention in gguf-py. It joins the base name, size label, fine-tune, version, encoding, and an optional type with hyphens, with the encoding in upper case. A repository then holds one file per quantization, for example:

Qwen2.5-Coder-0.5B-Instruct-BF16.gguf
Qwen2.5-Coder-0.5B-Instruct-Q8_0.gguf
Qwen2.5-Coder-0.5B-Instruct-Q6_K.gguf
Qwen2.5-Coder-0.5B-Instruct-Q4_K_M.gguf
Qwen2.5-Coder-0.5B-Instruct-IQ4_XS.gguf
Qwen2.5-Coder-0.5B-Instruct-Q4_K_M-00001-of-00002.gguf   (a split file, if needed)
imatrix.gguf
README.md

Tools depend on this pattern. llama-cli -hf my-user/Qwen2.5-Coder-0.5B-Instruct-GGUF:Q4_K_M picks the file by the quantization name in it. The -hf help text says the quant "is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist". The loop pull command of this course does the same.

The model card (README.md) starts with a YAML block (a small block of structured settings). The base_model field, part of ModelCardData in huggingface_hub, links the GGUF repository to the original model on the Hub. The card should state the source revision, the llama.cpp commit used, the calibration text of the imatrix, and the KL divergence table from Example 10. Model licenses often require the license text and attribution to travel with derived files, so the card also repeats the base model's license.

hf upload REPO_ID [LOCAL_PATH] [PATH_IN_REPO] uploads a file or a folder. For folders with many files it splits the upload into several commits and resumes when you run it again (from hf upload --help in huggingface_hub 2.1.1). hf repos create takes --private for a private repository.

Interview points

• The pipeline is: convert the safetensors checkpoint to a 16-bit GGUF, optionally build an imatrix, quantize from the 16-bit file, then measure against it.
• convert_hf_to_gguf.py --outtype auto picks BF16 or FP16 from the dtype of the first 2-dimensional tensor, and BF16 copies BF16 weights exactly.
• llama-quantize applies a per-tensor mix of types, keeps 1-dimensional tensors in F32, and falls back to 32-value block types for rows whose length is not divisible by 256.
• llama-imatrix collects the mean squared activation for each input column on calibration text. The result changes the quantizer's choices, not the file format.
• The lowest i-quants (IQ1_*, IQ2_*, IQ3_XXS) cannot be produced without an imatrix.
• Perplexity is comparable only on the same text, context size, and tokenizer, and differences of a few hundredths are often within the error.
• KL divergence (--kl-divergence-base first, then --kl-divergence) compares the full next-token distributions at every token. The numbers to report are the mean, the 99th percentile, and "same top p".
• A random-weight model has a perplexity close to its vocabulary size.
• llama-gguf-split splits on tensor boundaries. The first shard holds the metadata, and loading the first shard loads all of them.
• LoRA adapters are converted with convert_lora_to_gguf.py and applied at load time with --lora or --lora-scaled on top of a quantized base model.

Summary

• The conversion scripts and the binaries must come from the same llama.cpp checkout.
• requirements/requirements-convert_hf_to_gguf.txt installs the converter's dependencies, including CPU PyTorch.
• --outtype accepts f32, f16, bf16, q8_0, tq1_0, tq2_0, and auto.
• gguf-dump lists header fields, metadata, and tensors, with the dimensions written innermost first.
• general.file_type = 15 means Q4_K_M, and general.quantization_version = 2 is written by the converter and by llama-quantize.
• The tiny model's Q4_K_M file was 5.52 bits per weight because its two large tensors dominate, and its Q8_0 file was exactly 8.50 bits per weight.
• --pure, --tensor-type, --output-tensor-type, --token-embedding-type, and --dry-run control or preview the mix of types.
• llama-imatrix writes imatrix.gguf by default and records the calibration file name and the chunk count.
• llama-perplexity scores the second half of each -c-token chunk.
• The KL divergence logits file stores full-vocabulary log probabilities, so it grows with test length and vocabulary size.
• Q8_0 had a mean KLD of 0.000022 against the F32 base, and the 4-bit files were near 0.002.
• llama-gguf-split options are --split-max-tensors (default 128) and --split-max-size, and shards are named -0000K-of-0000N.gguf.
• --lora FNAME and --lora-scaled FNAME:SCALE apply adapters at load time.
• GGUF file names put the quantization type last so that -hf repo:QUANT and download tools can select files.