Lesson 28 of 73
The GGUF Format
A large language model is a very large collection of numbers, called weights, plus a description of how to use them. To run a model on your own computer, an inference engine such as llama.cpp (a program that runs a model to produce text) needs more than the numbers. It also needs the size of every layer, the rules for cutting text into tokens, and the template that formats a chat. GGUF is the file format llama.cpp uses to keep all of this together in one file.
The file holds the model's hyperparameters (fixed settings chosen when the model was designed, such as the number of layers), its tokenizer, its chat template, and every tensor (a block of numbers arranged in a grid, such as a matrix). The tensors can be quantized, which means stored with fewer bits per number to save memory, or left in full precision. Making, Quantizing, and Testing Your Own GGUF produced GGUF files with the llama.cpp tools. This lesson writes and reads them in Java, one byte at a time.
Reading a binary file means knowing which byte means what. A reader that gets one size wrong is off by a few bytes, and everything after that point is garbage. For that reason this lesson computes every position and size by hand before it trusts a program. After it, a reader can parse any GGUF header, compute each tensor's position and size, and explain every line that gguf-dump prints. gguf-dump is a command-line tool from the gguf-py Python package that lists the metadata and tensors of a GGUF file. The lesson ends with Project 7, GgufInfo, which loop show uses in section 4.
A GGUF file always has the five sections in the table below, in this order. Metadata here means information about the model, as opposed to the weights themselves.
| Section | Content |
|---|---|
| Header | magic GGUF, version, tensor count, key-value count |
| Metadata | key-value pairs: a string key, a value type, a value |
| Tensor infos | per tensor: name, number of dimensions, dimensions, ggml type, offset |
| Padding | zero bytes up to the next multiple of the alignment |
| Data | tensor bytes, each tensor padded to the alignment |
The constants come from gguf-py 0.19.0 (constants.py) and from ggml/include/gguf.h and ggml/src/gguf.cpp at the pinned llama.cpp commit. The file written in Example 6 was checked with gguf-dump, with gguf-py's GGUFReader and dequantizer, and with llama.cpp's own GGUF loader.
Example 1: the layout of a GGUF file
Before writing code that reads a file, a programmer needs a map: at which byte does each part start, and how long is it? This example builds that map for a tiny file with one metadata pair and two tensors. A tensor is a block of numbers arranged in a grid. A vector is a tensor with one dimension, and a matrix is a tensor with two.
The program adds up the field sizes given in gguf.h, the C header file that defines the format. In the output, each line starts where the previous one ended.
import java.util.Locale;
public class LayoutOverview {
public static void main(String[] args) {
int alignment = 32;
long header = 4 + 4 + 8 + 8; // magic, version, tensor count, kv count
long kv = (8 + "general.architecture".length()) + 4 + (8 + "llama".length()); // one string KV pair
long info1 = (8 + "output_norm.weight".length()) + 4 + 8 * 1 + 4 + 8; // name, n_dims, dims, type, offset
long info2 = (8 + "token_embd.weight".length()) + 4 + 8 * 2 + 4 + 8;
long beforeData = header + kv + info1 + info2;
long dataStart = (beforeData + alignment - 1) / alignment * alignment;
long t1 = 64 * 4, t2 = 3 * 2 * 34; // F32[64], Q8_0[64 x 3]
long t1Padded = (t1 + alignment - 1) / alignment * alignment;
String[][] rows = {
{"header", "0", String.valueOf(header)},
{"metadata (1 KV)", String.valueOf(header), String.valueOf(kv)},
{"tensor infos (2)", String.valueOf(header + kv), String.valueOf(info1 + info2)},
{"padding", String.valueOf(beforeData), String.valueOf(dataStart - beforeData)},
{"output_norm.weight", String.valueOf(dataStart), String.valueOf(t1)},
{"token_embd.weight", String.valueOf(dataStart + t1Padded), String.valueOf(t2)}};
for (String[] r : rows) System.out.println(String.format(Locale.ROOT, "%-20s at %4s, %4s bytes", r[0], r[1], r[2]));
System.out.println("tensor offsets inside the data section: 0 and " + t1Padded);
}
}header at 0, 24 bytes
metadata (1 KV) at 24, 45 bytes
tensor infos (2) at 69, 107 bytes
padding at 176, 16 bytes
output_norm.weight at 192, 256 bytes
token_embd.weight at 448, 204 bytes
tensor offsets inside the data section: 0 and 256Read the output from top to bottom. The header ends at byte 24, the metadata ends at 24 + 45 = 69, and the tensor infos end at 69 + 107 = 176. The padding then moves the start of the data from 176 to 192. The first tensor takes 256 bytes (64 values of 4 bytes each), so the second tensor starts at 192 + 256 = 448. The second tensor holds 6 blocks of 34 bytes, which is 204 bytes. Example 6 explains the blocks.
The size 107 for the two tensor infos comes from their names and dimensions:
info 1: (8 + 18) + 4 + 8 + 4 + 8 = 50 "output_norm.weight" has 18 characters, 1 dimension
info 2: (8 + 17) + 4 + 16 + 4 + 8 = 57 "token_embd.weight" has 17 characters, 2 dimensions
50 + 57 = 107The list below explains each size.
The comment at the top of gguf.h lists the structure, and the program adds up its field sizes:
- The header is 24 bytes: 4 bytes of magic, a
uint32version, and two 64-bit counts. - Each metadata pair is a string key, a 4-byte value type, and the value. A GGUF string is an 8-byte length followed by the bytes, so
general.architecture = "llama"takes(8 + 20) + 4 + (8 + 5) = 45bytes. - Each tensor info is a string name, a
uint32dimension count, one 8-byte size per dimension, a 4-byte type, and an 8-byte offset. The two infos take 107 bytes. - Padding fills up to the next multiple of the alignment, 32 by default: the infos end at byte 176, so the data starts at 192.
- The data section holds the tensors. Each tensor's offset is measured from the start of the data section, and each tensor is padded to the alignment too, so the second tensor starts at offset 256 even though the first is exactly 256 bytes, and would start at 256 if the first were 250.
Everything before the data section is metadata, and a reader can parse it without touching the tensors. That is how gguf-dump and llama-quantize --dry-run work on large files.
Example 2: magic, version, and counts
Every reader starts with the header, and the header answers two questions: is this a GGUF file at all, and which version of the layout does it use? The header is 24 bytes long. This program writes the header of a file with 291 tensors and 25 key-value pairs, prints the raw bytes, and reads them back.
Two terms matter here. A magic number is a fixed value at the start of a file that identifies its type, so a program can reject a wrong file early. GGUF uses the four ASCII letters GGUF. Endianness is the order in which the bytes of a multi-byte number are stored. Little-endian stores the least significant byte first, so the number 291 (hexadecimal 0x123) is stored as the bytes 23 01. Big-endian stores 01 23. The program sets ByteOrder.LITTLE_ENDIAN on its buffer because GGUF files use that order.
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
public class MagicVersion {
public static void main(String[] args) {
ByteBuffer out = ByteBuffer.allocate(24).order(ByteOrder.LITTLE_ENDIAN);
out.put("GGUF".getBytes(StandardCharsets.US_ASCII)).putInt(3).putLong(291).putLong(25);
StringBuilder hex = new StringBuilder();
for (byte b : out.array()) hex.append(String.format("%02x ", b));
System.out.println(hex.toString().trim());
ByteBuffer in = ByteBuffer.wrap(out.array()).order(ByteOrder.LITTLE_ENDIAN);
int magic = in.getInt();
System.out.printf("magic 0x%08x (%s), version %d, tensors %d, kv pairs %d%n",
magic, magic == 0x46554747 ? "GGUF" : "not GGUF", in.getInt(), in.getLong(), in.getLong());
int swapped = Integer.reverseBytes(3); // version 3 written big-endian
System.out.printf("big-endian version read as little-endian: %d (0x%08x), low 16 bits zero: %b%n",
swapped, swapped, (swapped & 0xFFFF) == 0);
}
}47 47 55 46 03 00 00 00 23 01 00 00 00 00 00 00 19 00 00 00 00 00 00 00
magic 0x46554747 (GGUF), version 3, tensors 291, kv pairs 25
big-endian version read as little-endian: 50331648 (0x03000000), low 16 bits zero: trueThe first output line is the 24 bytes of the header. They split into four fields:
bytes 0-3 47 47 55 46 "GGUF" in ASCII
bytes 4-7 03 00 00 00 version 3
bytes 8-15 23 01 00 00 00 00 00 00 tensor count: 0x0123 = 291
bytes 16-23 19 00 00 00 00 00 00 00 key-value count: 0x19 = 25The second line is the same bytes read back into numbers. The third line is explained after the next paragraph.
All numbers in a GGUF file are little-endian (unless the file was written for a big-endian machine, Example 16). The first four bytes are the ASCII letters G G U F (47 47 55 46); read as a little-endian uint32 they are 0x46554747, the value of GGUF_MAGIC in gguf-py. Version 3 is 03 00 00 00. The tensor count 291 is 23 01 00 ... (0x123) and the key-value count 25 is 19 00 ....
The third output line shows the trick both readers use to detect a file written in the other byte order: version 3 written big-endian reads as 0x03000000 (3 times 2^24, which is 50,331,648) on a little-endian machine. Real versions are small numbers, so a value whose low 16 bits are all zero signals swapped bytes ((ctx->version & 0x0000FFFF) == 0x00000000 in gguf.cpp, temp_version[0] & 65535 == 0 in GGUFReader). llama.cpp then stops with a message asking whether the host and model endianness match; gguf-py switches its byte order and continues.
Example 3: GGUF strings
Strings appear everywhere in GGUF: metadata keys, string values, the tokens of the vocabulary, and tensor names. A file format has to tell the reader where a string ends. The C language marks the end with a zero byte. GGUF does it the other way around: it writes the length first and then the bytes. This is called a length-prefixed string. The reader learns the size before it reads any text, so it can allocate exactly the right amount of memory and can skip a string without scanning it.
The characters are stored as UTF-8, an encoding that turns text into bytes. English letters take one byte each, and other characters take two or more. The program writes three strings, reads them back, and then reads one that was corrupted on purpose. In the output, look for the 8-byte length at the start of each line.
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
public class GgufStrings {
static byte[] write(String s) {
byte[] b = s.getBytes(StandardCharsets.UTF_8);
return ByteBuffer.allocate(8 + b.length).order(ByteOrder.LITTLE_ENDIAN).putLong(b.length).put(b).array();
}
static String read(ByteBuffer in) {
long n = in.getLong();
if (n > in.remaining()) throw new IllegalArgumentException("string length " + n + " exceeds remaining " + in.remaining());
byte[] b = new byte[(int) n];
in.get(b);
return new String(b, StandardCharsets.UTF_8);
}
public static void main(String[] args) {
for (String s : new String[] {"llama", "héllo", ""}) {
byte[] bytes = write(s);
StringBuilder hex = new StringBuilder();
for (byte b : bytes) hex.append(String.format("%02x ", b));
System.out.printf("%-7s %2d bytes: %s -> '%s'%n", "'" + s + "'", bytes.length, hex.toString().trim(),
read(ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN)));
}
try {
read(ByteBuffer.wrap(new byte[] {(byte) 0xff, (byte) 0xff, 0, 0, 0, 0, 0, 0, 'a'}).order(ByteOrder.LITTLE_ENDIAN));
} catch (IllegalArgumentException e) {
System.out.println(e.getMessage());
}
}
}'llama' 13 bytes: 05 00 00 00 00 00 00 00 6c 6c 61 6d 61 -> 'llama'
'héllo' 14 bytes: 06 00 00 00 00 00 00 00 68 c3 a9 6c 6c 6f -> 'héllo'
'' 8 bytes: 00 00 00 00 00 00 00 00 -> ''
string length 65535 exceeds remaining 1Line by line: llama has 5 bytes of text, so its length field holds 5 (05 00 00 00 00 00 00 00, little-endian) and the whole string takes 8 + 5 = 13 bytes. The empty string has length 0 and takes 8 bytes. In the last line the first 8 bytes are ff ff 00 00 00 00 00 00, which is the number 65,535, but only one byte (a) follows them.
gguf.h defines a string as "the string length (uint64_t) followed by the C string without the null terminator". The length counts bytes, not characters: héllo has 5 characters and 6 bytes, because é is two bytes in UTF-8 (c3 a9). An empty string is 8 zero bytes.
The length comes from the file, and a damaged or hostile file can claim any number. A reader that trusts it would try to allocate that much memory. The reader must therefore check the length before it allocates. The last line shows a string that claims 65,535 bytes when 1 remains. gguf.cpp checks every count and length against the bytes left and against SIZE_MAX before it allocates; a Java reader gets part of that protection from ByteBuffer, which throws BufferUnderflowException instead of reading past the end, but an explicit check gives a clearer error and avoids allocating a huge array first.
Example 4: the 13 metadata value types
Metadata values are not all text. A context length is a whole number, a normalization constant is a decimal number, and an architecture name is a string. So each value in the file is preceded by a 4-byte type id that tells the reader how many bytes follow and how to interpret them. GGUF defines 13 types. Twelve of them hold a single value. The thirteenth, the array, holds a list and comes in the next example.
Some types come in signed and unsigned pairs. A signed integer can be negative. An unsigned integer has no sign, starts at zero, and reaches twice as high. The same byte can mean different numbers: ff is 255 as UINT8 and -1 as INT8. Java has no unsigned integer types, so the reader code widens unsigned values into a larger type.
The program writes one value of each of the 12 single-value types, each prefixed by its type id, and decodes them. In the output, check that every decoded value equals the value that was written, and look at how many bytes each type takes.
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
public class ValueTypes {
static final String[] NAMES = {"UINT8", "INT8", "UINT16", "INT16", "UINT32", "INT32", "FLOAT32", "BOOL", "STRING", "ARRAY", "UINT64", "INT64", "FLOAT64"};
static void writeValue(ByteBuffer b, int type, Object v) {
switch (type) {
case 0, 1 -> b.put(((Number) v).byteValue());
case 2, 3 -> b.putShort(((Number) v).shortValue());
case 4, 5 -> b.putInt(((Number) v).intValue());
case 6 -> b.putFloat((Float) v);
case 7 -> b.put((byte) ((Boolean) v ? 1 : 0));
case 8 -> { byte[] s = ((String) v).getBytes(StandardCharsets.UTF_8); b.putLong(s.length).put(s); }
case 10, 11 -> b.putLong(((Number) v).longValue());
case 12 -> b.putDouble((Double) v);
default -> throw new IllegalArgumentException("type " + type);
}
}
static Object readValue(ByteBuffer b, int type) {
return switch (type) {
case 0 -> Byte.toUnsignedInt(b.get());
case 1 -> b.get();
case 2 -> Short.toUnsignedInt(b.getShort());
case 3 -> b.getShort();
case 4 -> Integer.toUnsignedLong(b.getInt());
case 5 -> b.getInt();
case 6 -> b.getFloat();
case 7 -> b.get() != 0;
case 8 -> { byte[] s = new byte[(int) b.getLong()]; b.get(s); yield new String(s, StandardCharsets.UTF_8); }
case 10 -> Long.toUnsignedString(b.getLong());
case 11 -> b.getLong();
case 12 -> b.getDouble();
default -> throw new IllegalArgumentException("type " + type);
};
}
public static void main(String[] args) {
Object[][] values = {{0, 200}, {1, -5}, {2, 60000}, {3, -1234}, {4, 4294967295L}, {5, -7}, {6, 1e-5f}, {7, true},
{8, "llama"}, {10, -1L}, {11, -9000000000L}, {12, 0.1}};
ByteBuffer b = ByteBuffer.allocate(256).order(ByteOrder.LITTLE_ENDIAN);
for (Object[] v : values) { b.putInt((Integer) v[0]); writeValue(b, (Integer) v[0], v[1]); }
b.flip();
System.out.println("encoded " + b.remaining() + " bytes");
while (b.hasRemaining()) {
int type = b.getInt();
int before = b.position();
Object v = readValue(b, type);
System.out.printf("%2d %-8s %d bytes %s%n", type, NAMES[type], b.position() - before, v);
}
System.out.println("type 9 (ARRAY) holds an element type, a count, and the elements: Example 5");
}
}encoded 104 bytes
0 UINT8 1 bytes 200
1 INT8 1 bytes -5
2 UINT16 2 bytes 60000
3 INT16 2 bytes -1234
4 UINT32 4 bytes 4294967295
5 INT32 4 bytes -7
6 FLOAT32 4 bytes 1.0E-5
7 BOOL 1 bytes true
8 STRING 13 bytes llama
10 UINT64 8 bytes 18446744073709551615
11 INT64 8 bytes -9000000000
12 FLOAT64 8 bytes 0.1
type 9 (ARRAY) holds an element type, a count, and the elements: Example 5The first line says 104 bytes were written. They add up like this:
12 values x 4-byte type id = 48 bytes
value bytes: 1 + 1 + 2 + 2 + 4 + 4 + 4 + 1 + 13 + 8 + 8 + 8 = 56 bytes
48 + 56 = 104 bytesThe STRING value takes 13 bytes because llama is 8 bytes of length plus 5 bytes of text. The UINT64 value that was written as -1 prints as 18,446,744,073,709,551,615, because all 64 bits are 1, and that is the largest unsigned 64-bit number.
The value types and their ids are fixed in enum gguf_type (gguf.h) and GGUFValueType (gguf-py):
| Id | Type | Bytes | Java reading |
|---|---|---|---|
| 0, 1 | UINT8, INT8 |
1 | Byte.toUnsignedInt(get()), get() |
| 2, 3 | UINT16, INT16 |
2 | Short.toUnsignedInt(getShort()), getShort() |
| 4, 5 | UINT32, INT32 |
4 | Integer.toUnsignedLong(getInt()), getInt() |
| 6 | FLOAT32 |
4 | getFloat() |
| 7 | BOOL |
1 | get() != 0 |
| 8 | STRING |
8 + length | Example 3 |
| 9 | ARRAY |
varies | Example 5 |
| 10, 11 | UINT64, INT64 |
8 | getLong() (unsigned needs Long.toUnsignedString) |
| 12 | FLOAT64 |
8 | getDouble() |
Java has no unsigned integer types, so unsigned values must be widened: the UINT8 200 would read as −56 with a plain get(), and the UINT32 4294967295 as −1. The output shows each value round-tripping correctly with the widening in place. gguf.h notes that "All bool values are stored as int8_t" and "All enums are stored as int32_t", which is why the value type and the tensor type are 4-byte fields. Most llama.cpp keys use UINT32 for counts and sizes and FLOAT32 for constants such as layer_norm_rms_epsilon; the FLOAT32 1e-5 prints as 9.999999747378752e-06 in gguf-dump because it is printed as a double.
Example 5: arrays, including a token array
A model's vocabulary is a list of tens or hundreds of thousands of tokens. A token is a piece of text, such as a word or part of a word, that the model reads and writes as one unit (Tokens, Vocabularies, and Special Tokens). The vocabulary is the full list of tokens, and the id of a token is its position in that list. GGUF therefore needs a way to store a list as one metadata value. That is the array type.
An array stores its element type once, then the number of elements, then the elements one after another. The program writes an array of 5 token strings and a matching INT32 array that holds one type number for each token, then reads both. The output shows the decoded lists and the total number of bytes.
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
public class GgufArrays {
static final int INT32 = 5, FLOAT32 = 6, STRING = 8, ARRAY = 9;
public static void main(String[] args) {
ByteBuffer b = ByteBuffer.allocate(512).order(ByteOrder.LITTLE_ENDIAN);
String[] tokens = {"<unk>", "<s>", "</s>", "▁the", "▁cat"};
int[] types = {2, 3, 3, 1, 1}; // UNKNOWN, CONTROL, CONTROL, NORMAL, NORMAL
b.putInt(ARRAY).putInt(STRING).putLong(tokens.length); // value type, element type, count
for (String t : tokens) { byte[] s = t.getBytes(StandardCharsets.UTF_8); b.putLong(s.length).put(s); }
b.putInt(ARRAY).putInt(INT32).putLong(types.length);
for (int t : types) b.putInt(t);
b.flip();
int total = b.remaining();
for (int k = 0; k < 2; k++) {
int valueType = b.getInt(), elemType = b.getInt();
long n = b.getLong();
List<Object> items = new ArrayList<>();
for (long i = 0; i < n; i++) {
if (elemType == STRING) { byte[] s = new byte[(int) b.getLong()]; b.get(s); items.add(new String(s, StandardCharsets.UTF_8)); }
else items.add(b.getInt());
}
System.out.println("type " + valueType + " of " + (elemType == STRING ? "STRING" : "INT32") + ", " + n + " items: " + items);
}
System.out.println(total + " bytes for both arrays");
}
}type 9 of STRING, 5 items: [<unk>, <s>, </s>, ▁the, ▁cat]
type 9 of INT32, 5 items: [2, 3, 3, 1, 1]
116 bytes for both arraysToken id 3 is the string at position 3 of the first array, ▁the, and its type is the number at position 3 of the second array, 1 (a normal token). The 116 bytes break down like this:
array 1: 4 (value type) + 4 (element type) + 8 (count) = 16 bytes of header
strings: 13 + 11 + 12 + 14 + 14 = 64 bytes (8-byte length + text; "▁" takes 3 bytes in UTF-8)
array 2: 16 bytes of header + 5 x 4 = 36 bytes
16 + 64 + 36 = 116 bytesAn array value is the type id 9, then the element type (4 bytes), the element count (8 bytes), and the elements one after another. Arrays of strings contain full GGUF strings, each with its own length, so a reader cannot jump to element i without reading the elements before it.
The two arrays are the core of a GGUF tokenizer: tokenizer.ggml.tokens holds the token texts in id order, and tokenizer.ggml.token_type holds one type per token. The type values come from TokenType in gguf-py: NORMAL = 1, UNKNOWN = 2, CONTROL = 3, USER_DEFINED = 4, UNUSED = 5, BYTE = 6. Here <unk> is unknown, <s> and </s> are control tokens, and the two words are normal. ▁ is the SentencePiece marker for a leading space (Tokens, Vocabularies, and Special Tokens). A real vocabulary of 150,000 tokens takes a few megabytes of metadata, which is why the data section of the tiny Q4_K_M model from lesson 24 starts at byte 5,931,072 (Example 17).
Example 6: writing a small GGUF in Java with metadata and two tensors
The strongest test of understanding a format is to write a file in it and see whether the real tools accept the file. This program writes a complete GGUF file with five metadata pairs and two tensors. The first tensor is a vector of 64 ones stored as F32 (32-bit floating-point numbers, 4 bytes each). The second is a matrix of 64 by 3 values stored as Q8_0.
Q8_0 is a quantization format. It cuts the values into blocks of 32 and stores each block as one scale (a number that sets the size of the steps, saved as a 16-bit float) plus 32 small signed integers of 1 byte each. A block therefore takes 2 + 32 = 34 bytes. The quantizer in the program is the one from Building Q8_0 and Q4_0 Like ggml, including its roundf helper. In the output, look at the file size and at the size of the Q8_0 tensor.
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
public class WriteGguf {
static final int UINT32 = 4, FLOAT32 = 6, STRING = 8, ARRAY = 9;
static final int GGML_F32 = 0, GGML_Q8_0 = 8;
// Little-endian output helper.
static final class Out {
final ByteArrayOutputStream bytes = new ByteArrayOutputStream();
Out u32(long v) { return raw(ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt((int) v).array()); }
Out u64(long v) { return raw(ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(v).array()); }
Out f32(float v) { return raw(ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putFloat(v).array()); }
Out str(String s) { byte[] b = s.getBytes(StandardCharsets.UTF_8); u64(b.length); return raw(b); }
Out raw(byte[] b) { bytes.write(b, 0, b.length); return this; }
int size() { return bytes.size(); }
}
record Tensor(String name, long[] dims, int type, byte[] data) {}
// C roundf: halves away from zero (lesson 18).
static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
static byte[] q8_0(float[] x) { // quantize_row_q8_0_ref, 34 bytes per 32 values
ByteBuffer out = ByteBuffer.allocate(x.length / 32 * 34).order(ByteOrder.LITTLE_ENDIAN);
for (int b = 0; b < x.length; b += 32) {
float amax = 0;
for (int j = 0; j < 32; j++) amax = Math.max(amax, Math.abs(x[b + j]));
float d = amax / 127, id = d != 0 ? 1 / d : 0;
out.putShort(Float.floatToFloat16(d));
for (int j = 0; j < 32; j++) out.put((byte) roundf(x[b + j] * id));
}
return out.array();
}
static byte[] gguf(Out kv, int kvCount, List<Tensor> tensors, int alignment) {
Out f = new Out();
f.raw("GGUF".getBytes(StandardCharsets.US_ASCII)).u32(3).u64(tensors.size()).u64(kvCount).raw(kv.bytes.toByteArray());
long offset = 0;
for (Tensor t : tensors) { // tensor infos
f.str(t.name()).u32(t.dims().length);
for (long d : t.dims()) f.u64(d);
f.u32(t.type()).u64(offset);
offset += (t.data().length + alignment - 1) / alignment * alignment;
}
while (f.size() % alignment != 0) f.raw(new byte[1]); // pad to the start of the data section
for (Tensor t : tensors) {
f.raw(t.data());
while (f.size() % alignment != 0) f.raw(new byte[1]); // pad each tensor
}
return f.bytes.toByteArray();
}
public static void main(String[] args) throws Exception {
Out kv = new Out();
kv.str("general.architecture").u32(STRING).str("llama");
kv.str("general.name").u32(STRING).str("java-written");
kv.str("llama.embedding_length").u32(UINT32).u32(64);
kv.str("llama.attention.layer_norm_rms_epsilon").u32(FLOAT32).f32(1e-5f);
kv.str("tokenizer.ggml.tokens").u32(ARRAY).u32(STRING).u64(3).str("<s>").str("hello").str("world");
int kvCount = 5;
float[] norm = new float[64];
java.util.Arrays.fill(norm, 1f);
ByteBuffer normBytes = ByteBuffer.allocate(64 * 4).order(ByteOrder.LITTLE_ENDIAN);
for (float v : norm) normBytes.putFloat(v);
float[] w = new float[64 * 3];
for (int i = 0; i < w.length; i++) w[i] = (float) Math.sin(i * 0.1) * 0.05f;
List<Tensor> tensors = new ArrayList<>(List.of(
new Tensor("output_norm.weight", new long[] {64}, GGML_F32, normBytes.array()),
new Tensor("token_embd.weight", new long[] {64, 3}, GGML_Q8_0, q8_0(w))));
byte[] file = gguf(kv, kvCount, tensors, 32);
Path path = args.length > 0 ? Path.of(args[0]) : Files.createTempFile("tiny", ".gguf");
Files.write(path, file);
System.out.println("wrote " + file.length + " bytes");
System.out.println("first 4 bytes: " + new String(file, 0, 4, StandardCharsets.US_ASCII) + ", Q8_0 tensor bytes: " + tensors.get(1).data().length);
}
}wrote 896 bytes
first 4 bytes: GGUF, Q8_0 tensor bytes: 204The output says the file has 896 bytes and the Q8_0 tensor takes 204 bytes. The code that produced them has three parts:
Outcollects little-endian bytes and writes GGUF strings.q8_0quantizes each 32-value block to an FP16 scale and 32 signed bytes, 34 bytes per block, exactly asquantize_row_q8_0_refdoes. The matrix has 3 rows of 64 values, 2 blocks per row, so 6 blocks of 34 bytes: 204 bytes.ggufwrites the header, the metadata, the tensor infos with offsets padded to the alignment, padding up to the data section, and each tensor followed by padding.
Run the program with a file name so that it keeps the file, then inspect the file with gguf-dump. The second gguf-dump command adds --data-offset, which prints the byte where the data section starts (the last line of the output, 416):
java WriteGguf.java tiny.gguf
gguf-dump tiny.gguf
gguf-dump --data-offset tiny.ggufOutput of the two gguf-dump commands (gguf-py 0.19.0):
INFO:gguf-dump:* Loading: tiny.gguf
* File is LITTLE endian, script is running on a LITTLE endian host.
* Dumping 8 key/value pair(s)
1: UINT32 | 1 | GGUF.version = 3
2: UINT64 | 1 | GGUF.tensor_count = 2
3: UINT64 | 1 | GGUF.kv_count = 5
4: STRING | 1 | general.architecture = 'llama'
5: STRING | 1 | general.name = 'java-written'
6: UINT32 | 1 | llama.embedding_length = 64
7: FLOAT32 | 1 | llama.attention.layer_norm_rms_epsilon = 9.999999747378752e-06
8: [STRING] | 3 | tokenizer.ggml.tokens = ['<s>', 'hello', 'world']
* Dumping 2 tensor(s)
1: 64 | 64, 1, 1, 1 | F32 | output_norm.weight
2: 192 | 64, 3, 1, 1 | Q8_0 | token_embd.weight
416How to read the gguf-dump output: the first three pairs (GGUF.version, GGUF.tensor_count, GGUF.kv_count) are header fields that the tool shows like metadata. That is why it reports 8 pairs: 3 header fields plus the 5 pairs the program wrote. Each tensor line shows the tensor number, the number of elements (64, and 192 = 64 x 3), the four dimensions (unused dimensions show as 1), the type, and the name.
The data section starts at byte 416. The F32 tensor occupies bytes 416 to 671 (256 bytes, already a multiple of 32), and the Q8_0 tensor starts at 672 and takes 204 bytes, padded to 224, so the file ends at 896. Loading the file with gguf-py's GGUFReader and dequantizing the Q8_0 tensor with gguf.quants.dequantize gave the original values with a maximum error of 0.0002, and llama-gguf-split --dry-run (which opens the file with llama.cpp's gguf_init_from_file) accepted it as well.
Example 7: tensor infos: name, dimensions, type, and offset
After the metadata, the file lists one tensor info for each tensor. A tensor info is a small record that tells a reader the tensor's name, its shape, its number type, and where its numbers are stored. A reader parses all the infos first. Later it can jump straight to any tensor by its offset without reading the others.
The shape of a tensor is the list of its sizes along each axis, also called its dimensions. A matrix with 64 values per row and 1,000 rows has the shape [64, 1000] here. The program encodes three infos, decodes them, and prints the byte position where each one starts and how many bytes it takes.
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
public class TensorInfos {
public static void main(String[] args) {
Object[][] infos = { // name, dims (innermost first), ggml type, offset
{"token_embd.weight", new long[] {64, 1000}, 8, 0L},
{"blk.0.attn_q.weight", new long[] {64, 64}, 1, 68000L},
{"output_norm.weight", new long[] {64}, 0, 76192L}};
ByteBuffer b = ByteBuffer.allocate(512).order(ByteOrder.LITTLE_ENDIAN);
for (Object[] t : infos) {
byte[] name = ((String) t[0]).getBytes(StandardCharsets.UTF_8);
long[] dims = (long[]) t[1];
b.putLong(name.length).put(name).putInt(dims.length);
for (long d : dims) b.putLong(d);
b.putInt((Integer) t[2]).putLong((Long) t[3]);
}
b.flip();
String[] typeNames = {"F32", "F16", "", "", "", "", "", "", "Q8_0"};
while (b.hasRemaining()) {
int start = b.position();
byte[] name = new byte[(int) b.getLong()];
b.get(name);
int nDims = b.getInt();
long[] dims = new long[nDims];
for (int i = 0; i < nDims; i++) dims[i] = b.getLong();
int type = b.getInt();
long offset = b.getLong();
System.out.printf("@%3d %-20s n_dims %d dims %-11s type %d (%s) offset %d, %d bytes of info%n", start,
new String(name, StandardCharsets.UTF_8), nDims, java.util.Arrays.toString(dims), type, typeNames[type], offset, b.position() - start);
}
}
}@ 0 token_embd.weight n_dims 2 dims [64, 1000] type 8 (Q8_0) offset 0, 57 bytes of info
@ 57 blk.0.attn_q.weight n_dims 2 dims [64, 64] type 1 (F16) offset 68000, 59 bytes of info
@116 output_norm.weight n_dims 1 dims [64] type 0 (F32) offset 76192, 50 bytes of infoThe first info takes 57 bytes. Its fields add up like this:
name length 8
name text 17 ("token_embd.weight")
n_dims 4
dimensions 16 (2 x 8 bytes)
ggml type 4
offset 8
total 57The other two infos have different names and dimension counts, so they take 59 and 50 bytes.
Each info has five fields. The dimension count is a uint32, at most 4 (GGML_MAX_DIMS; gguf.cpp rejects more). Each dimension is a 64-bit integer, listed innermost first: [64, 1000] means rows of 64 values, 1,000 of them. Example 13 explains the order.
The type is a ggml type id (the table in k-quants, i-quants, and the Importance Matrix): 0 is F32, 1 is F16, 8 is Q8_0. The offset is relative to the data section and must be a multiple of the alignment. gguf.cpp at the pinned commit is stricter: it requires each tensor's offset to equal the sum of the padded sizes of the tensors before it ("tensor '%s' has offset %" PRIu64 ", expected %zu"), so tensors must appear in the data section in the same order as their infos, without gaps.
The offsets here follow that rule: the Q8_0 matrix is 2 * 34 * 1000 = 68000 bytes, which is a multiple of 32, so the F16 matrix starts at 68000; it is 64 * 64 * 2 = 8192 bytes, so the norm starts at 76192. Every info is a different size (57, 59, 50 bytes) because the names differ in length, so a reader must parse the infos one after another.
Example 8: alignment and padding
Computer hardware reads memory fastest from addresses that are multiples of some size, for example 32. Alignment means starting each tensor at such a multiple. The price is a few unused zero bytes between tensors, called padding. This example shows how much padding appears and how the tensor positions change when the alignment is 32 and when it is 64. It also checks which values of the general.alignment key a reader accepts.
public class Alignment {
static long pad(long x, long a) { return (x + a - 1) / a * a; } // GGML_PAD
static boolean validAlignment(long a) { return a != 0 && (a & (a - 1)) == 0; }
public static void main(String[] args) {
long infosEnd = 470; // file offset where the tensor infos end
long[] sizes = {256, 204, 1000}; // bytes of three tensors
for (long alignment : new long[] {32, 64}) {
long dataStart = pad(infosEnd, alignment), offset = 0;
StringBuilder s = new StringBuilder();
for (long size : sizes) {
s.append(offset).append(' ');
offset += pad(size, alignment);
}
System.out.printf("alignment %2d: data starts at %d (%d padding bytes), tensor offsets %s, data section %d bytes%n",
alignment, dataStart, dataStart - infosEnd, s.toString().trim(), offset);
}
for (long a : new long[] {32, 48, 0, 4096})
System.out.println("general.alignment = " + a + " -> " + (validAlignment(a) ? "ok" : "rejected: not a power of 2"));
}
}alignment 32: data starts at 480 (10 padding bytes), tensor offsets 0 256 480, data section 1504 bytes
alignment 64: data starts at 512 (42 padding bytes), tensor offsets 0 256 512, data section 1536 bytes
general.alignment = 32 -> ok
general.alignment = 48 -> rejected: not a power of 2
general.alignment = 0 -> rejected: not a power of 2
general.alignment = 4096 -> okThe first output line works like this. To round 470 up to a multiple of 32, the program adds 31, divides with whole-number division, and multiplies back. The power-of-2 check uses the fact that a power of 2 has a single 1 bit, so a power of 2 AND the number one below it gives 0:
round 470 up to a multiple of 32: (470 + 31) / 32 = 15 (whole division), 15 x 32 = 480
padding: 480 - 470 = 10 bytes
is 32 a power of 2? 32 = 100000, 31 = 011111, 32 AND 31 = 000000, which is 0 -> yes
is 48 a power of 2? 48 = 110000, 47 = 101111, 48 AND 47 = 100000, which is not 0 -> nogeneral.alignment is an optional metadata key of type UINT32. Without it the alignment is 32 (GGUF_DEFAULT_ALIGNMENT). Both gguf.cpp and GGUFReader reject a key of another type and an alignment that is 0 or not a power of 2 ("alignment %zu is not a power of 2"), which the last four lines reproduce.
Alignment applies twice. The data section starts at the next multiple of the alignment after the tensor infos (480 at 32, 512 at 64, from byte 470). Each tensor's size is rounded up to a multiple of the alignment before the next tensor starts: the 204-byte tensor occupies 224 bytes at alignment 32 and 256 at 64. GGML_PAD(x, n) in ggml computes this rounding, which the program writes as (x + a - 1) / a * a.
Alignment exists for the reader's sake. llama.cpp maps the file into memory, and an aligned tensor can be used in place by SIMD code and handed to GPU copy routines that require aligned addresses. A larger alignment wastes a little space (here 32 bytes more data section) and helps hardware that wants larger alignment.
Example 9: a GgufReader that prints metadata
The earlier examples handled one part of the format at a time. This program puts them together into a complete reader. It parses the header, every key-value pair (including arrays), and every tensor info, computes where the data section starts, and prints the metadata in the style of gguf-dump.
Without arguments it reads a small file built in memory, so it runs anywhere. With a path it maps the file: FileChannel.map asks the operating system to make the file's bytes available as a ByteBuffer without copying the whole file into memory. The reader touches only the first part of the file, so even a multi-gigabyte model opens quickly.
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public class GgufMetadata {
record TensorInfo(String name, long[] dims, int type, long offset) {}
static final class GgufFile {
int version, alignment = 32; long dataOffset;
final Map<String, Object> kv = new LinkedHashMap<>(); final Map<String, String> kvType = new LinkedHashMap<>();
final List<TensorInfo> tensors = new ArrayList<>();
}
static final String[] TYPES = {"UINT8", "INT8", "UINT16", "INT16", "UINT32", "INT32", "FLOAT32", "BOOL", "STRING", "ARRAY", "UINT64", "INT64", "FLOAT64"};
static String str(ByteBuffer b) { byte[] s = new byte[Math.toIntExact(b.getLong())]; b.get(s); return new String(s, StandardCharsets.UTF_8); }
static Object value(ByteBuffer b, int type) {
return switch (type) {
case 0 -> Byte.toUnsignedInt(b.get()); case 1 -> b.get();
case 2 -> Short.toUnsignedInt(b.getShort()); case 3 -> b.getShort();
case 4 -> Integer.toUnsignedLong(b.getInt()); case 5 -> b.getInt();
case 6 -> b.getFloat(); case 7 -> b.get() != 0; case 8 -> str(b);
case 9 -> { int et = b.getInt(); long n = b.getLong(); List<Object> l = new ArrayList<>();
for (long i = 0; i < n; i++) l.add(value(b, et)); yield l; }
case 10, 11 -> b.getLong(); case 12 -> b.getDouble();
default -> throw new IllegalArgumentException("bad value type " + type);
};
}
static GgufFile read(ByteBuffer b) {
b.order(ByteOrder.LITTLE_ENDIAN);
if (b.getInt() != 0x46554747) throw new IllegalArgumentException("not a GGUF file");
GgufFile f = new GgufFile();
f.version = b.getInt();
if (f.version != 2 && f.version != 3) throw new IllegalArgumentException("unsupported version " + f.version);
long nTensors = b.getLong(), nKv = b.getLong();
for (long i = 0; i < nKv; i++) {
String key = str(b); int type = b.getInt();
f.kvType.put(key, type == 9 ? "[" + TYPES[b.getInt(b.position())] + "]" : TYPES[type]); // arrays: element type
f.kv.put(key, value(b, type));
}
for (long i = 0; i < nTensors; i++) {
String name = str(b); long[] dims = new long[b.getInt()];
for (int j = 0; j < dims.length; j++) dims[j] = b.getLong();
f.tensors.add(new TensorInfo(name, dims, b.getInt(), b.getLong()));
}
if (f.kv.get("general.alignment") instanceof Long a) f.alignment = a.intValue();
f.dataOffset = (b.position() + f.alignment - 1L) / f.alignment * f.alignment;
return f;
}
static String show(Object v) {
if (v instanceof List<?> l) return l.subList(0, Math.min(6, l.size())) + (l.size() > 6 ? " ... (" + l.size() + " items)" : "");
String s = String.valueOf(v);
return v instanceof String ? "'" + (s.length() > 40 ? s.substring(0, 40) + "..." : s) + "'" : s;
}
static byte[] demo() { // a tiny file: 3 KV pairs, no tensors
ByteBuffer b = ByteBuffer.allocate(256).order(ByteOrder.LITTLE_ENDIAN);
b.put("GGUF".getBytes(StandardCharsets.US_ASCII)).putInt(3).putLong(0).putLong(3);
for (String[] s : new String[][] {{"general.architecture", "llama"}}) {
b.putLong(s[0].length()).put(s[0].getBytes()).putInt(8).putLong(s[1].length()).put(s[1].getBytes());
}
b.putLong(22).put("llama.embedding_length".getBytes()).putInt(4).putInt(64);
b.putLong(21).put("tokenizer.ggml.tokens".getBytes()).putInt(9).putInt(8).putLong(2);
b.putLong(3).put("<s>".getBytes()).putLong(5).put("hello".getBytes());
return java.util.Arrays.copyOf(b.array(), b.position());
}
public static void main(String[] args) throws Exception {
ByteBuffer buf;
if (args.length > 0) try (FileChannel ch = FileChannel.open(Path.of(args[0]))) {
buf = ch.map(FileChannel.MapMode.READ_ONLY, 0, Math.min(ch.size(), Integer.MAX_VALUE));
} else buf = ByteBuffer.wrap(demo());
GgufFile f = read(buf);
System.out.println("GGUF v" + f.version + ", " + f.kv.size() + " KV pairs, " + f.tensors.size() + " tensors, data at " + f.dataOffset);
f.kv.forEach((k, v) -> System.out.printf("%-9s %-40s = %s%n", f.kvType.get(k), k, show(v)));
}
}GGUF v3, 3 KV pairs, 0 tensors, data at 192
STRING general.architecture = 'llama'
UINT32 llama.embedding_length = 64
[STRING] tokenizer.ggml.tokens = [<s>, hello]The first output line reports data at 192. The reader gets that number by adding up what it consumed:
header 24
general.architecture = "llama" 45 (8 + 20 + 4 + 8 + 5)
llama.embedding_length = 64 38 (8 + 22 + 4 + 4)
tokenizer.ggml.tokens = ["<s>", "hello"] 69 (8 + 21 + 4 + 4 + 8, then 11 + 13 for the two strings)
total 176 -> next multiple of 32 is 192The other three lines are the key-value pairs: the type name, the key, and the value. For arrays the program prints the first six elements and the total count, so a vocabulary of 128,256 tokens does not flood the screen.
The reader is the format in code:
strreads a GGUF string;Math.toIntExactthrows if a length does not fit in anintinstead of silently truncating it.valuehandles the 13 types, recursing for arrays. Arrays becomeList<Object>.readchecks the magic, accepts versions 2 and 3 (the same set as gguf-py'sREADER_SUPPORTED_VERSIONS), reads the key-value pairs and tensor infos in order, appliesgeneral.alignmentif present, and computes the data offset.- For arrays, the element type is read without consuming it (
b.getInt(b.position())) so the printout can show[STRING]likegguf-dump.
FileChannel.map maps up to 2 GiB of the file. That is enough for the metadata of any model, which is at most tens of megabytes even with a large vocabulary; the tensor data after it is never touched. The demo file has no tensors, so its "data at" value is only where a data section would start.
Example 10: the keys an engine needs
An engine cannot guess the shape of a model. It builds the model from numbers it reads out of the metadata: how many layers there are, how wide each layer is, how many attention heads there are, and so on. This example shows which keys an engine reads, which keys are required, and which have a default. It takes the metadata map that GgufMetadata returns for a Llama-architecture file and turns it into a configuration record. A missing required key stops the program with an error.
A few terms from earlier lessons appear in the keys. d_model is the number of values in the vector that represents one token inside the model. A layer (called a block in GGUF) is one repeated stage of the network. Attention heads split the attention computation into parallel parts, and KV heads are the heads that have their own keys and values in the KV cache (key-value cache, the memory that stores the keys and values of earlier tokens so the model does not recompute them). Memory Math uses these numbers to compute memory use.
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public class EngineKeys {
record ModelConfig(String arch, long ctx, long dModel, long layers, long ffn, long heads, long kvHeads, long headDim,
double ropeBase, double rmsEps, long vocab, long bos, long eos, boolean hasTemplate, long fileType) {}
static long num(Map<String, Object> kv, String key, long fallback) { return kv.get(key) instanceof Number n ? n.longValue() : fallback; }
static ModelConfig config(Map<String, Object> kv) {
String a = (String) kv.get("general.architecture"); // every other model key is prefixed with it
for (String k : List.of(".context_length", ".embedding_length", ".block_count", ".attention.head_count"))
if (!kv.containsKey(a + k)) throw new IllegalArgumentException("missing " + a + k);
long d = num(kv, a + ".embedding_length", 0), heads = num(kv, a + ".attention.head_count", 0);
return new ModelConfig(a, num(kv, a + ".context_length", 0), d, num(kv, a + ".block_count", 0),
num(kv, a + ".feed_forward_length", 0), heads, num(kv, a + ".attention.head_count_kv", heads),
num(kv, a + ".rope.dimension_count", d / heads),
kv.get(a + ".rope.freq_base") instanceof Number n ? n.doubleValue() : 10000.0,
((Number) kv.getOrDefault(a + ".attention.layer_norm_rms_epsilon", 1e-5f)).doubleValue(),
((List<?>) kv.get("tokenizer.ggml.tokens")).size(),
num(kv, "tokenizer.ggml.bos_token_id", -1), num(kv, "tokenizer.ggml.eos_token_id", -1),
kv.containsKey("tokenizer.chat_template"), num(kv, "general.file_type", -1));
}
public static void main(String[] args) {
Map<String, Object> kv = new LinkedHashMap<>(); // as GgufMetadata (Example 9) returns it
kv.put("general.architecture", "llama");
kv.put("llama.block_count", 32L); kv.put("llama.context_length", 8192L); kv.put("llama.embedding_length", 4096L);
kv.put("llama.feed_forward_length", 14336L); kv.put("llama.attention.head_count", 32L); kv.put("llama.attention.head_count_kv", 8L);
kv.put("llama.rope.freq_base", 500000.0f); kv.put("llama.attention.layer_norm_rms_epsilon", 1e-5f);
kv.put("general.file_type", 15L);
kv.put("tokenizer.ggml.model", "gpt2"); kv.put("tokenizer.ggml.tokens", java.util.Collections.nCopies(128256, "t"));
kv.put("tokenizer.ggml.bos_token_id", 128000L); kv.put("tokenizer.ggml.eos_token_id", 128009L);
ModelConfig c = config(kv);
System.out.println(c);
System.out.println("query heads per KV head: " + c.heads() / c.kvHeads() + ", KV cache bytes per token at F16: "
+ 2 * c.layers() * c.kvHeads() * c.headDim() * 2);
kv.remove("llama.block_count");
try { config(kv); } catch (IllegalArgumentException e) { System.out.println(e.getMessage()); }
}
}ModelConfig[arch=llama, ctx=8192, dModel=4096, layers=32, ffn=14336, heads=32, kvHeads=8, headDim=128, ropeBase=500000.0, rmsEps=9.999999747378752E-6, vocab=128256, bos=128000, eos=128009, hasTemplate=false, fileType=15]
query heads per KV head: 4, KV cache bytes per token at F16: 131072
missing llama.block_countRead the output in three steps. First, the record prints every field. headDim=128 was not in the file: the code used the default d_model / heads = 4096 / 32 = 128. rmsEps=9.999999747378752E-6 is the FLOAT32 value 1e-5 widened to a double, which shows the rounding of a 32-bit float. hasTemplate=false because the file has no tokenizer.chat_template key.
Second, the KV cache size per token comes straight from the keys:
2 (keys and values) x 32 layers x 8 KV heads x 128 head size x 2 bytes (F16) = 131,072 bytes per token
query heads per KV head = 32 / 8 = 4Third, after llama.block_count was removed, config threw missing llama.block_count.
Model keys are prefixed with the architecture name: general.architecture = "llama" means the engine reads llama.context_length, llama.embedding_length, and so on ({arch}.context_length in gguf-py's Keys.LLM). The keys and what they set:
| Key | Use |
|---|---|
{arch}.context_length |
Context the model was trained for |
{arch}.embedding_length |
d_model |
{arch}.block_count |
Number of layers |
{arch}.feed_forward_length |
FFN width |
{arch}.attention.head_count, .head_count_kv |
Query heads and KV heads (default: equal to query heads) |
{arch}.rope.dimension_count |
Dimensions rotated by RoPE (default: d_model / heads) |
{arch}.rope.freq_base |
RoPE base frequency |
{arch}.attention.layer_norm_rms_epsilon |
RMSNorm epsilon |
tokenizer.ggml.model, .tokens, .token_type, .merges, .scores |
Tokenizer type and data |
tokenizer.ggml.bos_token_id, .eos_token_id |
Special token ids |
tokenizer.chat_template |
Jinja chat template |
general.file_type |
The quant mix (LlamaFileType: 15 is MOSTLY_Q4_K_M) |
general.alignment, general.quantization_version |
Format details |
The numbers derived from them are the ones Memory Math needs: 32 query heads over 8 KV heads gives 4 query heads per KV head, and the KV cache costs 2 * 32 * 8 * 128 * 2 = 131072 bytes per token at F16. The record also shows a file without a chat template, which the engine must handle with a default template or an error. Removing llama.block_count makes config throw. A reader that guessed a layer count instead would build a model that loads and then produces wrong output.
Example 11: tensor naming and the Hugging Face names they come from
Most models are published on Hugging Face in the safetensors format, with tensor names chosen by the model's authors. GGUF uses its own short names that are the same for every architecture. A converter therefore has to rename every tensor. For example, model.layers.17.mlp.gate_proj.weight becomes blk.17.ffn_gate.weight: model.layers.17 is layer 17 (counting from 0), and blk is GGUF's short form of block.
The program applies the mapping rules for Llama-family names from gguf-py's tensor_mapping.py. It uses regular expressions, which are patterns for matching text. In a pattern, (\d+) captures the layer number, and $1 pastes the captured number into the new name.
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class TensorNames {
static final Map<Pattern, String> RULES = new LinkedHashMap<>();
static { // llama-hf names from gguf-py tensor_mapping.py
RULES.put(Pattern.compile("model\\.embed_tokens\\.(\\w+)"), "token_embd.$1");
RULES.put(Pattern.compile("model\\.norm\\.(\\w+)"), "output_norm.$1");
RULES.put(Pattern.compile("lm_head\\.(\\w+)"), "output.$1");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.input_layernorm\\.(\\w+)"), "blk.$1.attn_norm.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.self_attn\\.q_proj\\.(\\w+)"), "blk.$1.attn_q.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.self_attn\\.k_proj\\.(\\w+)"), "blk.$1.attn_k.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.self_attn\\.v_proj\\.(\\w+)"), "blk.$1.attn_v.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.self_attn\\.o_proj\\.(\\w+)"), "blk.$1.attn_output.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.post_attention_layernorm\\.(\\w+)"), "blk.$1.ffn_norm.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.mlp\\.gate_proj\\.(\\w+)"), "blk.$1.ffn_gate.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.mlp\\.up_proj\\.(\\w+)"), "blk.$1.ffn_up.$2");
RULES.put(Pattern.compile("model\\.layers\\.(\\d+)\\.mlp\\.down_proj\\.(\\w+)"), "blk.$1.ffn_down.$2");
}
static String map(String hf) {
for (var r : RULES.entrySet()) {
Matcher m = r.getKey().matcher(hf);
if (m.matches()) return m.replaceAll(r.getValue());
}
throw new IllegalArgumentException("cannot map tensor " + hf);
}
public static void main(String[] args) {
for (String hf : List.of("model.embed_tokens.weight", "model.layers.0.input_layernorm.weight",
"model.layers.0.self_attn.q_proj.weight", "model.layers.0.self_attn.q_proj.bias", "model.layers.0.self_attn.o_proj.weight",
"model.layers.0.post_attention_layernorm.weight", "model.layers.17.mlp.gate_proj.weight",
"model.layers.17.mlp.down_proj.weight", "model.norm.weight", "lm_head.weight", "model.rotary_emb.inv_freq")) {
String gguf;
try { gguf = map(hf); } catch (IllegalArgumentException e) { gguf = "error: " + e.getMessage(); }
System.out.printf("%-48s -> %s%n", hf, gguf);
}
}
}model.embed_tokens.weight -> token_embd.weight
model.layers.0.input_layernorm.weight -> blk.0.attn_norm.weight
model.layers.0.self_attn.q_proj.weight -> blk.0.attn_q.weight
model.layers.0.self_attn.q_proj.bias -> blk.0.attn_q.bias
model.layers.0.self_attn.o_proj.weight -> blk.0.attn_output.weight
model.layers.0.post_attention_layernorm.weight -> blk.0.ffn_norm.weight
model.layers.17.mlp.gate_proj.weight -> blk.17.ffn_gate.weight
model.layers.17.mlp.down_proj.weight -> blk.17.ffn_down.weight
model.norm.weight -> output_norm.weight
lm_head.weight -> output.weight
model.rotary_emb.inv_freq -> error: cannot map tensor model.rotary_emb.inv_freqThe output shows three kinds of results. Names without a layer number (token_embd, output_norm, output) are renamed directly. Names with a layer number keep the number and change the rest. The suffix (.weight or .bias) is copied unchanged. The last line is an error on purpose: model.rotary_emb.inv_freq matches no rule.
GGUF uses short names that are the same for every architecture: token_embd, blk.N.attn_norm, blk.N.attn_q, attn_k, attn_v, attn_output, blk.N.ffn_norm, ffn_gate, ffn_up, ffn_down, output_norm, and output (TENSOR_NAMES in gguf-py's constants.py). Each comes from many architecture-specific names; tensor_mapping.py lists them with comments naming the architectures that use each one. The .weight or .bias suffix is kept.
Two details matter when converting:
- Not every Hugging Face tensor is a weight.
model.rotary_emb.inv_freqis a buffer computed fromrope_theta; the converter skips such tensors, and the program reports it as unmappable. - The Llama converter in llama.cpp also changes the content of two tensors, not only their names:
LlamaModel.permutereorders the rows ofq_projandk_projso that the RoPE layout matches the one llama.cpp's kernels use. A converter that only renames tensors produces a model that loads and gives wrong attention.
Making, Quantizing, and Testing Your Own GGUF showed that llama-quantize chooses types by these names (attn_v, ffn_down, output), which is one reason the names are fixed.
Example 12: tensor byte size from block size and type size
To read one tensor out of a multi-gigabyte file, a reader must know how many bytes the tensor occupies. For an F32 tensor that is the number of values times 4. Quantized types store values in blocks: a block packs a fixed number of values (the block size) into a fixed number of bytes (the type size). Q8_0 packs 32 values into 34 bytes.
The program uses the (block size, type size) pairs from GGML_QUANT_SIZES in gguf-py and computes the size of a [4096, 14336] feed-forward matrix in eight types. That is the shape of a feed-forward weight in Llama 3 8B.
import java.util.Locale;
public class TensorBytes {
record GgmlType(String name, int id, int blockSize, int typeSize) {}
static final GgmlType[] TYPES = { // GGML_QUANT_SIZES in gguf-py constants.py
new GgmlType("F32", 0, 1, 4), new GgmlType("F16", 1, 1, 2), new GgmlType("BF16", 30, 1, 2),
new GgmlType("Q8_0", 8, 32, 34), new GgmlType("Q4_0", 2, 32, 18), new GgmlType("Q4_K", 12, 256, 144),
new GgmlType("Q6_K", 14, 256, 210), new GgmlType("IQ4_XS", 23, 256, 136)};
static long nbytes(long[] dims, GgmlType t) {
if (dims[0] % t.blockSize() != 0) throw new IllegalArgumentException(t.name() + ": ne0 " + dims[0] + " not divisible by " + t.blockSize());
long rows = 1;
for (int i = 1; i < dims.length; i++) rows *= dims[i];
return dims[0] / t.blockSize() * t.typeSize() * rows; // row size times number of rows
}
public static void main(String[] args) {
long[] ffn = {4096, 14336};
for (GgmlType t : TYPES)
System.out.println(String.format(Locale.ROOT, "%-6s (id %2d) [4096, 14336]: %,12d bytes = %7.2f MiB", t.name(), t.id(),
nbytes(ffn, t), nbytes(ffn, t) / 1048576.0));
System.out.println(String.format(Locale.ROOT, "Q6_K [256, 151936]: %.2f MiB", nbytes(new long[] {256, 151936}, TYPES[6]) / 1048576.0));
try { nbytes(new long[] {704, 256}, TYPES[5]); } catch (IllegalArgumentException e) { System.out.println(e.getMessage()); }
}
}F32 (id 0) [4096, 14336]: 234,881,024 bytes = 224.00 MiB
F16 (id 1) [4096, 14336]: 117,440,512 bytes = 112.00 MiB
BF16 (id 30) [4096, 14336]: 117,440,512 bytes = 112.00 MiB
Q8_0 (id 8) [4096, 14336]: 62,390,272 bytes = 59.50 MiB
Q4_0 (id 2) [4096, 14336]: 33,030,144 bytes = 31.50 MiB
Q4_K (id 12) [4096, 14336]: 33,030,144 bytes = 31.50 MiB
Q6_K (id 14) [4096, 14336]: 48,168,960 bytes = 45.94 MiB
IQ4_XS (id 23) [4096, 14336]: 31,195,136 bytes = 29.75 MiB
Q6_K [256, 151936]: 30.43 MiB
Q4_K: ne0 704 not divisible by 256Take the Q8_0 line as a worked case. A row has 4096 values, which is 4096 / 32 = 128 blocks of 34 bytes. Bits per weight (bpw) is the average number of bits used for one value.
Q8_0 row: 4096 / 32 = 128 blocks; 128 x 34 = 4,352 bytes
whole matrix: 4,352 x 14,336 rows = 62,390,272 bytes = 59.50 MiB
bits per weight: 34 x 8 / 32 = 8.5For any ggml type, a row of ne0 values takes ne0 / blockSize * typeSize bytes, and the tensor is that row size times the product of the other dimensions. ne0 must be a multiple of the block size, which gguf.cpp checks when it reads the tensor infos.
The output shows the ratios from Building Q8_0 and Q4_0 Like ggml and k-quants, i-quants, and the Importance Matrix: Q8_0 is 59.50 MiB against 224 MiB for F32 (8.5 bits per weight against 32), Q4_0 and Q4_K are both 4.5 bpw (31.50 MiB), Q6_K is 6.5625 bpw, and IQ4_XS is 4.25 bpw (29.75 MiB). The Q6_K line for [256, 151936] gives 30.43 MiB, the size llama-quantize printed for the tiny model's output tensor. The last line is the case behind the fallback warnings in the same log: a row of 704 values cannot be stored in 256-value blocks.
This computation is all a reader needs to find a tensor: its bytes are at dataOffset + offset and are nbytes long.
Example 13: dimension order against PyTorch shapes
A matrix is a grid with rows and columns, but a computer stores it as one long line of numbers. In row-major order, the numbers of row 0 come first, then row 1, and so on. A shape such as [3, 4] tells how to cut that line back into a grid.
PyTorch, the Python library most models are trained with, writes a shape starting with the outermost dimension. GGUF writes it starting with the innermost. This is why GGUF lists [4096, 14336] for a matrix that PyTorch calls [14336, 4096]. The program converts PyTorch shapes to GGUF dimensions and finds one element of a small matrix in both orders.
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Arrays;
public class DimensionOrder {
static long[] ggufDims(long[] torchShape) { // reverse: innermost (last) PyTorch dim first
long[] d = new long[torchShape.length];
for (int i = 0; i < d.length; i++) d[i] = torchShape[torchShape.length - 1 - i];
return d;
}
public static void main(String[] args) {
long[] torch = {3, 4}; // nn.Linear(4 -> 3): weight [out_features, in_features]
long[] dims = ggufDims(torch);
System.out.println("PyTorch shape " + Arrays.toString(torch) + " -> GGUF dims " + Arrays.toString(dims) + " (ne0 = in_features)");
ByteBuffer data = ByteBuffer.allocate(48).order(ByteOrder.LITTLE_ENDIAN);
for (int r = 0; r < 3; r++) for (int c = 0; c < 4; c++) data.putFloat(10 * r + c); // W[r][c] = 10r + c
long ne0 = dims[0];
int r = 2, c = 1;
System.out.println("W[2][1] at element " + (r * ne0 + c) + " = " + data.getFloat((int) (4 * (r * ne0 + c))));
System.out.println("expert tensor [8, 14336, 4096] -> " + Arrays.toString(ggufDims(new long[] {8, 14336, 4096})));
System.out.println("bias [4096] -> " + Arrays.toString(ggufDims(new long[] {4096})));
}
}PyTorch shape [3, 4] -> GGUF dims [4, 3] (ne0 = in_features)
W[2][1] at element 9 = 21.0
expert tensor [8, 14336, 4096] -> [4096, 14336, 8]
bias [4096] -> [4096]The matrix W in the program has 3 rows of 4 values. It is the weight of a linear layer, which multiplies an input of 4 numbers by this matrix to get an output of 3 numbers. In memory it is one line of 12 numbers:
memory: 0 1 2 3 | 10 11 12 13 | 20 21 22 23
row 0 row 1 row 2
W[2][1] is element 2 x 4 + 1 = 9, and element 9 holds 21PyTorch lists dimensions outermost first: a linear layer from 4 inputs to 3 outputs has weight.shape == [3, 4], 3 rows of 4 values, stored row-major. GGUF, following ggml, lists dimensions innermost first: ne0 is the number of values in a row (the dimension that changes fastest in memory), ne1 the number of rows. The bytes are the same; only the list is reversed.
So W[r][c] is element r * ne0 + c in both views: W[2][1] is element 2 * 4 + 1 = 9, value 21. For a linear layer ne0 is the input size, which is why ffn_down (14336 inputs to 4096 outputs) has ne0 = 14336 and ffn_gate has ne0 = 4096. A mixture-of-experts tensor [8, 14336, 4096] becomes [4096, 14336, 8]: one row of 4096 inputs, 14336 rows per expert, 8 experts. A reader that copies PyTorch shapes into GGUF without reversing them writes a valid file with a transposed model in it.
Example 14: a real model's metadata
Toy files show the format, and a real model shows its scale. This example reads the metadata of a real converted model. The llama.cpp source tree includes vocabulary-only GGUF files for its tokenizer tests; ggml-vocab-llama-bpe.gguf holds the metadata of meta-llama/Meta-Llama-3-8B (the repository listed for llama-bpe in convert_hf_to_gguf_update.py) and no tensors. The GgufMetadata program from Example 9 reads it:
java GgufMetadata.java models/ggml-vocab-llama-bpe.ggufOutput (from the file in the pinned llama.cpp tree):
GGUF v3, 20 KV pairs, 0 tensors, data at 7818144
STRING general.architecture = 'llama'
STRING general.name = 'llama-bpe'
UINT32 llama.block_count = 32
UINT32 llama.context_length = 8192
UINT32 llama.embedding_length = 4096
UINT32 llama.feed_forward_length = 14336
UINT32 llama.attention.head_count = 32
UINT32 llama.attention.head_count_kv = 8
FLOAT32 llama.rope.freq_base = 500000.0
FLOAT32 llama.attention.layer_norm_rms_epsilon = 1.0E-5
UINT32 general.file_type = 1
UINT32 llama.vocab_size = 128256
UINT32 llama.rope.dimension_count = 128
STRING tokenizer.ggml.model = 'gpt2'
STRING tokenizer.ggml.pre = 'llama-bpe'
[STRING] tokenizer.ggml.tokens = [!, ", #, $, %, &] ... (128256 items)
[INT32] tokenizer.ggml.token_type = [1, 1, 1, 1, 1, 1] ... (128256 items)
[STRING] tokenizer.ggml.merges = [Ġ Ġ, Ġ ĠĠĠ, ĠĠ ĠĠ, ĠĠĠ Ġ, i n, Ġ t] ... (280147 items)
UINT32 tokenizer.ggml.bos_token_id = 128000
UINT32 tokenizer.ggml.eos_token_id = 128001Reading the output: the first line says the file has 20 key-value pairs and no tensors, and that a data section would start at byte 7,818,144. The model keys (block_count, embedding_length, and the others) are the ones from Example 10. The three array entries show the first six items and the total count.
Every number matches the Llama 3 8B shapes used in lessons 19 and 22: 32 layers, 4096 dimensions, 14336 FFN width, 32 query heads and 8 KV heads, RoPE base 500,000, and a 128,256-token vocabulary. general.file_type = 1 is MOSTLY_F16, the type of the file the vocabulary was extracted from.
The tokenizer is gpt2, llama.cpp's name for byte-level BPE, with the pre-tokenizer llama-bpe, which selects the regular expression that splits text before BPE. The Ġ in the merges is the byte-level BPE stand-in for a space (Building a BPE Tokenizer). The metadata alone is 7.8 MB, almost all of it the 128,256 tokens, their types, and 280,147 merges. This file has no tokenizer.chat_template; a full Llama 3 Instruct GGUF has one.
Example 15: parameter count and quant mix from tensor infos
A parameter is one number in the model's weights. A model with 8 billion parameters has 8 billion such numbers. A quantized file often does not use one type for everything. The quantizer gives more bits to tensors where errors hurt most and fewer bits to the rest. The result is called the quant mix: how many tensors, parameters, and bytes each type holds. The file name, such as Q4_K_M, names the recipe but not the exact mix.
The program takes the tensor infos of a one-layer Llama-style model, as GgufMetadata returns them, and sums parameters and bytes per ggml type.
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
public class QuantMixFromInfos {
record TensorInfo(String name, long[] dims, int type, long offset) {}
static final Map<Integer, String> NAMES = Map.of(0, "F32", 8, "Q8_0", 12, "Q4_K", 14, "Q6_K", 6, "Q5_0");
static final Map<Integer, int[]> SIZES = Map.of(0, new int[] {1, 4}, 8, new int[] {32, 34}, 12, new int[] {256, 144},
14, new int[] {256, 210}, 6, new int[] {32, 22});
public static void main(String[] args) {
List<TensorInfo> infos = List.of( // the shapes of a 1-layer Llama-style model
new TensorInfo("token_embd.weight", new long[] {256, 1000}, 12, 0),
new TensorInfo("blk.0.attn_norm.weight", new long[] {256}, 0, 0),
new TensorInfo("blk.0.attn_q.weight", new long[] {256, 256}, 12, 0),
new TensorInfo("blk.0.attn_k.weight", new long[] {256, 64}, 12, 0),
new TensorInfo("blk.0.attn_v.weight", new long[] {256, 64}, 14, 0),
new TensorInfo("blk.0.attn_output.weight", new long[] {256, 256}, 12, 0),
new TensorInfo("blk.0.ffn_norm.weight", new long[] {256}, 0, 0),
new TensorInfo("blk.0.ffn_gate.weight", new long[] {256, 704}, 12, 0),
new TensorInfo("blk.0.ffn_up.weight", new long[] {256, 704}, 12, 0),
new TensorInfo("blk.0.ffn_down.weight", new long[] {704, 256}, 6, 0),
new TensorInfo("output_norm.weight", new long[] {256}, 0, 0),
new TensorInfo("output.weight", new long[] {256, 1000}, 14, 0));
Map<String, long[]> mix = new TreeMap<>(); // type -> {tensors, params, bytes}
long params = 0, bytes = 0;
for (TensorInfo t : infos) {
long n = 1;
for (long d : t.dims()) n *= d;
int[] s = SIZES.get(t.type());
long b = n / s[0] * s[1];
long[] m = mix.computeIfAbsent(NAMES.get(t.type()), k -> new long[3]);
m[0]++; m[1] += n; m[2] += b;
params += n; bytes += b;
}
for (var e : mix.entrySet())
System.out.println(String.format(Locale.ROOT, "%-5s %2d tensors %8d params %8d bytes %5.1f%%", e.getKey(), e.getValue()[0],
e.getValue()[1], e.getValue()[2], 100.0 * e.getValue()[2] / bytes));
System.out.println(String.format(Locale.ROOT, "total %d params, %d bytes, %.3f bits per weight", params, bytes, bytes * 8.0 / params));
}
}F32 3 tensors 768 params 3072 bytes 0.4%
Q4_K 6 tensors 763904 params 429696 bytes 55.1%
Q5_0 1 tensors 180224 params 123904 bytes 15.9%
Q6_K 2 tensors 272384 params 223440 bytes 28.6%
total 1217280 params, 780112 bytes, 5.127 bits per weightEach output row adds up several tensors. For one tensor, token_embd.weight with dimensions [256, 1000] in Q4_K, the arithmetic is:
parameters: 256 x 1000 = 256,000
bytes: 256,000 / 256 (block size) x 144 (type size) = 144,000The six Q4_K tensors add up to 763,904 parameters and 429,696 bytes. The last line divides all bytes by all parameters:
bits per weight = 780,112 bytes x 8 / 1,217,280 parameters = 5.127Parameters are the product of each tensor's dimensions, and bytes follow from Example 12. Grouping by type gives the quant mix: here 6 tensors in Q4_K hold 55 percent of the bytes, the two Q6_K tensors (attn_v and output, following lesson 22's rules) 29 percent, ffn_down falls back to Q5_0 because its rows have 704 values, and the norms stay F32. The average is 5.127 bits per weight.
This is how a tool answers "what is really inside this Q4_K_M file" without trusting the file name or general.file_type. The same loop over a real file's tensor infos is part of Project 7.
Example 16: version differences: v1, v2, and v3
File formats change over time, and old files stay around. A reader has to know which layout it is looking at before it can parse anything else, because an early version used smaller numbers in the header. The program writes headers in the v1, v2, and v3 layouts and in big-endian v3, and parses each one. In the output, compare the header sizes of v1 and v2.
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
public class GgufVersions {
static byte[] header(int version, ByteOrder order) {
ByteBuffer b = ByteBuffer.allocate(32).order(order);
b.put("GGUF".getBytes(StandardCharsets.US_ASCII)).putInt(version);
if (version == 1) b.putInt(7).putInt(12); // v1: 32-bit tensor and KV counts
else b.putLong(7).putLong(12); // v2, v3: 64-bit counts
return java.util.Arrays.copyOf(b.array(), b.position());
}
static String read(byte[] file) {
ByteBuffer b = ByteBuffer.wrap(file).order(ByteOrder.LITTLE_ENDIAN);
if (b.getInt() != 0x46554747) return "not GGUF";
int version = b.getInt();
if ((version & 0xFFFF) == 0) { // same test as gguf.cpp and GGUFReader
b.order(ByteOrder.BIG_ENDIAN);
version = Integer.reverseBytes(version);
if (version != 3) return "byte-swapped version " + version;
return "big-endian v3: tensors " + b.getLong() + ", kv " + b.getLong() + " (byte-swapped file)";
}
return switch (version) {
case 1 -> "v1: tensors " + Integer.toUnsignedLong(b.getInt()) + ", kv " + Integer.toUnsignedLong(b.getInt()) + " (llama.cpp: no longer supported)";
case 2, 3 -> "v" + version + ": tensors " + b.getLong() + ", kv " + b.getLong();
default -> "unsupported version " + version;
};
}
public static void main(String[] args) {
System.out.println(read(header(1, ByteOrder.LITTLE_ENDIAN)) + ", header " + header(1, ByteOrder.LITTLE_ENDIAN).length + " bytes");
System.out.println(read(header(2, ByteOrder.LITTLE_ENDIAN)) + ", header " + header(2, ByteOrder.LITTLE_ENDIAN).length + " bytes");
System.out.println(read(header(3, ByteOrder.LITTLE_ENDIAN)));
System.out.println(read(header(3, ByteOrder.BIG_ENDIAN)));
System.out.println(read(header(4, ByteOrder.LITTLE_ENDIAN)));
}
}v1: tensors 7, kv 12 (llama.cpp: no longer supported), header 16 bytes
v2: tensors 7, kv 12, header 24 bytes
v3: tensors 7, kv 12
big-endian v3: tensors 7, kv 12 (byte-swapped file)
unsupported version 4The five output lines show three layouts, one byte-swapped file, and one error. The v1 header is 16 bytes (4 + 4 + 4 + 4) and the v2 header is 24 bytes (4 + 4 + 8 + 8). The v3 line and the big-endian v3 line give the same counts, because the reader detects the swapped byte order (Example 2) and switches. Version 4 is rejected.
The GGUF specification (docs/gguf.md in the ggml repository, not part of the pinned tree) describes the history: version 1 stored the tensor count, the key-value count, and string lengths as 32-bit integers; version 2 changed them to 64-bit; version 3 added support for big-endian files, with the same layout in the other byte order. The pinned code shows the consequences:
- llama.cpp rejects version 1:
gguf.cpplogs "GGUFv1 is no longer supported, please use a more up-to-date version". The loader's version names are "GGUF V1 (support until nov 2023)", "GGUF V2", and "GGUF V3 (latest)". - gguf-py's
GGUFReaderaccepts versions 2 and 3 (READER_SUPPORTED_VERSIONS = [2, GGUF_VERSION]) and handles both byte orders. - A version higher than 3 is rejected by both:
gguf.cpplogs "this GGUF file is version N but this software only supports up to version 3", andGGUFReaderraises an error for any version outside its supported list.
Because v2 and v3 have the same little-endian layout, one reader handles both. The program's v1 header is 16 bytes instead of 24, so a reader that assumed 64-bit counts would read the KV count and the first key length as one number. Detecting the version first is what keeps the rest of the parse aligned. New files are version 3: gguf-py writes GGUF_VERSION = 3, and the vocabulary file in Example 14 is v3.
Example 17: Project 7: GgufInfo
This project turns the earlier examples into one tool that answers the question "what is in this file?" without loading the weights. GgufInfo prints a summary of any GGUF file: version, counts, alignment, data offset, architecture, file type, the main hyperparameters, the vocabulary size, and the quant mix with parameter and byte totals. It combines the reader from Example 9, the keys from Example 10, the sizes from Example 12, and the grouping from Example 15. Without arguments it reads a one-tensor file built in memory.
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
public class GgufInfo {
record TensorInfo(String name, long[] dims, int type, long offset) {}
// ggml type id -> {name, block size, type size}, from GGML_QUANT_SIZES in gguf-py
static final Map<Integer, Object[]> TYPES = Map.ofEntries(Map.entry(0, new Object[] {"F32", 1, 4}), Map.entry(1, new Object[] {"F16", 1, 2}),
Map.entry(30, new Object[] {"BF16", 1, 2}), Map.entry(2, new Object[] {"Q4_0", 32, 18}), Map.entry(6, new Object[] {"Q5_0", 32, 22}),
Map.entry(8, new Object[] {"Q8_0", 32, 34}), Map.entry(10, new Object[] {"Q2_K", 256, 84}), Map.entry(11, new Object[] {"Q3_K", 256, 110}),
Map.entry(12, new Object[] {"Q4_K", 256, 144}), Map.entry(13, new Object[] {"Q5_K", 256, 176}), Map.entry(14, new Object[] {"Q6_K", 256, 210}),
Map.entry(20, new Object[] {"IQ4_NL", 32, 18}), Map.entry(23, new Object[] {"IQ4_XS", 256, 136}));
static final Map<Long, String> FILE_TYPES = Map.of(0L, "ALL_F32", 1L, "MOSTLY_F16", 7L, "MOSTLY_Q8_0", 15L, "MOSTLY_Q4_K_M",
14L, "MOSTLY_Q4_K_S", 18L, "MOSTLY_Q6_K", 30L, "MOSTLY_IQ4_XS", 32L, "MOSTLY_BF16");
static String str(ByteBuffer b) { byte[] s = new byte[Math.toIntExact(b.getLong())]; b.get(s); return new String(s, StandardCharsets.UTF_8); }
static Object value(ByteBuffer b, int type) {
return switch (type) {
case 0, 1, 7 -> b.get(); case 2, 3 -> b.getShort(); case 4 -> Integer.toUnsignedLong(b.getInt()); case 5 -> b.getInt();
case 6 -> b.getFloat(); case 8 -> str(b); case 10, 11 -> b.getLong(); case 12 -> b.getDouble();
case 9 -> { int et = b.getInt(); long n = b.getLong(); List<Object> l = new ArrayList<>();
for (long i = 0; i < n; i++) l.add(value(b, et)); yield l; }
default -> throw new IllegalArgumentException("bad value type " + type);
};
}
static void info(ByteBuffer b, long fileSize) {
b.order(ByteOrder.LITTLE_ENDIAN);
if (b.getInt() != 0x46554747) throw new IllegalArgumentException("not a GGUF file");
int version = b.getInt();
long nTensors = b.getLong(), nKv = b.getLong();
Map<String, Object> kv = new LinkedHashMap<>();
for (long i = 0; i < nKv; i++) { String k = str(b); kv.put(k, value(b, b.getInt())); }
List<TensorInfo> tensors = new ArrayList<>();
for (long i = 0; i < nTensors; i++) {
String name = str(b); long[] dims = new long[b.getInt()];
for (int j = 0; j < dims.length; j++) dims[j] = b.getLong();
tensors.add(new TensorInfo(name, dims, b.getInt(), b.getLong()));
}
long alignment = kv.get("general.alignment") instanceof Long a ? a : 32;
long dataOffset = (b.position() + alignment - 1) / alignment * alignment;
String arch = (String) kv.get("general.architecture");
System.out.println("GGUF v" + version + ", " + nKv + " KV pairs, " + nTensors + " tensors, alignment " + alignment + ", data at byte " + dataOffset);
System.out.println("architecture " + arch + ", name " + kv.getOrDefault("general.name", "(none)") + ", file type "
+ FILE_TYPES.getOrDefault(kv.get("general.file_type"), String.valueOf(kv.get("general.file_type"))));
for (String k : new String[] {"context_length", "embedding_length", "block_count", "feed_forward_length", "attention.head_count", "attention.head_count_kv", "rope.freq_base"})
if (kv.containsKey(arch + "." + k)) System.out.println(" " + k + " = " + kv.get(arch + "." + k));
if (kv.get("tokenizer.ggml.tokens") instanceof List<?> tokens) System.out.println(" vocabulary " + tokens.size() + " tokens, chat template " + kv.containsKey("tokenizer.chat_template"));
Map<String, long[]> mix = new TreeMap<>();
long params = 0, bytes = 0;
for (TensorInfo t : tensors) {
Object[] ty = TYPES.get(t.type());
if (ty == null) throw new IllegalArgumentException(t.name() + ": unknown type " + t.type());
long n = 1; for (long d : t.dims()) n *= d;
long size = n / (Integer) ty[1] * (Integer) ty[2];
long[] m = mix.computeIfAbsent((String) ty[0], k -> new long[3]);
m[0]++; m[1] += n; m[2] += size;
params += n; bytes += size;
}
mix.forEach((t, m) -> System.out.println(String.format(Locale.ROOT, " %-6s %3d tensors %,14d params %,14d bytes", t, m[0], m[1], m[2])));
System.out.println(String.format(Locale.ROOT, "parameters %,d, tensor data %,d bytes, %.2f bpw, file %,d bytes", params, bytes, bytes * 8.0 / params, fileSize));
}
static byte[] demo() { // a 1-tensor file written the way Example 6 does it
ByteBuffer b = ByteBuffer.allocate(512).order(ByteOrder.LITTLE_ENDIAN);
b.put("GGUF".getBytes(StandardCharsets.US_ASCII)).putInt(3).putLong(1).putLong(3);
b.putLong(20).put("general.architecture".getBytes()).putInt(8).putLong(5).put("llama".getBytes());
b.putLong(22).put("llama.embedding_length".getBytes()).putInt(4).putInt(64);
b.putLong(17).put("general.file_type".getBytes()).putInt(4).putInt(7);
b.putLong(17).put("token_embd.weight".getBytes()).putInt(2).putLong(64).putLong(4).putInt(8).putLong(0);
while (b.position() % 32 != 0) b.put((byte) 0);
b.put(new byte[4 * 2 * 34]); // 4 rows x 2 Q8_0 blocks of zeros
return java.util.Arrays.copyOf(b.array(), b.position());
}
public static void main(String[] args) throws Exception {
if (args.length == 0) { byte[] d = demo(); info(ByteBuffer.wrap(d), d.length); return; }
try (FileChannel ch = FileChannel.open(Path.of(args[0]))) {
info(ch.map(FileChannel.MapMode.READ_ONLY, 0, Math.min(ch.size(), Integer.MAX_VALUE)), ch.size());
}
}
}GGUF v3, 3 KV pairs, 1 tensors, alignment 32, data at byte 224
architecture llama, name (none), file type MOSTLY_Q8_0
embedding_length = 64
Q8_0 1 tensors 256 params 272 bytes
parameters 256, tensor data 272 bytes, 8.50 bpw, file 496 bytesThe demo output checks the arithmetic on a small case: 256 parameters in Q8_0 are 8 blocks of 34 bytes, 272 bytes at 8.50 bpw, and the file is the 224-byte metadata section plus the data.
The 224 comes from the sections in front of the data:
header 24
general.architecture 45
llama.embedding_length 38
general.file_type 33
tensor info (token_embd) 57
total 197 -> padded up to 224
file size = 224 + 272 = 496Run on the Q4_K_M file from Making, Quantizing, and Testing Your Own GGUF (the tiny model with random weights):
java GgufInfo.java tiny-Q4_K_M.ggufOutput (from running the program on that file):
GGUF v3, 22 KV pairs, 39 tensors, alignment 32, data at byte 5931072
architecture llama, name tiny-random-llama, file type MOSTLY_Q4_K_M
context_length = 4096
embedding_length = 256
block_count = 4
feed_forward_length = 704
attention.head_count = 8
attention.head_count_kv = 2
rope.freq_base = 500000.0
vocabulary 151936 tokens, chat template true
F32 9 tensors 2,304 params 9,216 bytes
Q4_K 23 tensors 40,960,000 params 23,040,000 bytes
Q5_0 2 tensors 360,448 params 247,808 bytes
Q6_K 3 tensors 38,928,384 params 31,933,440 bytes
Q8_0 2 tensors 360,448 params 382,976 bytes
parameters 80,611,584, tensor data 55,613,440 bytes, 5.52 bpw, file 61,544,512 bytesRead this output from top to bottom. The first two lines give the header and the identity of the model. The indented lines are hyperparameters. The vocabulary line shows 151,936 tokens and a chat template. The quant mix lists five types, and most of the bytes are in Q4_K and Q6_K. The last line gives the totals.
The numbers agree with the llama.cpp tools: 80,611,584 parameters is the 80.61 M that llama-bench reported, 55,613,440 bytes is the "quant size = 53.04 MiB (5.52 BPW)" line of llama-quantize, and the mix matches its log: 2 Q5_0 and 2 Q8_0 ffn_down fallbacks, 3 Q6_K tensors (output and two attn_v), and 9 F32 norms. The file is 5.9 MB larger than the tensor data because of the tokenizer metadata in front of it.
The TYPES table covers the types used in this course; an unknown type id stops the program with an error rather than computing a wrong size. loop show in section 4 wraps this class in a picocli command and adds the safetensors path from Project 6.
Interview points
- A GGUF file is a 24-byte header (magic
GGUF, auint32version, and 64-bit tensor and key-value counts), then the key-value metadata, the tensor infos, padding, and an aligned data section. - A string is a 64-bit byte length followed by UTF-8 bytes with no terminator. An array is an element type, a 64-bit count, and the elements.
- There are 13 value types, with ids 0 to 12. Unsigned types must be widened in Java, because Java has no unsigned integers.
- A tensor info holds the name,
n_dims(at most 4), the dimensions with the innermost first, the ggml type id, and an offset measured from the start of the data section. - The alignment is
general.alignment(aUINT32that is a power of 2) or 32 when the key is absent. The data section and each tensor are padded to it. llama.cpp requires the offsets to follow the order of the infos with no gaps. - Model hyperparameters are stored under keys that start with the architecture name, such as
llama.block_count. The tokenizer is stored undertokenizer.ggml.*, and the chat template undertokenizer.chat_template. - GGUF tensor names such as
blk.N.attn_qandffn_downare mapped from Hugging Face names by gguf-py'stensor_mapping.py. The Llama converter also reorders the rows of the Q and K matrices to match llama.cpp's RoPE layout. - A row of a tensor takes
ne0 / blockSize * typeSizebytes, and the tensor takes that times the number of rows.ne0must be a multiple of the block size. - GGUF dimensions are the PyTorch shape reversed, so
ne0of a linear weight is its input size. - llama.cpp rejects GGUF version 1, and gguf-py reads versions 2 and 3. A version number whose low 16 bits are zero signals a file written in the other byte order.
Summary
GGUF_MAGICis0x46554747andGGUF_VERSIONis 3.- A file with one metadata pair and two tensors has its data section at byte 192 with the default alignment.
héllois a 14-byte GGUF string: 8 bytes of length and 6 bytes of UTF-8.BOOLtakes one byte, and enums are stored in 4 bytes.- The
TokenTypevalues are 1 normal, 2 unknown, 3 control, 4 user-defined, 5 unused, and 6 byte. - A Java writer of about 60 lines produces files that
gguf-dump,GGUFReader, and llama.cpp accept. Q8_0stores a 64 by 3 matrix in 204 bytes.GGML_PAD(x, n)roundsxup to a multiple ofn.- Reading the metadata never touches the tensor data, so the metadata of a multi-gigabyte file reads in milliseconds.
- A Llama 3 8B GGUF has 128,256 tokens and 280,147 merges in its metadata, about 7.8 MB.
- A
[4096, 14336]matrix takes 224 MiB inF32, 59.50 MiB inQ8_0, 31.50 MiB inQ4_K, and 29.75 MiB inIQ4_XS. GgufInforeports 80,611,584 parameters and 5.52 bits per weight for the tinyQ4_K_Mfile, which matchesllama-benchandllama-quantize.- The quant mix of a file comes from its tensor infos, not from its name.