Lesson 49 of 73
Loading GGUF with the FFM API
A language model lives in one big file. That file holds the settings of the model (how many layers, how wide each layer is, what the vocabulary is) and the weights, which are the millions or billions of numbers the model learned. Before the engine in this section can run a single calculation, it must open this file and find those pieces. This lesson writes that first part: a loader that opens a GGUF model file and hands out its metadata and its tensors. A tensor is a block of numbers with a shape, such as a table with 256 columns and 151,936 rows. The file format itself was taught in The GGUF Format.
Model files are large. A 7B model in Q4_K_M quantization is about 4.4 GB, and Java's old file tools cannot address more than 2 GB in one piece. This lesson therefore uses the Foreign Function and Memory (FFM) API, a Java API for memory outside the Java heap. It became final in Java 22 (JEP 454), so every program in this section needs JDK 22 or later. The course runs them on JDK 25. With it, the loader can map a model file of any size into memory, read every header field with an explicit byte order, take views of tensors without copying them, decode F32, F16, and Q8_0 values, and choose the right arena for each lifetime. The lesson ends with Project 16, the GgufModel class that the next two lessons use.
The table lists the ideas and the Java API behind each one. The example numbers tell you where each idea appears.
| Idea | Java API | Example |
|---|---|---|
| Off-heap memory with a lifetime | Arena, MemorySegment |
1, 9 |
| Byte order and alignment | ValueLayout.JAVA_INT_UNALIGNED.withOrder(...) |
1 |
| A file as memory | FileChannel.map(mode, offset, size, arena) |
2, 10 |
| Header, strings, arrays, tensor infos | a cursor over a segment | 3 to 5 |
| Tensor views without copies | asSlice |
6 |
| F32, F16, Q8_0 values | getAtIndex, Float.float16ToFloat |
7, 8 |
| Checks before use | magic, version, offsets, sizes | 11 |
| A real model file | the tiny random model | 12 |
| Project 16 | GgufModel |
13 |
The byte layout was taught in the GGUF lesson: a 24-byte header, key-value metadata, tensor infos, padding up to the alignment, and the data section. That lesson read files with ByteBuffer. This lesson reads the same layout with MemorySegment, because a ByteBuffer cannot address more than 2,147,483,647 bytes (Example 10 shows the failure). Two more lessons supply background. The Q8_0 block format comes from Building Q8_0 and Q4_0 Like ggml, and FP16 conversion from Number Formats. The Java Deep Dive lesson Foreign Function and Memory API covers the API in general, including calls to C functions, which this engine does not need.
Example 1: Arena, MemorySegment, and little-endian layouts
Java normally keeps data in the heap, the memory area managed by the garbage collector. The FFM API lets a program use memory outside the heap. Two ideas make this safe. A memory segment is a contiguous region of memory with a fixed size, and every read or write is checked against that size. An arena is the object that owns segments and decides when their memory is freed.
The program below allocates 16 bytes off the heap and writes the start of a GGUF header into them. Then it reads the values back with two byte orders and two alignments. Watch how the same four bytes turn into different numbers depending on the byte order, and which reads throw exceptions.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.util.HexFormat;
public class ArenaBasics {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfInt U32_BE = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
public static void main(String[] args) {
try (Arena arena = Arena.ofConfined()) {
MemorySegment seg = arena.allocate(16, 8); // 16 bytes, address a multiple of 8
seg.set(U32, 0, 0x46554747); // the GGUF magic as a uint32
seg.set(U32, 4, 3); // version 3
seg.set(U64, 8, 201L); // a 64-bit count
System.out.println("byteSize " + seg.byteSize() + ", native order " + ByteOrder.nativeOrder());
System.out.println("bytes " + HexFormat.ofDelimiter(" ").formatHex(seg.toArray(ValueLayout.JAVA_BYTE)));
System.out.println("first 4 bytes as text: " + new String(seg.asSlice(0, 4).toArray(ValueLayout.JAVA_BYTE), StandardCharsets.US_ASCII));
System.out.println("u32 at 4: little-endian " + seg.get(U32, 4) + ", big-endian " + seg.get(U32_BE, 4));
System.out.println("u64 at 8: " + seg.get(U64, 8));
System.out.println("alignment: JAVA_INT " + ValueLayout.JAVA_INT.byteAlignment() + ", JAVA_INT_UNALIGNED " + U32.byteAlignment());
System.out.println("u32 at 5 with the unaligned layout: " + seg.get(U32, 5));
try {
seg.get(ValueLayout.JAVA_INT, 5);
} catch (IllegalArgumentException e) {
System.out.println("JAVA_INT at 5: IllegalArgumentException (misaligned)");
}
try {
seg.get(U32, 14);
} catch (IndexOutOfBoundsException e) {
System.out.println("u32 at 14: IndexOutOfBoundsException (needs bytes 14 to 17)");
}
}
}
}byteSize 16, native order LITTLE_ENDIAN
bytes 47 47 55 46 03 00 00 00 c9 00 00 00 00 00 00 00
first 4 bytes as text: GGUF
u32 at 4: little-endian 3, big-endian 50331648
u64 at 8: 201
alignment: JAVA_INT 4, JAVA_INT_UNALIGNED 1
u32 at 5 with the unaligned layout: -922746880
JAVA_INT at 5: IllegalArgumentException (misaligned)
u32 at 14: IndexOutOfBoundsException (needs bytes 14 to 17)The program writes three values: the integer 0x46554747 at byte 0, the version 3 at byte 4, and the 64-bit count 201 at byte 8. To write or read a number, the program needs a value layout. A value layout describes how one Java value maps to bytes: its size, its byte order, and its alignment.
Two of those words need a definition. Byte order (also called endianness) is the order in which the bytes of a multi-byte number are stored. A little-endian machine stores the least significant byte first, so the number 3 as a 4-byte integer is 03 00 00 00. A big-endian machine stores the most significant byte first, which gives 00 00 00 03. Alignment is a rule that a value of size N must start at an address that is a multiple of N. Some processors and APIs insist on it.
JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN) is therefore a 4-byte integer, least significant byte first, that may sit at any address. GGUF stores every number little-endian, so every layout in this engine fixes the order explicitly instead of relying on the machine.
The output lines mean the following:
byteSize 16, native order LITTLE_ENDIAN: the segment holds 16 bytes, and this machine (x86-64) is little-endian. The explicit order makes the code correct on a big-endian machine as well.bytes 47 47 55 46 03 00 00 00 c9 00 ...: the integer0x46554747was stored lowest byte first:47 47 55 46. Those bytes are the ASCII codes ofG,G,U,F, which is why GGUF readers can compare the first four bytes with the textGGUF, or the firstuint32(unsigned 32-bit integer) with0x46554747. The version 3 is03 00 00 00, and 201 isc9followed by seven zero bytes.first 4 bytes as text: GGUF:asSlice(0, 4)gives a 4-byte view, andtoArray(JAVA_BYTE)copies it into abyte[]for theStringconstructor.u32 at 4: little-endian 3, big-endian 50331648: the same four bytes03 00 00 00read as big-endian are0x03000000, which is 50,331,648. A reader with the wrong byte order gets numbers like this and may use them as counts or sizes. For that reason the order is part of the layout and is never left implicit.alignment: JAVA_INT 4, JAVA_INT_UNALIGNED 1: the plainJAVA_INTlayout requires the address to be a multiple of 4, while the unaligned variant accepts any address.u32 at 5 with the unaligned layout: -922746880: bytes 5 to 8 are00 00 00 c9, the little-endian integer0xC9000000. Java'sintis signed, so the value prints as -922,746,880. As an unsigned 32-bit number it is 3,372,220,416. GGUF has unsigned types (UINT32,UINT64), and a Java reader must widen them, for example with& 0xFFFFFFFFL.JAVA_INT at 5: IllegalArgumentException: the same read with the aligned layout fails. The arena returned an address that is a multiple of 8, so the address plus 5 is not a multiple of 4. GGUF metadata has strings of any length between the numbers, so most metadata values sit at unaligned offsets. A GGUF reader therefore uses the_UNALIGNEDlayouts for metadata.u32 at 14: IndexOutOfBoundsException: a 4-byte read at offset 14 needs bytes 14 to 17, and the segment ends at 15. Every access is bounds-checked against the segment size. This is the main safety difference from raw pointers in C.
The try (Arena arena = Arena.ofConfined()) block frees the 16 bytes when it ends. The memory is outside the Java heap, so the garbage collector never moves or scans it.
Example 2: mapping a file with FileChannel.map(..., arena)
A model file can be several gigabytes. Reading all of it into Java arrays would be slow and would double the memory use. Mapping a file solves this. It asks the operating system to make the file's bytes appear at a range of memory addresses. The operating system reads a page (a small chunk, typically 4 KB) from disk the first time the program touches it, and keeps it in the page cache, a pool of recently used file data in RAM. Pages the program never touches are never read.
llama.cpp maps a model file by default (-lm, --load-mode is auto, which means mmap where the device supports it), and the Java engine does the same. This program writes a 16-byte file, maps it read-only into an arena, reads three values, tries to write, and then reads after the arena is closed. Look at the properties printed on the first line and at the two error messages.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
public class MapFile {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
public static void main(String[] args) throws Exception {
Path path = Files.createTempFile("map", ".bin");
Files.write(path, new byte[] {'G', 'G', 'U', 'F', 3, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0});
MemorySegment kept;
try (Arena arena = Arena.ofConfined(); FileChannel ch = FileChannel.open(path)) {
MemorySegment file = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena);
kept = file;
System.out.println("mapped " + file.byteSize() + " bytes, isMapped " + file.isMapped() + ", isReadOnly " + file.isReadOnly());
System.out.printf("magic 0x%08X, version %d, value at 8: %d%n", file.get(U32, 0), file.get(U32, 4), file.get(U32, 8));
try {
file.set(U32, 4, 4);
} catch (IllegalArgumentException e) {
System.out.println("write: " + e.getMessage());
}
}
try {
kept.get(U32, 0);
} catch (IllegalStateException e) {
System.out.println("read after the arena closed: " + e.getMessage());
}
Files.delete(path);
}
}mapped 16 bytes, isMapped true, isReadOnly true
magic 0x46554747, version 3, value at 8: 42
write: Attempt to write a read-only segment
read after the arena closed: Already closedFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena) returns a MemorySegment whose lifetime is the arena's lifetime. The first line shows its properties. byteSize equals the file size (16 bytes). isMapped() is true because the memory comes from a file mapping. isReadOnly() is true because of READ_ONLY. The second line shows the magic 0x46554747, the version 3, and the value 42. They come straight from the file's bytes, and nothing was copied into a Java array.
The attempt to write fails with IllegalArgumentException: Attempt to write a read-only segment. A read-only mapping protects the model file from a bug in the engine. Several processes can also map the same file read-only and share one copy of its pages in the page cache.
The last line comes from the variable kept, which still refers to the segment after the try block closed the arena. The read fails with IllegalStateException: Already closed instead of reading unmapped memory. Closing the arena unmapped the file, and the FFM API checks the arena's state on every access, so a stale segment cannot crash the JVM. The older MappedByteBuffer behaves differently: its mapping is released only when the garbage collector frees the buffer, and there is no way to unmap it at a chosen time.
Example 3: reading the GGUF header from a segment
The GGUF header is four fixed fields at fixed positions: the magic (4 bytes), the version (uint32), the tensor count (uint64), and the key-value count (uint64). The magic is a fixed value at the start of a file that identifies its type. This program writes such a header to a file, maps it, and reads it with a small cursor. A cursor is an object that holds a segment and a position, and advances the position after each read. The same cursor reads the variable-length parts in the next examples.
Look at the position printed at the start of each line. It is the byte where that field begins.
import java.io.ByteArrayOutputStream;
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
public class ReadHeader {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
// Reads values one after another and remembers the position.
static final class Cursor {
final MemorySegment s;
long pos;
Cursor(MemorySegment s) { this.s = s; }
int u32() { int v = s.get(U32, pos); pos += 4; return v; }
long u64() { long v = s.get(U64, pos); pos += 8; return v; }
}
static byte[] le(long v, int n) { byte[] b = new byte[n]; for (int i = 0; i < n; i++) b[i] = (byte) (v >>> (8 * i)); return b; }
public static void main(String[] args) throws Exception {
ByteArrayOutputStream out = new ByteArrayOutputStream(); // a header with 2 tensors and 5 key-value pairs
out.writeBytes("GGUF".getBytes(StandardCharsets.US_ASCII));
out.writeBytes(le(3, 4));
out.writeBytes(le(2, 8));
out.writeBytes(le(5, 8));
Path path = Files.createTempFile("header", ".gguf");
Files.write(path, out.toByteArray());
try (Arena arena = Arena.ofConfined(); FileChannel ch = FileChannel.open(path)) {
Cursor c = new Cursor(ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena));
int magic = c.u32();
System.out.printf("@0 magic 0x%08X (%s)%n", magic, magic == 0x46554747 ? "GGUF" : "not GGUF");
System.out.printf("@4 version %d%n", c.u32());
System.out.printf("@8 tensors %d%n", c.u64());
System.out.printf("@16 kv %d%n", c.u64());
System.out.println("header ends at " + c.pos);
}
Files.delete(path);
}
}@0 magic 0x46554747 (GGUF)
@4 version 3
@8 tensors 2
@16 kv 5
header ends at 24Each output line starts with the byte position where the field begins. The magic at byte 0 is 0x46554747, so the program prints GGUF. The version at byte 4 is 3, the tensor count at byte 8 is 2, and the key-value count at byte 16 is 5. After four reads the cursor stands at byte 24, where the first key-value pair would start. The header is always 24 bytes: 4 + 4 + 8 + 8.
Why a cursor? Everything after the header has variable length. A key is a string of any length, a value can be an array of any size, and a tensor name can be long or short. A reader cannot compute where the tenth key starts without reading the nine before it. The Cursor keeps that running position in one long, so positions past 2 GB work without any change.
The counts are uint64 in the file and long in Java. A real model has a few hundred tensors and a few dozen keys, so the values never come close to the signed limit. Positions inside the file use long from the start.
Example 4: strings and arrays from a segment
After the header come the key-value pairs, and their values are strings and arrays. A GGUF string is a uint64 byte length followed by that many UTF-8 bytes, without a terminating zero. UTF-8 is the common text encoding in which one character takes between 1 and 4 bytes. A GGUF array is a uint32 element type, a uint64 element count, and then the elements.
This program builds a string, an array of four token strings, and an array of four INT32 token types in a segment, then reads them with the cursor. Compare the number of characters in the first string with the number of bytes it takes. Also compare how the two arrays are read.
import java.io.ByteArrayOutputStream;
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class StringsAndArrays {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final class Cursor {
final MemorySegment s;
long pos;
Cursor(MemorySegment s) { this.s = s; }
int u32() { int v = s.get(U32, pos); pos += 4; return v; }
long u64() { long v = s.get(U64, pos); pos += 8; return v; }
String str() { // uint64 byte length, then UTF-8 bytes, no terminator
long n = u64();
String v = new String(s.asSlice(pos, n).toArray(ValueLayout.JAVA_BYTE), StandardCharsets.UTF_8);
pos += n;
return v;
}
}
static byte[] le(long v, int n) { byte[] b = new byte[n]; for (int i = 0; i < n; i++) b[i] = (byte) (v >>> (8 * i)); return b; }
static void str(ByteArrayOutputStream o, String s) { byte[] b = s.getBytes(StandardCharsets.UTF_8); o.writeBytes(le(b.length, 8)); o.writeBytes(b); }
public static void main(String[] args) {
ByteArrayOutputStream o = new ByteArrayOutputStream();
str(o, "世界"); // a plain string
o.writeBytes(le(8, 4)); o.writeBytes(le(4, 8)); // array: element type 8 (STRING), 4 elements
for (String t : List.of("<s>", "Ġworld", "Ċ", "ä¸ĸçķĮ")) str(o, t);
o.writeBytes(le(5, 4)); o.writeBytes(le(4, 8)); // array: element type 5 (INT32), 4 elements
for (int t : new int[] {3, 1, 1, 1}) o.writeBytes(le(t, 4));
try (Arena arena = Arena.ofConfined()) {
MemorySegment seg = arena.allocate(o.size());
MemorySegment.copy(o.toByteArray(), 0, seg, ValueLayout.JAVA_BYTE, 0, o.size());
Cursor c = new Cursor(seg);
String s = c.str();
System.out.println("string \"" + s + "\": " + s.length() + " chars, ends at byte " + c.pos);
int type = c.u32();
long n = c.u64();
List<String> tokens = new ArrayList<>();
for (long i = 0; i < n; i++) tokens.add(c.str()); // strings: walk one by one
System.out.println("array type " + type + ", " + n + " strings " + tokens + ", ends at byte " + c.pos);
type = c.u32();
int[] types = new int[(int) c.u64()];
MemorySegment.copy(seg, U32, c.pos, types, 0, types.length); // fixed-size elements: one bulk copy
c.pos += 4L * types.length;
System.out.println("array type " + type + ", " + types.length + " ints " + Arrays.toString(types) + ", ends at byte " + c.pos);
System.out.println("segment size " + seg.byteSize());
}
}
}string "世界": 2 chars, ends at byte 14
array type 8, 4 strings [<s>, Ġworld, Ċ, ä¸ĸçķĮ], ends at byte 82
array type 5, 4 ints [3, 1, 1, 1], ends at byte 110
segment size 110The first line reads the string 世界. It prints 2 chars but ends at byte 14: 8 bytes of length plus 6 bytes of UTF-8, because each of these two characters takes 3 bytes in UTF-8. The length field always counts bytes, never characters. The reader takes n bytes from the segment and decodes them. Counting characters would read the wrong number of bytes for any non-ASCII text.
The second line is a string array (element type 8) of four tokens. The texts are vocabulary entries in the byte-level form from Building a BPE Tokenizer. In that form, Ġworld is a space plus world, Ċ is a newline, and ä¸ĸçķĮ is 世界 written through the byte-to-unicode map. Each string element carries its own length, so the loop reads them one by one. The array ends at byte 82.
The third line is an INT32 array (element type 5) with the token types [3, 1, 1, 1], a control token followed by three normal tokens. Elements of a fixed size need no loop. MemorySegment.copy(seg, U32, c.pos, types, 0, types.length) copies all of them into an int[] in one call, and converts the byte order on the way. The real Qwen2 vocabulary has 151,936 token types, and the bulk copy reads them in one call instead of 151,936 separate reads. The segment is 110 bytes, and the cursor ends exactly at its end, which confirms that every byte was accounted for.
Example 5: tensor infos as records, and the data offset
The metadata is followed by one tensor info per tensor. A tensor info describes a tensor without containing its numbers. It has the name, the number of dimensions, the size of each dimension (innermost first), the ggml type id (which says how the numbers are stored, such as F32 or Q8_0), and an offset. The offset is relative to the start of the data section, where the actual numbers live.
A Java record is a compact immutable class with named fields, which fits this description exactly. This program parses two small headers that differ only in the optional key general.alignment. It builds TensorInfo records and computes where the data section begins. Pay attention to how the alignment changes the numbers.
import java.io.ByteArrayOutputStream;
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public class TensorInfos {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
record TensorInfo(String name, long[] dims, int type, long offset) {}
static final class Cursor {
final MemorySegment s;
long pos;
Cursor(MemorySegment s) { this.s = s; }
int u32() { int v = s.get(U32, pos); pos += 4; return v; }
long u64() { long v = s.get(U64, pos); pos += 8; return v; }
String str() { long n = u64(); String v = new String(s.asSlice(pos, n).toArray(ValueLayout.JAVA_BYTE), StandardCharsets.UTF_8); pos += n; return v; }
Object value(int type) { // only the types this file uses
return switch (type) {
case 4 -> u32() & 0xFFFFFFFFL; // UINT32, widened to long
case 8 -> str(); // STRING
default -> throw new IllegalStateException("value type " + type);
};
}
}
static byte[] le(long v, int n) { byte[] b = new byte[n]; for (int i = 0; i < n; i++) b[i] = (byte) (v >>> (8 * i)); return b; }
static void str(ByteArrayOutputStream o, String s) { byte[] b = s.getBytes(StandardCharsets.UTF_8); o.writeBytes(le(b.length, 8)); o.writeBytes(b); }
static byte[] file(Long alignment) {
ByteArrayOutputStream o = new ByteArrayOutputStream();
o.writeBytes("GGUF".getBytes(StandardCharsets.US_ASCII)); o.writeBytes(le(3, 4)); o.writeBytes(le(2, 8));
o.writeBytes(le(alignment == null ? 1 : 2, 8));
str(o, "general.architecture"); o.writeBytes(le(8, 4)); str(o, "llama");
if (alignment != null) { str(o, "general.alignment"); o.writeBytes(le(4, 4)); o.writeBytes(le(alignment, 4)); }
str(o, "token_embd.weight"); o.writeBytes(le(2, 4)); o.writeBytes(le(64, 8)); o.writeBytes(le(5, 8)); o.writeBytes(le(8, 4)); o.writeBytes(le(0, 8));
str(o, "output_norm.weight"); o.writeBytes(le(1, 4)); o.writeBytes(le(64, 8)); o.writeBytes(le(0, 4)); o.writeBytes(le(alignment == null ? 352 : 384, 8));
return o.toByteArray(); // header only: enough to compute offsets
}
public static void main(String[] args) {
for (Long alignment : Arrays.asList(null, 64L)) {
byte[] bytes = file(alignment);
try (Arena arena = Arena.ofConfined()) {
MemorySegment seg = arena.allocate(bytes.length);
MemorySegment.copy(bytes, 0, seg, ValueLayout.JAVA_BYTE, 0, bytes.length);
Cursor c = new Cursor(seg);
c.pos = 8;
long nTensors = c.u64(), nKv = c.u64();
Map<String, Object> kv = new LinkedHashMap<>();
for (long i = 0; i < nKv; i++) { String key = c.str(); kv.put(key, c.value(c.u32())); }
List<TensorInfo> infos = new ArrayList<>();
for (long i = 0; i < nTensors; i++) {
String name = c.str();
long[] dims = new long[c.u32()];
for (int d = 0; d < dims.length; d++) dims[d] = c.u64();
infos.add(new TensorInfo(name, dims, c.u32(), c.u64()));
}
long align = kv.get("general.alignment") instanceof Long a ? a : 32;
long dataStart = (c.pos + align - 1) / align * align;
System.out.println("metadata " + kv);
for (TensorInfo t : infos)
System.out.println(" " + t.name() + " dims " + Arrays.toString(t.dims()) + " type " + t.type() + " offset " + t.offset() + " -> file byte " + (dataStart + t.offset()));
System.out.println(" infos end at " + c.pos + ", alignment " + align + ", data section starts at " + dataStart);
}
}
}
}metadata {general.architecture=llama}
token_embd.weight dims [64, 5] type 8 offset 0 -> file byte 192
output_norm.weight dims [64] type 0 offset 352 -> file byte 544
infos end at 176, alignment 32, data section starts at 192
metadata {general.architecture=llama, general.alignment=64}
token_embd.weight dims [64, 5] type 8 offset 0 -> file byte 256
output_norm.weight dims [64] type 0 offset 384 -> file byte 640
infos end at 209, alignment 64, data section starts at 256The first file has one metadata key and no alignment key. Its tensor infos end at byte 176, and the data section starts at the next multiple of 32, byte 192. The second file adds general.alignment = 64. Its infos end at byte 209 (the extra key took 33 bytes), and the data section starts at 256, the next multiple of 64.
The two TensorInfo lines of each file read as follows:
token_embd.weight dims [64, 5] type 8: aQ8_0tensor (type 8) with rows of 64 values and 5 rows. A Q8_0 row of 64 values is 2 blocks of 34 bytes, so the tensor is5 * 2 * 34 = 340bytes.output_norm.weight dims [64] type 0 offset 352in the first file andoffset 384in the second: each tensor's size is rounded up to the alignment before the next one starts. 340 rounded up to a multiple of 32 is 352, and rounded up to a multiple of 64 is 384.-> file byte 544and-> file byte 640: the absolute position of a tensor is the data-section start plus the tensor's offset,192 + 352and256 + 384.
Here is the rounding formula with real numbers. The formula is (pos + align - 1) / align * align with integer division. For the first file, pos = 176 and align = 32: (176 + 31) / 32 = 207 / 32 = 6 (the remainder is dropped), and 6 * 32 = 192. If pos were already a multiple of 32, such as 192, the result would stay 192. The formula moves a position up to the next multiple and leaves multiples unchanged.
The alignment value is a UINT32 in the file. The cursor widens it to long with & 0xFFFFFFFFL. The pattern kv.get("general.alignment") instanceof Long a ? a : 32 falls back to GGUF_DEFAULT_ALIGNMENT, which is 32 in ggml/include/gguf.h. The rounding formula is GGML_PAD from the GGUF lesson. Getting this one number wrong shifts every tensor by the same amount. The result is a model that loads without error and outputs garbage, so Example 11 checks it.
Example 6: tensor views with asSlice, without copying
The engine needs to hand each tensor to the code that does the math. Copying a 300 MB tensor into a new array for every use would waste time and memory. A view solves this. A view is a segment that shares memory with another segment instead of copying it. asSlice(offset, size) returns such a view, with its own bounds. The engine uses one view per tensor, so a 300 MB tensor costs no memory beyond the mapping.
This program maps a 256-byte file read-write, takes a view of an 8-float tensor at data offset 32, and shows that writes through either segment are visible through the other. It also shows what a real copy looks like. The output prints address differences instead of addresses, because the differences are the same on every run.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
public class TensorViews {
static final ValueLayout.OfFloat F32 = ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
public static void main(String[] args) throws Exception {
Path path = Files.createTempFile("views", ".bin");
Files.write(path, new byte[256]); // pretend the data section starts at 64
long dataStart = 64, offset = 32, bytes = 4 * 8; // a tensor of 8 floats at offset 32
try (Arena arena = Arena.ofConfined();
FileChannel ch = FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.WRITE)) {
MemorySegment file = ch.map(FileChannel.MapMode.READ_WRITE, 0, ch.size(), arena);
MemorySegment tensor = file.asSlice(dataStart + offset, bytes);
System.out.println("view: " + tensor.byteSize() + " bytes, starts " + (tensor.address() - file.address()) + " bytes into the file");
file.set(F32, dataStart + offset + 4 * 3, 2.5f); // write through the whole-file segment
System.out.println("element 3 through the view: " + tensor.get(F32, 4 * 3));
tensor.set(F32, 0, -1f); // write through the view
System.out.println("file float at " + (dataStart + offset) + ": " + file.get(F32, dataStart + offset));
float[] copy = tensor.toArray(F32); // an explicit copy to the Java heap
tensor.set(F32, 4 * 3, 7f);
System.out.println("copy[3] " + copy[3] + ", view[3] " + tensor.get(F32, 4 * 3));
MemorySegment row1 = tensor.asSlice(16, 16); // a view of a view: elements 4 to 7
System.out.println("row view starts " + (row1.address() - file.address()) + " bytes into the file");
try {
tensor.asSlice(16, 32);
} catch (IndexOutOfBoundsException e) {
System.out.println("slice past the end of the view: IndexOutOfBoundsException");
}
}
Files.delete(path);
}
}view: 32 bytes, starts 96 bytes into the file
element 3 through the view: 2.5
file float at 96: -1.0
copy[3] 2.5, view[3] 7.0
row view starts 112 bytes into the file
slice past the end of the view: IndexOutOfBoundsExceptionThe view is 32 bytes (8 floats) and starts 96 bytes into the file: the data section at 64 plus the tensor offset 32. The program prints tensor.address() - file.address() instead of the addresses themselves, because the addresses change from run to run and the difference does not.
The next two lines prove that the view and the file segment are the same memory. A float written through the file segment at byte 64 + 32 + 12 appears as element 3 of the view (2.5). A float written through the view at its byte 0 appears in the file segment at byte 96 (-1.0). With a read-write mapping, those writes also reach the file on disk. The engine maps read-only, and the read-write mode here exists only to show the sharing.
toArray(F32) is the opposite of a view: it copies the 8 floats into a new float[] on the Java heap. After the copy, the program writes 7.0 into element 3 of the view. The copy still holds 2.5, and the view shows 7.0, so the copy is independent. The engine copies only when it needs a float[] for arithmetic (The Forward Pass in Java dequantizes weights into arrays). Everything else stays a view.
A view of a view works the same way. tensor.asSlice(16, 16) covers elements 4 to 7 and starts 112 bytes into the file. Its bounds are those of the parent view, so tensor.asSlice(16, 32) fails with IndexOutOfBoundsException. The reason is that 16 + 32 is past the view's 32 bytes, even though the file itself has those bytes. Each tensor view therefore protects the neighboring tensors from an indexing bug.
Example 7: reading F32 and F16 values
Now the loader must read actual numbers. A GGUF tensor with dimensions [ne0, ne1] stores ne1 rows of ne0 values one after another. Element (row, col) is therefore at index row * ne0 + col. For [4, 3], element (2, 1) is at index 2 * 4 + 1 = 9.
getAtIndex(layout, index) multiplies the index by the layout's size, 4 bytes for F32 and 2 bytes for F16, and reads there. F32 is the 32-bit floating point format Java calls float. F16 (also called FP16 or half precision) uses 16 bits, which halves the size and costs some precision. This program fills a [4, 3] tensor in both formats with the same values and prints them side by side. Compare the F32 and F16 columns digit by digit.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.util.Locale;
public class F32AndF16 {
static final ValueLayout.OfFloat F32 = ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfShort F16 = ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
public static void main(String[] args) {
int ne0 = 4, ne1 = 3; // GGUF dims [4, 3]: 3 rows of 4 values
try (Arena arena = Arena.ofConfined()) {
MemorySegment f32 = arena.allocate(4L * ne0 * ne1), f16 = arena.allocate(2L * ne0 * ne1);
for (int i = 0; i < ne0 * ne1; i++) {
float v = 0.1f * i - 0.35f;
f32.setAtIndex(F32, i, v); // index i -> byte offset 4 * i
f16.setAtIndex(F16, i, Float.floatToFloat16(v)); // index i -> byte offset 2 * i
}
for (int row = 0; row < ne1; row++) {
StringBuilder a = new StringBuilder(), b = new StringBuilder();
for (int col = 0; col < ne0; col++) {
long index = (long) row * ne0 + col; // element (row, col) of a [ne0, ne1] tensor
a.append(String.format(Locale.ROOT, "%10.6f", f32.getAtIndex(F32, index)));
b.append(String.format(Locale.ROOT, "%10.6f", Float.float16ToFloat(f16.getAtIndex(F16, index))));
}
System.out.println("row " + row + " F32" + a + " F16" + b);
}
short bits = f16.getAtIndex(F16, 1);
System.out.printf(Locale.ROOT, "element 1: F16 bits 0x%04X, %.8f as F16, %.8f as F32%n",
bits, Float.float16ToFloat(bits), f32.getAtIndex(F32, 1));
System.out.println("bytes: F32 " + f32.byteSize() + ", F16 " + f16.byteSize());
}
}
}row 0 F32 -0.350000 -0.250000 -0.150000 -0.050000 F16 -0.350098 -0.250000 -0.150024 -0.049988
row 1 F32 0.050000 0.150000 0.250000 0.350000 F16 0.049988 0.150024 0.250000 0.350098
row 2 F32 0.450000 0.550000 0.650000 0.750000 F16 0.449951 0.549805 0.649902 0.750000
element 1: F16 bits 0xB400, -0.25000000 as F16, -0.25000000 as F32
bytes: F32 48, F16 24The F32 columns show the stored values exactly as written: -0.35, -0.25, and so on in steps of 0.1. The F16 columns show the same values after a round trip through 16 bits: -0.350098 instead of -0.35, 0.049988 instead of 0.05, 0.549805 instead of 0.55. FP16 keeps 10 fraction bits, so values between 0.25 and 0.5 are spaced about 0.000244 apart, and each value lands on the nearest representable one. -0.25 and 0.75 are exact in both formats, because they are short binary fractions.
Float.float16ToFloat(short) and Float.floatToFloat16(float) (Java 20 and later) do the conversion. They use the same round-to-nearest-even rule that ggml's FP16 conversion uses (Number Formats). The line for element 1 shows the raw bits 0xB400: sign 1, exponent 01101, fraction 0, which is -2^-2 = -0.25.
The last line is the reason F16 exists in model files: the same 12 values take 48 bytes in F32 and 24 in F16. The index formula is the same for both types, and only the element size differs. getAtIndex applies the size. Row 2 of the table is indices 8 to 11, which is row * ne0 + col with ne0 = 4.
Example 8: Q8_0 blocks read from a segment
Q8_0 is a quantization format, a way to store each weight in about one byte instead of four. The tensor is cut into blocks of 32 values. Each block stores one shared scale and 32 small integers, and the decoded value is the scale times the integer. A Q8_0 tensor therefore has no fixed size per element. It stores blocks of 34 bytes: an FP16 scale d (2 bytes) and 32 signed bytes qs (the quants, one per value). Value j of a block is d * qs[j], as taught in Building Q8_0 and Q4_0 Like ggml.
To find element i, first find its block: block i / 32, which starts at byte 34 * (i / 32). Inside the block, the element is quant i % 32. For i = 70: block 70 / 32 = 2, starting at byte 68, and quant 70 % 32 = 6, at byte 68 + 2 + 6 = 76.
This program quantizes 128 values (2 rows of 64) into a segment with the reference quantizer and reads them back block by block and element by element. Look at the block start positions, at how the scales differ between block 0 and the other blocks, and at the maximum error per row.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.util.Locale;
public class Q8Blocks {
static final ValueLayout.OfShort F16 = ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final int QK = 32, BLOCK_BYTES = 34;
// C roundf: halves away from zero (from the Q8_0 lesson).
static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
// quantize_row_q8_0_ref: per 32 values, an FP16 scale d = amax / 127 and 32 signed bytes.
static void quantize(float[] x, MemorySegment out) {
for (int b = 0; b < x.length / QK; b++) {
float amax = 0;
for (int j = 0; j < QK; j++) amax = Math.max(amax, Math.abs(x[b * QK + j]));
float d = amax / 127, id = d != 0 ? 1 / d : 0;
out.set(F16, (long) b * BLOCK_BYTES, Float.floatToFloat16(d));
for (int j = 0; j < QK; j++) out.set(ValueLayout.JAVA_BYTE, (long) b * BLOCK_BYTES + 2 + j, (byte) roundf(x[b * QK + j] * id));
}
}
// Element i of a Q8_0 tensor: block i / 32, quant i % 32, scaled by that block's d.
static float get(MemorySegment t, long i) {
long block = i / QK * BLOCK_BYTES;
return Float.float16ToFloat(t.get(F16, block)) * t.get(ValueLayout.JAVA_BYTE, block + 2 + i % QK);
}
public static void main(String[] args) {
int ne0 = 64, ne1 = 2; // 2 rows of 64 values: 4 blocks
float[] x = new float[ne0 * ne1];
for (int i = 0; i < x.length; i++) x[i] = (float) Math.sin(i * 0.3) * (i < 32 ? 0.1f : 1.0f);
try (Arena arena = Arena.ofConfined()) {
MemorySegment t = arena.allocate((long) x.length / QK * BLOCK_BYTES);
quantize(x, t);
System.out.println("tensor bytes " + t.byteSize() + " for " + x.length + " values");
for (int b = 0; b < x.length / QK; b++) {
long at = (long) b * BLOCK_BYTES;
System.out.printf(Locale.ROOT, "block %d at byte %3d: d = %.6f, qs[0..3] = %d %d %d %d%n", b, at,
Float.float16ToFloat(t.get(F16, at)), t.get(ValueLayout.JAVA_BYTE, at + 2), t.get(ValueLayout.JAVA_BYTE, at + 3),
t.get(ValueLayout.JAVA_BYTE, at + 4), t.get(ValueLayout.JAVA_BYTE, at + 5));
}
for (int row = 0; row < ne1; row++) {
double maxErr = 0;
for (int col = 0; col < ne0; col++) maxErr = Math.max(maxErr, Math.abs(get(t, (long) row * ne0 + col) - x[row * ne0 + col]));
System.out.printf(Locale.ROOT, "row %d: element 5 = %.6f (original %.6f), max error %.6f%n",
row, get(t, (long) row * ne0 + 5), x[row * ne0 + 5], maxErr);
}
}
}
}tensor bytes 136 for 128 values
block 0 at byte 0: d = 0.000786, qs[0..3] = 0 38 72 100
block 1 at byte 34: d = 0.007866, qs[0..3] = -22 -58 -89 -112
block 2 at byte 68: d = 0.007874, qs[0..3] = 44 77 103 121
block 3 at byte 102: d = 0.007874, qs[0..3] = -64 -93 -115 -126
row 0: element 5 = 0.099861 (original 0.099749), max error 0.004095
row 1: element 5 = 0.960571 (original 0.961153), max error 0.003906128 values make 4 blocks, and 4 blocks of 34 bytes are 136 bytes. The block lines show the layout: block 0 starts at byte 0, block 1 at 34, block 2 at 68, block 3 at 102. These offsets are not multiples of 4, which is one more reason the FP16 scale is read with an unaligned layout.
The first 32 values were multiplied by 0.1 in the input. Block 0 therefore has the small scale d = 0.000786 (its largest absolute value divided by 127), while the other blocks have scales near 0.00787. Each block's quants use the full range from -127 to 127 relative to its own maximum, so small and large regions of a tensor keep similar relative precision. qs[0..3] = 0 38 72 100 in block 0 are the first four quants: sin(0) = 0 gives 0, and the following samples grow.
The row lines read individual elements through get(t, i). Row 0, element 5 decodes to 0.099861 against the original 0.099749. Row 1, element 5 decodes to 0.960571 against 0.961153. The maximum errors, 0.004095 and 0.003906, are close to half of the larger block scales (0.00787 / 2 = 0.0039). The reason is that rounding to the nearest step costs at most d / 2. The scale itself is stored in FP16, and its rounding error, multiplied by a quant near 127, adds a few ten-thousandths. This is why row 0 lands slightly above 0.0039.
This element-by-element reader is slow for a forward pass, because it decodes the scale for every element. Lesson 51 dequantizes whole rows at a time. Quantized Kernels, the Vector API, and Threads computes dot products directly on the blocks without producing floats at all.
Example 9: arena lifetimes: confined, shared, auto, and global
A model file is mapped once and used for a long time, possibly by many threads. Someone must decide when the mapping ends and which threads may read it. The arena answers both questions. Java has four kinds of arena:
- A confined arena belongs to one thread and is closed explicitly.
- A shared arena can be used by any thread and is closed explicitly.
- An auto arena is freed by the garbage collector when no segment of it is reachable.
- The global arena is never freed.
This program tries each one from a second thread and after closing. Read the output line by line against the list above.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.util.concurrent.atomic.AtomicReference;
public class ArenaLifetimes {
// Reads the segment on a second thread and reports what happened.
static String fromOtherThread(MemorySegment s) throws InterruptedException {
AtomicReference<String> result = new AtomicReference<>();
Thread t = new Thread(() -> {
try {
result.set("read " + s.get(ValueLayout.JAVA_INT, 0));
} catch (RuntimeException e) {
result.set(e.getClass().getSimpleName());
}
});
t.start();
t.join();
return result.get();
}
static String read(MemorySegment s) {
try {
return "read " + s.get(ValueLayout.JAVA_INT, 0);
} catch (RuntimeException e) {
return e.getClass().getSimpleName() + ": " + e.getMessage();
}
}
public static void main(String[] args) throws Exception {
Arena confined = Arena.ofConfined();
MemorySegment a = confined.allocate(ValueLayout.JAVA_INT);
a.set(ValueLayout.JAVA_INT, 0, 1);
System.out.println("confined, owner thread: " + read(a) + "; other thread: " + fromOtherThread(a));
confined.close();
System.out.println("confined, after close: " + read(a));
Arena shared = Arena.ofShared();
MemorySegment b = shared.allocate(ValueLayout.JAVA_INT);
b.set(ValueLayout.JAVA_INT, 0, 2);
System.out.println("shared, other thread: " + fromOtherThread(b));
shared.close();
System.out.println("shared, after close: " + read(b));
MemorySegment c = Arena.ofAuto().allocate(ValueLayout.JAVA_INT);
c.set(ValueLayout.JAVA_INT, 0, 3);
System.out.println("auto, other thread: " + fromOtherThread(c));
try {
Arena.ofAuto().close();
} catch (UnsupportedOperationException e) {
System.out.println("auto, close(): UnsupportedOperationException");
}
MemorySegment d = Arena.global().allocate(ValueLayout.JAVA_INT);
System.out.println("global: " + read(d) + ", scope alive " + d.scope().isAlive());
}
}confined, owner thread: read 1; other thread: WrongThreadException
confined, after close: IllegalStateException: Already closed
shared, other thread: read 2
shared, after close: IllegalStateException: Already closed
auto, other thread: read 3
auto, close(): UnsupportedOperationException
global: read 0, scope alive trueThe output lines map to the four kinds:
confined, owner thread: read 1; other thread: WrongThreadException: a confined arena's segments are usable only by the thread that created the arena. The check happens on every access, and it is cheap, because the owner never changes.confined, after close: IllegalStateException: Already closed: afterclose(), the memory is freed and every segment of the arena becomes unusable.shared, other thread: read 2andshared, after close: IllegalStateException: any thread can read a shared arena's segments, and closing it invalidates them for all threads at once. To close safely, the JVM must make sure no other thread is in the middle of an access, so closing a shared arena involves more coordination than closing a confined one.auto, other thread: read 3andauto, close(): UnsupportedOperationException: an auto arena is shared between threads, but nobody can close it. The garbage collector frees its memory at some point after the last segment becomes unreachable, so the time of release is unknown.global: read 0, scope alive true: the global arena is shared and lives until the JVM exits. It reads 0 because memory allocated by an arena is zero-filled.
The choice for a model loader follows from these rules, by elimination:
- The mapped file must stay valid while any thread runs the forward pass, and the later lesson on quantized kernels splits matrix rows across threads. A confined arena is ruled out.
- The file should be unmapped at a known time, for example when
loop runswitches models. An auto arena is ruled out as well, because a 4 GB mapping released "eventually" keeps 4 GB of address space and page cache busy. The global arena is ruled out for the same reason. - A shared arena, closed by
GgufModel.close(), satisfies both rules.
Short-lived scratch memory used by one thread fits a confined arena.
Example 10: files over 2 GB: long offsets, and why ByteBuffer is not enough
ByteBuffer and MappedByteBuffer are indexed by int, and a Java int cannot go above Integer.MAX_VALUE, which is 2,147,483,647 (about 2 GiB). A 7B model in Q4_K_M is about 4.4 GB, and a single tensor of a large model can pass 2 GiB. A loader built on ByteBuffer therefore cannot hold such a file in one piece.
This program shows the problem without needing a big model. It creates a 3 GiB sparse file (a file whose unwritten regions take no disk space), writes four bytes at 2.5 GiB, and maps the whole file both ways. Compare what each mapping style does with the 3 GiB size.
import java.io.RandomAccessFile;
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
public class BigFile {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
public static void main(String[] args) throws Exception {
Path path = Files.createTempFile("big", ".gguf");
long size = 3L << 30, at = 2_684_354_560L; // a 3 GiB file; a value at 2.5 GiB
try (RandomAccessFile f = new RandomAccessFile(path.toFile(), "rw")) {
f.setLength(size); // sparse: no disk blocks for the zeros
f.seek(at);
f.write(new byte[] {0x78, 0x56, 0x34, 0x12});
}
System.out.println("file size " + Files.size(path) + ", Integer.MAX_VALUE " + Integer.MAX_VALUE);
try (FileChannel ch = FileChannel.open(path); Arena arena = Arena.ofConfined()) {
try {
ch.map(FileChannel.MapMode.READ_ONLY, 0, size); // the MappedByteBuffer variant
} catch (IllegalArgumentException e) {
System.out.println("map to MappedByteBuffer: " + e.getMessage());
}
MemorySegment file = ch.map(FileChannel.MapMode.READ_ONLY, 0, size, arena);
System.out.println("map to MemorySegment: byteSize " + file.byteSize());
System.out.printf("u32 at offset %d: 0x%08X%n", at, file.get(U32, at));
System.out.printf("u32 at offset %d: 0x%08X%n", size - 4, file.get(U32, size - 4));
try {
file.asByteBuffer();
} catch (IllegalStateException e) {
System.out.println("asByteBuffer: " + e.getMessage());
}
System.out.println("a 1 GiB slice as ByteBuffer: capacity " + file.asSlice(at - (1L << 30), 1L << 30).asByteBuffer().capacity());
} finally {
Files.delete(path);
}
}
}file size 3221225472, Integer.MAX_VALUE 2147483647
map to MappedByteBuffer: Size exceeds Integer.MAX_VALUE
map to MemorySegment: byteSize 3221225472
u32 at offset 2684354560: 0x12345678
u32 at offset 3221225468: 0x00000000
asByteBuffer: Segment is too large to wrap as ByteBuffer. Size: 3221225472
a 1 GiB slice as ByteBuffer: capacity 1073741824The file is 3,221,225,472 bytes, larger than Integer.MAX_VALUE. RandomAccessFile.setLength extends it without writing zeros, so the test needs almost no disk space, and the mapping reads the untouched regions as zeros.
ch.map(READ_ONLY, 0, size), the variant that returns a MappedByteBuffer, fails with Size exceeds Integer.MAX_VALUE. A ByteBuffer-based loader must therefore map a large model in several pieces of at most 2 GiB. It must also handle values and tensors that cross the boundary between pieces.
ch.map(READ_ONLY, 0, size, arena) maps the whole file as one segment with byteSize 3,221,225,472. The read at offset 2,684,354,560 (2.5 GiB) returns 0x12345678, the four bytes 78 56 34 12 read little-endian. The read at the last four bytes returns 0 from the sparse region. All offsets are long, so the code is the same at byte 10 and at byte 3,000,000,000.
The last two lines show the boundary between the two APIs. asByteBuffer() on the whole segment fails, because a ByteBuffer cannot describe 3 GiB. A slice of 1 GiB converts without trouble (capacity 1,073,741,824), which helps when a library accepts only ByteBuffer. The engine never needs that conversion, because it reads every tensor through segments.
Example 11: validating a file before trusting it
A loader runs on files from the internet. A damaged or hostile file must fail with a clear message, not read outside the file or produce wrong numbers. Checking the file first costs a few comparisons. Skipping the checks costs hours of debugging when the failure shows up deep inside the forward pass.
This program builds a valid one-tensor file and five broken variants. It checks each one in the order a loader should: size, magic, version, offsets, and the end of each tensor against the file size. Look at which check rejects each broken file.
import java.io.ByteArrayOutputStream;
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
public class Validate {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static byte[] le(long v, int n) { byte[] b = new byte[n]; for (int i = 0; i < n; i++) b[i] = (byte) (v >>> (8 * i)); return b; }
// A file with no metadata and one F32 tensor [8] at the given offset, followed by dataBytes bytes of data.
static byte[] file(String magic, int version, long offset, int dataBytes) {
ByteArrayOutputStream o = new ByteArrayOutputStream();
o.writeBytes(magic.getBytes(StandardCharsets.US_ASCII)); o.writeBytes(le(version, 4)); o.writeBytes(le(1, 8)); o.writeBytes(le(0, 8));
o.writeBytes(le(1, 8)); o.writeBytes("w".getBytes(StandardCharsets.US_ASCII)); o.writeBytes(le(1, 4)); o.writeBytes(le(8, 8));
o.writeBytes(le(0, 4)); o.writeBytes(le(offset, 8));
while (o.size() % 32 != 0) o.write(0);
o.writeBytes(new byte[dataBytes]);
return o.toByteArray();
}
static String check(MemorySegment s) {
if (s.byteSize() < 24) return "rejected: shorter than the 24-byte header";
if (s.get(U32, 0) != 0x46554747) return String.format("rejected: magic 0x%08X is not GGUF", s.get(U32, 0));
int version = s.get(U32, 4);
if (version != 3) return "rejected: version " + version + " (this loader reads 3)";
long pos = 24, nameLen = s.get(U64, pos); // the one tensor info
pos += 8 + nameLen;
int nDims = s.get(U32, pos);
long elements = 1;
for (int d = 0; d < nDims; d++) elements *= s.get(U64, pos + 4 + 8L * d);
pos += 4 + 8L * nDims;
long offset = s.get(U64, pos + 4), bytes = elements * 4; // type 0 = F32, 4 bytes each
long dataStart = (pos + 12 + 31) / 32 * 32;
if (offset != 0) return "rejected: first tensor offset " + offset + ", expected 0";
if (dataStart + offset + bytes > s.byteSize())
return "rejected: tensor needs bytes up to " + (dataStart + offset + bytes) + ", file has " + s.byteSize();
return "ok: tensor at " + (dataStart + offset) + ", " + bytes + " bytes";
}
public static void main(String[] args) {
byte[][] files = {file("GGUF", 3, 0, 32), file("GGUF", 3, 0, 16), file("GGML", 3, 0, 32),
file("GGUF", 2, 0, 32), file("GGUF", 3, 64, 32), Arrays.copyOf(file("GGUF", 3, 0, 32), 20)};
String[] labels = {"good file", "truncated data", "wrong magic", "version 2", "wrong offset", "20 bytes"};
try (Arena arena = Arena.ofConfined()) {
for (int i = 0; i < files.length; i++) {
MemorySegment s = arena.allocate(files[i].length);
MemorySegment.copy(files[i], 0, s, ValueLayout.JAVA_BYTE, 0, files[i].length);
System.out.printf("%-15s %s%n", labels[i], check(s));
}
}
}
}good file ok: tensor at 64, 32 bytes
truncated data rejected: tensor needs bytes up to 96, file has 80
wrong magic rejected: magic 0x4C4D4747 is not GGUF
version 2 rejected: version 2 (this loader reads 3)
wrong offset rejected: first tensor offset 64, expected 0
20 bytes rejected: shorter than the 24-byte headerThe good file has its single F32 tensor of 8 values (32 bytes) at byte 64, and the file has those bytes, so the check prints ok. Each broken variant fails at the first check that catches it:
truncated data: the tensor needs bytes up to 96, but the file has 80. Without this check, the read would hitIndexOutOfBoundsExceptiondeep inside the forward pass, far from the cause. llama.cpp performs the same check at load time and fails with "tensor '%s' data is not within the file bounds, model is corrupted or incomplete" (src/llama-model-loader.h).wrong magic:GGMLinstead ofGGUFreads as0x4C4D4747. The old pre-GGUF formats of llama.cpp (ggml,ggjt) start with other magics, and this check rejects them with a clear message.version 2: the format changed between versions (v1 used 32-bit counts), so a reader written for v3 rejects other versions instead of misreading them. The GGUF lesson covers the version differences.wrong offset: the first tensor claims offset 64. The rule comes fromgguf.cppat the pinned commit (shown in The GGUF Format): each tensor's offset must equal the padded sizes of the tensors before it, so the first one must be at 0. A reader that skips this check would read the wrong bytes as weights.20 bytes: shorter than the 24-byte header, so not even the counts can be read.
The GgufModel class in Example 13 performs the magic, version, and size checks when it opens a file. Every read also stays bounds-checked by the segment. The worst outcome of a missed check is therefore an exception, never a read of foreign memory.
Example 12: the loader on the tiny random model
The earlier examples used tiny hand-made files. This one runs the same steps on a real file: the tiny random-weight model from the reference build. It has the Llama architecture, 4 layers, a 256-wide residual stream (the vector that carries each token's information from layer to layer), the 151,936-token Qwen2 vocabulary, and all tensors in F32. Its weights are random, so its text output is meaningless, but its file layout is the same as that of a real model.
The program needs the 328 MB file, so it is marked java,ignore, which tells the example checker to skip it. It was run with java InspectModel.java tiny-random-qwen2vocab-f32.gguf. It skips arrays and reports their length only. Read the code for the same cursor, record, and view pieces from Examples 3 to 6.
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
public class InspectModel {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfFloat F32 = ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
record TensorInfo(String name, long[] dims, int type, long offset) {}
static final class Cursor {
final MemorySegment s;
long pos;
Cursor(MemorySegment s) { this.s = s; }
int u32() { int v = s.get(U32, pos); pos += 4; return v; }
long u64() { long v = s.get(U64, pos); pos += 8; return v; }
String str() { long n = u64(); String v = new String(s.asSlice(pos, n).toArray(ValueLayout.JAVA_BYTE), StandardCharsets.UTF_8); pos += n; return v; }
Object value(int type) {
return switch (type) {
case 4 -> u32() & 0xFFFFFFFFL;
case 5 -> u32();
case 6 -> { float f = s.get(F32, pos); pos += 4; yield f; }
case 7 -> s.get(ValueLayout.JAVA_BYTE, pos++) != 0;
case 8 -> str();
case 9 -> { // arrays: keep only the element count
int elementType = u32();
long n = u64();
for (long i = 0; i < n; i++) value(elementType);
yield "array of " + n;
}
default -> throw new IllegalStateException("value type " + type);
};
}
}
public static void main(String[] args) throws Exception {
try (Arena arena = Arena.ofShared(); FileChannel ch = FileChannel.open(Path.of(args[0]))) {
MemorySegment file = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena);
Cursor c = new Cursor(file);
c.pos = 8;
long nTensors = c.u64(), nKv = c.u64();
Map<String, Object> kv = new LinkedHashMap<>();
for (long i = 0; i < nKv; i++) { String key = c.str(); kv.put(key, c.value(c.u32())); }
List<TensorInfo> infos = new ArrayList<>();
for (long i = 0; i < nTensors; i++) {
String name = c.str();
long[] dims = new long[c.u32()];
for (int d = 0; d < dims.length; d++) dims[d] = c.u64();
infos.add(new TensorInfo(name, dims, c.u32(), c.u64()));
}
long dataStart = (c.pos + 31) / 32 * 32;
System.out.println("file " + file.byteSize() + " bytes, " + nKv + " keys, " + nTensors + " tensors, metadata ends at " + c.pos + ", data at " + dataStart);
kv.forEach((k, v) -> { if (!k.equals("tokenizer.chat_template")) System.out.println(" " + k + " = " + v); });
TensorInfo embd = infos.get(0);
MemorySegment view = file.asSlice(dataStart + embd.offset(), embd.dims()[0] * embd.dims()[1] * 4);
long row = 9707 * embd.dims()[0]; // token 9707 is "Hello" in this vocabulary
System.out.printf(Locale.ROOT, "%s %s: row 9707 starts %.4f %.4f %.4f, ends %.4f %.4f %.4f%n", embd.name(),
java.util.Arrays.toString(embd.dims()), view.getAtIndex(F32, row), view.getAtIndex(F32, row + 1), view.getAtIndex(F32, row + 2),
view.getAtIndex(F32, row + 253), view.getAtIndex(F32, row + 254), view.getAtIndex(F32, row + 255));
TensorInfo last = infos.get(infos.size() - 1);
System.out.println("last tensor " + last.name() + " ends at byte " + (dataStart + last.offset() + last.dims()[0] * last.dims()[1] * 4));
}
}
}Output of the run on the tiny random model (the values are fixed by the file, not by the machine):
file 328377376 bytes, 21 keys, 39 tensors, metadata ends at 5931011, data at 5931040
general.architecture = llama
general.name = tiny-random-llama
llama.context_length = 4096
llama.embedding_length = 256
llama.block_count = 4
llama.feed_forward_length = 704
llama.attention.head_count = 8
llama.attention.head_count_kv = 2
llama.rope.dimension_count = 32
llama.attention.layer_norm_rms_epsilon = 1.0E-5
llama.rope.freq_base = 500000.0
general.file_type = 0
tokenizer.ggml.model = gpt2
tokenizer.ggml.pre = qwen2
tokenizer.ggml.tokens = array of 151936
tokenizer.ggml.token_type = array of 151936
tokenizer.ggml.merges = array of 151387
tokenizer.ggml.eos_token_id = 151643
tokenizer.ggml.padding_token_id = 151643
tokenizer.ggml.bos_token_id = 151643
token_embd.weight [256, 151936]: row 9707 starts 0.0296 -0.0015 -0.0031, ends -0.0259 0.0172 -0.0251
last tensor output.weight ends at byte 328377376The first line gives the file layout: 328,377,376 bytes, 21 keys, 39 tensors, metadata ending at byte 5,931,011, and the data section at 5,931,040 (the next multiple of 32). gguf-dump --data-offset from gguf-py 0.19.0 prints the same 5,931,040 for this file. Almost all of the 5.9 MB of metadata is the vocabulary: 151,936 token strings, 151,936 token types, and 151,387 merge strings. The tokenizer.chat_template string is skipped in the printout because it is long.
The key-value lines are the model's configuration:
llama.embedding_length = 256is the width of the residual stream.llama.block_count = 4is the number of layers.llama.attention.head_count = 8withhead_count_kv = 2means grouped-query attention with 4 query heads per key/value head.llama.rope.freq_base = 500000.0is the RoPE base.tokenizer.ggml.pre = qwen2selects the pre-tokenizer regex, and the three token ids name the end-of-text token.
A Real Tokenizer from GGUF Metadata turns the tokenizer.ggml.* entries into a tokenizer. Lesson 51 turns the llama.* entries into a model configuration.
The embedding line reads row 9707 of token_embd.weight through a view. In the Qwen2 vocabulary, token 9707 is Hello. The six values 0.0296 -0.0015 -0.0031 ... -0.0259 0.0172 -0.0251 are exactly the first row that llama-eval-callback prints for the tensor embd when it runs this model on the prompt Hello world (lesson 51, Example 11 shows that tool). The Java loader and llama.cpp therefore find the same bytes for the same tensor.
The last line checks the end of the data. output.weight (256 by 151,936 F32 values, 155,582,464 bytes) ends at byte 328,377,376, which is exactly the file size. Every byte of the file is accounted for by the header, metadata, padding, and 39 tensors.
The run reads 5.9 MB of metadata and 24 bytes from one tensor. The other 322 MB of weights are mapped but never touched, so the operating system never reads them from disk. A loader that reads metadata through a mapping starts in milliseconds even for a 40 GB file.
Example 13: Project 16, the GgufModel class
Project 16 collects Examples 3 to 11 into one reusable class, GgufModel. It does the following:
- It opens a file with a shared arena.
- It parses all 13 GGUF value types.
- It keeps tensor infos as records.
- It checks magic, version, and tensor sizes.
- It hands out zero-copy views with
tensorData. - It dequantizes F32, F16, and Q8_0 tensors to
float[]withfloats. To dequantize means to turn stored quantized values back into ordinary floats.
A nested Writer (the writer from The GGUF Format, condensed) produces test files. The main method writes a file with three copies of the same 64 values in three types, opens it, compares the values, and then opens a truncated copy. In the output, compare the byte sizes and the maximum error of the three tensors.
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public final class GgufModel implements AutoCloseable {
static final ValueLayout.OfInt U32 = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfLong U64 = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfShort U16 = ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfFloat F32 = ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final ValueLayout.OfDouble F64 = ValueLayout.JAVA_DOUBLE_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
static final int GGML_F32 = 0, GGML_F16 = 1, GGML_Q8_0 = 8;
public record TensorInfo(String name, long[] dims, int type, long offset) {
public long elements() { long n = 1; for (long d : dims) n *= d; return n; }
public long byteSize() {
return switch (type) {
case GGML_F32 -> elements() * 4;
case GGML_F16 -> elements() * 2;
case GGML_Q8_0 -> elements() / 32 * 34;
default -> throw new IllegalArgumentException("unsupported ggml type " + type + " for " + name);
};
}
}
// Reads little-endian values one after another from a segment.
static final class Cursor {
final MemorySegment s;
long pos;
Cursor(MemorySegment s, long pos) { this.s = s; this.pos = pos; }
int u32() { int v = s.get(U32, pos); pos += 4; return v; }
long u64() { long v = s.get(U64, pos); pos += 8; return v; }
String str() {
long n = u64();
byte[] b = s.asSlice(pos, n).toArray(ValueLayout.JAVA_BYTE);
pos += n;
return new String(b, StandardCharsets.UTF_8);
}
Object value(int type) {
return switch (type) {
case 0 -> (int) s.get(ValueLayout.JAVA_BYTE, pos++) & 0xFF; // UINT8
case 1 -> (int) s.get(ValueLayout.JAVA_BYTE, pos++); // INT8
case 2 -> { int v = s.get(U16, pos) & 0xFFFF; pos += 2; yield v; } // UINT16
case 3 -> { int v = s.get(U16, pos); pos += 2; yield v; } // INT16
case 4 -> u32() & 0xFFFFFFFFL; // UINT32
case 5 -> u32(); // INT32
case 6 -> { float v = s.get(F32, pos); pos += 4; yield v; } // FLOAT32
case 7 -> s.get(ValueLayout.JAVA_BYTE, pos++) != 0; // BOOL
case 8 -> str(); // STRING
case 9 -> array(); // ARRAY
case 10, 11 -> u64(); // UINT64, INT64
case 12 -> { double v = s.get(F64, pos); pos += 8; yield v; } // FLOAT64
default -> throw new IllegalStateException("bad value type " + type + " at " + pos);
};
}
Object array() {
int type = u32();
int n = Math.toIntExact(u64());
if (type == 5) { int[] a = new int[n]; for (int i = 0; i < n; i++) a[i] = u32(); return a; }
if (type == 6) { float[] a = new float[n]; for (int i = 0; i < n; i++) { a[i] = s.get(F32, pos); pos += 4; } return a; }
List<Object> list = new ArrayList<>(n);
for (int i = 0; i < n; i++) list.add(value(type));
return list;
}
}
private final Arena arena;
private final MemorySegment file;
private final Map<String, Object> metadata = new LinkedHashMap<>();
private final Map<String, TensorInfo> tensors = new LinkedHashMap<>();
private final long dataStart;
private final int version;
private GgufModel(Arena arena, MemorySegment file) {
this.arena = arena;
this.file = file;
Cursor c = new Cursor(file, 0);
if (file.byteSize() < 24 || c.u32() != 0x46554747) throw new IllegalArgumentException("not a GGUF file");
version = c.u32();
if (version != 3) throw new IllegalArgumentException("unsupported GGUF version " + version);
long tensorCount = c.u64(), kvCount = c.u64();
for (long i = 0; i < kvCount; i++) {
String key = c.str();
metadata.put(key, c.value(c.u32()));
}
for (long i = 0; i < tensorCount; i++) {
String name = c.str();
long[] dims = new long[c.u32()];
for (int d = 0; d < dims.length; d++) dims[d] = c.u64();
tensors.put(name, new TensorInfo(name, dims, c.u32(), c.u64()));
}
long alignment = metadata.get("general.alignment") instanceof Long a ? a : 32;
dataStart = (c.pos + alignment - 1) / alignment * alignment;
for (TensorInfo t : tensors.values())
if (dataStart + t.offset() + t.byteSize() > file.byteSize()) throw new IllegalArgumentException("tensor " + t.name() + " runs past the end of the file");
}
public static GgufModel open(Path path) throws IOException {
Arena arena = Arena.ofShared();
try (FileChannel ch = FileChannel.open(path, StandardOpenOption.READ)) {
return new GgufModel(arena, ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena));
} catch (RuntimeException | IOException e) {
arena.close();
throw e;
}
}
public int version() { return version; }
public long dataStart() { return dataStart; }
public Map<String, Object> metadata() { return metadata; }
public Map<String, TensorInfo> tensors() { return tensors; }
public String string(String key) { return (String) metadata.get(key); }
public long integer(String key) { return ((Number) metadata.get(key)).longValue(); }
public float f32(String key) { return ((Number) metadata.get(key)).floatValue(); }
// A view of the tensor's bytes inside the mapped file: no copy.
public MemorySegment tensorData(String name) {
TensorInfo t = tensors.get(name);
if (t == null) throw new IllegalArgumentException("no tensor " + name);
return file.asSlice(dataStart + t.offset(), t.byteSize());
}
// Dequantizes a whole tensor to float[] (F32, F16, Q8_0).
public float[] floats(String name) {
TensorInfo t = tensors.get(name);
MemorySegment s = tensorData(name);
float[] out = new float[Math.toIntExact(t.elements())];
switch (t.type()) {
case GGML_F32 -> MemorySegment.copy(s, F32, 0, out, 0, out.length);
case GGML_F16 -> { for (int i = 0; i < out.length; i++) out[i] = Float.float16ToFloat(s.get(U16, 2L * i)); }
case GGML_Q8_0 -> {
for (int b = 0; b < out.length / 32; b++) {
long base = 34L * b;
float d = Float.float16ToFloat(s.get(U16, base));
for (int j = 0; j < 32; j++) out[b * 32 + j] = d * s.get(ValueLayout.JAVA_BYTE, base + 2 + j);
}
}
default -> throw new IllegalArgumentException("unsupported type " + t.type());
}
return out;
}
@Override public void close() { arena.close(); }
// A small GGUF writer for tests (the writer from the GGUF format lesson, condensed).
public static final class Writer {
private final ByteArrayOutputStream kv = new ByteArrayOutputStream();
private final List<Object[]> tensors = new ArrayList<>();
private int kvCount;
private static byte[] le(long v, int n) { byte[] b = new byte[n]; for (int i = 0; i < n; i++) b[i] = (byte) (v >>> (8 * i)); return b; }
private static void put(ByteArrayOutputStream o, byte[] b) { o.write(b, 0, b.length); }
private static void str(ByteArrayOutputStream o, String s) { byte[] b = s.getBytes(StandardCharsets.UTF_8); put(o, le(b.length, 8)); put(o, b); }
public Writer string(String key, String v) { str(kv, key); put(kv, le(8, 4)); str(kv, v); kvCount++; return this; }
public Writer u32(String key, long v) { str(kv, key); put(kv, le(4, 4)); put(kv, le(v, 4)); kvCount++; return this; }
public Writer f32(String key, float v) { str(kv, key); put(kv, le(6, 4)); put(kv, le(Float.floatToIntBits(v), 4)); kvCount++; return this; }
public Writer bool(String key, boolean v) { str(kv, key); put(kv, le(7, 4)); kv.write(v ? 1 : 0); kvCount++; return this; }
public Writer strings(String key, List<String> v) {
str(kv, key); put(kv, le(9, 4)); put(kv, le(8, 4)); put(kv, le(v.size(), 8));
for (String s : v) str(kv, s);
kvCount++; return this;
}
public Writer ints(String key, int[] v) {
str(kv, key); put(kv, le(9, 4)); put(kv, le(5, 4)); put(kv, le(v.length, 8));
for (int x : v) put(kv, le(x, 4));
kvCount++; return this;
}
public Writer tensor(String name, int type, byte[] data, long... dims) { tensors.add(new Object[] {name, type, data, dims}); return this; }
public byte[] bytes() {
ByteArrayOutputStream f = new ByteArrayOutputStream();
put(f, "GGUF".getBytes(StandardCharsets.US_ASCII)); put(f, le(3, 4)); put(f, le(tensors.size(), 8)); put(f, le(kvCount, 8));
put(f, kv.toByteArray());
long offset = 0;
for (Object[] t : tensors) {
long[] dims = (long[]) t[3];
str(f, (String) t[0]); put(f, le(dims.length, 4));
for (long d : dims) put(f, le(d, 8));
put(f, le((Integer) t[1], 4)); put(f, le(offset, 8));
offset += (((byte[]) t[2]).length + 31) / 32 * 32;
}
while (f.size() % 32 != 0) f.write(0);
for (Object[] t : tensors) { put(f, (byte[]) t[2]); while (f.size() % 32 != 0) f.write(0); }
return f.toByteArray();
}
public static byte[] f32(float[] x) { byte[] b = new byte[x.length * 4]; for (int i = 0; i < x.length; i++) System.arraycopy(le(Float.floatToIntBits(x[i]), 4), 0, b, 4 * i, 4); return b; }
public static byte[] f16(float[] x) { byte[] b = new byte[x.length * 2]; for (int i = 0; i < x.length; i++) System.arraycopy(le(Float.floatToFloat16(x[i]), 2), 0, b, 2 * i, 2); return b; }
static int roundf(float v) { return (int) Math.copySign(Math.floor(Math.abs((double) v) + 0.5), v); }
public static byte[] q8_0(float[] x) {
byte[] out = new byte[x.length / 32 * 34];
for (int blk = 0; blk < x.length / 32; blk++) {
float amax = 0;
for (int j = 0; j < 32; j++) amax = Math.max(amax, Math.abs(x[blk * 32 + j]));
float d = amax / 127, id = d != 0 ? 1 / d : 0;
System.arraycopy(le(Float.floatToFloat16(d), 2), 0, out, blk * 34, 2);
for (int j = 0; j < 32; j++) out[blk * 34 + 2 + j] = (byte) roundf(x[blk * 32 + j] * id);
}
return out;
}
}
public static void main(String[] args) throws Exception {
float[] w = new float[64];
for (int i = 0; i < w.length; i++) w[i] = (float) Math.sin(i * 0.37) * 0.5f;
byte[] bytes = new Writer()
.string("general.architecture", "llama").u32("llama.embedding_length", 64).f32("llama.rope.freq_base", 10000f)
.strings("tokenizer.ggml.tokens", List.of("<s>", "a", "b"))
.tensor("w.f32", GGML_F32, Writer.f32(w), 32, 2)
.tensor("w.f16", GGML_F16, Writer.f16(w), 32, 2)
.tensor("w.q8_0", GGML_Q8_0, Writer.q8_0(w), 32, 2)
.bytes();
Path path = Files.createTempFile("model", ".gguf");
Files.write(path, bytes);
try (GgufModel m = GgufModel.open(path)) {
System.out.println("version " + m.version() + ", " + m.metadata().size() + " keys, " + m.tensors().size() + " tensors, data at " + m.dataStart() + ", file " + bytes.length + " bytes");
System.out.println("architecture " + m.string("general.architecture") + ", embedding " + m.integer("llama.embedding_length") + ", freq_base " + m.f32("llama.rope.freq_base"));
System.out.println("tokens " + m.metadata().get("tokenizer.ggml.tokens"));
for (TensorInfo t : m.tensors().values()) {
float[] got = m.floats(t.name());
double maxErr = 0;
for (int i = 0; i < w.length; i++) maxErr = Math.max(maxErr, Math.abs(got[i] - w[i]));
System.out.printf(java.util.Locale.ROOT, "%-7s dims %-7s offset %3d, %3d bytes, max error %.6f%n", t.name(), Arrays.toString(t.dims()), t.offset(), t.byteSize(), maxErr);
}
Files.write(path, Arrays.copyOf(bytes, bytes.length - 40)); // cut off the end of the last tensor
try (GgufModel cut = GgufModel.open(path)) {
System.out.println("opened a truncated file");
} catch (IllegalArgumentException e) {
System.out.println("truncated file: " + e.getMessage());
}
} finally {
Files.delete(path);
}
}
}version 3, 4 keys, 3 tensors, data at 384, file 864 bytes
architecture llama, embedding 64, freq_base 10000.0
tokens [<s>, a, b]
w.f32 dims [32, 2] offset 0, 256 bytes, max error 0.000000
w.f16 dims [32, 2] offset 256, 128 bytes, max error 0.000121
w.q8_0 dims [32, 2] offset 384, 68 bytes, max error 0.002091
truncated file: tensor w.q8_0 runs past the end of the fileThe first line shows the parsed header: version 3, 4 keys, 3 tensors, the data section at byte 384, and an 864-byte file. The second and third lines read metadata through the typed helpers: string, integer (which widens the UINT32 to long), and f32. The token array comes back as a List.
The three tensor lines decode the same 64 input values from each type:
w.f32at offset 0 is 256 bytes (64 times 4) with error 0: F32 stores the values exactly.w.f16at offset 256 is 128 bytes with a maximum error of 0.000121, the FP16 rounding error for values up to 0.5.w.q8_0at offset 384 is 68 bytes (2 blocks of 34) with a maximum error of 0.002091, about half of a block scale of 0.0039.
The sizes show the trade-off of each format: 256, 128, and 68 bytes for the same 64 values, in exchange for errors of 0, 0.000121, and 0.002091.
Opening the same file with gguf-py's GGUFReader gives the same data offset (384) and the same maximum errors after gguf.quants.dequantize. The class and the reference Python reader therefore agree on the bytes.
The last line is the size check at work. The truncated copy lost its last 40 bytes, which belong to the Q8_0 tensor, and open rejects the file with tensor w.q8_0 runs past the end of the file. The exception happens inside open, so the class closes the arena before rethrowing. A failed open leaves no mapping behind.
The next lessons use the class like this. Save the program as GgufModel.java. Since Java 22 (JEP 458), the source launcher compiles other classes from .java files in the same directory on demand. A program Generate.java next to GgufModel.java can therefore call GgufModel.open(path) and run with java Generate.java, without a build tool. Lesson 50 adds GgufTokenizer.java and lesson 51 adds Llama.java the same way. The methods the engine uses are:
| Method | Returns |
|---|---|
open(Path) |
a model backed by a shared arena; close() unmaps the file |
metadata() |
all key-value pairs; arrays of INT32 and FLOAT32 as int[] and float[], other arrays as List |
tensors() |
TensorInfo records in file order |
tensorData(name) |
a read-only view of the tensor's bytes |
floats(name) |
the tensor dequantized to a new float[] (F32, F16, Q8_0) |
Interview points
- The FFM API (final in Java 22) gives a Java program off-heap memory with 64-bit offsets, explicit byte order, bounds checks on every access, and deterministic release through arenas.
ValueLayout.JAVA_INT_UNALIGNED.withOrder(LITTLE_ENDIAN)is the right layout for GGUF metadata, because the data is little-endian and unaligned. The alignedJAVA_INTthrows on misaligned offsets.FileChannel.map(mode, offset, size, arena)maps a whole file as one segment. TheMappedByteBuffervariant is limited to 2 GiB and cannot be unmapped on demand.- A GGUF reader needs a cursor because strings, arrays, and tensor infos have variable length. The header is the only fixed-position part.
- String lengths in GGUF are byte counts, not character counts. Fixed-size arrays can be bulk-copied with
MemorySegment.copy. - The data section starts at the end of the tensor infos rounded up to
general.alignment(default 32). A wrong alignment shifts every tensor and loads silently wrong weights. asSlicecreates a bounds-checked view without copying, andtoArraycopies. A loader keeps views and copies only for arithmetic.- Confined arenas are single-threaded. Shared arenas allow multi-threaded kernels and explicit unmapping. Auto arenas release memory at an unknown time.
- Accessing a segment after its arena closes throws
IllegalStateExceptioninstead of crashing, unlike a dangling C pointer. - A loader validates magic, version, offsets, and that every tensor ends inside the file before the forward pass touches any weight.
- Mapping touches only the pages that are read, so reading the metadata of a multi-gigabyte file costs milliseconds.
Summary
0x46554747written little-endian is the bytes47 47 55 46, the textGGUF.- The bytes
03 00 00 00read as big-endian give 50,331,648 instead of 3. - A GGUF string
世界takes 14 bytes: 8 for the length and 6 for UTF-8. - A Q8_0 tensor of 64 by 5 values is 340 bytes, padded to 352 at alignment 32 and 384 at alignment 64.
- Element (row, col) of a tensor with dims
[ne0, ne1]is at indexrow * ne0 + col. - Q8_0 element
iis in the 34-byte blocki / 32, quanti % 32, multiplied by the block's FP16 scale. - F16 stores 12 values in 24 bytes, and 0.05 becomes 0.049988.
- A confined arena throws
WrongThreadExceptionon other threads, a closed arena throwsIllegalStateException: Already closed, and an auto arena cannot be closed. - Mapping a 3 GiB file as a
MappedByteBufferfails withSize exceeds Integer.MAX_VALUE, while the segment mapping succeeds. - The tiny random model has 21 keys, 39 tensors, and its data section at byte 5,931,040, as
gguf-dumpreports. - Row 9707 of its
token_embd.weightstarts0.0296 -0.0015 -0.0031, the same valuesllama-eval-callbackprints. GgufModelreads F32, F16, and Q8_0 and agrees with gguf-py on the data offset and dequantized values.- Since Java 22,
java Main.javacompiles classes from other.javafiles in the same directory, so the engine's classes need no build tool.