llama.cpp Architecture

llama.cpp is a program and a library that runs a language model on your own machine. It reads a model file, turns your text into numbers, does the model's arithmetic, and turns the result back into text. Programs such as llama-server, which the rest of this course uses, are thin layers on top of that engine. Knowing how the engine is organized makes its logs readable and its flags predictable.

This lesson follows a model from the GGUF file to the first generated token. A GGUF file is the single file that holds a model's weights and settings (The GGUF Format reads it byte by byte). Along the way the lesson explains every line of the startup log and shows where each megabyte of memory goes. It also says which program to use for which job.

After the lesson, a reader can read a llama.cpp startup log without guessing, estimate the memory a run needs, pick the right binary for a task, and find the source code that does each step.

The table lists the parts of the engine. Each part is explained in the examples that follow.

Every fact in this lesson comes from the llama.cpp source at commit 0c1e57098 (ggml 0.25.3) and from runs of the programs built from it. The runs use the course's tiny test model. It has the Llama architecture, 4 layers, 256 numbers per token embedding, 8 query heads, 2 key-value heads, the Qwen2 tokenizer with 151,936 tokens, and 80.6 million parameters (the numbers a model learns during training). Its weights are stored as F32, which means 32-bit floating-point numbers of 4 bytes each, so the file is about 80.6 million times 4 bytes, or 307 MiB (a MiB is 1,048,576 bytes).

The model has random weights, so the text it generates is meaningless. Its logs, buffer sizes, and timings are real, and those are what this lesson studies. The Java programs run as shown and print the output below them.

Example 1: the repository layout

Before looking at any mechanism, it helps to know where the code lives. The commands below list the top-level directories, the tools, and the ggml backend directories of the pinned checkout. A backend is the code that runs the math on one kind of hardware (Example 5 explains it).

cd llama.cpp
ls -d */
ls tools
ls ggml/src | grep '^ggml-' | grep -v '\.'

The first command lists every directory at the top of the repository. The second lists the programs under tools/. The third lists the backend folders under ggml/src/ and keeps only the names, not the files. Output from the pinned checkout (the build directory removed from the first line):

app/ benches/ ci/ cmake/ common/ conversion/ docs/ examples/ ggml/ gguf-py/ grammars/ include/ licenses/ media/ models/ pocs/ requirements/ scripts/ skills/ src/ tests/ tools/ vendor/
CMakeLists.txt batched-bench cli completion cvector-generator export-lora fit-params gguf-split imatrix llama-bench mtmd perplexity quantize results rpc server tokenize tts tuning ui
ggml-blas ggml-cann ggml-cpu ggml-cuda ggml-et ggml-hexagon ggml-hip ggml-metal ggml-musa ggml-opencl ggml-openvino ggml-rpc ggml-sycl ggml-virtgpu ggml-vulkan ggml-webgpu ggml-zdnn ggml-zendnn

The first line shows the main folders: ggml, src, include, common, tools, examples, and some folders for Python scripts, tests, and documentation. The second line shows the programs that can be built. The third line shows one folder per kind of hardware: ggml-cuda for NVIDIA GPUs, ggml-metal for Apple GPUs, ggml-vulkan for GPUs with a Vulkan driver, and so on. The CPU code lives in ggml-cpu.

The project has three layers, and each depends only on the one below it. The tools use libllama, and libllama uses ggml. Nothing in ggml knows about language models, and nothing in libllama knows about command-line flags.

The split tells a reader where to look when a question comes up:

• A question about a tensor operation, such as how a matrix multiply works on a CPU, is answered in ggml/.
• A question about a model, such as where the KV cache is allocated, is answered in src/.
• A question about a flag, such as what -ub defaults to, is answered in common/arg.cpp, which is also where the --help text comes from.

Example 2: ggml tensors, sizes and byte strides

Every number a model uses is stored in a tensor: a grid of numbers with up to 4 dimensions. A vector is a tensor with one dimension, and a matrix (rows and columns) has two. Computer memory is a single long row of bytes, so a grid has to be laid out in that row. The engine then needs a way to find the address of any element.

Take a matrix with 3 rows of 4 floats each. If the rows are stored one after another, a float is 4 bytes, so a row takes 16 bytes and the whole matrix takes 48 bytes.

byte offset   0  4  8 12 | 16 20 24 28 | 32 36 40 44
value         0  1  2  3 |  4  5  6  7 |  8  9 10 11
              row 0      | row 1       | row 2

element (column 2, row 1): 2 * 4 + 1 * 16 = 24  ->  the value 6

To find an element, multiply each index by "how many bytes to jump to move one step in that direction" and add the results. That jump size is called a stride. Moving one column right jumps 4 bytes. Moving one row down jumps 16 bytes.

ggml stores this in the ggml_tensor struct as two arrays. ne holds the number of elements in each dimension. nb holds the stride in bytes for each dimension. ggml allows at most 4 dimensions (GGML_MAX_DIMS is 4 in ggml.h). The comment in ggml.h gives the rule: nb[0] is the type size, nb[1] = nb[0] * (ne[0] / block size), and each later nb[i] = nb[i-1] * ne[i-1]. The block size is 1 for ordinary types. It is larger for quantized types, which the last part of the example covers.

The Java program below applies the rule to the 3-by-4 matrix and finds one element's address. It then builds a transposed view (rows and columns swapped) by changing only ne and nb, without copying data. Last, it computes the strides and byte sizes of a real weight tensor of the tiny model in three storage types. In the output, look at the byte offset of the element, the contiguous = false flag of the view, and how much smaller the q8_0 tensor is.

import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Arrays;
import java.util.Locale;

public class GgmlStrides {
    record GgmlType(String name, int blockSize, int typeSize) {}

    // nb[0] = type size, nb[1] = nb[0] * (ne[0] / block size), nb[i] = nb[i-1] * ne[i-1]
    static long[] strides(GgmlType t, long[] ne) {
        long[] nb = new long[4];
        nb[0] = t.typeSize();
        nb[1] = nb[0] * (ne[0] / t.blockSize());
        for (int i = 2; i < 4; i++) nb[i] = nb[i - 1] * ne[i - 1];
        return nb;
    }

    static long offset(long[] nb, long i0, long i1) {
        return i0 * nb[0] + i1 * nb[1];
    }

    public static void main(String[] args) {
        GgmlType f32 = new GgmlType("f32", 1, 4);
        long[] ne = {4, 3, 1, 1};                        // 3 rows of 4 floats
        long[] nb = strides(f32, ne);
        System.out.println("ne = " + Arrays.toString(ne) + "  nb = " + Arrays.toString(nb));

        ByteBuffer data = ByteBuffer.allocate(48).order(ByteOrder.LITTLE_ENDIAN);
        for (int i = 0; i < 12; i++) data.putFloat(i * 4, i);
        System.out.println("element (i0=2, i1=1) at byte " + offset(nb, 2, 1)
                + " = " + data.getFloat((int) offset(nb, 2, 1)));

        // a transposed view: swap ne[0]/ne[1] and nb[0]/nb[1], keep the same bytes
        long[] neT = {3, 4, 1, 1};
        long[] nbT = {nb[1], nb[0], nb[2], nb[3]};
        System.out.println("view ne = " + Arrays.toString(neT) + "  nb = " + Arrays.toString(nbT)
                + "  contiguous = " + (nbT[0] == f32.typeSize()));
        for (int i1 = 0; i1 < neT[1]; i1++) {
            StringBuilder row = new StringBuilder("  view row " + i1 + ":");
            for (int i0 = 0; i0 < neT[0]; i0++) {
                row.append(String.format(Locale.ROOT, " %4.1f", data.getFloat((int) offset(nbT, i0, i1))));
            }
            System.out.println(row);
        }

        GgmlType f16 = new GgmlType("f16", 1, 2);
        GgmlType q8 = new GgmlType("q8_0", 32, 34);
        long[] gate = {256, 704, 1, 1};                  // blk.0.ffn_gate.weight
        for (GgmlType t : new GgmlType[] {f32, f16, q8}) {
            long[] s = strides(t, gate);
            System.out.printf(Locale.ROOT, "ffn_gate {256, 704} as %-5s nb = %s, bytes = %d%n",
                    t.name(), Arrays.toString(s), s[3] * gate[3]);
        }
    }
}

ne = [4, 3, 1, 1]  nb = [4, 16, 48, 48]
element (i0=2, i1=1) at byte 24 = 6.0
view ne = [3, 4, 1, 1]  nb = [16, 4, 48, 48]  contiguous = false
  view row 0:  0.0  4.0  8.0
  view row 1:  1.0  5.0  9.0
  view row 2:  2.0  6.0 10.0
  view row 3:  3.0  7.0 11.0
ffn_gate {256, 704} as f32   nb = [4, 1024, 720896, 720896], bytes = 720896
ffn_gate {256, 704} as f16   nb = [2, 512, 360448, 360448], bytes = 360448
ffn_gate {256, 704} as q8_0  nb = [34, 272, 191488, 191488], bytes = 191488

The first line describes the 3-by-4 matrix. ggml lists dimensions innermost first: ne[0] = 4 is the length of a row, and ne[1] = 3 is the number of rows. The strides follow from the rule: one float is 4 bytes, one row of 4 floats is 16 bytes, and the whole matrix is 48 bytes. nb[3] equals nb[2] because dimensions 2 and 3 have size 1, so they add nothing.

The second line computes an address. Element (i0 = 2, i1 = 1) is at 2 * 4 + 1 * 16 = 24 bytes, and the float stored there is 6.0, the seventh value written (counting from zero). Every ggml kernel (a function that does one kind of math over tensors) computes addresses this way. The comment in ggml.h says that "all tensor operations have to take the stride into account and not assume that the tensor is contiguous in memory".

The third block shows why strides exist. A view is a tensor that reuses the bytes of another tensor with different ne, nb, or starting offset. Swapping ne[0] with ne[1] and nb[0] with nb[1] gives the transpose without copying a byte. View row 0 reads 0, 4, 8, which is column 0 of the original. This is what ggml_transpose and ggml_permute do.

The view is not contiguous, which means its elements are not next to each other in memory in index order. The program detects this because nb[0] is 16 instead of the type size 4. An operation that needs contiguous input inserts a ggml_cont node that copies the data into the right order first.

The last three lines apply the rule to a real tensor of the tiny model, blk.0.ffn_gate.weight with ne = {256, 704}, which is 704 rows of 256 values. For a quantized type, nb[0] is the size of one block, not of one value. A block is a group of values stored together with a shared scale (a number that says how to turn the small stored integers back into real values). A q8_0 block holds 32 values in 34 bytes: 32 one-byte integers plus a 2-byte scale. A row of 256 values is therefore 8 blocks, so nb[1] = 8 * 34 = 272 bytes. The matrix takes 191,488 bytes instead of 720,896 for F32, which is about 27% of the size. This is also why ne[0] of a quantized tensor must be a multiple of the block size: a row must be made of whole blocks.

Example 3: compute graphs, build first and compute later

Suppose an engine ran each operation the moment it was called. It could not know how much working memory the whole computation needs, and it could not decide in advance which device should run which step. ggml avoids this by splitting the work into two phases. First it writes down the whole computation. Then it runs it.

A cooking analogy fits. Writing the recipe is the first phase, and cooking is the second. Each recipe step names its ingredients and the action to take. The same recipe can be cooked again with different ingredients. The analogy stops at one point: ggml's recipe is a list of tensors, not text, and the engine uses it to plan memory before it starts cooking.

In ggml, a call such as ggml_add(ctx, a, b) creates a new tensor whose op field is GGML_OP_ADD and whose src array points to a and b. It does not add anything. Only after the whole computation is described does ggml_build_forward_expand collect the tensors into a compute graph (ggml_cgraph), which is the ordered list of steps. A separate call then runs the graph. The header comment of ggml.h uses the function f(x) = a*x^2 + b to show this.

The Java program below is a toy version with the same two phases. It builds f = a*x^2 + b and prints how many operations ran after building (expected: none). It then collects the steps in order and runs them twice with different inputs. Look at the first output line, the node order, and the final operation count.

import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Set;

public class ToyGraph {
    // A tensor is either a leaf (an input or a weight) or the result of an op on source tensors.
    static final class Tensor {
        final String name, op;
        final Tensor[] src;
        float value;
        Tensor(String name, String op, Tensor... src) { this.name = name; this.op = op; this.src = src; }
    }

    static int opsExecuted = 0;

    static Tensor leaf(String name) { return new Tensor(name, "NONE"); }
    static Tensor mul(String name, Tensor a, Tensor b) { return new Tensor(name, "MUL", a, b); }
    static Tensor add(String name, Tensor a, Tensor b) { return new Tensor(name, "ADD", a, b); }

    // like ggml_build_forward_expand: visit sources first, so nodes end up in execution order
    static void expand(Tensor t, Set<Tensor> leafs, Set<Tensor> nodes) {
        if (leafs.contains(t) || nodes.contains(t)) return;
        for (Tensor s : t.src) expand(s, leafs, nodes);
        if (t.op.equals("NONE")) leafs.add(t); else nodes.add(t);
    }

    static void compute(List<Tensor> nodes) {
        for (Tensor n : nodes) {
            float a = n.src[0].value, b = n.src[1].value;
            n.value = n.op.equals("MUL") ? a * b : a + b;
            opsExecuted++;
        }
    }

    public static void main(String[] args) {
        Tensor x = leaf("x"), a = leaf("a"), b = leaf("b");
        Tensor x2 = mul("x2", x, x);
        Tensor f = add("f", mul("ax2", a, x2), b);      // f = a*x^2 + b, nothing computed yet
        System.out.println("ops executed after building: " + opsExecuted);

        Set<Tensor> leafs = new LinkedHashSet<>(), nodes = new LinkedHashSet<>();
        expand(f, leafs, nodes);
        List<Tensor> order = new ArrayList<>(nodes);
        System.out.println("leafs: " + leafs.stream().map(t -> t.name).toList());
        for (int i = 0; i < order.size(); i++) {
            Tensor n = order.get(i);
            System.out.printf(Locale.ROOT, "node %d: %-3s = %s(%s, %s)%n", i, n.name, n.op, n.src[0].name, n.src[1].name);
        }

        a.value = 3; b.value = 4;
        for (float in : new float[] {2, 5}) {
            x.value = in;
            compute(order);                              // same graph, new input
            System.out.printf(Locale.ROOT, "x = %.1f -> f = %.1f%n", in, f.value);
        }
        System.out.println("ops executed after two computes: " + opsExecuted);
    }
}

ops executed after building: 0
leafs: [a, x, b]
node 0: x2  = MUL(x, x)
node 1: ax2 = MUL(a, x2)
node 2: f   = ADD(ax2, b)
x = 2.0 -> f = 16.0
x = 5.0 -> f = 79.0
ops executed after two computes: 6

The first line shows that building the expression ran no arithmetic: the program only created objects that remember their operation and sources. The expansion then walks from f back to the inputs and records two kinds of tensors:

• Leafs have no operation. They are inputs and weights (a, x, b).
• Nodes are results of operations.

The walk visits the sources of a tensor before the tensor itself, so every node lands after the nodes it depends on. The list is therefore already a valid execution order: x2 before ax2 before f.

The compute phase runs the list front to back. With a = 3 and b = 4, x = 2 gives 3 * 4 + 4 = 16, and x = 5 gives 3 * 25 + 4 = 79. The same graph ran twice with a new input. llama.cpp does the same for every decode step (the step that produces one new token). Six operations ran in total, three per compute.

Separating the two phases gives the engine three things:

• It knows the full list of operations before running any, so it can compute the peak memory once and allocate it up front. This is the "compute buffer" of Example 11.
• It can assign each node to a device before running (Example 6).
• It can reuse a graph whose shape has not changed. The timing summary reports this as graphs reused.

Example 4: the real graph of one decode step

The toy graph had three nodes. The real graph for one token of the tiny model has 130. The program llama-eval-callback prints them. It is an example program that registers a callback (a function the engine calls) on every node. For each node it prints the name, the operation, the shapes of the sources, and the shape of the result. This is the quickest way to see what a forward pass (one run of a token through the whole model) consists of.

The commands build the program and run it on the prompt "hi", generating one token with a 256-cell context.

cmake --build build --target llama-eval-callback -j4
llama-eval-callback -m tiny-random-qwen2vocab-f32.gguf -p "hi" -n 1 --seed 1 -c 256

The output below shows layer 0 only. It leaves out the tensor values that the program prints under each line, and several reshape, view, permute, and normalization lines. Each line has the form name = (type) OPERATION(source{shape}, ...) = {result shape}. The shapes in braces list ne, innermost dimension first, as in Example 2.

common_debug_cb_eval:                     embd = (f32)   GET_ROWS(token_embd.weight{256, 151936, 1, 1}, inp_tokens{1, 1, 1, 1}}) = {256, 1, 1, 1}
common_debug_cb_eval:                   norm-0 = (f32)   RMS_NORM(embd{256, 1, 1, 1}, }) = {256, 1, 1, 1}
common_debug_cb_eval:              attn_norm-0 = (f32)        MUL(norm-0{256, 1, 1, 1}, blk.0.attn_norm.weight{256, 1, 1, 1}}) = {256, 1, 1, 1}
common_debug_cb_eval:                   Qcur-0 = (f32)    MUL_MAT(blk.0.attn_q.weight{256, 256, 1, 1}, attn_norm-0{256, 1, 1, 1}}) = {256, 1, 1, 1}
common_debug_cb_eval:                   Qcur-0 = (f32)    RESHAPE(Qcur-0{256, 1, 1, 1}, }) = {32, 8, 1, 1}
common_debug_cb_eval:                   Qcur-0 = (f32)       ROPE(Qcur-0{32, 8, 1, 1}, inp_pos{1, 1, 1, 1}}) = {32, 8, 1, 1}
common_debug_cb_eval:                   Kcur-0 = (f32)    MUL_MAT(blk.0.attn_k.weight{256, 64, 1, 1}, attn_norm-0{256, 1, 1, 1}}) = {64, 1, 1, 1}
common_debug_cb_eval:        cache_k_l0 (view) = (f16)   SET_ROWS(Kcur-0 (view){64, 1, 1, 1}, attn_inp_k_idxs{1, 1, 1, 1}}) = {64, 256, 1, 1}
common_debug_cb_eval:                  node_21 = (f32) FLASH_ATTN_EXT(Qcur-0 (view) (permuted){32, 1, 8, 1}, cache_k_l0 (view) (permuted){32, 256, 2, 1}}) = {32, 8, 1, 1}
common_debug_cb_eval:               attn_out-0 = (f32)    MUL_MAT(blk.0.attn_output.weight{256, 256, 1, 1}, kqv_out-0{256, 1, 1, 1}}) = {256, 1, 1, 1}
common_debug_cb_eval:                ffn_inp-0 = (f32)        ADD(attn_out-0{256, 1, 1, 1}, embd{256, 1, 1, 1}}) = {256, 1, 1, 1}
common_debug_cb_eval:               ffn_gate-0 = (f32)    MUL_MAT(blk.0.ffn_gate.weight{256, 704, 1, 1}, ffn_norm-0{256, 1, 1, 1}}) = {704, 1, 1, 1}
common_debug_cb_eval:                 ffn_up-0 = (f32)    MUL_MAT(blk.0.ffn_up.weight{256, 704, 1, 1}, ffn_norm-0{256, 1, 1, 1}}) = {704, 1, 1, 1}
common_debug_cb_eval:             ffn_swiglu-0 = (f32)     SWIGLU(ffn_gate-0{704, 1, 1, 1}, ffn_up-0{704, 1, 1, 1}}) = {704, 1, 1, 1}
common_debug_cb_eval:                ffn_out-0 = (f32)    MUL_MAT(blk.0.ffn_down.weight{704, 256, 1, 1}, ffn_swiglu-0{704, 1, 1, 1}}) = {256, 1, 1, 1}
common_debug_cb_eval:                  l_out-0 = (f32)        ADD(ffn_out-0{256, 1, 1, 1}, ffn_inp-0{256, 1, 1, 1}}) = {256, 1, 1, 1}

These lines are the transformer block from The Transformer Block and the Full Model, written as ggml operations. A transformer block is one repeated unit of the model that mixes information between tokens (attention) and then transforms each token (the MLP, a small feed-forward network). Reading the lines in order:

• GET_ROWS reads one row of token_embd.weight for the input token. It is the embedding lookup, which turns a token id into a vector of 256 numbers. It reads 1 of 151,936 rows, not the whole matrix.
• RMS_NORM followed by MUL with attn_norm.weight is RMSNorm, a step that rescales the vector to a standard size, followed by a learned scale.
• MUL_MAT with attn_q.weight {256, 256} gives the query, a vector of 256 values that says what this token is looking for. RESHAPE turns it into {32, 8}, which is 8 heads of 32 dimensions each. ROPE rotates it by the position stored in inp_pos, which tells the model where the token sits in the text.
• The key projection gives {64, 1} because the model has 2 key-value heads of 32 dimensions. SET_ROWS writes the key into row attn_inp_k_idxs of the layer-0 K cache. That cache is a {64, 256} F16 tensor: 64 values per token and 256 cells because the context is 256.
• FLASH_ATTN_EXT computes attention of the 8 query heads against the cached keys and values in one fused operation (Example 11 explains flash attention).
• MUL_MAT with attn_output.weight, then ADD with embd, is the output projection followed by the residual connection. A residual connection adds the block's input back to its output, so the block only has to learn a change.
• ffn_gate, ffn_up, SWIGLU, and ffn_down are the SwiGLU MLP, and the last ADD is the second residual connection.

The shapes follow ggml's MUL_MAT convention. For a weight {K, M} (M rows of length K) and an input {K, N}, the result is {M, N}. As a check, ffn_gate.weight {256, 704} times a 256-value input gives 704 values, which matches the ffn_gate-0 line. The full graph repeats the block for layers 1 to 3 and ends with the output norm and the output matrix. Example 11 shows its node count, 130.

Example 5: backends, devices, and buffer types

A model can run on a CPU, on a GPU, or on both. The engine needs words for the three things involved.

• A backend is a ggml implementation of the tensor operations for one kind of hardware. The CPU backend is in ggml/src/ggml-cpu/, the CUDA backend in ggml/src/ggml-cuda/, and so on.
• A device is one piece of hardware that a backend drives. CUDA0 is the first NVIDIA GPU.
• A buffer type says where a block of memory lives and which backend can read it directly.

An analogy: backends are kinds of workers (a CPU worker, a GPU worker), a device is one particular worker, and a buffer type is the warehouse where material is stored, together with a list of workers who can reach it. The analogy stops at one point: copying between warehouses costs time, and in llama.cpp that copy is a real data transfer between memories.

The commands below ask a CPU-only build which devices it has, and then load the model with -v (verbose) to print the placement decisions. The output of the first command and selected lines of the second follow, both from the pinned CPU build with timestamps removed.

llama-cli --list-devices
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 4 -no-cnv -v

The first block is the device list. The second block is selected lines of the verbose run:

Available devices:
  (none)

D load_tensors: layer   0 assigned to device CPU, is_swa = 0
D load_tensors: layer   1 assigned to device CPU, is_swa = 0
D load_tensors: layer   2 assigned to device CPU, is_swa = 0
D load_tensors: layer   3 assigned to device CPU, is_swa = 0
D load_tensors: layer   4 assigned to device CPU, is_swa = 0
D done_getting_tensors: tensor 'token_embd.weight' (f32) (and 38 others) cannot be used with preferred buffer type AMX, using CPU instead
I load_tensors:   CPU_Mapped model buffer size =   307.51 MiB
I system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

--list-devices lists offload devices (devices that can take work away from the CPU), so a CPU-only build prints (none). The CPU always exists and is not listed.

The -v run shows the placement decisions that llama-model.cpp makes. The model has 4 layers, but the log lists layers 0 to 4. The fifth "layer" is the output layer, which holds the output norm and the output matrix. The input layer (token_embd) is not in this list at all. The source assigns it to the CPU always, with the comment "there is very little benefit to offloading the input layer". The reason is that the input step only reads one row of the embedding matrix, which is almost no work.

The done_getting_tensors line shows the buffer type preference list at work. For each weight, libllama tries the buffer types in order and takes the first one that supports the operation that will use the weight. This CPU has AMX (Intel's matrix extension), so the AMX buffer type comes first. It does not accept F32 weights, so all 39 tensors fall back to the plain CPU type. The other CPU buffer types are:

• CPU_REPACK stores some quantized types in an interleaved layout that the CPU code reads faster (REPACK = 1 in system_info, and the --repack and --no-repack flags).
• CPU_Mapped is a CPU buffer whose memory is the memory-mapped model file (Example 9 and Example 12).

The system_info line lists the CPU features that the CPU backend was compiled with, followed by the thread counts (Example 13). These features are instruction sets, such as AVX2 and AVX512, that let the CPU process several numbers per instruction.

A GPU build adds one section per backend before CPU :. With CUDA, the device lines come from ggml_cuda_init in ggml/src/ggml-cuda/ggml-cuda.cu. The format below is from that file, with an illustrative device, not a capture. VRAM is the video memory on the GPU card.

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24080 MiB):
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24080 MiB

Buffer types on a GPU build then include CUDA0 (VRAM of the first GPU) and CUDA_Host. CUDA_Host is pinned host memory: ordinary memory that the operating system promises not to move, so the GPU can copy from it faster than from regular memory. CPUs, GPUs, and Memory for Inference explains why VRAM bandwidth decides generation speed.

Example 6: the backend scheduler and graph splits

When some weights are on a GPU and others on the CPU, one graph has to run on two devices. The backend scheduler (ggml_backend_sched in ggml/src/ggml-backend.cpp) handles this. It does three things:

1. It assigns every node to a backend.
2. It cuts the node list into splits, which are runs of consecutive nodes on the same backend.
3. It copies tensors between devices at each cut.

Its comment in ggml-backend.h states the rule. A node goes to "the backend that supports the operation" and to "the location of the pre-allocated tensors (e.g. the weights)". In plain words, an operation runs where its weights live, and operations without weights join a neighbor. The source does this in six passes. The first two are "assign backends to ops with pre-allocated inputs" and "expand current backend assignments".

The Java program below models those two passes for a 2-layer model. It prints, for four settings, how many splits and device copies result and which nodes sit in each split. In the output, compare the split count and copy count across the four lines.

import java.util.ArrayList;
import java.util.List;
import java.util.Set;

public class ToyScheduler {
    // weightLayer: the layer whose weights the op reads; -1 for the input layer, -2 for an op without weights
    record Node(String name, int weightLayer, boolean ffn) {}

    static List<Node> graph(int nLayer) {
        List<Node> g = new ArrayList<>(List.of(new Node("embd", -1, false)));
        for (int il = 0; il < nLayer; il++) {
            g.add(new Node("attn-" + il, il, false));
            g.add(new Node("add-" + il, -2, false));
            g.add(new Node("ffn-" + il, il, true));
            g.add(new Node("out-" + il, -2, false));
        }
        g.add(new Node("logits", nLayer, false));       // the output layer has index nLayer
        return g;
    }

    static void schedule(int nLayer, int ngl, Set<Integer> ffnOnCpu) {
        int gpuStart = Math.max(nLayer + 1 - ngl, 0);   // same formula as llama-model.cpp
        List<Node> g = graph(nLayer);
        String[] dev = new String[g.size()];
        for (int i = 0; i < g.size(); i++) {             // pass 1: ops follow their weights
            Node n = g.get(i);
            if (n.weightLayer() == -1) dev[i] = "CPU";
            else if (n.weightLayer() >= 0) {
                boolean gpu = n.weightLayer() >= gpuStart && !(n.ffn() && ffnOnCpu.contains(n.weightLayer()));
                dev[i] = gpu ? "GPU" : "CPU";
            }
        }
        for (int i = 1; i < g.size(); i++) {             // pass 2: ops without weights join their neighbour
            if (dev[i] == null) dev[i] = dev[i - 1];
        }
        StringBuilder splits = new StringBuilder();
        int nSplits = 0, copies = 0;
        for (int i = 0; i < g.size(); i++) {
            if (i == 0 || !dev[i].equals(dev[i - 1])) {
                nSplits++;
                if (i > 0) copies++;                     // the previous output crosses devices
                splits.append(splits.length() == 0 ? "" : " | ").append(dev[i]).append(":");
            }
            splits.append(' ').append(g.get(i).name());
        }
        System.out.printf("ngl=%d ffnOnCpu=%s -> splits=%d copies=%d%n  %s%n", ngl, ffnOnCpu, nSplits, copies, splits);
    }

    public static void main(String[] args) {
        schedule(2, 0, Set.of());
        schedule(2, 3, Set.of());
        schedule(2, 2, Set.of());
        schedule(2, 3, Set.of(1));
    }
}

ngl=0 ffnOnCpu=[] -> splits=1 copies=0
  CPU: embd attn-0 add-0 ffn-0 out-0 attn-1 add-1 ffn-1 out-1 logits
ngl=3 ffnOnCpu=[] -> splits=2 copies=1
  CPU: embd | GPU: attn-0 add-0 ffn-0 out-0 attn-1 add-1 ffn-1 out-1 logits
ngl=2 ffnOnCpu=[] -> splits=2 copies=1
  CPU: embd attn-0 add-0 ffn-0 out-0 | GPU: attn-1 add-1 ffn-1 out-1 logits
ngl=3 ffnOnCpu=[1] -> splits=4 copies=3
  CPU: embd | GPU: attn-0 add-0 ffn-0 out-0 attn-1 add-1 | CPU: ffn-1 out-1 | GPU: logits

The layer-to-device rule is copied from llama-model.cpp: i_gpu_start = max(n_layer + 1 - n_gpu_layers, 0), and every layer with an index at or above it goes to the GPU. Layers are counted from the end. As a worked case, with 2 layers and ngl = 2, i_gpu_start = 2 + 1 - 2 = 1, so layer 1 and the output layer go to the GPU, not layers 0 and 1. With ngl = 3 (all layers plus the output), i_gpu_start = 0 and the whole model is offloaded.

The four results show how placement changes the graph:

• With no layers offloaded, everything runs on the CPU in one split.
• Full offload still gives 2 splits, because the embedding lookup stays on the CPU. Its 256-value result is the only tensor copied to the GPU per token. Real logs of fully offloaded models report splits = 2 for this reason.
• Partial offload (ngl=2) also gives 2 splits here. The CPU runs the first layer, one activation vector (the intermediate result passed from layer to layer) crosses to the GPU, and the GPU runs the rest. The cost of partial offload is not the copy, which is small. It is the CPU layers, which run at CPU speed (Tuning llama.cpp measures it).
• Moving one layer's FFN weights (the MLP weights) to the CPU while the rest stays on the GPU gives 4 splits and 3 copies. This is the shape that tensor overrides for MoE experts produce. MoE (mixture of experts) models hold many alternative MLP blocks, and a tensor override pins chosen tensors to a chosen device. Each such override adds two device transfers per layer per step.

The real scheduler also handles operations that a backend does not support (passes 3 and 4) and copies of inputs that several splits need. It also supports "pipeline parallelism" across several GPUs, which llama-context.cpp enables only when the model is fully offloaded to more than one device in layer split mode.

Example 7: CMake options and a CPU build

llama.cpp is built with CMake, a tool that reads a project's build description and produces the build. Which hardware backends the engine supports is decided when it is built. The options are all declared in ggml/CMakeLists.txt. A build without CUDA contains no CUDA code, whatever flags are given later.

The first command below configures the build for a release (optimized) binary. The second compiles it with 8 parallel jobs. The configure output that follows is from the pinned checkout on an x86-64 Linux machine with GCC 13.3, with compiler detection lines removed. Look for the lines that name the CPU backend and the -march=native flag.

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 8

The configure step prints the following lines:

-- llama.cpp version: 0.5.0-dev
CMAKE_BUILD_TYPE=Release
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Found OpenMP: TRUE (found version "4.5")
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native
-- ggml version: 0.25.3
-- ggml commit:  0c1e570-dirty
-- Configuring done (1.3s)

The CPU backend is always included. -march=native comes from GGML_NATIVE, which is on by default (off when cross-compiling or when SOURCE_DATE_EPOCH is set). It tells the compiler to use every instruction set of the build machine. The binary is then fast on that machine, and it may crash with "illegal instruction" on an older CPU. OpenMP, found here, is a library for running code on several threads. It provides the CPU thread pool (OPENMP = 1 in system_info).

The table lists the main options with their defaults from ggml/CMakeLists.txt.

A CUDA build is cmake -B build -DGGML_CUDA=ON (from docs/build.md). The same file adds that -DGGML_NATIVE=OFF builds for all CUDA GPUs instead of only the ones in the build machine. Several backends can be on at once, for example CUDA and Vulkan, and the CPU backend is always built alongside them. The pinned build used in this course is CPU-only, which is why --list-devices printed (none). The version command confirms which commit the binary was built from:

llama-cli --version

version: 0.5.0-dev (build 1, commit 0c1e570)
built with GNU 13.3.0 for Linux x86_64

Example 8: startup log, part 1, model loading and metadata

Every llama.cpp tool prints the same startup sequence, because they all load models through libllama. The log has four parts, and the next four examples go through them in order. Part 1 describes the model file.

By default the tools show only their own messages. The libllama messages appear at verbosity 4. common/log.cpp maps libllama's info level to LOG_LEVEL_TRACE (4), and the default threshold is 3, so -lv 4 is needed to see them.

The command below loads the tiny model with a 4096-cell context and generates 4 tokens. The output shows only the first part of the log. Timestamps are removed, and most print_info lines and the tokenizer arrays are shortened.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 4 -c 4096 --seed 1 -no-cnv -lv 4

The log starts with raw metadata, then a summary of the file, then the tokenizer's end tokens, then the settings that libllama derived:

I llama_model_loader: loaded meta data with 21 key-value pairs and 39 tensors from tiny-random-qwen2vocab-f32.gguf (version GGUF V3 (latest))
I llama_model_loader: - kv   0:                       general.architecture str              = llama
I llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
I llama_model_loader: - kv   3:                     llama.embedding_length u32              = 256
I llama_model_loader: - kv   4:                          llama.block_count u32              = 4
I llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 8
I llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 2
I llama_model_loader: - kv  14:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
I llama_model_loader: - kv  20:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
I llama_model_loader: - type  f32:   39 tensors
I print_info: file format = GGUF V3 (latest)
I print_info: file type   = all F32
I print_info: file size   = 307.51 MiB (32.00 BPW)
W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
I load: printing all EOG tokens:
I load:   - 128247 ('</s>')
I load:   - 151643 ('<|endoftext|>')
I load:   - 151645 ('<|im_end|>')
I print_info: arch                  = llama
I print_info: n_ctx_train           = 4096
I print_info: n_embd                = 256
I print_info: n_layer               = 4
I print_info: n_head                = 8
I print_info: n_head_kv             = 2
I print_info: n_embd_head_k         = 32
I print_info: n_gqa                 = 4
I print_info: n_embd_k_gqa          = 64
I print_info: n_ff                  = 704
I print_info: rope type             = 0
I print_info: freq_base_train       = 500000.0
I print_info: model params          = 80.61 M
I print_info: vocab type            = BPE
I print_info: n_vocab               = 151936
I print_info: BOS token             = 151643 '<|endoftext|>'
I print_info: EOT token             = 151645 '<|im_end|>'

The log has three groups.

The llama_model_loader lines are the raw GGUF metadata, one line per key, in file order. These are the keys that The GGUF Format reads byte by byte. Arrays are shown as their element type and length (arr[str,151936]) followed by the first few elements. The last line counts tensors per type: all 39 are F32. A quantized model shows one such line per ggml type, which is the quant mix of the file.

The print_info: file ... lines summarize the file. BPW (bits per weight) is the file size in bits divided by the parameter count. It is 32.00 for F32 and lower for quantized files. For the tiny model, 4 bytes per weight is 32 bits per weight, as the log says.

The load: lines come from the tokenizer setup in llama-vocab.cpp. The EOG list (end of generation) is the set of tokens that make the model stop. llama-vocab.cpp builds it from the EOS and EOT ids in the metadata (end of sequence and end of turn) plus every vocabulary entry whose text is on a fixed list of known end tokens (<|endoftext|>, </s>, <|eom_id|>, <end_of_turn>, and others). The warning comes from that list. Token 128247 has the text </s> but is not marked as a control token in the file, so libllama marks it as one.

The remaining print_info lines are the hyperparameters (the model's size settings) after libllama interpreted the metadata. Several are derived from the others, and each one has a short calculation:

• n_embd_head_k = 256 / 8 = 32 dimensions per head.
• n_gqa = 8 / 2 = 4 query heads share each key-value head (GQA, grouped-query attention).
• n_embd_k_gqa = 32 * 2 = 64 values of K per token per layer.

These derived numbers size the KV cache in Example 10. rope type = 0 is the normal RoPE layout of Llama models. NEOX-style models such as Qwen2 show 2. model params = 80.61 M is computed from the tensor shapes. It adds up to two 38.9-million-parameter matrices (the embedding and the output matrix) and 0.7 million per layer.

Example 9: startup log, part 2, tensor loading and buffers

The second part of the log reports how the tensors were loaded and where their memory is. This example explains the words CPU_Mapped and mmap and what the buffer size means.

The lines below follow the previous ones in the same run, with timestamps removed.

I load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
I load_tensors:   CPU_Mapped model buffer size =   307.51 MiB
......

load_mode = mmap means libllama memory-mapped the file. mmap is the operating system call that makes a file's contents appear as memory. Reading an address loads the matching page of the file from disk on first access, and later reads come from the operating system's page cache (memory the system keeps for recently read file pages). The default --load-mode auto uses mmap unless a device does not support it.

CPU_Mapped model buffer size = 307.51 MiB says that all weights are in one CPU buffer whose memory is the mapped file itself. Nothing was copied. The 307.51 MiB is the tensor data region of the file. The dots after it are a progress bar, one dot per step.

On a GPU build with layers offloaded, this part has one line per buffer type, for example a CUDA0 model buffer for the offloaded layers and a CPU_Mapped model buffer for the rest. The offload lines come before them. Their formats are in llama-model.cpp. The lines below use illustrative values for a 32-layer model fully offloaded. They are not a capture.

load_tensors: offloading output layer to GPU
load_tensors: offloading 31 repeating layers to GPU
load_tensors: offloaded 33/33 layers to GPU
load_tensors:        CUDA0 model buffer size =  4403.49 MiB
load_tensors:   CPU_Mapped model buffer size =   281.81 MiB

The count 33/33 includes the output layer, as in Example 5: 32 repeating layers plus the output layer. Even with every layer offloaded, a CPU buffer remains. It holds token_embd, which stays on the CPU, and its size is the embedding matrix in the file's type.

With -v (all messages), these lines appear twice. Before the real load, the tools "fit" the parameters to free memory. To do this, libllama loads the model once without reading tensor data, which measures the buffers. The fitting step sends that pass's messages to the debug level (Tuning llama.cpp, Example 12).

Example 10: startup log, part 3, the context and the KV cache

After the weights, the tool creates a context (llama_context). The context holds the KV cache, the batch sizes, and the state of the requests in progress. The weights are read-only and the same for every request. The context is the part that changes while text is generated.

The KV cache (key-value cache) stores, for every earlier token, the key and value vectors that attention needs. Without it, the model would recompute them for the whole text at every step. The cache has a fixed number of cells, one per token position, and its size follows from numbers in the first part of the log.

The lines below follow the previous ones in the same run. Timestamps are removed, and the n_rs_seq, n_outputs_max_per_seq, and two attn_rot lines are removed.

I llama_context: constructing llama_context
I llama_context: n_seq_max             = 1
I llama_context: n_ctx                 = 4096
I llama_context: n_ctx_seq             = 4096
I llama_context: n_batch               = 2048
I llama_context: n_ubatch              = 512
I llama_context: causal_attn           = 1
I llama_context: flash_attn            = auto
I llama_context: kv_unified            = false
I llama_context: freq_base             = 500000.0
I llama_context: freq_scale            = 1
I llama_context: n_outputs_max         = 2048
I llama_context:        CPU  output buffer size =     0.58 MiB
I llama_kv_cache:        CPU KV buffer size =     4.00 MiB
I llama_kv_cache: size =    4.00 MiB (  4096 cells,   4 layers,  1/1 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB

Each line in the output means the following:

• n_seq_max = 1: the context holds one sequence. A server with several parallel slots shows a larger number (Running llama.cpp: Core Flags, Example 8).
• n_ctx = 4096: the number of cells (token positions) in the KV cache, from -c 4096. libllama rounds it up to a multiple of 256 (GGML_PAD(n_ctx, 256) in llama-context.cpp).
• n_ctx_seq: the cells one sequence can use. It equals n_ctx with one sequence.
• n_batch = 2048: the logical batch, the most tokens one llama_decode call accepts. n_ubatch = 512: the micro-batch (physical batch), the most tokens one graph computes at once. A 2,000-token prompt is accepted in one call and computed as four graphs of up to 512 tokens.
• causal_attn = 1: each token attends only to earlier tokens, as in every decoder-only model.
• flash_attn = auto: the default. After a test, libllama decides whether flash attention is supported on the device of every layer. The next part of the log reports the result.
• freq_base and freq_scale: the RoPE parameters in use. They differ from the model's values only when RoPE scaling flags are given (Tuning llama.cpp, Example 8).
• output buffer size = 0.58 MiB: room for one row of logits, which is 151,936 floats. Logits are the model's raw scores for every token in the vocabulary.
• The llama_kv_cache lines give the cache: 4096 cells for 4 layers and 1 sequence, with keys and values in F16 (16-bit floats of 2 bytes), 2 MiB each.

The KV numbers can be computed from print_info. For K, each token in each layer stores n_embd_k_gqa = 64 values. That gives 4 layers times 64 values times 4096 cells times 2 bytes, which is 2,097,152 bytes, or 2 MiB. V has the same size. The Java program below does this arithmetic for an F16 cache and for a q8_0 cache, and it also computes the output and logits buffers that Example 11 uses. In the output, compare the first two lines with the log above.

import java.util.Locale;

public class ContextBuffers {
    static final double MIB = 1024.0 * 1024.0;

    // bytes for n values of a ggml type: whole blocks of blockSize values, typeSize bytes each
    static long bytes(long n, int blockSize, int typeSize) {
        return n / blockSize * typeSize;
    }

    public static void main(String[] args) {
        // values printed by print_info for the tiny model
        int nLayer = 4, nEmbdKGqa = 64, nEmbdVGqa = 64, nVocab = 151936;
        int nCtx = 4096;

        long kF16 = nLayer * bytes((long) nEmbdKGqa * nCtx, 1, 2);
        long vF16 = nLayer * bytes((long) nEmbdVGqa * nCtx, 1, 2);
        System.out.printf(Locale.ROOT, "KV f16 : K %.2f MiB + V %.2f MiB = %.2f MiB%n", kF16 / MIB, vF16 / MIB, (kF16 + vF16) / MIB);
        long kQ8 = nLayer * bytes((long) nEmbdKGqa * nCtx, 32, 34);
        System.out.printf(Locale.ROOT, "KV q8_0: K %.2f MiB + V %.2f MiB = %.2f MiB%n", kQ8 / MIB, kQ8 / MIB, 2 * kQ8 / MIB);

        long logitsRow = (long) nVocab * 4;              // one row of F32 logits
        System.out.printf(Locale.ROOT, "output buffer, 1 sequence : %.2f MiB%n", logitsRow / MIB);
        System.out.printf(Locale.ROOT, "output buffer, 4 sequences: %.2f MiB%n", 4 * logitsRow / MIB);
        for (int ubatch : new int[] {512, 128, 32}) {
            System.out.printf(Locale.ROOT, "logits for a full ubatch of %3d: %6.2f MiB%n", ubatch, ubatch * logitsRow / MIB);
        }
    }
}

KV f16 : K 2.00 MiB + V 2.00 MiB = 4.00 MiB
KV q8_0: K 1.06 MiB + V 1.06 MiB = 2.13 MiB
output buffer, 1 sequence : 0.58 MiB
output buffer, 4 sequences: 2.32 MiB
logits for a full ubatch of 512: 296.75 MiB
logits for a full ubatch of 128:  74.19 MiB
logits for a full ubatch of  32:  18.55 MiB

The first line reproduces the log: 4 layers times 64 values times 4096 cells times 2 bytes is 2 MiB for K, and the same for V. The second line predicts a q8_0 cache, which uses 34 bytes per 32 values, so about 1.06 bytes per value instead of 2. The pinned build prints exactly that when started with -ctk q8_0 -ctv q8_0: K (q8_0): 1.06 MiB, V (q8_0): 1.06 MiB.

The output buffer line matches 0.58 MiB, which is one row of 151,936 floats at 4 bytes each. A server context with 4 sequences prints 2.32 MiB, four rows. The last three lines belong to the next example. Memory Math uses the same per-token formula for real model sizes.

Example 11: startup log, part 4, graph splits and compute buffers

The last part of the startup log reserves memory for the compute graph, the working memory that holds the intermediate results of one forward pass. The weights and the KV cache stay for the whole run. The compute buffer is scratch space that the steps of the graph reuse. This example explains what decides its size and what the graph line counts.

The lines below follow the previous ones in the same run. Timestamps are removed, and four resolve_fused_ops lines about fused operations of another architecture are removed.

I sched_reserve: reserving ...
I resolve_fused_ops: Flash Attention enabled
I sched_reserve:        CPU compute buffer size =   297.75 MiB
I sched_reserve: graph: nodes = 130, splits = 1, input objects = 4, input tensors = 6
I sched_reserve: reserve took 1.03 ms, sched copies = 1
I cmn          init: llama threadpool init, n_threads = 4
I cmn  common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)

sched_reserve prepares for the worst case. It builds the largest graph the context can run (a full micro-batch of 512 tokens) and the smallest (1 token). It runs the allocator on them without computing anything, and it allocates a compute buffer large enough for the worst case. Intermediate tensors are placed in this buffer and reused as soon as no later node needs them. The buffer size is therefore the largest set of tensors that are alive at the same time.

resolve_fused_ops: Flash Attention enabled is the result of flash_attn = auto. Flash attention computes attention in tiles, so it never stores the full matrix of attention scores. ggml implements it as the single FLASH_ATTN_EXT node seen in Example 4. libllama reserved a test graph, checked that the fused operation landed on the same device as each layer, and enabled it. A later lesson on attention optimizations explains the algorithm.

The compute buffer is 297.75 MiB, almost as large as the weights. The previous program explains why. At reserve time, libllama sizes the graph for the worst case in which every token of the micro-batch requests logits (perplexity and embedding runs do that). 512 rows of 151,936 F32 logits are 512 * 151,936 * 4 bytes, which is 296.75 MiB. The remaining 1 MiB is everything else.

The pinned build confirms this. With -ub 128 the log prints CPU compute buffer size = 74.44 MiB, and with -ub 32 it prints 18.61 MiB. Each is about 0.3% above the logits estimate. A model with a large vocabulary and a small hidden size, like this one, has a compute buffer dominated by logits. A 7B model has a hidden size of 4,096, and there the attention and MLP intermediates matter as well. llama-server requests logits only for the last token of each sequence. It reserved 10.14 MiB for the same model with 4 slots.

The graph line counts the graph built for the reserve:

• nodes = 130: operations in one forward pass of 4 layers. With -fa off, the pinned build reports nodes = 150. Without the fused operation, llama-eval-callback shows attention in layer 0 as separate nodes. These are kq-0 (MUL_MAT of keys and queries, giving the {256, 1, 8} score matrix: 256 cells, 1 token, 8 heads), kq_soft_max-0 (SOFT_MAX with the mask), kqv-0 (MUL_MAT with the values), a PERMUTE, and a CONT copy. There are also extra reshapes because the V cache is stored transposed. That is 5 more nodes per layer, and 4 layers give 20 more nodes, which matches 150 minus 130.
• splits = 1: one backend ran the whole graph. A fully offloaded GPU run reports 2 (Example 6), and a partial offload at least 2.
• input objects and input tensors: tensors that the tool fills before each run, such as inp_tokens, inp_pos, the attention mask, and the cache indices seen in Example 4.
• sched copies = 1: the scheduler keeps one copy of the inputs. Pipeline parallelism across several GPUs uses up to GGML_SCHED_MAX_COPIES (4) copies, so that one GPU can work on the next micro-batch while another finishes the previous one.

The thread pool starts with 4 threads. The warmup run (common_init in common/common.cpp) decodes a batch of the BOS and EOS tokens once and then clears the cache. This way the first real request does not pay for page faults (the first load of a memory page) and first-use setup.

At exit, the tools print a memory summary that adds the three parts:

I common_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
I common_memory_breakdown_print: |   - Host               |                  609 =   307 +       4 +     297                |

self is the memory this process allocated: 307 MiB of model weights, 4 MiB of context (the KV cache), and 297 MiB of compute buffer, in whole MiB. The three add to the total of 609 MiB. On a GPU build there is one row per device with total and free filled in.

Example 12: mmap, the page cache, and resident memory

Example 9 said that mmap lets the model file act as memory. This example measures what that changes. --load-mode selects the loading method. It replaces the separate --mmap, --no-mmap, and --mlock flags of older versions, and none of those three is accepted at the pinned commit.

The run below starts llama-server twice on a free port, once with -lm mmap and once with -lm none. It waits for /health to answer, and it reads the process's memory counters from /proc/<pid>/status, a Linux file that reports them. The lines to compare are RssAnon and RssFile.

llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port "$PORT" -lm mmap -c 512 -ub 32 &
# wait until curl -s 127.0.0.1:$PORT/health returns ok, then:
grep -E 'VmRSS|RssAnon|RssFile' /proc/$!/status
kill $!

The two outputs follow, first with -lm mmap and then with -lm none. Values differ slightly between runs.

VmRSS:	  385784 kB
RssAnon:	   55828 kB
RssFile:	  329956 kB

VmRSS:	  385960 kB
RssAnon:	  370752 kB
RssFile:	   15208 kB

Resident memory (VmRSS) is the physical memory a process uses now. Linux splits it into two kinds:

• Anonymous memory (RssAnon) belongs only to this process, such as heap allocations.
• File-backed memory (RssFile) holds pages of files mapped into the process.

The totals differ by less than 1 MiB, because every weight is read during warmup and is therefore resident in both runs. The split differs:

• With mmap, about 322 MiB of the 376 MiB is file-backed. The weights are pages of the GGUF file in the operating system's page cache. Anonymous memory is about 55 MiB: the KV cache, the compute buffer (small here because of -ub 32), and the program itself.
• With none, libllama allocated a 307.51 MiB CPU buffer and read the file into it, so the weights are anonymous memory. The startup log shows CPU model buffer instead of CPU_Mapped model buffer.

The difference matters in three situations:

• A second process that maps the same file shares the same physical pages, so two servers with one model use the weight memory once.
• When memory runs low, the kernel can drop file-backed pages without writing them anywhere, because it can read them from the file again. Anonymous pages must go to swap (disk space used as spill-over memory) or stay.
• A restart is faster with mmap, because the file is still in the page cache.

The other modes work as follows. mlock reads the file into memory and locks it so it cannot be paged out (the log shows load_mode = mlock and a CPU model buffer). mmap+mlock maps and locks. dio reads with direct I/O, bypassing the page cache. Running llama.cpp: Core Flags shows when to choose each.

Example 13: CPU threads for prompt processing and generation

The CPU backend splits each operation across a pool of threads, so several CPU cores work on one operation together. Two flags control the pool. -t sets the threads for generation, which handles one token per step. -tb sets the threads for batches, which is prompt processing. -tb defaults to -t, and -t defaults to -1, which the system_info line resolved to 4 on this 4-core machine.

llama-bench measures both phases. pp512 is prompt processing of 512 tokens. tg128 is text generation of 128 tokens. t/s is tokens per second, shown with its standard deviation (the spread) over the repetitions. The command below runs the benchmark with 1, 2, and 4 threads, three repetitions each. The question is how each phase scales with threads.

llama-bench -m tiny-random-qwen2vocab-f32.gguf -t 1,2,4 -r 3

Representative output follows. It comes from the pinned build on a shared 4-core machine, and numbers differ by machine and run.

| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       1 |           pp512 |    6279.24 ± 1061.71 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       1 |           tg128 |         82.76 ± 4.50 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       2 |           pp512 |    12762.86 ± 220.71 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       2 |           tg128 |        252.22 ± 7.53 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |           pp512 |    23152.16 ± 793.76 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |           tg128 |       374.93 ± 94.24 |

Prompt processing reaches thousands of tokens per second, and generation reaches hundreds. The difference suggests that the two phases use the machine differently. The program below turns the table into the two quantities that the roofline model uses. For prompt processing it computes floating-point operations per second (FLOP/s, a measure of arithmetic speed). For generation it computes bytes read per second (a measure of memory speed). In the output, compare how each speed grows from 1 to 4 threads.

import java.util.Locale;

public class ThreadScaling {
    public static void main(String[] args) {
        // tiny model: parameters per part, F32 = 4 bytes each
        long output = 256L * 151936, layers = 4L * 705_024 + 256;
        long bytesPerDecodeToken = (output + layers) * 4;   // token_embd is a row lookup, not read in full
        long flopsPerPromptToken = 2 * layers;              // only the last prompt token computes logits
        System.out.printf(Locale.ROOT, "weights read per generated token: %.1f MB%n", bytesPerDecodeToken / 1e6);
        System.out.printf(Locale.ROOT, "FLOPs per prompt token (matmuls): %.2f M%n", flopsPerPromptToken / 1e6);

        int[] threads = {1, 2, 4};
        double[] pp = {6279.24, 12762.86, 23152.16};        // llama-bench pp512, t/s
        double[] tg = {82.76, 252.22, 374.93};              // llama-bench tg128, t/s
        for (int i = 0; i < threads.length; i++) {
            System.out.printf(Locale.ROOT, "t=%d  pp %6.1f GFLOP/s (x%.2f)   tg %5.1f GB/s (x%.2f)%n",
                    threads[i], pp[i] * flopsPerPromptToken / 1e9, pp[i] / pp[0],
                    tg[i] * bytesPerDecodeToken / 1e9, tg[i] / tg[0]);
        }
    }
}

weights read per generated token: 166.9 MB
FLOPs per prompt token (matmuls): 5.64 M
t=1  pp   35.4 GFLOP/s (x1.00)   tg  13.8 GB/s (x1.00)
t=2  pp   72.0 GFLOP/s (x2.03)   tg  42.1 GB/s (x3.05)
t=4  pp  130.6 GFLOP/s (x3.69)   tg  62.6 GB/s (x4.53)

The two phases do different work. A generated token reads the output matrix (38.9 million floats, because every token needs logits) and all layer weights, 166.9 MB per token. The work is 2.8 million multiply-adds in the layers plus 38.9 million in the output matrix. That is about one multiply-add for every 4 bytes read, so generation waits on memory. A prompt token in a 512-token batch reuses each weight 512 times and computes logits only for the last token, so prompt processing waits on arithmetic.

Prompt processing scales almost linearly with threads: 2.03 times with 2 and 3.69 times with 4. Each thread adds arithmetic units, so more threads mean more arithmetic per second.

Generation scaled even faster than the thread count here, at 3.05 times with 2 threads. One core cannot keep enough memory requests in flight to use the memory system fully. Adding cores therefore adds usable bandwidth, until the memory system is the limit. The single-thread value also had the largest spread across runs on this shared machine. A later sweep with -t 1,2,3,4 -r 5, run while other processes were busy, gave 61.36 ± 11.68 t/s for tg128 with 4 threads. Thread counts are therefore chosen from repeated measurements, not from one run. Tuning llama.cpp does this with sweeps.

The rules that follow from the two bottlenecks are:

• For generation, threads beyond the point where memory bandwidth is used fully add nothing. They can slow things down by competing for the same memory.
• Threads on efficiency cores or hyper-threads of busy cores make every step wait for the slowest thread.
• For prompt processing, more physical cores help until the cores run out.

That is why llama.cpp has separate -t and -tb. A setting that follows from the two rules is -t equal to the number of performance cores and -tb equal to all cores, confirmed by a measurement.

Example 14: the binaries and when to use each

The pinned build has one binary per tool in tools/ and one per program in examples/. Choosing the right one saves time, because each is built for one job. The table lists them with their source folders and their use.

cmake --build build --target llama-simple builds one example program alone. For loop, two binaries matter: llama-server, which loop serve starts and supervises, and llama-bench, which measures a machine before choosing flags. llama-cli itself follows the same split as loop: a terminal client in front of an HTTP server.

Example 15: the libllama C API flow

Every tool above, and every language binding, calls the C API in include/llama.h. A C API is a set of plain functions that any language can call. The smallest complete program that uses it is examples/simple/simple.cpp. The excerpt below keeps its calls in order and removes argument parsing and error checks. Read it top to bottom as a recipe with six steps, which the list after it names.

llama_backend_init();

llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = ngl;
llama_model * model = llama_model_load_from_file(model_path.c_str(), model_params);
const llama_vocab * vocab = llama_model_get_vocab(model);

// count the prompt tokens (negative return value), then tokenize
const int n_prompt = -llama_tokenize(vocab, prompt.c_str(), prompt.size(), NULL, 0, true, true);
std::vector<llama_token> prompt_tokens(n_prompt);
llama_tokenize(vocab, prompt.c_str(), prompt.size(), prompt_tokens.data(), prompt_tokens.size(), true, true);

llama_context_params ctx_params = llama_context_default_params();
ctx_params.n_ctx   = n_prompt + n_predict - 1;
ctx_params.n_batch = n_prompt;
llama_context * ctx = llama_init_from_model(model, ctx_params);

llama_sampler * smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());
llama_sampler_chain_add(smpl, llama_sampler_init_greedy());

llama_batch_ext * batch = llama_batch_ext_init(ctx);
batch_set_tokens(batch, prompt_tokens.data(), n_prompt, 0);   // logits only for the last token
for (int n_pos = 0; n_pos + n_tokens < n_prompt + n_predict; ) {
    llama_process(ctx, LLAMA_PROCESS_TYPE_DECODE, batch);
    n_pos += n_tokens;
    llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);
    if (llama_vocab_is_eog(vocab, new_token_id)) break;
    // print llama_token_to_piece(vocab, new_token_id, ...)
    batch_set_tokens(batch, &new_token_id, 1, n_pos);
    n_tokens = 1;
}

llama_batch_ext_free(batch);
llama_sampler_free(smpl);
llama_free(ctx);
llama_model_free(model);

The flow has six steps, each with a matching object:

1. llama_backend_init initializes ggml and the backends.
2. llama_model_load_from_file reads the GGUF file into a llama_model (Examples 8 and 9). llama_model_params carries the load options, such as n_gpu_layers. The model is read-only and can be shared by several contexts.
3. llama_tokenize turns text into token ids with the model's vocabulary. Called with no output buffer, it returns the negative of the required length. The program uses this to size its token array first.
4. llama_init_from_model creates a llama_context with the KV cache and compute buffers (Examples 10 and 11). n_ctx and n_batch are set from the prompt length here.
5. A sampler chain holds samplers in order. A sampler is the code that picks the next token from the model's scores. This program uses only the greedy sampler, which always takes the most likely token. The tools build the chain from Penalties, Logit Bias, and Sampler Chains.
6. The loop decodes a batch, samples, checks for an end-of-generation token, and feeds the sampled token back as a one-token batch at the next position. A batch carries a group of tokens into the model at once.

At the pinned commit, examples/simple uses the extended batch API (llama_batch_ext_init, llama_batch_ext_add_token, llama_batch_ext_set_pos, llama_batch_ext_set_output_logits, and llama_process). The older llama_batch struct with llama_batch_init, llama_batch_get_one, and llama_decode is still declared in llama.h. The header says llama_process returns "the same as llama_decode()":

• 0 for success.
• 1 when no KV slot was found for the batch.
• 2 when the call was aborted.
• Negative values for errors.

A batch carries for each token its id, its position, its sequence id, and whether logits are wanted. Requesting logits for only the last prompt token is what keeps the output small.

The program is built as llama-simple. The command below runs it on the tiny model with a prompt of four words and a limit of 8 new tokens. Look at the prompt token count, the number of decode runs, and the graphs reused line.

llama-simple -m tiny-random-qwen2vocab-f32.gguf -n 8 "Hello my name is"

Representative output follows. It comes from the pinned build. The weights are random, so the text is meaningless, and timings differ by machine.

Hello my name isAssertion.Shared.Sharedアウト pourquoiAssertion.Shared pourquoi
main: decoded 8 tokens in 0.03 s, speed: 271.82 t/s

llama_perf_sampler_print:    samplers time =       0.96 ms /     8 runs
llama_perf_context_print:        load time =     207.22 ms
llama_perf_context_print: prompt eval time =       5.38 ms /     4 tokens (    1.34 ms per token,   744.05 tokens per second)
llama_perf_context_print:        eval time =      20.29 ms /     7 runs   (    2.90 ms per token,   345.08 tokens per second)
llama_perf_context_print:       total time =     231.29 ms /    11 tokens
llama_perf_context_print:    graphs reused =          6

The prompt "Hello my name is" is 4 tokens and was decoded in one batch (prompt eval ... / 4 tokens). Then 7 single-token decodes followed (eval ... / 7 runs). The eighth sampled token needed no further decode because the loop stopped.

graphs reused = 6 shows Example 3 at work. The 7 decode steps have the same graph shape, so libllama built the graph once and reused it 6 times, changing only the input tensors. The context was sized from the prompt (11 cells, rounded up to 256) and the batch is 4, so this program's compute buffer was 2.33 MiB, against 297.75 MiB with the defaults of Example 11.

A Java program can run the same flow in-process by calling these C functions through the Foreign Function and Memory API, the Java feature for calling native code. A later lesson on the engine landscape calls libllama from Java, and section 7 of the course builds the whole flow in pure Java.

Interview points

• llama.cpp has three layers. ggml holds tensors, operations, graphs, and backends. libllama (src/, include/llama.h) holds models, the KV cache, batches, and samplers. common/ and tools/ hold the flags, logging, and binaries.
• A ggml tensor has ne (elements per dimension, innermost first) and nb (byte strides). Views such as transposes change ne and nb without copying data. Kernels must honour strides, and ggml_cont makes a copy when contiguous data is needed.
• For a quantized type, nb[0] is the size of one block (34 bytes for 32 q8_0 values), so a row length must be a multiple of the block size.
• ggml builds a graph first and computes later. This allows one up-front allocation of the compute buffer, per-node device assignment, and graph reuse across decode steps.
• The backend scheduler assigns nodes to backends by weight location and operation support, and it cuts the graph into splits. Every split boundary is a device copy. Full GPU offload gives 2 splits because the embedding lookup stays on the CPU.
• -ngl N offloads the last N layers, counting the output layer, with i_gpu_start = n_layer + 1 - N.
• The startup log has four parts: metadata (llama_model_loader, print_info), tensor buffers (load_tensors), context and KV cache (llama_context, llama_kv_cache), and compute buffers and graph (sched_reserve). The libllama info lines need -lv 4.
• The compute buffer is sized for the worst-case micro-batch. With a large vocabulary it is dominated by n_ubatch * n_vocab * 4 bytes of logits, so -ub changes it directly.
• mmap keeps weights as file-backed page-cache pages. These pages are shared between processes, can be dropped under memory pressure, and make restarts fast. --load-mode none copies the weights into anonymous memory instead.
• Prompt processing is compute-bound and scales with cores. Generation is bandwidth-bound and stops scaling once memory bandwidth is saturated. This is why -t and -tb are separate flags.
• The C API flow is: backend init, load model, tokenize, create context, build a sampler chain, then decode, sample, and feed the token back in a loop.

Summary

• The pinned source is commit 0c1e57098 with ggml 0.25.3. llama-cli --version prints version: 0.5.0-dev (build 1, commit 0c1e570).
• ggml/ holds the tensor library and one directory per backend. src/ holds libllama with one file per architecture under src/models/. common/arg.cpp defines every flag.
• The strides follow one rule: nb[0] is the type size, nb[1] = nb[0] * ne[0] / block size, and nb[i] = nb[i-1] * ne[i-1].
• MUL_MAT of a weight {K, M} and an input {K, N} gives {M, N}.
• One forward pass of the tiny model is 130 graph nodes with flash attention and 150 without.
• --list-devices prints only offload devices, so a CPU-only build prints (none).
• The CMake options GGML_CUDA, GGML_METAL, GGML_VULKAN, GGML_HIP, and GGML_BLAS select backends. GGML_NATIVE (on by default) compiles for the build machine's CPU.
• print_info shows derived hyperparameters such as n_embd_head_k, n_gqa, and n_embd_k_gqa, which size the KV cache.
• The K part of the KV cache is n_layer * n_embd_k_gqa * n_ctx * bytes per value, and V has the same size. For the tiny model at 4096 cells in F16, K and V together are 4 MiB.
• n_ctx is rounded up to a multiple of 256. n_batch (default 2048) is the logical batch, and n_ubatch (default 512) is the physical batch.
• The tiny model's compute buffer is 297.75 MiB at -ub 512, 74.44 MiB at -ub 128, and 18.61 MiB at -ub 32.
• --load-mode accepts auto, none, mmap, mlock, mmap+mlock, and dio. The flags --mmap, --no-mmap, and --mlock do not exist at this commit.
• With mmap, the tiny model's weights showed up as 322 MiB of RssFile. With none, they showed up as anonymous memory.
• llama-cli at this commit runs llama-server in a thread and talks to it over HTTP. One-shot completion is llama-completion.
• examples/simple uses llama_batch_ext and llama_process. llama_batch and llama_decode remain in the API with the same return codes.