Tuning llama.cpp

llama.cpp starts with sensible defaults, but a default cannot know your hardware, your model, or how long your conversations are. A coding agent sends long prompts and waits for long answers, so a few settings decide whether the model fits in memory and how fast it answers. This lesson changes those settings one at a time and measures what each change does.

The method matters as much as the settings. Changing five flags at once and noticing that the model got faster teaches nothing, because you cannot tell which flag helped. Here, every example changes one thing, looks at the memory or speed numbers, and explains why the numbers moved.

After the lesson, you can run llama-bench (llama.cpp's benchmark tool) over several settings, tell a real difference from random noise, choose settings from the results, decide which weights live on the CPU and which on the GPU, stretch the context length, turn on speculative decoding, and write a tuned command for each of the four hardware tiers used in this course.

Flags and log formats come from the pinned llama.cpp commit 0c1e57098. Runs use the course's tiny test model with random weights (Llama architecture, 4 layers, 256 dimensions, 151,936-token vocabulary, F32) on a shared 4-core CPU machine, so speeds differ between runs and machines. GPU log lines are labelled as representative and built from the source's format strings. llama.cpp Architecture and Running llama.cpp: Core Flags introduce the startup log and the basic flags used here.

A few terms appear in almost every example:

• A token is a small piece of text (a word or part of a word) that the model reads and writes. Speeds are measured in tokens per second (t/s).
• The KV cache (key-value cache) is memory where the model keeps notes about every earlier token, so it does not recompute them for each new token. It grows with the context length.
• The context is the number of tokens the model can hold in view at once, set with -c.
• VRAM is the memory on a graphics card (GPU). System memory (RAM) is the memory the CPU uses. Weights in VRAM are read much faster than weights in system memory.
• Offloading means moving work or weights from the CPU to the GPU.
• The compute buffer is scratch memory that llama.cpp reserves for the intermediate results of one step of the model.
• The micro-batch (-ub) is how many prompt tokens llama.cpp processes in one go.

Example 1: flash attention, on, off, and auto

Problem: to produce each token, the model compares that token with every earlier token in the context. When it reads a prompt of many tokens at once, it builds a table of scores with one number for every pair of tokens. For a long context, that table is large, and storing it costs memory and time.

Idea: you can compute the answer in small tiles and keep only a running result, so the whole table never exists at once. This technique is called flash attention.

-fa (--flash-attn) takes on, off, or auto, "default: 'auto'". Flash attention computes attention in tiles, keeping a running maximum and sum for the softmax, so the full matrix of attention scores is never stored; ggml implements it as one fused node, FLASH_ATTN_EXT. A later lesson on attention optimizations derives the algorithm. The question is what the flag changes in the log and in memory.

The command below starts the tiny model four times: flash attention on and off, each at a context of 4,096 and of 32,768 tokens. It prints only the lines about the context size, the flash attention state, the compute buffer, and the size of the computation graph. A node is one operation the model performs, so fewer nodes means fewer separate steps.

for fa in on off; do for c in 4096 32768; do
  llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 2 -no-cnv -lv 4 -ub 32 -c "$c" -fa "$fa" 2>&1 \
    | grep -E 'n_ctx  |flash_attn  |compute buffer size =|graph:'
done; done

Output from the pinned build (timestamps removed; blank lines separate the four runs):

I llama_context: n_ctx                 = 4096
I llama_context: flash_attn            = enabled
I sched_reserve:        CPU compute buffer size =    18.61 MiB
I sched_reserve: graph: nodes = 130, splits = 1, input objects = 4, input tensors = 6

I llama_context: n_ctx                 = 32768
I llama_context: flash_attn            = enabled
I sched_reserve:        CPU compute buffer size =    18.61 MiB
I sched_reserve: graph: nodes = 130, splits = 1, input objects = 4, input tensors = 6

I llama_context: n_ctx                 = 4096
I llama_context: flash_attn            = disabled
I sched_reserve:        CPU compute buffer size =    18.64 MiB
I sched_reserve: graph: nodes = 150, splits = 1, input objects = 4, input tensors = 6

I llama_context: n_ctx                 = 32768
I llama_context: flash_attn            = disabled
I sched_reserve:        CPU compute buffer size =    36.17 MiB
I sched_reserve: graph: nodes = 150, splits = 1, input objects = 4, input tensors = 6

Compare the compute buffer lines first. With flash attention it stays the same when the context grows eight times. Without it, the buffer doubles. Now the explanation with numbers.

With flash attention, the compute buffer does not depend on the context: 18.61 MiB at 4096 and at 32,768 cells. Without it, the buffer grows with the context, from 18.64 MiB to 36.17 MiB. The reason is the score matrix: a micro-batch of 32 tokens attending to 32,768 cells with 8 heads in F32 is 32 * 32768 * 8 * 4 bytes, exactly 32 MiB, and without flash attention that matrix must exist in memory at once. At 4096 cells it is 4 MiB, smaller than the 18.55 MiB of logits that already set the buffer size, so it barely shows. For a real model at a long context with the default -ub 512, the same matrix is 512 * 32768 * 32 heads * 4 bytes, 2 GiB, which is why flash attention matters most at long contexts.

The graph has 130 nodes with flash attention and 150 without: the separate attention operations (MUL_MAT for the scores, SOFT_MAX, MUL_MAT for the values, a PERMUTE, and a CONT copy) replace the one fused node in each of the 4 layers.

With auto, llama-context.cpp decides at startup: it reserves a test graph with flash attention, checks for each layer that the FLASH_ATTN_EXT node landed on the same device as the layer, and logs resolve_fused_ops: Flash Attention enabled. If a backend lacks the kernel for the model's head size or cache types, the node falls back to another device, and the log says instead:

resolve_fused_ops: layer 0 is assigned to device CUDA0 but Flash Attention is assigned to device CPU (usually due to missing support)
resolve_fused_ops: Flash Attention not supported, set to disabled

These two lines are the format strings from llama-context.cpp with an illustrative device name; they were not captured from a run. Forcing -fa on in that situation keeps the fused operation on the device that supports it, which adds copies between devices in every layer. auto avoids that case; a benchmark decides whether on or off is faster on a given machine.

Example 2: KV cache types and the flash-attention requirement

Problem: the KV cache grows with every token in the context. For a long agent session it can take several gigabytes, as much as the model weights themselves.

Idea: store each number in the cache with fewer bits. A number stored in 16 bits becomes a number stored in 8 or 4 bits, so it is rounded and takes less space. Rounding loses a little precision, which is the trade. Storing numbers with fewer bits is called quantization, and the cache type names (f16, q8_0, q4_0) say how many bits each number uses.

-ctk TYPE and -ctv TYPE set the data type of the K and V caches: "allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16)". A quantized cache stores keys and values in the same block formats as quantized weights (Building Q8_0 and Q4_0 Like ggml). The question is what each combination costs and which ones are allowed.

The three commands below try three cache settings. Look at the cache size in the first and third runs, and at the error in the second.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 2 -no-cnv -lv 4 -c 4096 -ctk q8_0 -ctv q8_0
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 2 -no-cnv -lv 4 -c 4096 -ctk q8_0 -ctv q8_0 -fa off
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 2 -no-cnv -lv 4 -c 4096 -ctk q4_0 -fa off

Selected lines of the three runs (from the pinned build; timestamps removed):

I llama_init_from_model: enabling flash_attn since it is required for quantized V cache
I llama_context: flash_attn            = enabled
I llama_kv_cache: size =    2.12 MiB (  4096 cells,   4 layers,  1/1 seqs), K (q8_0):    1.06 MiB, V (q8_0):    1.06 MiB

E llama_init_from_model: quantized V cache requires flash_attn to be enabled
E llama_completion: error: unable to create context

I llama_context: flash_attn            = disabled
I llama_kv_cache: size =    2.56 MiB (  4096 cells,   4 layers,  1/1 seqs), K (q4_0):    0.56 MiB, V (f16):    2.00 MiB

In the first run, the cache is 2.12 MiB, about half of the 4 MiB that the default would need. The second run is an error, and the third run shows a smaller K cache working. Here are the reasons.

The first run halves the cache: q8_0 stores 32 values in 34 bytes, 1.0625 bytes per value against 2 for F16, so 2.12 MiB instead of 4. With -fa auto, libllama turns flash attention on by itself, because llama-context.cpp requires it for a quantized V cache: without flash attention, the V cache is stored transposed and read by a plain matrix product, which has no quantized path. The second run forbids flash attention and fails at context creation with that reason. The third run shows that only V is restricted: a q4_0 K cache (18 bytes per 32 values, 0.56 MiB) works with flash attention off.

The table turns bytes per value into the size of the whole cache for a real 8B model at a context of 32,768 tokens. The calculation is the F16 size times the new bytes per value, divided by 2. For q8_0, that is 4.00 GiB x 1.0625 / 2, about 2.13 GiB.

What the type changes:

The 8B column uses Memory Math's 128 KiB per token in F16. The quality cost of a quantized cache is measured, not assumed: llama-perplexity with --kl-divergence against an F16-cache run, as in Making, Quantizing, and Testing Your Own GGUF, compares the two on the same text.

On CUDA, a second restriction applies at build time. docs/build.md lists GGML_CUDA_FA_QUANTS with the default q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16: flash-attention kernels are compiled only for these K/V pairs, and "Combinations that were not compiled fall back to f16-f16 kernel with a warning". A mixed pair such as -ctk q8_0 -ctv q4_0 therefore needs a build with that pair added (or all), and matched pairs are the ones that work on a default CUDA build.

Example 3: llama-bench basics

Problem: running a program once and timing it with a stopwatch is unreliable. The first run is slow because files load from disk, and other programs on the machine disturb the result. A benchmark tool fixes this by warming up, repeating the test, and reporting the average and the spread.

llama.cpp measures two different things. Prompt processing (also called prefill) reads the whole prompt in large batches, which is fast per token. Generation (also called decode) produces one new token at a time, which is slower per token. llama-bench reports them separately.

llama-bench runs fixed tests and prints one row per test and parameter combination. The defaults, from its --help: -p 512 (a prompt-processing test of 512 tokens, named pp512), -n 128 (a generation test of 128 tokens, tg128), -r 5 repetitions, and -o md (a markdown table). The question is what one default run reports and how to read it.

One command with no options runs both default tests. Read the t/s column, which is tokens per second, and the ± part, which says how much the repetitions disagreed.

llama-bench -m tiny-random-qwen2vocab-f32.gguf

Representative output (from the pinned build; numbers differ by machine and run):

| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |           pp512 |   21220.36 ± 1329.27 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |           tg128 |       405.33 ± 61.69 |

build: 0c1e570 (1)

The model column joins the architecture, a size label (?B because libllama does not recognize this layer count as a known model size), and the file type. pp512 decodes a 512-token prompt in batches and reports prompt tokens per second. tg128 generates 128 tokens one at a time and reports generated tokens per second. Each value is the mean over 5 repetitions after a warmup run, and ± is the standard deviation. A spread of 15% on tg128 (61.69 on 405.33) says that this machine was busy during the run: one repetition alone could have been off by that much. The last line names the commit, which belongs in every published benchmark.

The other output formats carry every parameter and every sample. Part of one -o json record for a pp512 test:

    "n_batch": 2048,
    "n_ubatch": 512,
    "n_threads": 4,
    "type_k": "f16",
    "type_v": "f16",
    "n_gpu_layers": -1,
    "flash_attn": -1,
    "n_prompt": 512,
    "n_gen": 0,
    "avg_ns": 27648492,
    "stddev_ns": 5761921,
    "avg_ts": 18929.239306,
    "stddev_ts": 3944.837108,
    "samples_ns": [ 31722786, 23574198 ],
    "samples_ts": [ 16139.8, 21718.7 ]

avg_ns is the mean time of one test and avg_ts the mean tokens per second; samples_ts lists each repetition (this run used -r 2). flash_attn: -1 encodes auto. Each test measures one phase: n_prompt: 512, n_gen: 0 is a pp test, and a tg test has n_prompt: 0. -o csv gives the same fields as columns, -o jsonl one JSON object per line, and -o sql SQL insert statements. -oe FORMAT writes a second format to stderr, so one run can print a table for a person and JSON lines for a program.

Other test parameters: -pg 512,128 runs a combined test (a 512-token prompt followed by 128 generated tokens, timed together), -d N (--n-depth) fills the KV cache with N tokens before each test, and --delay waits between tests to let a hot machine cool down.

Example 4: sweeps over threads, micro-batch, flash attention, and depth

Problem: you want to know which settings are best on your machine, and the answer is often not obvious. More threads help until the CPU runs out of cores. A bigger micro-batch helps prompt processing but costs memory.

Idea: a sweep is a benchmark that tries every combination of several values and prints one row per combination. In the command below, there are 2 thread counts, 2 micro-batch sizes, and 2 flash attention settings, so 8 combinations. Each combination runs a prompt test and a generation test, which gives 16 rows.

A thread is one worker that runs in parallel on a CPU core. -t 2 uses two cores for generation and -t 4 uses four.

Every llama-bench parameter accepts several values, "separated by ','", and ranges such as 1-8 or 1-16*2. The tool runs every combination. The question is what a sweep shows that single runs do not.

The first command runs the sweep and also saves the results as JSON lines in sweep.jsonl for Example 5.

llama-bench -m tiny-random-qwen2vocab-f32.gguf -t 2,4 -ub 128,512 -fa 0,1 -p 512 -n 64 -r 3 -o md -oe jsonl 2> sweep.jsonl

Representative output (from the pinned build on a shared 4-core machine; the model, size, params, and backend columns removed):

| threads | n_ubatch |  fa |            test |                  t/s |
| ------: | -------: | --: | --------------: | -------------------: |
|       2 |      128 |   0 |           pp512 |     9298.35 ± 335.10 |
|       2 |      128 |   0 |            tg64 |       232.96 ± 21.05 |
|       4 |      128 |   0 |           pp512 |   10789.64 ± 6840.88 |
|       4 |      128 |   0 |            tg64 |       423.51 ± 34.63 |
|       2 |      128 |   1 |           pp512 |    13519.85 ± 100.97 |
|       2 |      128 |   1 |            tg64 |        269.31 ± 7.02 |
|       4 |      128 |   1 |           pp512 |    24985.95 ± 235.74 |
|       4 |      128 |   1 |            tg64 |       498.61 ± 10.41 |
|       2 |      512 |   0 |           pp512 |     9437.46 ± 146.70 |
|       2 |      512 |   0 |            tg64 |        269.32 ± 2.91 |
|       4 |      512 |   0 |           pp512 |    12143.10 ± 951.28 |
|       4 |      512 |   0 |            tg64 |      358.84 ± 118.10 |
|       2 |      512 |   1 |           pp512 |   12566.65 ± 1039.32 |
|       2 |      512 |   1 |            tg64 |        243.55 ± 8.01 |
|       4 |      512 |   1 |           pp512 |   23094.58 ± 1821.60 |
|       4 |      512 |   1 |            tg64 |      173.70 ± 252.40 |

To read the table, compare two rows that differ in one setting only, and check the ± value of each. A difference smaller than the ± values is not a real difference. Machine noise has several causes: other programs, the operating system moving work between cores, and the CPU changing its clock speed when it gets hot.

Three things stand out.

• Prompt processing is 1.3 to 2.3 times faster with flash attention in all four pairs (for example 13,520 against 9,298 with 2 threads and -ub 128). On this CPU, the one fused operation is faster than the five separate ones it replaces.
• Two rows have a spread larger than half their mean: pp512 with 4 threads, -ub 128, flash attention off (± 6,841), and tg64 with 4 threads, -ub 512, flash attention on (173.70 ± 252.40). The JSON samples of the second show why: 464.7, 41.6, and 14.7 tokens per second. Two of three repetitions ran while another process took the CPU. A mean with such a spread is not a measurement.
• -ub cannot affect generation, because a generation step processes one token. Yet the four tg64 rows with 2 threads range from 233 to 269. That 15% range is the noise of this machine, and any difference smaller than it says nothing.

The fourth test parameter, depth, shows how speed changes as a conversation grows:

The first sweep ran with an empty KV cache. A real conversation has thousands of tokens in the cache, and attention must read all of them for each new token. The -d flag (depth) fills the cache first. This command skips the prompt test (-p 0) and measures 32 generated tokens at three depths.

llama-bench -m tiny-random-qwen2vocab-f32.gguf -p 0 -n 32 -d 0,1024,3072 -fa 0,1 -r 3

Representative output (columns removed as above):

|  fa |            test |                  t/s |
| --: | --------------: | -------------------: |
|   0 |            tg32 |       342.38 ± 33.88 |
|   0 |    tg32 @ d1024 |       324.55 ± 68.64 |
|   0 |    tg32 @ d3072 |       197.23 ± 62.55 |
|   1 |            tg32 |       471.26 ± 20.29 |
|   1 |    tg32 @ d1024 |       297.08 ± 14.89 |
|   1 |    tg32 @ d3072 |       175.30 ± 20.21 |

tg32 @ d3072 generates 32 tokens after 3,072 tokens are already in the cache. Everything else in the graph is identical to depth 0, so the drop, to about 40% of the depth-0 speed with flash attention, is the cost of attention over the cached cells, which grows linearly with depth. A coding agent works at depths of tens of thousands of tokens, so a benchmark at depth 0 overstates the speed it sees; -d measures the case that matters.

Example 5: choosing settings from sweep results

Problem: a table of 16 rows is hard to read by eye, and it is easy to pick the largest number without noticing that its spread is huge. A small program can apply the same rule every time.

Idea: for each row, divide the spread (the standard deviation) by the average. This ratio is the coefficient of variation. For example, a row of 10,789.64 t/s with a spread of 6,840.88 has a ratio of 6,840.88 / 10,789.64 = 0.634, or 63.4%. A row with a ratio above 10% gets marked rerun.

The JSON lines from Example 4's -oe jsonl can be read by a program that sorts each test by speed and marks results whose spread is too large to trust. The program below parses the 16 records (trimmed to the fields it needs) with Jackson. The question is which settings to choose for prompt processing and for generation.

The program holds the 16 JSON records as a text block, turns each into a Result record, and prints each test's rows from fastest to slowest with the spread as a percentage. Look at the last column of the output, which says stable or rerun.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Locale;

public class BenchPick {
    // llama-bench -t 2,4 -ub 128,512 -fa 0,1 -p 512 -n 64 -r 3 -oe jsonl, fields trimmed
    static final String JSONL = """
        {"n_threads": 2, "n_ubatch": 128, "flash_attn": 0, "n_prompt": 512, "n_gen": 0, "avg_ts": 9298.35, "stddev_ts": 335.10}
        {"n_threads": 2, "n_ubatch": 128, "flash_attn": 0, "n_prompt": 0, "n_gen": 64, "avg_ts": 232.96, "stddev_ts": 21.05}
        {"n_threads": 4, "n_ubatch": 128, "flash_attn": 0, "n_prompt": 512, "n_gen": 0, "avg_ts": 10789.64, "stddev_ts": 6840.88}
        {"n_threads": 4, "n_ubatch": 128, "flash_attn": 0, "n_prompt": 0, "n_gen": 64, "avg_ts": 423.51, "stddev_ts": 34.63}
        {"n_threads": 2, "n_ubatch": 128, "flash_attn": 1, "n_prompt": 512, "n_gen": 0, "avg_ts": 13519.85, "stddev_ts": 100.97}
        {"n_threads": 2, "n_ubatch": 128, "flash_attn": 1, "n_prompt": 0, "n_gen": 64, "avg_ts": 269.31, "stddev_ts": 7.02}
        {"n_threads": 4, "n_ubatch": 128, "flash_attn": 1, "n_prompt": 512, "n_gen": 0, "avg_ts": 24985.95, "stddev_ts": 235.74}
        {"n_threads": 4, "n_ubatch": 128, "flash_attn": 1, "n_prompt": 0, "n_gen": 64, "avg_ts": 498.61, "stddev_ts": 10.41}
        {"n_threads": 2, "n_ubatch": 512, "flash_attn": 0, "n_prompt": 512, "n_gen": 0, "avg_ts": 9437.46, "stddev_ts": 146.70}
        {"n_threads": 2, "n_ubatch": 512, "flash_attn": 0, "n_prompt": 0, "n_gen": 64, "avg_ts": 269.32, "stddev_ts": 2.91}
        {"n_threads": 4, "n_ubatch": 512, "flash_attn": 0, "n_prompt": 512, "n_gen": 0, "avg_ts": 12143.10, "stddev_ts": 951.28}
        {"n_threads": 4, "n_ubatch": 512, "flash_attn": 0, "n_prompt": 0, "n_gen": 64, "avg_ts": 358.84, "stddev_ts": 118.10}
        {"n_threads": 2, "n_ubatch": 512, "flash_attn": 1, "n_prompt": 512, "n_gen": 0, "avg_ts": 12566.65, "stddev_ts": 1039.32}
        {"n_threads": 2, "n_ubatch": 512, "flash_attn": 1, "n_prompt": 0, "n_gen": 64, "avg_ts": 243.55, "stddev_ts": 8.01}
        {"n_threads": 4, "n_ubatch": 512, "flash_attn": 1, "n_prompt": 512, "n_gen": 0, "avg_ts": 23094.58, "stddev_ts": 1821.60}
        {"n_threads": 4, "n_ubatch": 512, "flash_attn": 1, "n_prompt": 0, "n_gen": 64, "avg_ts": 173.70, "stddev_ts": 252.40}
        """;

    record Result(int threads, int ubatch, int fa, String test, double avg, double sd) {
        double cv() { return sd / avg; }
        String config() { return String.format(Locale.ROOT, "-t %d -ub %d -fa %s", threads, ubatch, fa == 1 ? "on" : "off"); }
    }

    public static void main(String[] args) throws Exception {
        ObjectMapper mapper = new ObjectMapper();
        List<Result> all = new ArrayList<>();
        for (String line : JSONL.strip().split("\n")) {
            JsonNode n = mapper.readTree(line);
            String test = n.get("n_prompt").asInt() > 0 ? "pp" + n.get("n_prompt").asInt() : "tg" + n.get("n_gen").asInt();
            all.add(new Result(n.get("n_threads").asInt(), n.get("n_ubatch").asInt(), n.get("flash_attn").asInt(),
                    test, n.get("avg_ts").asDouble(), n.get("stddev_ts").asDouble()));
        }
        for (String test : List.of("pp512", "tg64")) {
            System.out.println(test + " (stable = spread under 10% of the mean):");
            all.stream().filter(r -> r.test().equals(test))
               .sorted(Comparator.comparingDouble(Result::avg).reversed())
               .forEach(r -> System.out.printf(Locale.ROOT, "  %-22s %9.1f t/s  +-%5.1f%%  %s%n",
                       r.config(), r.avg(), 100 * r.cv(), r.cv() < 0.10 ? "stable" : "rerun"));
        }
    }
}

pp512 (stable = spread under 10% of the mean):
  -t 4 -ub 128 -fa on      24986.0 t/s  +-  0.9%  stable
  -t 4 -ub 512 -fa on      23094.6 t/s  +-  7.9%  stable
  -t 2 -ub 128 -fa on      13519.9 t/s  +-  0.7%  stable
  -t 2 -ub 512 -fa on      12566.7 t/s  +-  8.3%  stable
  -t 4 -ub 512 -fa off     12143.1 t/s  +-  7.8%  stable
  -t 4 -ub 128 -fa off     10789.6 t/s  +- 63.4%  rerun
  -t 2 -ub 512 -fa off      9437.5 t/s  +-  1.6%  stable
  -t 2 -ub 128 -fa off      9298.4 t/s  +-  3.6%  stable
tg64 (stable = spread under 10% of the mean):
  -t 4 -ub 128 -fa on        498.6 t/s  +-  2.1%  stable
  -t 4 -ub 128 -fa off       423.5 t/s  +-  8.2%  stable
  -t 4 -ub 512 -fa off       358.8 t/s  +- 32.9%  rerun
  -t 2 -ub 512 -fa off       269.3 t/s  +-  1.1%  stable
  -t 2 -ub 128 -fa on        269.3 t/s  +-  2.6%  stable
  -t 2 -ub 512 -fa on        243.6 t/s  +-  3.3%  stable
  -t 2 -ub 128 -fa off       233.0 t/s  +-  9.0%  stable
  -t 4 -ub 512 -fa on        173.7 t/s  +-145.3%  rerun

The output shows how noise changes the ranking. The pp512 row with 4 threads, -ub 128, and flash attention off sits in the middle by speed, but its 63.4% spread means the number cannot be trusted. The rows marked rerun were measured while the machine was busy. The spread as a fraction of the mean is the coefficient of variation. A 10% threshold marks three results as unusable; they need a rerun before any decision.

The choice follows from the stable rows and from what each flag can affect:

• Threads: 4 threads beat 2 in both tests by close to 2 times; on this 4-core machine, -t 4 and -tb 4.
• Flash attention: on is faster for prompt processing in every pair. For generation, the stable pairs differ by less than the noise found in Example 4, so the generation results neither support nor contradict it. Keep -fa auto, which turns it on here.
• Micro-batch: -ub 128 and -ub 512 are within 8% for prompt processing with flash attention, inside the spread of the -ub 512 rows. -ub 128 has a compute buffer of 74 MiB against 298 MiB, so it gives the same speed for less memory. Generation results cannot choose -ub, because -ub has no effect on generation.

The general procedure is the same on any machine: sweep only the flags that can affect a test, repeat until the spread is small, discard unstable rows, prefer the cheaper setting when two are within the noise, and confirm the choice with a run at the depth the application uses. A later lesson on benchmarking inference adds percentiles and load tests against a server.

Example 6: tensor overrides and MoE expert offload

Problem: a model may be larger than your VRAM. The -ngl flag moves whole layers, but some models have a better split: a small set of weights that every token reads (attention) and a large set that each token reads only a part of (the experts of a Mixture of Experts model). The ideal split puts the first set in VRAM and the second in system memory.

Idea: every weight in a model file has a name, such as blk.0.ffn_up.weight (layer 0, the feed-forward part, the "up" matrix). You can write a pattern that matches names and tell llama.cpp where to put the matching weights. A buffer type is a place in memory where weights are stored, for example CPU (system memory) or CUDA0 (the first NVIDIA GPU).

-ot <tensor name pattern>=<buffer type>,... (--override-tensor) places individual weights on a chosen buffer type, overriding the layer rule of -ngl. The question is how patterns match tensor names, and what --cpu-moe and --n-cpu-moe add. A run on the CPU build with a pattern that moves the FFN weights of layers 0 and 1 to CPU shows the matching:

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 1 -no-cnv -c 512 -v -ot "blk\.[0-1]\.ffn_.*=CPU"

Selected lines from the pinned build (timestamps removed):

D tensor blk.0.ffn_norm.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.0.ffn_gate.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.0.ffn_down.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.0.ffn_up.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.1.ffn_norm.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.1.ffn_gate.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.1.ffn_down.weight (0 MiB f32) buffer type overridden to CPU
D tensor blk.1.ffn_up.weight (0 MiB f32) buffer type overridden to CPU
W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance

Each D line is a debug message that confirms one tensor was moved. The pattern blk\.[0-1]\.ffn_.* matches the names of layers 0 and 1 whose part starts with ffn_ (a regular expression is a text pattern; .* means any characters, and \. means a literal dot). Eight tensors matched: four in each of the two layers.

The pattern is a regular expression, and llama-model-loader.cpp tests it with std::regex_search, so it matches anywhere in the name: ffn_.* caught ffn_norm too. Patterns are tried in order and the first match decides. The size is printed in whole MiB, and these tensors are all under 1 MiB, so they show 0. A buffer type must exist in the build: -ot "ffn=CUDA0" on this CPU build fails with "unknown buffer type" after printing the available types. The warning recommends --load-mode none whenever weights are overridden to the CPU while the file is memory-mapped.

A Mixture of Experts (MoE) model replaces the MLP of each layer with many smaller MLPs, the experts, and a router that picks a few of them for each token. All experts must be stored, but each token reads only the chosen ones. Their weights are stored per layer in tensors named ffn_up_exps, ffn_gate_exps, and ffn_down_exps (one tensor holds all experts of the layer). --cpu-moe adds the override \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU from common/common.h, and --n-cpu-moe N adds blk\.<i>\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU for layers 0 to N-1. The program below applies these patterns, with the same first-match rule, to a toy 3-layer MoE whose tensor sizes are made up for the example:

The program below builds a toy MoE model with 3 layers and applies the same pattern rules that llama.cpp uses. For each setting it prints how many MiB stay on the GPU and how many go to the CPU. The sizes are made up so that the arithmetic is easy to follow.

import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.regex.Pattern;

public class MoeOverrides {
    // patterns from common/common.h at the pinned commit
    static final String EXPS = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";

    record Override(String pattern, String bufferType) {}

    // the first pattern that matches (regex search, not full match) decides; otherwise the layer's device
    static String place(String tensor, List<Override> overrides, String layerDevice) {
        for (Override o : overrides) {
            if (Pattern.compile(o.pattern()).matcher(tensor).find()) return o.bufferType();
        }
        return layerDevice;
    }

    static void show(String label, Map<String, Integer> tensors, List<Override> overrides) {
        long cpu = 0, gpu = 0;
        List<String> onCpu = new ArrayList<>();
        for (Map.Entry<String, Integer> t : tensors.entrySet()) {
            String where = place(t.getKey(), overrides, "CUDA0");
            if (where.equals("CPU")) { cpu += t.getValue(); onCpu.add(t.getKey()); } else gpu += t.getValue();
        }
        System.out.printf(Locale.ROOT, "%-26s GPU %4d MiB  CPU %4d MiB  on CPU: %d tensors%n", label, gpu, cpu, onCpu.size());
    }

    public static void main(String[] args) {
        Map<String, Integer> tensors = new LinkedHashMap<>();     // a toy 3-layer MoE, sizes in MiB
        for (int il = 0; il < 3; il++) {
            tensors.put("blk." + il + ".attn_q.weight", 20);
            tensors.put("blk." + il + ".ffn_gate_inp.weight", 1);  // the router
            tensors.put("blk." + il + ".ffn_up_exps.weight", 300);
            tensors.put("blk." + il + ".ffn_gate_exps.weight", 300);
            tensors.put("blk." + il + ".ffn_down_exps.weight", 300);
            tensors.put("blk." + il + ".ffn_up_shexp.weight", 10);  // the shared expert
        }
        show("all on GPU", tensors, List.of());
        show("--cpu-moe", tensors, List.of(new Override(EXPS, "CPU")));
        List<Override> nCpuMoe = new ArrayList<>();
        for (int il = 0; il < 2; il++) nCpuMoe.add(new Override("blk\\." + il + EXPS, "CPU"));
        show("--n-cpu-moe 2", tensors, nCpuMoe);
        show("-ot \"blk\\.2\\.ffn_down=CPU\"", tensors, List.of(new Override("blk\\.2\\.ffn_down", "CPU")));
        show("-ot \"exps=CPU\"", tensors, List.of(new Override("exps", "CPU")));
    }
}

all on GPU                 GPU 2793 MiB  CPU    0 MiB  on CPU: 0 tensors
--cpu-moe                  GPU   93 MiB  CPU 2700 MiB  on CPU: 9 tensors
--n-cpu-moe 2              GPU  993 MiB  CPU 1800 MiB  on CPU: 6 tensors
-ot "blk\.2\.ffn_down=CPU" GPU 2493 MiB  CPU  300 MiB  on CPU: 1 tensors
-ot "exps=CPU"             GPU   93 MiB  CPU 2700 MiB  on CPU: 9 tensors

Check the numbers by hand. Each layer holds 20 + 1 + 300 + 300 + 300 + 10 = 931 MiB, so 3 layers hold 2,793 MiB. The experts are 3 tensors of 300 MiB in each layer, so 9 tensors and 2,700 MiB in all. That leaves 2,793 - 2,700 = 93 MiB (attention, router, and shared expert) on the GPU. With --n-cpu-moe 2, only the experts of two layers move: 6 x 300 = 1,800 MiB.

--cpu-moe moves all 9 expert tensors, 2,700 of 2,793 MiB, and leaves attention, the router (ffn_gate_inp), and the shared expert (ffn_up_shexp, an expert that every token uses, present in some MoE designs) on the GPU, because the pattern requires _exps. This is the placement that makes large MoE models run on a small GPU: the GPU holds the weights every token reads, and the CPU holds the experts, of which each token reads only a few. --n-cpu-moe 2 keeps the experts of the last layer on the GPU and is the knob for filling the remaining VRAM layer by layer. A hand-written -ot can move a single tensor, and the short pattern exps=CPU matches the same 9 tensors as --cpu-moe because regex search finds exps anywhere in the name.

Example 7: partial offload and its effect

Problem: you have a dense model (every weight is used for every token) that is a little bigger than your VRAM. What happens to the speed if you keep some layers on the CPU?

Idea: think of a relay race where one runner is 15 times slower than the others. Every layer must finish before the next starts, so the total time is the sum of all the legs, and the slow runner's legs dominate. The analogy holds for generation, where each token passes through all layers in order. It does not hold for prompt processing, which handles many tokens together and can hide some of the cost.

When a dense model does not fit in VRAM, -ngl N keeps the first layers on the CPU. The question is how generation speed changes with N. A generation step reads every weight once, so its time is the GPU part's bytes divided by GPU bandwidth plus the CPU part's bytes divided by CPU bandwidth. The program below evaluates that model for an 8B-shaped model; the sizes, bandwidths, and the 70% efficiency are assumptions written in the code, not measurements.

The formula is: time per token = (GB on GPU) / (GPU speed) + (GB on CPU) / (CPU speed), and speed in tokens per second is 1 divided by that time. The program applies it for six values of -ngl, the number of layers placed on the GPU. Watch how little the speed grows at first.

import java.util.Locale;

public class PartialOffload {
    public static void main(String[] args) {
        // assumptions: a 32-layer 8B model at about 4.5 bits per weight
        double layerGB = 0.135, outputGB = 0.30;      // per repeating layer, output layer
        double gpuBW = 900, cpuBW = 60;               // GB/s: a discrete GPU, dual-channel DDR5
        double efficiency = 0.7;                      // share of peak bandwidth a decode step reaches
        int nLayer = 32;
        System.out.println(" ngl  GPU GB  CPU GB  tg t/s (model)");
        for (int ngl : new int[] {0, 8, 16, 24, 32, 33}) {
            int gpuStart = Math.max(nLayer + 1 - ngl, 0);
            double gpu = 0, cpu = 0;
            for (int il = 0; il <= nLayer; il++) {
                double size = il == nLayer ? outputGB : layerGB;
                if (il >= gpuStart) gpu += size; else cpu += size;
            }
            double seconds = gpu / (gpuBW * efficiency) + cpu / (cpuBW * efficiency);
            System.out.printf(Locale.ROOT, "%4d  %6.2f  %6.2f  %8.1f%n", ngl, gpu, cpu, 1 / seconds);
        }
    }
}

 ngl  GPU GB  CPU GB  tg t/s (model)
   0    0.00    4.62       9.1
   8    1.25    3.37      12.1
  16    2.33    2.30      17.1
  24    3.40    1.22      29.1
  32    4.48    0.14      96.8
  33    4.62    0.00     136.4

Check one row by hand. For -ngl 16, the GPU holds 2.33 GB at 900 x 0.7 = 630 GB/s, which takes 2.33 / 630 = 3.7 ms. The CPU holds 2.30 GB at 60 x 0.7 = 42 GB/s, which takes 2.30 / 42 = 54.8 ms. The total is 58.5 ms, so 1 / 0.0585 = 17.1 tokens per second, as the table shows.

The curve is not a straight line. Offloading half the model (-ngl 16) only doubles the speed, because the 2.3 GB still on the CPU take 55 ms per token at 42 GB/s effective, against 4 ms for the GPU half. Even one layer on the CPU (-ngl 32) costs 30%: 0.135 GB at CPU speed takes as long as 2.0 GB at GPU speed. The last few layers matter most, so freeing VRAM for them (a smaller context, a quantized KV cache, a smaller quantization) is worth more than any CPU tuning.

The real effect is measured with llama-bench -ngl 0,8,16,24,32,99, which needs a GPU build. Its log for a partial offload contains the lines from Running llama.cpp: Core Flags, Example 5, and two model buffers. The format below is from llama-model.cpp; the values are illustrative for -ngl 24 on such a model, not a capture:

load_tensors: offloading output layer to GPU
load_tensors: offloading 23 repeating layers to GPU
load_tensors: offloaded 24/33 layers to GPU
load_tensors:        CUDA0 model buffer size =  3244.00 MiB
load_tensors:   CPU_Mapped model buffer size =  1430.00 MiB

The CPU buffer holds the 9 CPU layers and the token embedding. llama.cpp Architecture, Example 6, models how the graph is split at the point where the CPU layers hand over to the GPU layers. For MoE models, Example 6's expert placement replaces partial offload: keeping all layers on the GPU and only experts on the CPU leaves far fewer bytes per token on the slow side.

Example 8: RoPE scaling and YaRN for longer contexts

Problem: every model is trained with a maximum context length, such as 4,096 tokens. If you ask for a longer context, the model sees positions it never saw during training, and its answers get worse.

Idea: a model tells tokens apart by position using RoPE (rotary position embedding). RoPE rotates each token's vectors by an angle that grows with the token's position, so the model can tell position 5 from position 6 by the difference in rotation. If position 10,000 produces rotations the model never learned, you can shrink the positions so that 10,000 positions fit inside the learned range. This shrinking is RoPE scaling.

A model trained on 4,096 positions has never seen RoPE rotations for position 10,000. RoPE scaling compresses positions so that a longer context maps into the trained range. Linear scaling (position interpolation) divides every position by a factor. YaRN scales low-frequency dimensions and leaves high-frequency ones alone, with a ramp between them and an attention temperature correction; a later lesson on attention and long context derives it. The flags: --rope-scaling {none,linear,yarn}, --rope-scale N ("expands context by a factor of N"), --rope-freq-scale N (the same as 1/N), --rope-freq-base N, and --yarn-orig-ctx, --yarn-ext-factor, --yarn-attn-factor, --yarn-beta-slow, --yarn-beta-fast. The question is what the log shows with and without them.

Both runs ask for a context of 16,384 on a model trained on 4,096. The second adds scaling flags. Compare the freq_scale line and the warnings in the two outputs.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 1 -no-cnv -lv 4 -c 16384
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 1 -no-cnv -lv 4 -c 16384 \
  --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 4096

Selected lines of both runs (from the pinned build; timestamps removed):

I print_info: n_ctx_train           = 4096
I print_info: rope scaling          = linear
I print_info: freq_base_train       = 500000.0
I print_info: freq_scale_train      = 1
I print_info: n_ctx_orig_yarn       = 4096
I llama_context: n_ctx                 = 16384
I llama_context: freq_scale            = 1
W llama_context: n_ctx_seq (16384) > n_ctx_train (4096) -- possible training context overflow
W llama_completion: model was trained on only 4096 context tokens (16384 specified)

W llama_completion: warning: scaling RoPE frequency by 0.25.
I print_info: n_ctx_train           = 4096
I print_info: rope scaling          = linear
I print_info: freq_base_train       = 500000.0
I print_info: freq_scale_train      = 1
I print_info: n_ctx_orig_yarn       = 4096
I llama_context: n_ctx                 = 16384
I llama_context: freq_scale            = 0.25
W llama_context: n_ctx_seq (16384) > n_ctx_train (4096) -- possible training context overflow
I llama_init_from_model: custom YaRN scaling detected, re-adjusting n_ctx_train(4096)...
I llama_init_from_model: n_ctx_train adjusted to 16384

The print_info lines describe the model file: no scaling in the metadata (freq_scale_train = 1; rope scaling = linear with a scale of 1 is no change), trained on 4,096 positions. In the first run, the context is 16,384 but freq_scale stays 1, so positions 4,096 to 16,383 get rotations the model never saw. libllama and the tool both warn, and output quality degrades once a conversation passes the trained length.

In the second run, --rope-scale 4 sets freq_scale = 0.25: every position is multiplied by 0.25 inside RoPE, so position 16,383 rotates like position 4,095.75. YaRN, selected by --rope-scaling yarn with the original length 4,096, then applies its per-frequency ramp around that scale. libllama raises the effective training context to 16,384 (n_ctx_train adjusted to 16384), and the tool's warning about the trained length disappears; only the general llama_context warning, printed before the adjustment, remains.

Scaling trades some quality at short contexts for usable long contexts, so it is set only when the context exceeds the trained length. Many current models ship with YaRN parameters in their configuration, which the converter writes into the GGUF metadata (<arch>.rope.scaling.type, <arch>.rope.scaling.factor, <arch>.rope.scaling.original_context_length); llama.cpp then applies them without flags, and print_info shows rope scaling = yarn and a freq_scale_train below 1. The flags are for models without that metadata. The model card states which applies.

Example 9: context shift, --keep, and sliding-window caches

Problem: a conversation keeps growing, and the context has a fixed size. When it is full, something must be dropped, or generation must stop.

Idea: context shift drops the middle of the conversation to make room. It keeps the first tokens (the system prompt, which holds the instructions) and removes older messages after them. It is like erasing the oldest lines on a full whiteboard but keeping the title at the top. The analogy stops where the whiteboard erases whole lines, while llama.cpp cuts at a token count and may cut a message in half.

When the cache is full, the tools stop by default (Running llama.cpp: Core Flags, Example 4). --context-shift instead discards old tokens: it keeps the first --keep N tokens ("number of tokens to keep from the initial prompt (default: 0, -1 = all)"), removes half of the rest, and moves the remaining cells back by the removed count. The question is which tokens survive a shift. The run below fills a 256-cell context with --keep 4:

The command below makes the tiny model generate 300 tokens into a context of only 256 cells, with context shift on and the first 4 tokens protected. Look for the line that says the context is full.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "one two three four five" -n 300 -no-cnv -c 256 \
  --seed 1 --ignore-eos -v --context-shift --keep 4

Selected lines from the pinned build (timestamps removed; timings differ by machine):

I generate: n_ctx = 256, n_batch = 2048, n_predict = 300, n_keep = 4
D context full, swapping: n_past = 255, n_left = 251, n_ctx = 256, n_keep = 4, n_discard = 125
I common_perf_print:        eval time =     687.43 ms /   299 runs   (    2.30 ms per token,   434.95 tokens per second)

The second line reports the shift: 255 cells were in use, 4 were kept, 251 were left, and half of those 251 (125, rounded down) were discarded. The third line shows that all 299 generation steps ran.

Without --context-shift, the same command stopped after 250 decode runs. With it, all 299 runs completed: at 255 used cells, the tool removed 125 cells after the 4 kept ones and continued. The program below performs the same arithmetic as tools/completion/completion.cpp on a 15-token conversation, so the effect on content is visible:

The program below repeats the same steps on a tiny conversation of 15 tokens, written as words so you can see which ones survive.

import java.util.ArrayList;
import java.util.List;

public class ContextShift {
    // the arithmetic of tools/completion/completion.cpp when the context is full
    static List<String> shift(List<String> cells, int nKeep) {
        int nPast = cells.size();
        int nLeft = nPast - nKeep;
        int nDiscard = nLeft / 2;
        System.out.printf("context full: n_past = %d, n_keep = %d, n_left = %d, n_discard = %d%n",
                nPast, nKeep, nLeft, nDiscard);
        List<String> kept = new ArrayList<>(cells.subList(0, nKeep));             // llama_memory_seq_rm
        kept.addAll(cells.subList(nKeep + nDiscard, nPast));                     // llama_memory_seq_add(-n_discard)
        return kept;
    }

    public static void main(String[] args) {
        List<String> cells = new ArrayList<>();
        for (String t : "SYS rules: be brief . Q1 A1 Q2 A2 Q3 A3 Q4 A4 Q5 A5".split(" ")) cells.add(t);
        System.out.println("before (" + cells.size() + " cells): " + cells);
        List<String> after = shift(cells, 5);
        System.out.println("after  (" + after.size() + " cells): " + after);
        System.out.println("free cells in a 16-cell context: " + (16 - after.size()));

        // the real run: -c 256 --keep 4 printed n_past = 255, n_left = 251, n_discard = 125
        int nPast = 255, nKeep = 4;
        System.out.println("tiny model run: n_discard = " + (nPast - nKeep) / 2 + ", n_past after = " + (nPast - (nPast - nKeep) / 2));
    }
}

before (15 cells): [SYS, rules:, be, brief, ., Q1, A1, Q2, A2, Q3, A3, Q4, A4, Q5, A5]
context full: n_past = 15, n_keep = 5, n_left = 10, n_discard = 5
after  (10 cells): [SYS, rules:, be, brief, ., A3, Q4, A4, Q5, A5]
free cells in a 16-cell context: 6
tiny model run: n_discard = 125, n_past after = 130

--keep 5 protected the 5-token system prompt. Of the 10 remaining tokens, 5 were discarded: Q1, A1, Q2, A2, and Q3. The cut ignores message boundaries, so A3 survived without its question. The remaining cells were moved back by 5 positions; llama.cpp does this by adding -n_discard to their positions (llama_memory_seq_add), which re-rotates the stored keys instead of recomputing them. The last line reproduces the real run: 125 discarded, 130 cells in use afterwards. The flag is off by default, and the example shows the consequence of turning it on: a model that loses the middle of a conversation can produce answers that ignore earlier instructions or questions. A client that trims or summarizes whole messages before the context fills keeps the conversation coherent instead; a later lesson on context management in coding agents does that.

Some models save cache memory in a different way, built into the model itself.

Sliding-window attention (SWA) is a model design in which some layers attend only to the last W tokens. For those layers, llama.cpp allocates a smaller cache: llama-kv-cache-iswa.cpp sizes it as GGML_PAD(min(n_ctx, n_swa * (unified ? n_seq_max : 1) + n_ubatch), 256) cells. For a model with n_swa = 1024, one sequence, -ub 512, and -c 32768, that is 1,536 cells for the windowed layers against 32,768 for the others. The price is that tokens outside the window are gone from those layers, so the server cannot reuse an old prompt prefix from them. --swa-full ("use full-size SWA cache") gives the windowed layers full-size caches again and logs using full-size SWA cache. The tiny model has no sliding window (n_swa = 0 in print_info), and llama-server started with --swa-full on it logs swa_full is not supported by this model, it will be disabled. Instead of full caches, llama-server saves context checkpoints for such models (-ctxcp N, default 32 per slot): copies of the state at points in the prompt, from which a later request with the same beginning can resume. The source comment in server-context.cpp says checkpoints are created only when the model uses SWA without swa_full, or when its memory cannot be rolled back to an earlier position.

Example 10: speculative decoding with a draft model

Problem: generation produces one token per pass through the model, and each pass must read all the weights from memory. For a large model, that memory reading is the bottleneck, so the speed is limited even when the processor is mostly idle.

Idea: a junior writer drafts several words quickly, and a senior editor reads the whole draft at once and accepts the correct prefix. Reading many words at once costs about the same as reading one, because the cost is loading the weights. The junior is the small draft model, the senior is the large target model, and checking one batch replaces several slow single steps. The analogy stops at one point: the editor never takes a wrong word, so the final text is exactly what the target would have written alone.

Speculative decoding uses a small draft model to propose several tokens, and the main (target) model checks all of them in one batch. Accepted tokens cost one target pass together instead of one pass each; the first rejected token is replaced by the target's own choice. A later lesson on speculative decoding derives the acceptance rule. The question is what llama.cpp reports, using the tiny model as its own draft, which is the case where every proposal should be accepted.

The command below uses the same tiny model as both draft and target. That makes every draft token correct, which shows the best case. Look at the accept line at the end.

llama-speculative-simple -m tiny-random-qwen2vocab-f32.gguf -md tiny-random-qwen2vocab-f32.gguf \
  --spec-type draft-simple -p "def add(a, b):" -n 64 --seed 1 -c 512 --spec-draft-n-max 8 --temp 0

Selected lines from the pinned build (timestamps removed):

I common_speculative_init_result: loading draft model 'tiny-random-qwen2vocab-f32.gguf'
I n_draft   = 8
I n_predict = 72
I n_drafted = 64
I n_accept  = 64
I accept    = 100.000%

The acceptance rate is n_accept / n_drafted, here 64 / 64 = 100%.

With greedy sampling (--temp 0) and identical models, the target agrees with every proposal: 64 of 64 drafted tokens were accepted. Each round drafts up to 8 tokens (--spec-draft-n-max 8, default 3) and the target verifies them in one batch; 8 rounds of 8 accepted tokens plus the target's own next token give 72 predicted tokens, more than the 64 requested, because the tool stops only after a full round. Without --spec-type draft-simple, the same command fails with failed to initialize speculative decoding: at this commit, --spec-type (default none) selects the method, and -md (--spec-draft-model) only provides the model.

The same flags work in llama-server, which reports acceptance per request:

The server can do the same and reports acceptance for every request.

llama-server -m tiny-random-qwen2vocab-f32.gguf -md tiny-random-qwen2vocab-f32.gguf --spec-type draft-simple \
  --spec-draft-n-max 8 --host 127.0.0.1 --port "$PORT" -np 1 -c 1024
curl -s "127.0.0.1:$PORT/completion" -d '{"prompt":"def add(a, b):","n_predict":64,"temperature":0}'

The server log line and the matching timings fields of the response (from the pinned build):

I slot print_timing: id  0 | task 0 | draft acceptance = 1.00000 (   56 accepted /    56 generated), mean len =  9.00

"predicted_n": 64, "draft_n": 56, "draft_n_accepted": 56

The server stops exactly at 64 tokens, so it drafted 56. Mean length 9.00 is the tokens gained per target pass: 8 accepted drafts plus the target's own token.

The same tool with --temp 0.8 gives a different picture: n_drafted = 250, n_accept = 1, accept = 0.400%. The draft proposes its most likely token (common/speculative.cpp takes the first candidate after a top-k of 10), and the target samples from its distribution at temperature 0.8. With random weights over 151,936 tokens, that distribution is nearly flat, so the sampled token almost never equals the most likely one. Real models produce peaked distributions on predictable text such as code; the example statistics in docs/speculative.md show acceptance rates of 0.58 and 0.70. The speedup depends on that rate and on how much cheaper the draft is than the target; with a draft as expensive as the target, as here, speculation saves nothing even at 100%. --spec-type also accepts draft-free methods, such as ngram-simple and ngram-mod, which propose tokens by finding the current text earlier in the context; they suit tasks such as editing code, where the output repeats the input.

Example 11: --jinja, template files, and reasoning flags

Problem: a chat model does not see "messages". It sees one long string with special markers that show who is speaking. The program that builds this string is the chat template (Chat Templates, Roles, and Special Tokens explains the format). If the template is wrong, the model gets a confusing prompt and answers badly.

Some models also reason: they write their thinking inside <think> and </think> tags before the answer. Thinking improves answers on hard questions but costs extra tokens, so llama.cpp has flags to turn it on or off.

--jinja renders the chat template with llama.cpp's Jinja engine, and --chat-template-file FILE replaces the template from the GGUF metadata. The reasoning flags apply to models that write their thinking between tags before the answer: -rea on|off|auto (--reasoning, "Use reasoning/thinking in the chat", default auto, detected from the template), --reasoning-format none|deepseek|deepseek-legacy (whether thinking stays in content or moves to reasoning_content), --reasoning-budget N ("token budget for thinking: -1 for unrestricted, 0 for immediate end"), and --reasoning-effort LEVEL (a value given to templates that support it). The question is what -rea changes in the prompt. llama-server has an /apply-template endpoint that returns the rendered prompt without generating (llama-server: Endpoints, Slots, and Prompt Cache covers it); the runs load the Qwen3 template from the repository's models/templates/ directory into the tiny model.

The command starts the server with a template file and --reasoning on; the second run changes only that flag to off. The curl line then asks the server to show the prompt it would send to the model for a one-word user message. Compare the two outputs at the end of the string.

llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port "$PORT" -c 1024 -np 1 \
  --chat-template-file models/templates/Qwen-Qwen3-0.6B.jinja --reasoning on      # second run: --reasoning off
curl -s "127.0.0.1:$PORT/apply-template" -d '{"messages":[{"role":"user","content":"Hi"}]}'

Responses from the pinned build, with --reasoning on and then --reasoning off:

{"prompt":"<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"}

{"prompt":"<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"}

The template file replaced the model's own template, and --jinja (on by default in llama-server) ran it. With reasoning on, the prompt ends where the assistant turn begins, and the model is free to start with <think>. With reasoning off, the template itself appends an empty <think>\n\n</think> block, so the model continues after the thinking section and answers directly. This is how Qwen3's template implements its enable_thinking switch, and llama.cpp passes the switch to the template from -rea. A model whose template has no such switch ignores the flag, which is why the effect is checked in the rendered prompt.

--reasoning-budget N caps thinking at generation time instead: after N thinking tokens, llama.cpp forces the end-of-thinking tag (preceded by --reasoning-budget-message if set), and 0 ends thinking immediately. --reasoning-format deepseek parses the output and returns the thinking in a separate reasoning_content field, so a client such as loop chat can show it apart from the answer. A template file is also the fix when a GGUF ships a template with a bug: the corrected template is loaded with --chat-template-file, and /apply-template confirms the result before any generation.

Example 12: automatic parameter fitting

Problem: if the settings need more memory than the machine has, the program crashes with an out-of-memory error, and you fix it by trial and error. Each trial means loading a large model, which takes time.

Idea: llama.cpp can compute the memory need before loading the real weights, compare it with the free memory, and lower the settings that you did not choose yourself. This is the fit step. Host memory below means the system RAM of the machine.

The fit step runs before loading in every tool ("-fit, --fit [on|off] ... default: 'on'"). It measures the memory of the requested settings by loading the model without data, and if they do not fit the free memory minus a margin (-fitt, default 1024 MiB per device), it reduces settings that the user did not set: first the context (not below -fitc, default 4096), then the GPU layers, then it moves MoE experts to the CPU. llama-fit-params runs the same step and prints the resulting flags. The question is what it prints and how it decides.

The three commands below ask the fit tool for its result with default settings, for its memory estimate at a context of 32,768, and for a forced reduction. In the third, -fitt 15600 sets a very large safety margin so that the tool has to reduce something.

llama-fit-params -m tiny-random-qwen2vocab-f32.gguf
llama-fit-params -m tiny-random-qwen2vocab-f32.gguf --fit-print on -c 32768
llama-fit-params -m tiny-random-qwen2vocab-f32.gguf -fitc 256 -fitt 15600 -lv 4

Output of the three runs from the pinned build (stderr and stdout together; timestamps removed):

I llama_fit_params: printing fitted CLI arguments to stdout...
-c 4096 -ngl -1

I llama_fit_params: printing estimated memory in MiB to stdout (device, model, context, compute) ...
Host 307 32 297

I common_params_fit_impl: getting device memory data for initial parameters:
I common_params_fit_impl: projected to use 609 MiB of host memory vs. 16094 MiB of total host memory
I common_params_fit_impl: cannot meet free memory target of 15600 MiB, need to reduce device memory by 114 MiB
I common_params_fit_impl: context size reduced from 4096 to 1024 -> need 122 MiB less memory in total
I common_params_fit_impl: entire model can be fit by reducing context
I common_fit_params: successfully fit params to free device memory
I llama_fit_params: printing fitted CLI arguments to stdout...
-c 1024 -ngl -1

The first run found that the defaults fit: the full training context (4096) and -ngl -1 (all layers, which on a CPU build means nothing to offload). The second, with --fit-print on, prints the estimate itself: 307 MiB of weights, 32 MiB of KV cache for 32,768 cells, and 297 MiB of compute buffer.

The third run forces a reduction on a CPU-only machine. With no GPU, the fit compares against host memory: 609 MiB projected out of 16,094 MiB total. A margin of 15,600 MiB leaves 494 MiB, 114 MiB short. The fit lowered the context from 4,096 toward the minimum of 256 and chose 1,024. Its estimate of the saving, 122 MiB, comes from a straight line between the memory at 4,096 cells and at 256 cells (fit.cpp comments it as "linear interpolation between minimum and maximum context size"), and the result is rounded down to a multiple of 256. For this model the line overstates the saving: at 256 cells the micro-batch is also capped at 256, which halves the 297 MiB logits buffer, but at 1,024 cells the micro-batch is back to 512. The fit is an estimate that errs on the cheap side when the compute buffer depends on the context; the startup log of the real run shows the actual sizes.

On a GPU build, the same function also distributes layers. The lines below follow the format strings of common/fit.cpp with illustrative values for a 30B MoE model on a 24 GB GPU, not a capture:

common_params_fit_impl: projected to use 61807 MiB of device memory vs. 24077 MiB of free device memory
common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 38754 MiB
common_params_fit_impl: context size reduced from 40960 to 4096 -> need 3456 MiB less memory in total
common_params_fit_impl: with only dense weights in device memory there is a total surplus of 16164 MiB
common_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next device/system memory:
common_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 4090): 48 layers (34 overflowing),  19187 MiB used,   1199 MiB free

The sequence is the decision order: reduce the context, check whether the dense weights alone fit (they do, with 16 GB to spare), then put whole layers on the GPU and let the experts of 34 layers "overflow" to system memory. llama-fit-params then prints -c 4096 -ngl 48 -ot "..." with one exps=CPU override per overflowing layer, the same placement Example 6 builds by hand. The fit changes only settings left at their defaults: an explicit -c, -ngl, -ts, or -ot makes the step keep that value (or abort with a message such as n_gpu_layers already set by user to 20, abort), and --fit off disables it. Because the fit runs before every start, loop serve can either rely on it or run llama-fit-params once and store the printed flags, which makes later starts independent of the memory free at that moment.

Example 13: a tuning procedure and commands for four hardware tiers

The examples above combine into a procedure. The question is what a tuned command looks like on each hardware tier of the course, for a 7B to 8B coding model in Q4_K_M used by one person with a coding agent.

Each tier has a different limit. On a CPU, the limit is how fast memory can be read, so extra threads help only up to a point. On Apple Silicon, the CPU and GPU share one pool of memory, so the context and cache type decide whether the model fits. On a single NVIDIA GPU, VRAM is the limit, so the main question is what must stay in system memory. With several GPUs, the question is how to split the model across them.

The procedure:

1. Estimate memory before downloading: weights plus KV cache for the target context (Memory Math), or llama-fit-params --fit-print on once the file is present.
2. Start from the defaults, which already fit memory, offload all layers, and enable flash attention when it is supported.
3. Set the context to what the application needs, not the training maximum; quantize the KV cache to q8_0 when the context does not fit otherwise.
4. Sweep only the flags that can matter on the hardware (-t on CPUs, -ub for prompt-heavy work, -ngl or --n-cpu-moe when VRAM is short) with llama-bench -r 5 at depth 0 and at the working depth with -d.
5. Discard unstable rows, keep the cheaper of equal settings, and confirm the final command with llama-bench and one real request.

Tier 1, CPU only with 16 GB of RAM and 16 cores, 8 of them performance cores:

llama-server -m qwen2.5-coder-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 \
  -c 16384 -ctk q8_0 -ctv q8_0 -t 8 -tb 16 -np 1

Generation is bound by memory bandwidth, so -t stops at the performance cores, while -tb uses every core for prompts. The quantized cache halves the KV memory of a long agent session. One slot gives the whole context to one conversation.

Tier 2, Apple Silicon with 16 GB or more of unified memory:

llama-server -m qwen2.5-coder-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 \
  -c 32768 -ctk q8_0 -ctv q8_0 -np 1

Metal is on by default in Apple builds, -ngl defaults to all layers, and -fa auto enables flash attention where the Metal kernels support the model. The GPU shares system memory, and the Metal backend reports recommendedMaxWorkingSetSize as the device's total memory (ggml-metal-device.m), so the fit step plans against that value, which the Metal startup log prints, not against the full RAM size.

Tier 3, Linux with one NVIDIA GPU of 12 to 24 GB:

llama-server -m qwen2.5-coder-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 \
  -c 32768 -np 2 -ngl all
# a larger MoE model that does not fit: keep the experts of the first layers in system memory
llama-server -m <moe-model>-q4_k_m.gguf --host 127.0.0.1 --port 8080 -c 32768 -ngl all --n-cpu-moe 20

With the model fully in VRAM, two slots let an agent and a chat share the server (each slot gets 16,384 tokens unless -kvu is added). For an MoE model larger than VRAM, --n-cpu-moe is lowered until the fit step reports free memory above the margin; the fit step without any flag reaches a similar placement automatically.

Tier 4, several GPUs or a rented cloud GPU:

llama-server -m <large-model>-q4_k_m.gguf --host 127.0.0.1 --port 8080 -c 65536 -np 4 -ngl all -sm layer -ts 1,1

Layer split (-sm layer, the default) places whole layers on each GPU in the given proportions and pipelines micro-batches between them; -sm row and the experimental -sm tensor split each weight. For many concurrent users on data-center GPUs, vLLM (the lesson vLLM Architecture and Installation covers it) is built for batching at that scale, and a later lesson on multi-GPU inference compares the approaches.

Interview points

• -fa auto turns flash attention on only when the fused operation can run on the same device as its layer. Forcing on when a kernel is missing moves attention to the CPU, which is slower than off.
• Without flash attention, the table of attention scores (n_ubatch * n_ctx * n_head * 4 bytes) must fit in the compute buffer. With flash attention, the buffer does not grow with the context.
• A quantized V cache requires flash attention. With auto, llama.cpp turns it on. With off, context creation fails. The K cache can be quantized alone. CUDA builds compile flash-attention kernels only for a list of K/V pairs.
• llama-bench reports prompt processing (pp) and generation (tg) separately, each with a standard deviation. A coefficient of variation above about 10% means the row needs a rerun, and a difference smaller than the noise says nothing.
• Sweep only the flags that can affect a test. For example, -ub cannot change generation speed, because generation processes one token per step.
• -d measures generation at a given depth. Speed falls as depth grows because attention reads more cached cells.
• -ot patterns are regular expression searches, and the first match wins. --cpu-moe and --n-cpu-moe add patterns that match only expert tensors, so attention, the router, and shared experts stay on the GPU.
• With partial offload, the layers on the CPU dominate the time per token. Even one layer on the CPU costs a large share of the speed.
• RoPE scaling (--rope-scale, YaRN) is needed only when the context exceeds the trained length and the model metadata has no scaling of its own.
• Context shift keeps the first --keep tokens and drops half of the rest without regard to message boundaries. It is off by default.
• Speculative decoding needs --spec-type at this commit. Acceptance depends on how peaked the target's probabilities are and on the sampling temperature.
• The fit step changes only settings that the user left unset. It reduces the context first, then the GPU layers, then moves MoE experts to the CPU.

Summary

• On the tiny model at -ub 32, the compute buffer was 18.61 MiB with flash attention at both 4,096 and 32,768 cells, and 36.17 MiB without it at 32,768.
• Flash attention reduced the computation graph from 150 to 130 nodes for the 4 layers.
• q8_0 K and V caches took 2.12 MiB against 4.00 MiB in F16 at 4,096 cells.
• The combination -ctk q8_0 -ctv q8_0 -fa off fails with "quantized V cache requires flash_attn to be enabled".
• The default CUDA flash-attention K/V pairs are q4_0-q4_0, q8_0-q8_0, f16-f16, and bf16-bf16.
• llama-bench defaults to -p 512, -n 128, -r 5, and -o md. The -oe flag writes a second format to stderr, and parameter values accept lists and ranges.
• In the sweep, flash attention made prompt processing 1.3 to 2.3 times faster, and three of the 16 results had spreads too large to use.
• --cpu-moe uses the pattern \.ffn_(up|down|gate|gate_up)_(ch|)exps, and --n-cpu-moe N applies it to layers 0 to N-1.
• --rope-scale 4 sets freq_scale = 0.25. With YaRN, libllama raises n_ctx_train to the scaled length.
• A context shift at 255 of 256 cells with --keep 4 discarded 125 tokens.
• The sliding-window cache size is GGML_PAD(min(n_ctx, n_swa * streams + n_ubatch), 256), and --swa-full makes it full size.
• The tiny model as its own draft with greedy sampling accepted 100% of its drafts, and at temperature 0.8 it accepted 0.4%.
• --fit is on by default with a 1024 MiB margin and a minimum context of 4096. llama-fit-params prints the fitted -c, -ngl, and -ot values.
• The tier commands differ mainly in thread counts (CPU), context and KV type (memory), slots, and expert placement (VRAM).