Running llama.cpp: Core Flags

A llama.cpp program does nothing useful until you tell it what to run and how. You do that with command-line flags: short options such as -m (which model file) or -c (how much text the model can keep in mind). A handful of flags decide most of what you see: how much memory a run needs, how fast it is, when it stops, and how many users a server can handle at once.

This lesson goes through those flags one group at a time. For each group it first explains the problem the flags solve, then shows a real command and the log lines it produces, so you can see the effect of each flag instead of taking it on trust. The previous lesson, llama.cpp Architecture, explains the four parts of the startup log that this lesson refers to.

After the lesson, you can write a llama-server or llama-completion command for a given model and machine, predict the log lines it prints, and turn the command into a reproducible launch script.

Every flag, default, and help text below was checked against common/arg.cpp at commit 0c1e57098 and against the --help output of the binaries built from it. The programs share one flag parser, but some defaults differ per program. For example, --jinja is on in llama-server and off in llama-completion. The text names the program when it matters.

The runs use the course's tiny test model. Its weights are random, so the text it generates is meaningless, but its logs are real.

Example 1: choosing a model file with -m, -hf, and --hf-file

The first thing a program needs is a model. There are two ways to give it one. -m FNAME loads a GGUF file that is already on your disk. -hf <user>/<model>[:quant] downloads the file from a Hugging Face repository first (Hugging Face is a website that hosts model files). -hff FILE (--hf-file) names the exact file inside the repository.

A repository often holds the same model in many quantizations. A quantization is a way of storing the weights with fewer bits to save memory, and a repository file name usually ends with its type, such as Q4_K_M or Q8_0. This example answers which file -hf picks when you do not name one.

llama-server -m ./models/qwen2.5-coder-7b-instruct-q4_k_m.gguf
llama-server -hf ggml-org/gemma-3-1b-it-GGUF            # default quantization
llama-server -hf ggml-org/gemma-3-1b-it-GGUF:Q8_0       # requested quantization
llama-server -hf ggml-org/gemma-3-1b-it-GGUF -hff gemma-3-1b-it-Q4_K_M.gguf

The --help text of -hf says: "quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist". The function find_best_model in common/download.cpp adds three details:

• Without a tag, it tries Q4_K_M and then Q8_0.
• A tag matches when it is followed by . or - in the file name.
• Only the first shard of a split model is selected. A shard is one piece of a model that was cut into several files.

The Java program below repeats that logic on a fixed list of file names, so you can see the rules at work without a network. It tries five requests against one pretend repository, and then one more repository that has neither default quantization. Look at which file each request returns.

import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class HfFileSelect {
    static final Pattern SPLIT = Pattern.compile("^(.+)-([0-9]{5})-of-([0-9]{5})$", Pattern.CASE_INSENSITIVE);

    static boolean isModel(String path) {
        String f = path.substring(path.lastIndexOf('/') + 1);
        return f.endsWith(".gguf") && !f.contains("mmproj") && !f.contains("imatrix");
    }

    static boolean isFirstShardOrSingle(String path) {
        Matcher m = SPLIT.matcher(path.substring(0, path.length() - ".gguf".length()));
        return !m.matches() || Integer.parseInt(m.group(2)) == 1;
    }

    // the order of common/download.cpp find_best_model: requested tag, else Q4_K_M then Q8_0, else first model file
    static String select(List<String> files, String tag) {
        List<String> tags = tag.isEmpty() ? List.of("Q4_K_M", "Q8_0") : List.of(tag);
        for (String t : tags) {
            Pattern p = Pattern.compile(Pattern.quote(t) + "[.-]", Pattern.CASE_INSENSITIVE);
            for (String f : files) {
                if (isModel(f) && p.matcher(f).find() && isFirstShardOrSingle(f)) return f;
            }
        }
        if (tag.isEmpty()) {
            for (String f : files) if (isModel(f) && isFirstShardOrSingle(f)) return f;
        }
        return "(no match)";
    }

    public static void main(String[] args) {
        List<String> repo = List.of(
            "README.md",
            "mmproj-model-f16.gguf",
            "model-Q2_K.gguf",
            "model-q4_k_m.gguf",
            "model-Q8_0.gguf",
            "BF16/model-BF16-00001-of-00002.gguf",
            "BF16/model-BF16-00002-of-00002.gguf");
        for (String tag : new String[] {"", "Q8_0", "bf16", "Q5_K_M"}) {
            System.out.printf("%-8s -> %s%n", tag.isEmpty() ? "(none)" : tag, select(repo, tag));
        }
        System.out.println("fallback -> " + select(List.of("model-IQ4_XS.gguf", "model-Q6_K.gguf"), ""));
    }
}

(none)   -> model-q4_k_m.gguf
Q8_0     -> model-Q8_0.gguf
bf16     -> BF16/model-BF16-00001-of-00002.gguf
Q5_K_M   -> (no match)
fallback -> model-IQ4_XS.gguf

Without a tag, the lower-case model-q4_k_m.gguf wins because matching ignores case. Q8_0 selects its file directly. bf16 selects the first of two shards. llama.cpp downloads the other shards itself and loads them together, so a split model is always named by its first file (Making, Quantizing, and Testing Your Own GGUF explains the -00001-of-00002 naming).

A tag that does not exist gives no match. The real tool then logs "Available GGUF files:" with the list and stops. The last line shows the fallback: a repository without Q4_K_M or Q8_0 files gives the first model file in listing order.

The real filter also skips files whose names contain mtp-, eagle3-, dflash-, or dspark-. These are speculative-decoding companions, small helper models that speed up generation (a later lesson on speculative decoding explains them). -hf also downloads a matching mmproj file for multimodal models (models that accept images or audio) unless --no-mmproj is given.

-hf needs HTTPS, the secure web protocol. The pinned build used in this course was configured without OpenSSL, and -hf fails at once with error: HTTPS is not supported. Please rebuild with one of: -DLLAMA_BUILD_BORINGSSL=ON .... Release builds and package-manager builds include HTTPS. -hft TOKEN (--hf-token, default: the HF_TOKEN variable) is needed for gated repositories, which are repositories that require you to log in and accept terms (The Hugging Face Hub).

Example 2: the download cache directory

A downloaded model can be several gigabytes, so llama.cpp saves it in a cache directory and reuses it on the next run. The files go into the Hugging Face hub cache, with the same models--<org>--<repo>/blobs and snapshots/<commit> layout that The Hub HTTP API and Cache Layout in Java builds. Models downloaded by the hf command or Python libraries are therefore found by llama.cpp too, and the other way round.

The question is which directory is used. get_cache_directory() in common/hf-cache.cpp checks environment variables (named settings that the operating system passes to every program) in a fixed order, and the first one that is set wins. The program below copies that order. It takes the environment as a parameter instead of reading the real one, so its output does not depend on the machine. It tries four environments and prints the cache folder for one repository in each.

import java.nio.file.Path;
import java.util.List;
import java.util.Map;

public class HfCacheDir {
    record Rule(String variable, String suffix) {}

    // order of get_cache_directory() in common/hf-cache.cpp
    static final List<Rule> RULES = List.of(
        new Rule("LLAMA_CACHE", ""),
        new Rule("HF_HUB_CACHE", ""),
        new Rule("HUGGINGFACE_HUB_CACHE", ""),
        new Rule("HF_HOME", "hub"),
        new Rule("XDG_CACHE_HOME", "huggingface/hub"),
        new Rule("HOME", ".cache/huggingface/hub"));

    static Path cacheDir(Map<String, String> env) {
        for (Rule r : RULES) {
            String base = env.get(r.variable());
            if (base != null && !base.isEmpty()) {
                return r.suffix().isEmpty() ? Path.of(base) : Path.of(base, r.suffix());
            }
        }
        throw new IllegalStateException("Failed to determine HF cache directory");
    }

    static String repoFolder(String repoId) {
        return "models--" + repoId.replace("/", "--");
    }

    public static void main(String[] args) {
        List<Map<String, String>> cases = List.of(
            Map.of("HOME", "/home/ana"),
            Map.of("HOME", "/home/ana", "XDG_CACHE_HOME", "/data/cache"),
            Map.of("HOME", "/home/ana", "HF_HOME", "/models/hf"),
            Map.of("HOME", "/home/ana", "HF_HOME", "/models/hf", "LLAMA_CACHE", "/fast/llama"));
        String repo = "ggml-org/gemma-3-1b-it-GGUF";
        for (Map<String, String> env : cases) {
            System.out.println(cacheDir(env).resolve(repoFolder(repo)));
        }
    }
}

/home/ana/.cache/huggingface/hub/models--ggml-org--gemma-3-1b-it-GGUF
/data/cache/huggingface/hub/models--ggml-org--gemma-3-1b-it-GGUF
/models/hf/hub/models--ggml-org--gemma-3-1b-it-GGUF
/fast/llama/models--ggml-org--gemma-3-1b-it-GGUF

Read the output line by line against the order in the program:

• With only HOME set, the cache is ~/.cache/huggingface/hub, the same default as the Python huggingface_hub library.
• XDG_CACHE_HOME moves it to <dir>/huggingface/hub.
• HF_HOME is the Hugging Face root, so hub is appended.
• LLAMA_CACHE wins over everything and is used as is, without a hub subdirectory.

The repository folder replaces / with -- and adds the models-- prefix. On Windows the last fallback is USERPROFILE instead of HOME.

Two flags work with the cache. -cl (--cache-list) prints the cached models and exits. On an empty cache the pinned build prints number of models in cache: 0. --offline "forces use of cache, prevents network access", so a server that must not reach the internet can still start from -hf names that were downloaded earlier. loop keeps its own model registry (built in Workshop: loop pull, list, show, rm, and fit) and passes -m with a full path, which avoids network access at engine start.

Example 3: the context size -c and its default

The context is the amount of text, counted in tokens, that the model can hold in mind at once: the prompt plus everything it has generated so far. -c N (--ctx-size) sets it. Technically, it sets the number of cells in the KV cache, the memory that stores one entry for each token. Each cell holds one token, so 4096 cells means at most 4096 tokens.

The help text says "default: 0, 0 = loaded from model". The model file records how long a text it was trained on, called the training context. This example shows what value a run ends up with, and what libllama does with values that are too small or too large. The three commands below run the same model with no -c, with -c 64, and with -c 16384.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 1 -no-cnv -lv 4            # no -c
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 1 -no-cnv -lv 4 -c 64
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 1 -no-cnv -lv 4 -c 16384

Selected lines from the three runs of the pinned build (timestamps removed). Each block is one run, in the order above.

I llama_context: n_ctx                 = 4096

I llama_context: n_ctx                 = 256

I llama_context: n_ctx                 = 16384
I llama_context: n_ctx_seq             = 16384
W llama_context: n_ctx_seq (16384) > n_ctx_train (4096) -- possible training context overflow
W llama_completion: model was trained on only 4096 context tokens (16384 specified)

Without -c, the context is the model's training context, llama.context_length = 4096 from the metadata (n_ctx_train in print_info). For a current model that value is often 32,768 or 131,072, and the KV cache for it can be larger than the weights (Memory Math, Example 4). Two mechanisms limit the damage:

• The fit step (--fit, on by default) reduces an unset context until the model fits in device memory, but not below --fit-ctx (default 4096).
• -c 0 given explicitly means "the full training context, do not reduce". The handler in arg.cpp sets the fit step's minimum context to the maximum value when it sees 0.

Tuning llama.cpp runs the fit step.

-c 64 gave 256 cells. llama-context.cpp rounds the context up to a multiple of 256 (GGML_PAD(cparams.n_ctx, 256)), so a small value costs the same as 256. The completion tool also has a lower bound: below 8 it logs "minimum context size is 8, using minimum size".

-c 16384 is accepted on a model trained for 4096 positions, with two warnings. The model never saw positions beyond its training length. RoPE (the rotation that tells the model where a token sits in the text) produces attention patterns at those positions that the model did not learn, so quality drops. RoPE scaling flags change that (Tuning llama.cpp, Example 8).

The KV cache grows with -c whether or not the cells are used. The 16,384-cell cache of this model is 16 MiB, four times the 4096-cell cache, because 16,384 is four times 4096.

Example 4: -n and the ways generation stops

A model can keep generating forever, so a run needs a rule for stopping. -n N (--n-predict) caps the number of generated tokens. The help text of llama-completion gives the values: "-1 = infinity, -2 = until context filled". llama-server accepts only the first two meanings (a positive number or -1).

There are four ways a run can end: the -n limit, an end-of-generation token, a full context, or a stop string. This example shows the first and the third. The two commands below use -c 256, which gives 256 cells, and a fixed seed (a number that makes random choices repeatable). The first stops at 8 tokens. The second uses -n -2 and runs until the context is full.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "one two three four five" -no-cnv -c 256 --seed 1 -n 8
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "one two three four five" -no-cnv -c 256 --seed 1 -n -2

Selected lines from the pinned build (timestamps removed; timings differ by machine). The first block is the -n 8 run and the second is the -n -2 run.

I generate: n_ctx = 256, n_batch = 2048, n_predict = 8, n_keep = 0
I common_perf_print: prompt eval time =       4.37 ms /     5 tokens (    0.87 ms per token,  1144.95 tokens per second)
I common_perf_print:        eval time =      22.50 ms /     7 runs   (    3.21 ms per token,   311.07 tokens per second)

I generate: n_ctx = 256, n_batch = 2048, n_predict = -2, n_keep = 0
llama_completion: context full and context shift is disabled => stopping
I common_perf_print: prompt eval time =       4.06 ms /     5 tokens (    0.81 ms per token,  1230.62 tokens per second)
I common_perf_print:        eval time =     700.71 ms /   250 runs   (    2.80 ms per token,   356.78 tokens per second)

With -n 8, the prompt of 5 tokens was decoded in one batch, and 7 more decode runs produced the rest. The first new token comes from the prompt's logits (the model's scores at the end of the prompt), so 8 tokens need only 7 extra runs. Prompt tokens are processed at about 1,100 per second and generated tokens at about 300 per second. These are the two speeds that llama.cpp Architecture, Example 13, explains.

With -n -2, generation ran until the cache was full. The 5 prompt tokens plus 250 decode runs fill 255 of the 256 cells, and the next token would not fit. The completion tool checks n_past + new tokens >= n_ctx before each decode and stops with the warning shown. It stops because context shift is off by default at the pinned commit. Context shift means discarding old tokens to make room for new ones (--context-shift). llama-server has the same flag, also off by default. llama-server: Endpoints, Slots, and Prompt Cache shows its error response for a request that does not fit the context.

The third way to stop is an end-of-generation token (EOG). This is a special token that the model produces when it considers its answer finished. It comes from the EOG list printed at load time (</s>, <|endoftext|>, <|im_end|> for this vocabulary). The sampled token is checked against the list after every step.

At -lv 4, the startup log shows lines such as added <|im_end|> logit bias = -inf for every EOG token. common_init in common/common.cpp prepares that list of biases for every run, but adds it to the active sampler only when --ignore-eos is given ("ignore end of stream token and continue generating (implies --logit-bias EOS-inf)"). A logit bias adds a fixed amount to one token's score before a token is picked. A bias of minus infinity makes the token's probability zero. With --ignore-eos the model can therefore never stop by itself. Benchmarks use this to generate an exact number of tokens.

The fourth way is a stop string. The help text of -r PROMPT (--reverse-prompt) reads "halt generation at PROMPT, return control in interactive mode". Servers take stop strings per request instead (the stop field). Generation, Prefill, Decode, and the KV Cache shows how a stop string that is split across two tokens is detected.

Example 5: GPU offload with -ngl, -sm, -mg, -ts, and -dev

A GPU is much faster than a CPU at the matrix math that models need, but its own memory (VRAM) is limited. Offload means moving some or all of the model's layers into VRAM so the GPU runs them. -ngl N (--n-gpu-layers) says how many layers to offload. The help text reads "either an exact number, 'auto', or 'all' (default: auto)".

This example answers three questions: what the flag does on a CPU-only build, which layers a number selects on a GPU build, and what the log reports. The command asks for 99 layers on a build that has no GPU support.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -ngl 99 -p "hi" -n 1 -no-cnv

Output from the pinned CPU-only build (stderr, the stream where programs print warnings):

warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

The three lines are printed by the flag handler in arg.cpp when llama_supports_gpu_offload() is false. The run then continues on the CPU. The same warning is a common first sign of a build without CUDA or Metal.

On a GPU build, src/llama-model.cpp places layers from the end of the model. Layer index il goes to the GPU when il >= n_layer + 1 - ngl, where index n_layer is the output layer. For example, in a model with 32 layers and -ngl 20, the threshold is 32 + 1 - 20 = 13, so layers 13 to 32 go to the GPU.

The program below applies that rule to a 32-layer model, and it prints the three load_tensors lines with the exact formulas from the source. The output is therefore what a GPU build prints for these flags. It tries -ngl values 0, 1, 20, 32, 33, and 99.

public class OffloadLines {
    // the three load_tensors lines of src/llama-model.cpp, for a model with nLayer repeating layers
    static void print(int nLayer, int nGpuLayers) {
        int iGpuStart = Math.max(nLayer + 1 - nGpuLayers, 0);
        int nGpu = Math.min(nGpuLayers, nLayer);
        int nRepeating = nGpu;
        System.out.printf("-ngl %d: GPU %s, CPU %s%n", nGpuLayers,
                range(iGpuStart, nLayer), range(0, Math.min(iGpuStart, nLayer + 1) - 1));
        if (nRepeating > 0) {
            System.out.println("  load_tensors: offloading output layer to GPU");
            nRepeating--;
        }
        System.out.printf("  load_tensors: offloading %d repeating layers to GPU%n", nRepeating);
        System.out.printf("  load_tensors: offloaded %d/%d layers to GPU%n", Math.min(nGpuLayers, nLayer + 1), nLayer + 1);
    }

    static String range(int from, int to) {
        return from > to ? "none" : "layers " + from + ".." + to;
    }

    public static void main(String[] args) {
        int nLayer = 32;                 // a Llama-3-8B-shaped model
        for (int ngl : new int[] {0, 1, 20, 32, 33, 99}) print(nLayer, ngl);
    }
}

-ngl 0: GPU none, CPU layers 0..32
  load_tensors: offloading 0 repeating layers to GPU
  load_tensors: offloaded 0/33 layers to GPU
-ngl 1: GPU layers 32..32, CPU layers 0..31
  load_tensors: offloading output layer to GPU
  load_tensors: offloading 0 repeating layers to GPU
  load_tensors: offloaded 1/33 layers to GPU
-ngl 20: GPU layers 13..32, CPU layers 0..12
  load_tensors: offloading output layer to GPU
  load_tensors: offloading 19 repeating layers to GPU
  load_tensors: offloaded 20/33 layers to GPU
-ngl 32: GPU layers 1..32, CPU layers 0..0
  load_tensors: offloading output layer to GPU
  load_tensors: offloading 31 repeating layers to GPU
  load_tensors: offloaded 32/33 layers to GPU
-ngl 33: GPU layers 0..32, CPU none
  load_tensors: offloading output layer to GPU
  load_tensors: offloading 31 repeating layers to GPU
  load_tensors: offloaded 33/33 layers to GPU
-ngl 99: GPU layers 0..32, CPU none
  load_tensors: offloading output layer to GPU
  load_tensors: offloading 31 repeating layers to GPU
  load_tensors: offloaded 33/33 layers to GPU

A 32-layer model has 33 offloadable layers: 32 repeating blocks (indices 0 to 31) and the output layer (index 32). The token embedding stays on the CPU in every case. Reading the results:

• -ngl 1 offloads only the output layer, the largest single matrix in small models.
• -ngl 20 offloads layers 13 to 31 and the output layer. The first 13 blocks run on the CPU.
• -ngl 32 leaves only block 0 on the CPU, and -ngl 33 offloads everything.
• Values above 33 behave like 33, which is why commands often use -ngl 99. At this commit -ngl all says the same thing directly.

The middle line has a quirk. It counts at most n_layer layers and subtracts one for the output layer, so with 33 or more it still says "31 repeating layers" although all 32 blocks are on the GPU. The third line, offloaded 33/33, is the reliable count. -v also prints one layer N assigned to device ... line per layer.

The default auto (the value -1) means all layers. The fit step then lowers the count until the model fits in free VRAM with a margin (--fit-target, default 1024 MiB per device). An explicit number turns that part of fitting off.

When a machine has several GPUs, more flags decide how the model is divided. The table lists them from --help.

A later lesson on multi-GPU inference compares layer and tensor splitting.

Example 6: CPU threads with -t and -tb

A CPU has several cores, and each thread is one line of work that a core can run. llama.cpp splits each operation across threads. -t N sets the threads for generation and -tb N the threads for batch and prompt processing. -tb defaults to "same as --threads", and -t defaults to -1.

This example shows what -1 resolves to and how the log confirms a setting. The command uses 2 threads for generation and 4 for the 8-token prompt.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "one two three four five six seven eight" -n 16 -no-cnv -c 256 --seed 1 -t 2 -tb 4

Selected lines from the pinned build (timestamps removed; the CPU feature list shortened):

I cmn          init: llama threadpool init, n_threads = 2
I system_info: n_threads = 2 (n_threads_batch = 4) / 4 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 ...

system_info prints both counts and, after the slash, the number of logical CPUs the program sees (4). The thread pool line shows the generation count, 2. The 8-token prompt ran with 4 threads, and each of the 15 decode steps ran with 2.

With -t -1, common_cpu_get_num_math() in common/common.cpp picks the count. On most systems it is the number of physical cores. On x86-64 Linux with a hybrid CPU (performance and efficiency cores), it pins a thread to each CPU in turn and counts only performance cores, skipping hyper-threads. The source comments give the reasons: "efficiency cores harm lockstep threading" and "hyperthreading isn't useful for linear algebra". Lockstep here means that every thread of an operation must finish before the next operation starts, so the slowest thread sets the pace. A slow efficiency core therefore makes all the other threads wait.

llama.cpp Architecture, Example 13, measured how generation and prompt processing scale with threads. The flags for finer control are:

• -C and -Cr: a CPU affinity mask or range, which limits the threads to chosen CPUs.
• --cpu-strict: pin threads strictly to those CPUs.
• --prio: process priority.
• --poll: "use polling level to wait for work", default 50. Threads spin briefly between operations instead of sleeping, which trades CPU time for lower latency.

Each of these has a -batch variant for the batch threads.

Example 7: batch and micro-batch with -b and -ub

A prompt can hold thousands of tokens. The engine does not feed them one by one. It processes them in groups, and two flags set the group sizes. -b N is the "logical maximum batch size (default: 2048)". -ub N is the "physical maximum batch size (default: 512)". A prompt is fed in batches of up to -b tokens, and each batch is computed as graphs of up to -ub tokens.

This example shows how -ub changes prompt speed and memory. llama-bench accepts the same flags, with a list of values to compare. The command below measures a 512-token prompt (-p 512) with four -ub values, no generation (-n 0), and three repetitions each.

llama-bench -m tiny-random-qwen2vocab-f32.gguf -ub 8,32,128,512 -p 512 -n 0 -r 3

Representative output (from the pinned build on a shared 4-core machine; numbers differ by machine and run). The t/s column is tokens per second, so larger is faster.

| model                          |       size |     params | backend    | threads | n_ubatch |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | -------: | --------------: | -------------------: |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |        8 |           pp512 |     2910.02 ± 197.35 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |       32 |           pp512 |      3404.71 ± 56.28 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |      128 |           pp512 |   18287.23 ± 1814.68 |
| llama ?B all F32               | 307.51 MiB |    80.61 M | CPU        |       4 |      512 |           pp512 |    21644.78 ± 367.27 |

A 512-token prompt with -ub 8 runs 64 graphs (512 divided by 8), and each graph reads every layer weight once to process 8 tokens. With -ub 512, one graph reads each weight once for all 512 tokens. The work per token is the same, but the bytes read per token fall by a factor of 64. Prompt processing therefore moves from memory-bound toward compute-bound (FLOPs, Bandwidth, and the Roofline Model calls this the arithmetic intensity, the amount of math done per byte read). Memory-bound means the speed is limited by how fast data arrives, and compute-bound means it is limited by how fast the processor calculates.

Above 128 the gain is small on this CPU, because the matrix kernels are already busy. On GPUs the useful range is often larger, and the table above is the method to find it.

The cost of a larger -ub is memory. llama.cpp Architecture, Example 11, measured the tiny model's compute buffer: 297.75 MiB at -ub 512, 74.44 MiB at 128, and 18.61 MiB at 32. Generation does not use -ub at all, because each step processes one token per sequence.

-b limits how many tokens one decode call accepts. libllama caps the batch at the context size (n_batch = min(n_ctx, n_batch) for causal models) and the micro-batch at the batch (n_ubatch = min(n_batch, n_ubatch)). A run with -b 64 printed n_batch = 64 and n_ubatch = 64. In llama-server, -b is the budget of tokens per step shared by all slots. Embedding models need the whole input in one micro-batch. For that reason llama-server sets -b equal to -ub when embeddings are enabled, and it logs a warning that says so.

Example 8: loading with --load-mode

Before a model can run, its weights must get from the file into memory. There are several ways to do it, each with different effects on start-up time, memory use, and sharing. At the pinned commit, one flag controls it: -lm MODE (--load-mode), with the modes auto, none, mmap, mlock, mmap+mlock, and dio.

Older releases and many tutorials use --mmap, --no-mmap, and --mlock. None of those three exists in common/arg.cpp at this commit, and passing one is an error. This example shows what each mode prints and when to choose it. The loop below starts the program once per mode and keeps only the lines about loading.

for lm in mmap none mlock mmap+mlock dio; do
  llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 2 -no-cnv -c 512 -lv 4 -lm "$lm" 2>&1 | grep -E 'load_mode|model buffer'
done

Output from the pinned build (timestamps removed). There are two lines per mode, in the order of the loop:

I load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
I load_tensors:   CPU_Mapped model buffer size =   307.51 MiB
I load_tensors: loading model tensors, this can take a while... (load_mode = none)
I load_tensors:          CPU model buffer size =   307.51 MiB
I load_tensors: loading model tensors, this can take a while... (load_mode = mlock)
I load_tensors:          CPU model buffer size =   307.51 MiB
I load_tensors: loading model tensors, this can take a while... (load_mode = mmap+mlock)
I load_tensors:   CPU_Mapped model buffer size =   307.51 MiB
I load_tensors: loading model tensors, this can take a while... (load_mode = dio)
I load_tensors:          CPU model buffer size =   307.51 MiB

The buffer type shows the difference. CPU_Mapped appears for the two modes that map the file. A plain CPU buffer appears for the three that read the file into allocated memory. All modes give the same weights and the same output. The table explains each mode.

When most layers are offloaded to a GPU, the weights are copied into VRAM either way. The mode then mainly changes load time and how much host memory stays in use afterwards. Locking memory can fail when the user's lock limit (ulimit -l) is lower than the model size. llama.cpp Architecture, Example 12, measured the resident memory of mmap and none. The related -lzm (--lazy-mode) reads some very large tensors, such as per-layer embeddings, row by row on demand. Its default auto applies this only to tensors larger than 4 GiB.

Example 9: parallel slots with -np and the unified KV cache -kvu

A server may get several requests at the same time. A slot in llama-server is one sequence the server can work on at the same time, which means one request in progress. -np N (--parallel) sets the number of slots ("default: -1, -1 = auto"). Each slot needs its own part of the KV cache.

-kvu (--kv-unified) makes all slots share one KV buffer, "default: enabled if number of slots is auto". The question is how the cells of -c are divided between slots. The three server starts below all use -c 4096: with no -np, with -np 2, and with -np 2 -kvu.

llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port "$PORT" -c 4096 -lv 4
llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port "$PORT" -c 4096 -lv 4 -np 2
llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port "$PORT" -c 4096 -lv 4 -np 2 -kvu

Selected lines of each start from the pinned build (timestamps removed). The blocks are in the order of the commands:

I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
I llama_context: n_seq_max             = 4
I llama_context: n_ctx_seq             = 4096
I llama_kv_cache: size =    4.00 MiB (  4096 cells,   4 layers,  4/1 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB
I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 4096, kv_unified = 'true'

I llama_context: n_seq_max             = 2
I llama_context: n_ctx_seq             = 2048
I llama_context: n_ctx_seq (2048) < n_ctx_train (4096) -- the full capacity of the model will not be utilized
I llama_kv_cache: size =    4.00 MiB (  2048 cells,   4 layers,  2/2 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB
I srv    load_model: initializing, n_slots = 2, n_ctx_slot = 2048, kv_unified = 'false'

I llama_context: n_seq_max             = 2
I llama_context: n_ctx_seq             = 4096
I llama_kv_cache: size =    4.00 MiB (  4096 cells,   4 layers,  2/1 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB
I srv    load_model: initializing, n_slots = 2, n_ctx_slot = 4096, kv_unified = 'true'

All three caches are 4 MiB, because -c sets the total and the slot settings only decide how it is shared. The seqs field in the cache line reads as sequences over KV streams, for example 4/1 seqs is 4 sequences in 1 stream. A stream is one separate block of cells.

• Auto (no -np): tools/server/server.cpp chooses 4 slots with a unified cache. All 4 slots draw cells from one pool of 4096 (4/1 seqs), and each slot may use up to 4096 if the others are idle.
• -np 2 without -kvu: the cache is split into 2 streams of 2048 cells (2/2 seqs). Each slot is limited to 2048 tokens even when the other slot is idle, and libllama warns that the model's 4096 positions cannot be used.
• -np 2 -kvu: 2 slots share the pool of 4096 (2/1 seqs).

A unified cache lets one long request use the whole pool, and lets requests with a common prefix share cells. A split cache guarantees each slot its share, so one long request cannot take cells another slot needs. A request that finds no free slot is deferred. server-context.cpp puts it back in the task queue ("no slot is available, defer task") until a slot is released.

The program below computes the per-slot context with the rules of llama-context.cpp, including the rounding to 256. It tries six combinations of -c, -np, and -kvu.

public class SlotContext {
    static int pad(int n, int multiple) {             // GGML_PAD
        return (n + multiple - 1) / multiple * multiple;
    }

    // n_ctx and n_ctx_seq as llama-context.cpp computes them
    static void show(int ctxFlag, int nSeq, boolean kvUnified) {
        int nCtx = pad(ctxFlag, 256);
        int nCtxSeq = nCtx;
        if (!kvUnified) {
            nCtxSeq = pad(nCtx / nSeq, 256);
            nCtx = nCtxSeq * nSeq;                     // "rounding" warning when this changes n_ctx
        }
        System.out.printf("-c %5d -np %d %-4s -> n_ctx = %5d, n_ctx_seq = %5d, streams = %d%n",
                ctxFlag, nSeq, kvUnified ? "-kvu" : "", nCtx, nCtxSeq, kvUnified ? 1 : nSeq);
    }

    public static void main(String[] args) {
        show(64, 1, false);
        show(4096, 1, false);
        show(4096, 2, false);
        show(4096, 2, true);
        show(4096, 4, true);
        show(5000, 3, false);
    }
}

-c    64 -np 1      -> n_ctx =   256, n_ctx_seq =   256, streams = 1
-c  4096 -np 1      -> n_ctx =  4096, n_ctx_seq =  4096, streams = 1
-c  4096 -np 2      -> n_ctx =  4096, n_ctx_seq =  2048, streams = 2
-c  4096 -np 2 -kvu -> n_ctx =  4096, n_ctx_seq =  4096, streams = 1
-c  4096 -np 4 -kvu -> n_ctx =  4096, n_ctx_seq =  4096, streams = 1
-c  5000 -np 3      -> n_ctx =  5376, n_ctx_seq =  1792, streams = 3

The rows with 4096 match the three server starts. The last row shows the rounding at work. 5000 becomes 5120 (the next multiple of 256). A third of that is 1706, which rounds up to 1792 per slot, and 3 times 1792 is 5376 cells in total. The pinned build prints the same numbers for -c 5000 -np 3, with the warning n_ctx is not divisible by n_seq_max - rounding down to 5376. The message says "down", although the value went up.

The practical rule is to choose -c as the per-slot context times the slot count, a multiple of 256, when the cache is not unified.

Example 10: continuous batching and --cache-reuse

Imagine a server with two requests that arrive together. It could finish the first before starting the second, or it could work on both in the same steps. Continuous batching is the second option. The server adds a new request to the running batch at the next step instead of waiting for the current requests to finish. -cb (--cont-batching) is "enabled" by default in llama-server, and -nocb turns it off.

This example shows what the log says when two requests arrive together. The commands start a server with 2 slots and then send two completion requests at the same time. Each asks for 16 new tokens.

llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port "$PORT" -np 2 -c 2048 --cache-reuse 256 &
for k in 1 2; do
  curl -s "127.0.0.1:$PORT/completion" \
    -d "{\"prompt\":\"def fib(n): return n if n < 2 else fib(n-1) + fib(n-2) # request $k\",\"n_predict\":16,\"seed\":$k}" &
done
wait

Server log lines for the two requests (from the pinned build; timestamps removed; timings differ by machine):

I slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1
I slot launch_slot_: id  1 | task 0 | processing task, is_child = 0
I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
I slot launch_slot_: id  0 | task 1 | processing task, is_child = 0
I slot print_timing: id  0 | task 1 | prompt eval time =       9.21 ms /    27 tokens (    0.34 ms per token,  2930.96 tokens per second)
I slot print_timing: id  0 | task 1 |        eval time =      79.75 ms /    16 tokens (    5.32 ms per token,   188.09 tokens per second)
I slot print_timing: id  0 | task 1 |       total time =      88.96 ms /    43 tokens
I slot print_timing: id  0 | task 1 |    graphs reused =         15
I slot      release: id  0 | task 1 | stop processing: n_tokens = 42, truncated = 0
I slot print_timing: id  1 | task 0 | prompt eval time =       9.94 ms /    27 tokens (    0.37 ms per token,  2717.66 tokens per second)
I slot print_timing: id  1 | task 0 |        eval time =      79.64 ms /    16 tokens (    5.31 ms per token,   188.36 tokens per second)
I slot print_timing: id  1 | task 0 |       total time =      89.57 ms /    43 tokens
I slot print_timing: id  1 | task 0 |    graphs reused =         15
I slot      release: id  1 | task 0 | stop processing: n_tokens = 42, truncated = 0

Both requests were assigned before either finished. Task 0 went to slot 1 and task 1 to slot 0. Each was chosen as the least recently used (LRU) free slot, and t_last = -1 means the slot was never used. Their total times are almost equal (88.96 ms and 89.57 ms) for requests sent at the same time. This means the 16 generation steps of both requests ran in the same decode calls: each step put one token of each slot into one batch.

graphs reused = 15 shows that the step graph was built once and reused for the other 15 steps. Each slot ended with n_tokens = 42. That is 27 prompt tokens plus 16 generated, minus the last sampled token, which is returned but never decoded.

A later lesson on batching and scheduling explains why decoding two sequences together costs little more than one: the weights are read once per step for both.

Prompt caching keeps a slot's KV cells after a request, so a later request with the same beginning skips the shared part. It is controlled by --cache-prompt and is enabled by default. --cache-reuse N extends it: "min chunk size to attempt reusing from the cache via KV shifting, requires prompt caching to be enabled (default: 0)". With it, chunks of at least N tokens that match the cache at a different position are moved instead of recomputed. This helps when text in the middle of a long prompt changes, as when a coding agent edits one file in a long context. llama-server: Endpoints, Slots, and Prompt Cache measures both with cache_n.

Two related server flags exist:

• -cram N (--cache-ram, default 8192 MiB) sets the host memory for saved prompt caches.
• -sps (--slot-prompt-similarity, default 0.10) sets how similar a prompt must be to a slot's cached prompt for the server to prefer that slot over the least recently used one.

Example 11: sampling flags and --samplers

The model does not output one token. It outputs a score for every token in its vocabulary, and a sampler picks one. A chain of samplers can first remove unlikely tokens and then draw from the rest. The sampling flags set the default parameters of the sampler chain from Penalties, Logit Bias, and Sampler Chains:

• --temp (default 0.80): how random the choice is. Lower is more predictable.
• --top-k (40): keep only the 40 highest-scoring tokens.
• --top-p (0.95): keep the smallest group of tokens whose probabilities add up to 0.95.
• --min-p (0.05): drop tokens far less likely than the best one.
• The penalties, DRY, and XTC settings, which discourage repetition.
• -s (--seed, default -1 for a random seed).

--samplers sets the order and selection of samplers with names separated by ;. --sampler-seq does the same with one letter per sampler. This example shows what chain each setting builds. The three commands below pick a custom chain by name, use temperature 0, and use a letter sequence together with penalty flags.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 4 -no-cnv -c 512 --seed 42 \
  --samplers "top_k;min_p;temperature" --top-k 20 --min-p 0.1 --temp 0.6
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 4 -no-cnv -c 512 --seed 42 --temp 0
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 4 -no-cnv -c 512 --seed 42 \
  --sampler-seq kmt --repeat-penalty 1.1 --dry-multiplier 0.8

The sampler chain line of each run, and the parameter block of the first (from the pinned build; timestamps removed). The three blocks after it are in the order of the commands, with the parameter block shown first:

I sampler seed: 42
I sampler params:
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 64
	top_k = 20, top_p = 0.950, min_p = 0.100, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
I sampler chain: logits -> top-k -> min-p -> temp-ext -> dist

I sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist

I sampler chain: logits -> top-k -> min-p -> temp-ext -> dist

The first run lists exactly the three named samplers, in the given order, followed by dist. dist is the final sampler. It draws a token from the remaining distribution with the seeded random generator. temperature appears as temp-ext, the temperature sampler with dynamic-temperature support (--dynatemp-range).

The second run uses the default order from --help: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature. A ? before a name marks a placeholder. llama-sampler.cpp returns an empty sampler named ?penalties, ?dry, and so on when the sampler's parameters disable it (repeat penalty 1.0, DRY multiplier 0, and so on). The chain then shows its position, but it does nothing.

--temp 0 does not remove any sampler. The temperature sampler with a temperature of 0 or less keeps only the token with the highest logit and sets every other logit to minus infinity, which makes dist always pick that token. This is greedy decoding inside the normal chain.

The third run shows a trap. kmt stands for top-k, min-p, temperature (k, m, t in common_sampler_type_to_chr), so the chain contains only those three. The penalties given on the same command line (--repeat-penalty 1.1, --dry-multiplier 0.8) are set in the parameter block but have no sampler in the chain, so they have no effect. A custom --samplers or --sampler-seq must include every sampler whose flags are set. The letters are e penalties, d DRY, s top-n-sigma, k top-k, y typical, p top-p, m min-p, x XTC, t temperature.

llama-server uses these flags as defaults for requests that do not set the fields themselves. A request's temperature, top_k, and other fields override them for that request.

Example 12: conversation flags and chat templates

A chat model expects its input in a fixed layout that marks who is speaking: system, user, or assistant. That layout is the model's chat template, and it is stored in the GGUF file. llama-completion switches to conversation mode when the model has a chat template. It then wraps your input in the template's roles and does not print special tokens.

The flags are:

• -cnv and -no-cnv force conversation mode on or off.
• -sys sets the system prompt.
• -st (--single-turn) runs one turn and exits.
• --verbose-prompt prints the token ids.

The question is which prompt the model really receives. The command below gives a system prompt "You are terse." and the user message "Hi".

llama-completion -m tiny-random-qwen2vocab-f32.gguf -sys "You are terse." -p "Hi" -st -n 4 --seed 3 -c 512 --verbose-prompt

Selected lines from the pinned build (timestamps and level letters removed; a token piece that is a line feed appears as a line break). First comes the example conversation that the tool renders from the template, then the real prompt as token ids and pieces:

llama_completion: chat template is available, enabling conversation mode (disable it with -no-cnv)
llama_completion: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

llama_completion: prompt: 'Hi'
llama_completion: number of tokens in prompt = 18
151644 -> '<|im_start|>'
  8948 -> 'system'
   198 -> '
'
  2610 -> 'You'
   525 -> ' are'
 50537 -> ' terse'
    13 -> '.'
151645 -> '<|im_end|>'
   198 -> '
'
151644 -> '<|im_start|>'
   872 -> 'user'
   198 -> '
'
 13048 -> 'Hi'
151645 -> '<|im_end|>'
   198 -> '
'
151644 -> '<|im_start|>'
 77091 -> 'assistant'
   198 -> '
'

The tool detected the template in the GGUF metadata (tokenizer.chat_template) and printed an example conversation rendered with it. This is the ChatML format of Qwen models. The real prompt has 18 tokens: the system message, the user message Hi, and the opening of the assistant turn, each role between <|im_start|> (151644) and <|im_end|> (151645). A two-character user input became 18 tokens, which is the overhead Chat Templates, Roles, and Special Tokens explains.

A run with -p and no -sys prints the warning "User-specified prompt will pre-start conversation, did you mean to set --system-prompt (-sys) instead?". In conversation mode, -p becomes the first user message.

How the template is rendered depends on --jinja. Jinja is a template language, and many model authors write their chat templates in it.

• With --jinja, the template text is run by llama.cpp's Jinja engine, as the model authors wrote it. Tool definitions and reasoning options are passed to it.
• Without it, llama.cpp matches the template against its list of built-in formats and renders the matching one in C++ code. The --chat-template help lists 54 names at this commit, among them chatml, llama3, mistral-v7, and gemma.

The default differs by program: --jinja is enabled in llama-server and llama-cli and disabled in llama-completion. --chat-template NAME selects a built-in format or, after --jinja, takes template text. --chat-template-file FILE reads a template from a file, which fixes models whose embedded template is wrong. Tool calling requires the Jinja path, because the built-in formats have no tool syntax.

The reasoning flags (-rea on|off|auto, --reasoning-format, --reasoning-budget) apply to models that think before answering. Tuning llama.cpp, Example 11, shows their effect on a template.

Example 13: logging flags

Every program logs through common/log.cpp. Sometimes you want more detail to debug a problem, less noise in a script, or logs that another program can read. This example shows how to get each.

The four commands below capture the logs in four ways. The first prints the first three log lines with the default format. The second lowers the verbosity and removes timestamps and prefixes. The third writes the log to a file. The fourth switches to JSON. In the commands, ... stands for the same model and prompt flags as the first command, and 2>&1 >/dev/null is shell syntax that sends the log stream to the pipe and discards the generated text.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 2 -no-cnv -c 512 --seed 1 2>&1 >/dev/null | head -3
llama-completion ... -lv 2 --no-log-timestamps --no-log-prefix 2>&1 >/dev/null
llama-completion ... --log-file run.log; wc -l run.log; head -1 run.log
llama-completion ... --log-jsonl 2>/dev/null | head -4

Output of the four runs from the pinned build. Each block is one run, in the order of the commands.

0.00.001.505 I llama_completion: llama backend init
0.00.004.001 I llama_completion: load the model and apply lora adapter, if any
0.00.563.176 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden

load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden

26 run.log
0.00.001.554 I llama_completion: llama backend init

{"type":"log","time":1214,"level":"info","msg":"llama_completion: llama backend init\n"}
{"type":"log","time":3142,"level":"info","msg":"llama_completion: load the model and apply lora adapter, if any\n"}
{"type":"fit_memory_breakdown","data":{"unit":"MiB","rows":[{"kind":"host","name":"Host","self":326,"model":307,"context":0,"compute":18}]}}
{"type":"log","time":527197,"level":"warn","msg":"load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden\n"}

The default format has a timestamp and a level letter. The timestamp is minutes, seconds, milliseconds, and microseconds since the program started. 0.00.563.176 is 563.176 ms, from the format %d.%02d.%03d.%03d in log.cpp. The letters are I info, W warning, E error, D debug.

-lv N (--verbosity) sets the threshold:

• 0: output only.
• 1: errors.
• 2: warnings.
• 3: info (the default).
• 4: trace.
• 5: debug.

-v sets it to "infinity". libllama's own info messages (the startup log) count as trace, which is why earlier examples used -lv 4. With -lv 2, only the warning remained, and --no-log-timestamps --no-log-prefix removed the timestamp and the letter.

--log-file FNAME writes the log to a file in the same format. --log-jsonl writes one JSON object per line to stdout instead. Besides log records, it emits structured records such as fit_memory_breakdown. This is the memory estimate of the fit step, here for the tiny model with -c 512: 307 MiB of weights and 18 MiB of compute buffer. time is microseconds since start. A supervisor such as loop serve can parse these lines with Jackson (a Java JSON library) instead of matching text with regular expressions. --log-colors on|off|auto controls ANSI colors, auto meaning colors only when writing to a terminal.

Example 14: environment variables and precedence

Long commands are hard to manage in containers and services, where you want to set values outside the command line. Most flags have an environment variable, shown in --help as (env: LLAMA_ARG_...). Examples are LLAMA_ARG_CTX_SIZE for -c, LLAMA_ARG_N_GPU_LAYERS for -ngl, LLAMA_ARG_THREADS for -t, LLAMA_ARG_UBATCH for -ub, LLAMA_ARG_MODEL for -m, and LLAMA_ARG_PORT for --port. A few use other names, such as HF_TOKEN for -hft and LLAMA_API_KEY for --api-key.

The question is which value wins when the same setting comes from several places. The first command below sets a context size and a micro-batch by variable only. The second sets the context size by variable and by flag, with different values.

LLAMA_ARG_CTX_SIZE=1024 LLAMA_ARG_UBATCH=64 \
  llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 1 -no-cnv -lv 4
LLAMA_ARG_CTX_SIZE=1024 \
  llama-completion -m tiny-random-qwen2vocab-f32.gguf -p hello -n 1 -no-cnv -lv 4 -c 2048

Selected lines of the two runs (from the pinned build; timestamps removed). The first block is the first run and the second block is the second run.

I llama_context: n_ctx                 = 1024
I llama_context: n_ubatch              = 64

warn: LLAMA_ARG_CTX_SIZE environment variable is set, but will be overwritten by command line argument -c
I llama_context: n_ctx                 = 2048

The variables set the values exactly as the flags would. When both are given, the command line wins, and the parser prints a warning so that a stale variable in a shell profile does not go unnoticed. common_params_parse_ex in arg.cpp applies three sources in order, each overriding the one before:

1. A configuration file in INI format. It is /etc/llama.cpp/config.ini, then config.ini in the user configuration directory ($XDG_CONFIG_HOME/llama.cpp/ or ~/.config/llama.cpp/ on Linux and macOS, %APPDATA%\llama.cpp\ on Windows). The source comment says "the same config file is shared by all programs", so unknown keys are ignored.
2. Environment variables.
3. Command-line flags.

Boolean flags read their variable with is_truthy and is_falsey in arg.cpp. The values on, enabled, true, and 1 mean true. The values off, disabled, false, and 0 mean false. LLAMA_ARG_JINJA=0 turns Jinja off in llama-server. Environment variables suit containers, where the image fixes the command and the deployment sets the values. They also suit process supervisors that pass settings without building command lines.

Example 15: a reproducible launch script

A reproducible launch records every setting that affects the run, so the same command gives the same server next week or on another machine. A Java program, such as the loop serve command that a later workshop builds, has to produce that record. Two forms are useful: a shell script that sets environment variables, and an argument list for ProcessBuilder (the Java class that starts another program).

The program below produces both from one map of settings. The variable names are the ones --help prints at the pinned commit. Look at how the two outputs list the same settings in the same order.

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

public class LaunchScript {
    // flag -> environment variable, as printed by --help "(env: ...)" at the pinned commit
    static final Map<String, String> ENV_NAMES = Map.of(
        "--model", "LLAMA_ARG_MODEL",
        "--ctx-size", "LLAMA_ARG_CTX_SIZE",
        "--n-gpu-layers", "LLAMA_ARG_N_GPU_LAYERS",
        "--threads", "LLAMA_ARG_THREADS",
        "--parallel", "LLAMA_ARG_N_PARALLEL",
        "--host", "LLAMA_ARG_HOST",
        "--port", "LLAMA_ARG_PORT");

    static String quote(String s) {
        return "'" + s.replace("'", "'\\''") + "'";
    }

    public static void main(String[] args) {
        Map<String, String> settings = new TreeMap<>(Map.of(
            "--model", "/models/qwen2.5-coder-7b-instruct-q4_k_m.gguf",
            "--ctx-size", "16384",
            "--n-gpu-layers", "all",
            "--threads", "8",
            "--parallel", "2",
            "--host", "127.0.0.1",
            "--port", "8080"));

        // 1. a script that pins every setting in the environment
        StringBuilder script = new StringBuilder("#!/bin/sh\nset -eu\n");
        settings.forEach((flag, value) -> script.append("export ").append(ENV_NAMES.get(flag))
                .append('=').append(quote(value)).append('\n'));
        script.append("exec llama-server --log-file server.log\n");
        System.out.print(script);

        // 2. the same settings as an argument list for ProcessBuilder
        List<String> argv = new ArrayList<>(List.of("llama-server"));
        settings.forEach((flag, value) -> { argv.add(flag); argv.add(value); });
        System.out.println(argv);
    }
}

#!/bin/sh
set -eu
export LLAMA_ARG_CTX_SIZE='16384'
export LLAMA_ARG_HOST='127.0.0.1'
export LLAMA_ARG_MODEL='/models/qwen2.5-coder-7b-instruct-q4_k_m.gguf'
export LLAMA_ARG_N_GPU_LAYERS='all'
export LLAMA_ARG_N_PARALLEL='2'
export LLAMA_ARG_PORT='8080'
export LLAMA_ARG_THREADS='8'
exec llama-server --log-file server.log
[llama-server, --ctx-size, 16384, --host, 127.0.0.1, --model, /models/qwen2.5-coder-7b-instruct-q4_k_m.gguf, --n-gpu-layers, all, --parallel, 2, --port, 8080, --threads, 8]

The settings live in a TreeMap, which keeps keys sorted, so both outputs list them in the same order every time. A diff between two launch scripts then shows only real changes.

The script has four details worth copying:

• It quotes each value with single quotes (and escapes single quotes inside), so a path with spaces survives.
• It uses set -eu to stop on errors and on undefined variables.
• It ends with exec, so llama-server replaces the shell and receives signals from the supervisor directly.
• It uses -ngl all instead of a number, which stays correct when the model changes.

-c 16384 with 2 slots and no -kvu gives each slot 8,192 tokens (Example 9).

The argument list is the form ProcessBuilder takes: one element per argument, with no shell and no quoting, so a value can never be split or interpreted. A launcher that records the exact list, the binary's --version output, and the GGUF file's hash has everything needed to repeat a run. Workshop: loop serve, run, and chat builds this command line from the model's metadata and the memory estimate of loop fit.

Interview points

• -hf repo[:quant] picks Q4_K_M, then Q8_0, then the first model file. It matches tags case-insensitively before . or -, and it downloads into the Hugging Face hub cache, which LLAMA_CACHE overrides.
• -c defaults to the training context. The fit step may reduce an unset context down to --fit-ctx (4096), while an explicit -c 0 keeps the full length. libllama rounds n_ctx up to a multiple of 256.
• Generation stops at the -n limit, an EOG token, a stop string, or a full context (context shift is off by default). --ignore-eos adds minus-infinity biases for all EOG tokens.
• -ngl counts layers from the end, including the output layer. -ngl 99 and -ngl all offload everything. The default auto offloads all layers and lets the fit step reduce the number.
• -t is for generation, which is bandwidth-bound, and -tb is for prompt batches, which are compute-bound. -t -1 uses physical cores, or performance cores on hybrid x86 Linux.
• -ub sets the micro-batch. Larger values raise prompt speed through weight reuse and cost compute-buffer memory. -b caps tokens per decode call.
• At the pinned commit, --load-mode replaces --mmap, --no-mmap, and --mlock.
• -np slots share -c. The cache is split into equal streams when -np is given, and it is one unified pool with -kvu or with auto slots (4 slots).
• A custom --samplers list removes every sampler not named, including penalties whose flags are set.
• --jinja is on by default in llama-server and llama-cli and off in llama-completion. Tool calls need it.
• Settings come from config.ini, then LLAMA_ARG_* variables, then flags, each overriding the previous one. A flag that overrides a variable triggers a warning.

Summary

• -m loads a local file. -hf downloads from Hugging Face and needs a build with HTTPS. -hff names an exact file, -cl lists the cache, and --offline uses only the cache.
• The cache directory order is LLAMA_CACHE, HF_HUB_CACHE, HUGGINGFACE_HUB_CACHE, HF_HOME/hub, XDG_CACHE_HOME/huggingface/hub, ~/.cache/huggingface/hub.
• -c 64 gives 256 cells. A -c larger than the training context logs two warnings.
• -n -2 generates until the context is full. llama-server accepts only -n -1 or a positive limit.
• A CPU-only build ignores -ngl and prints three warning: lines.
• -sm has the modes none, layer (default), row, and tensor (experimental).
• system_info prints n_threads = T (n_threads_batch = TB) / logical CPUs.
• -b defaults to 2048 and -ub to 512. On the tiny model, -ub 512 processed a 512-token prompt about 7 times faster than -ub 8.
• --load-mode takes auto, none, mmap, mlock, mmap+mlock, or dio. The mapped modes print CPU_Mapped model buffer.
• llama-server with no -np uses 4 slots and a unified KV cache.
• Continuous batching (-cb) and prompt caching (--cache-prompt) are on by default. --cache-reuse defaults to 0.
• The default sampling settings are temperature 0.80, top-k 40, top-p 0.95, min-p 0.05, and seed -1. The default order is penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature.
• A ? before a sampler name in the chain marks a sampler disabled by its parameters.
• The default verbosity is 3. libllama's startup lines need -lv 4. --log-jsonl writes JSON lines to stdout.
• Every flag with an (env: ...) note can be set by that environment variable.