Memory Math: Weights, KV Cache, and What Fits

Before you download a 5 GB model file, you want to know whether it will run on your machine. The answer depends on more than the file size. A running model needs memory for three things at once: its weights (the numbers learned in training), its KV cache (key-value cache, the memory where the model remembers the conversation so far), and the working buffers the engine uses while computing. If the total is larger than the memory available, the model either fails to start or runs very slowly.

The good news is that all three parts can be calculated with school arithmetic from a few numbers in the model's configuration. This lesson does each calculation by hand, then in Java, and then checks the results against real startup logs from llama.cpp. The lesson ends with Project 4, a MemoryCalculator program that combines everything.

After this lesson, a reader can say whether a model fits a machine at a given context length and number of parallel users, before downloading it, and can choose how many layers to put on the GPU when it does not all fit.

The table lists the parts of memory and the examples that cover them.

The lesson uses ideas from earlier lessons, with a short reminder of each:

• The model configuration fields (hidden_size, num_hidden_layers, num_key_value_heads, and so on) come from The Transformer Block and the Full Model. They describe the shape of the model: how wide it is and how many layers it has.
• Grouped-query attention, from Self-Attention Step by Step, is a design where several query heads share one key-value head, which makes the cache smaller.
• The KV cache, from Generation, Prefill, Decode, and the KV Cache, stores the keys and values of every earlier token so the model does not recompute them for each new token.

The captured log lines come from the pinned llama.cpp build running the course's tiny model, which has random weights. It cannot write useful text, but its memory numbers are real.

Example 1: weight memory from parameters and bits per weight

Weights are the biggest fixed cost. The rule is simple: the memory for weights equals the number of parameters times the bytes each parameter takes.

weight bytes = parameters * bitsPerWeight / 8

The division by 8 converts bits to bytes. For an 8-billion-parameter model in FP16 (16 bits), that is 8.03e9 * 16 / 8 = 16.06e9 bytes, about 16 GB. At 4.5 bits per weight it is 8.03e9 * 4.5 / 8 = 4.52e9 bytes.

The "bits per weight" (bpw) of a quantized format is a little higher than the bit width of its codes. Quantized formats store numbers in small blocks (for example 32 weights), and each block carries a scale. The scale's bytes are shared by the weights in the block, so the cost per weight is the block's total bytes times 8, divided by the values per block. Number Formats showed this for several formats.

The program holds a table of block sizes for ggml types (the values in gguf-py's GGML_QUANT_SIZES). For each type it computes bpw and the weight memory of an 8B model, both in GB (10^9 bytes) and in GiB (2^30 bytes).

import java.util.Locale;

public class WeightMemory {
    record QuantType(String name, int valuesPerBlock, int bytesPerBlock) {
        double bpw() { return bytesPerBlock * 8.0 / valuesPerBlock; }
    }

    public static void main(String[] args) {
        QuantType[] types = {
            new QuantType("F32", 1, 4), new QuantType("F16", 1, 2), new QuantType("BF16", 1, 2),
            new QuantType("Q8_0", 32, 34), new QuantType("Q6_K", 256, 210), new QuantType("Q5_K", 256, 176),
            new QuantType("Q4_K", 256, 144), new QuantType("Q4_0", 32, 18), new QuantType("Q3_K", 256, 110),
            new QuantType("Q2_K", 256, 84),
        };
        double params = 8.03e9;
        System.out.println("type   block  bytes   bpw      GB      GiB");
        for (QuantType t : types) {
            double bytes = params * t.bpw() / 8;
            System.out.printf(Locale.ROOT, "%-5s %6d %6d %6.4f %7.2f %8.2f%n",
                    t.name(), t.valuesPerBlock(), t.bytesPerBlock(), t.bpw(), bytes / 1e9, bytes / (1L << 30));
        }
    }
}

type   block  bytes   bpw      GB      GiB
F32        1      4 32.0000   32.12    29.91
F16        1      2 16.0000   16.06    14.96
BF16       1      2 16.0000   16.06    14.96
Q8_0      32     34 8.5000    8.53     7.95
Q6_K     256    210 6.5625    6.59     6.13
Q5_K     256    176 5.5000    5.52     5.14
Q4_K     256    144 4.5000    4.52     4.21
Q4_0      32     18 4.5000    4.52     4.21
Q3_K     256    110 3.4375    3.45     3.21
Q2_K     256     84 2.6250    2.63     2.45

Look at the bpw column first. Q8_0 stores 32 one-byte values plus a 2-byte scale, so 34 bytes per 32 weights is 8.5 bpw, not 8. Q4_K stores 256 weights in 144 bytes, which is 144 * 8 / 256 = 4.5 bpw.

Then compare the GB and GiB columns. A GB is 1,000,000,000 bytes, and a GiB is 1,073,741,824 bytes, so the same memory is a smaller number in GiB. The two differ by about 7%. A vendor's "8 GB" is 7.45 GiB in llama.cpp's units, so check which unit a tool reports before comparing numbers.

A file named "Q4_K_M" is not all Q4_K. Its quantization mix keeps some tensors in higher-precision types such as Q6_K, so its average is above 4.5 bpw. Lesson 22 computes these mixes from the pinned llama-quant.cpp. Once a model file exists, its size on disk is the exact weight memory.

Example 2: parameter count from a config

Example 1 used "8.03 billion" as a given. Where does that number come from? It comes from the model's configuration, which lists the sizes of every weight matrix. Counting parameters is multiplying out the sizes of the matrices, as lesson 10 showed. Knowing the count per layer also matters later, because layers are what get moved between GPU and CPU.

The configuration used here is the one published for Llama 3.1 8B in its config.json:

• hidden size 4096 (the width of the vectors that flow through the model)
• 32 layers
• 32 query heads and 8 KV heads
• head dimension 128
• feed-forward network (FFN) size 14336
• vocabulary 128,256 (the number of different tokens)
• untied embeddings (the input embedding table and the output projection are separate matrices)

Each layer has an attention part and an MLP (multi-layer perceptron, the feed-forward part). The attention part has four matrices: query, key, value, and output. The MLP has three: gate, up, and down. Each layer also has two small norm vectors. The program adds these up per layer and then for the whole model.

import java.util.Locale;

public class ParamCount {
    record Config(long hidden, long layers, long heads, long kvHeads, long headDim, long ffn, long vocab, boolean tied) {
        long attention() { return hidden * heads * headDim * 2 + hidden * kvHeads * headDim * 2; }  // q,o + k,v
        long mlp() { return 3 * hidden * ffn; }                                                       // gate, up, down
        long norms() { return 2 * hidden; }
        long perLayer() { return attention() + mlp() + norms(); }
        long embedding() { return vocab * hidden; }
        long output() { return tied ? 0 : vocab * hidden; }
        long total() { return layers * perLayer() + embedding() + output() + hidden; }   // + final norm
    }

    public static void main(String[] args) {
        Config c = new Config(4096, 32, 32, 8, 128, 14336, 128_256, false);
        System.out.println("attention per layer = " + c.attention());
        System.out.println("mlp per layer       = " + c.mlp());
        System.out.println("per layer           = " + c.perLayer());
        System.out.println("embedding           = " + c.embedding());
        System.out.println("output (lm_head)    = " + c.output());
        System.out.printf(Locale.ROOT, "total               = %d (%.2f B)%n", c.total(), c.total() / 1e9);
    }
}

attention per layer = 41943040
mlp per layer       = 176160768
per layer           = 218112000
embedding           = 525336576
output (lm_head)    = 525336576
total               = 8030261248 (8.03 B)

Check the first line by hand. The query and output matrices are each 4096 * 32 * 128 = 16,777,216 numbers, so together 33,554,432. The key and value matrices are each 4096 * 8 * 128 = 4,194,304, so together 8,388,608. The sum is 41,943,040, which matches the output.

The total, 8.03 billion, matches the model's name. Each layer holds 218 million parameters, and 81% of them are in the MLP (176 of 218 million). The embedding and the output projection hold 525 million each, because each is a table of 128,256 tokens times 4096 numbers.

Placement treats these two differently. llama.cpp keeps the token embedding on the CPU, because only one row is read per token. It treats the output projection as one extra layer that can be moved to the GPU (Example 10).

Example 3: KV cache bytes per token

When a model generates text, each new token needs to look at all the earlier tokens. Recomputing the earlier tokens every time would be very slow, so the model stores what it computed for them. For every token, every layer saves two vectors per KV head: a key vector and a value vector. These saved vectors are the KV cache.

The size of the cache for one token follows from counting the vectors:

KV bytes per token = 2 * layers * kvHeads * headDim * bytesPerValue

The factor 2 is for keys and values. With grouped-query attention, the number of KV heads (num_key_value_heads) is smaller than the number of query heads, so the cache is smaller than the number of query heads would suggest.

By hand, for the 8B shape in F16 (2 bytes per value): 2 * 32 * 8 * 128 * 2 = 131,072 bytes, which is 128 KiB for every token. The program computes this and also the same for the course's tiny model, which has 4 layers, 2 KV heads, and head dimension 32.

public class KvPerToken {
    static long kvBytesPerToken(long layers, long kvHeads, long headDim, double bytesPerValue) {
        return (long) (2 * layers * kvHeads * headDim * bytesPerValue);
    }

    public static void main(String[] args) {
        long llama = kvBytesPerToken(32, 8, 128, 2);
        System.out.println("Llama 3.1 8B shape, F16: " + llama + " bytes per token (" + llama / 1024 + " KiB)");

        long tiny = kvBytesPerToken(4, 2, 32, 2);
        long cells = 4096;
        System.out.println("tiny model, F16: " + tiny + " bytes per token");
        System.out.println("tiny model, 4096 cells: " + tiny * cells / (1024 * 1024) + " MiB total, "
                + tiny * cells / 2 / (1024 * 1024) + " MiB for K, " + tiny * cells / 2 / (1024 * 1024) + " MiB for V");
    }
}

Llama 3.1 8B shape, F16: 131072 bytes per token (128 KiB)
tiny model, F16: 1024 bytes per token
tiny model, 4096 cells: 4 MiB total, 2 MiB for K, 2 MiB for V

The 8B shape costs 128 KiB per token. The tiny model costs 1,024 bytes per token, so 4,096 token positions need 4,096 * 1,024 = 4 MiB, half for K and half for V.

To check this prediction, the tiny model was started with a context of 4,096 tokens (-c 4096):

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 4 -c 4096 -no-cnv -lv 4

Representative output (from the pinned llama.cpp build; timestamps removed):

llama_kv_cache:        CPU KV buffer size =     4.00 MiB
llama_kv_cache: size =    4.00 MiB (  4096 cells,   4 layers,  1/1 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB

The log reports exactly 4.00 MiB, with 2.00 MiB for K and 2.00 MiB for V. The calculation matches the real program.

A cell is one token position in llama.cpp's KV cache. The log says "4096 cells, 4 layers". The default KV type is f16 for both K and V (llama-server --help shows default: f16 for -ctk and -ctv).

Example 4: the KV cache from 4K to 128K tokens

The cache size grows in direct proportion to the context length, which is the number of tokens the model keeps in view. Twice the context needs twice the cache. This matters most for coding agents, because they put whole files and long tool outputs into the context, and use long contexts all the time.

The program prints the cache size of the 8B config at six context lengths. It also compares each cache with the weights of the Q4_K model (from Example 1), expressed as a percentage.

import java.util.Locale;

public class KvByContext {
    public static void main(String[] args) {
        long perToken = 2L * 32 * 8 * 128 * 2;          // 131072 bytes, F16
        double weightsGiB = 8.03e9 * 4.5 / 8 / (1L << 30);
        System.out.printf(Locale.ROOT, "weights (Q4_K) = %.2f GiB%n", weightsGiB);
        System.out.println("context    KV GiB   KV / weights");
        for (int ctx : new int[] {4096, 8192, 16384, 32768, 65536, 131072}) {
            double gib = (double) perToken * ctx / (1L << 30);
            System.out.printf(Locale.ROOT, "%7d  %8.2f  %10.0f%%%n", ctx, gib, 100 * gib / weightsGiB);
        }
    }
}

weights (Q4_K) = 4.21 GiB
context    KV GiB   KV / weights
   4096      0.50          12%
   8192      1.00          24%
  16384      2.00          48%
  32768      4.00          95%
  65536      8.00         190%
 131072     16.00         380%

At 128 KiB per token, the arithmetic gives round numbers: 8K tokens is 1 GiB, 32K is 4 GiB, and 128K is 16 GiB. Follow the percentage column. At 32K tokens the cache (4 GiB) is about as large as the 4-bit weights (4.21 GiB). At 128K it is almost four times larger.

A model advertised with a 128K context therefore does not run at 128K on a 12 GB GPU, even though its 4.2 GiB of weights fit. The weights are small, but the cache needs 16 GiB.

The option -c 0 makes llama.cpp load the training context length from the model. The --fit option (on by default at the pinned commit) then reduces the context when it would not fit in device memory, down to --fit-ctx (default 4096).

Example 5: GQA against full multi-head attention

The KV head count is the biggest lever on cache size that the model's designers have. In the original multi-head attention (MHA), every query head has its own key and value head. In grouped-query attention (GQA), several query heads share one KV head. In the extreme case, multi-query attention (MQA), all query heads share a single KV head.

Only the KV heads are stored in the cache, so fewer KV heads means a smaller cache. The program compares the 8B shape, which has 8 KV heads for 32 query heads, with the same model built with full MHA (32 KV heads) and with MQA (1 KV head). It prints the cache per token and the cache at 32K tokens.

import java.util.Locale;

public class GqaVsMha {
    public static void main(String[] args) {
        int layers = 32, headDim = 128, heads = 32, ctx = 32768;
        System.out.println("variant          kvHeads  KiB/token  GiB at 32K");
        String[] names = {"MHA (full)", "GQA (Llama 3)", "MQA"};
        int[] kvHeads = {32, 8, 1};
        for (int i = 0; i < names.length; i++) {
            long perToken = 2L * layers * kvHeads[i] * headDim * 2;
            System.out.printf(Locale.ROOT, "%-16s %7d  %9d  %10.2f   (%d query heads per KV head)%n",
                    names[i], kvHeads[i], perToken / 1024, perToken * (double) ctx / (1L << 30), heads / kvHeads[i]);
        }
    }
}

variant          kvHeads  KiB/token  GiB at 32K
MHA (full)            32        512       16.00   (1 query heads per KV head)
GQA (Llama 3)          8        128        4.00   (4 query heads per KV head)
MQA                    1         16        0.50   (32 query heads per KV head)

Each row differs only in the number of KV heads, so the cache scales directly with it: 32 heads cost 512 KiB per token, 8 heads cost 128 KiB, and 1 head costs 16 KiB. With full MHA, a 32K context would need 16 GiB of cache for this shape. GQA with 8 KV heads cuts that to 4 GiB, a factor of 4. The last column shows the sharing: with GQA, 4 query heads read from each KV head.

This is why nearly every current open model uses GQA, and why the KV head count, not the query head count, goes into the memory formula. Sliding-window layers (which only keep the most recent tokens) and multi-head latent attention (which stores a compressed form of keys and values) save cache in other ways and need model-specific formulas. The lesson on attention optimizations and long context later in this course covers both.

Example 6: quantized KV cache, q8_0 and q4_0

The cache stores numbers, so the idea from the number formats lesson applies to it too: store the numbers in fewer bits. llama.cpp can store the cache in a quantized type, chosen with -ctk for keys and -ctv for values. The allowed values at the pinned commit are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1.

A quantized cache uses the same block formats as weights, so its bytes per value are the block bytes divided by 32. For q8_0 that is 34 / 32 = 1.0625 bytes. For q4_0 it is 18 / 32 = 0.5625 bytes. Compare this with 2 bytes for f16.

The program predicts the cache size of the tiny model at 16,384 cells with three types. It also prints the saving compared with f16.

import java.util.Locale;

public class QuantizedKv {
    public static void main(String[] args) {
        String[] types = {"f16", "q8_0", "q4_0"};
        double[] bytesPerValue = {2.0, 34 / 32.0, 18 / 32.0};
        long valuesPerToken = 2L * 4 * 2 * 32;                  // tiny model: K and V, 4 layers, 2 heads, dim 32
        long cells = 16384;
        for (int i = 0; i < types.length; i++) {
            double mib = valuesPerToken * cells * bytesPerValue[i] / (1 << 20);
            System.out.printf(Locale.ROOT, "%-5s %.4f bytes/value  total %5.2f MiB  (K %.2f, V %.2f)  saving %.0f%%%n",
                    types[i], bytesPerValue[i], mib, mib / 2, mib / 2, 100 * (1 - bytesPerValue[i] / 2));
        }
    }
}

f16   2.0000 bytes/value  total 16.00 MiB  (K 8.00, V 8.00)  saving 0%
q8_0  1.0625 bytes/value  total  8.50 MiB  (K 4.25, V 4.25)  saving 47%
q4_0  0.5625 bytes/value  total  4.50 MiB  (K 2.25, V 2.25)  saving 72%

The numbers follow from the formula. The tiny model has 2 * 4 * 2 * 32 = 512 values per token (K and V, 4 layers, 2 heads, dimension 32). With 16,384 cells that is 8,388,608 values. At 2 bytes each that is exactly 16 MiB, and at 1.0625 bytes it is 8.5 MiB.

To check the predictions, the real program was run with each cache type:

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 2 -c 16384 -ctk q8_0 -ctv q8_0 -no-cnv -lv 4

Representative output (from the pinned llama.cpp build; one line per run, three runs with f16, q8_0, and q4_0):

llama_kv_cache: size =   16.00 MiB ( 16384 cells,   4 layers,  1/1 seqs), K (f16):    8.00 MiB, V (f16):    8.00 MiB
llama_kv_cache: size =    8.50 MiB ( 16384 cells,   4 layers,  1/1 seqs), K (q8_0):    4.25 MiB, V (q8_0):    4.25 MiB
llama_kv_cache: size =    4.50 MiB ( 16384 cells,   4 layers,  1/1 seqs), K (q4_0):    2.25 MiB, V (q4_0):    2.25 MiB

The three logged sizes (16.00, 8.50, 4.50 MiB) match the predictions. A q8_0 cache is 47% smaller than f16, and q4_0 is 72% smaller.

The cost is precision. The stored keys and values are rounded, which slightly changes the attention scores. The lesson on tuning llama.cpp later in this course measures the quality effect.

There is one more requirement. A quantized V cache requires flash attention in llama.cpp. Flash attention is a fused attention kernel that never stores the full score matrix, and the lesson on attention optimizations later in this course covers it. With the default -fa auto, llama.cpp turns flash attention on when the V cache is quantized. The q8_0 run above printed enabling flash_attn since it is required for quantized V cache and flash_attn = enabled. With -fa off, the source raises the error quantized V cache was requested, but this requires Flash Attention instead.

Example 7: parallel sequences and how llama.cpp divides the context

A server usually has several users at once. llama-server handles this with slots. A slot is one place for one conversation (one parallel sequence). The number of slots is set with -np (--parallel).

The question is how the context memory is shared among the slots. There are two designs:

• A split cache divides the context equally. Each slot gets its own fixed part, and no slot can use another's part.
• A unified cache (-kvu) is one pool of cells. Any slot can use any free cell, up to the whole pool.

The pinned source, src/llama-context.cpp, does this:

1. Round n_ctx (the total context, in tokens) up to a multiple of 256.
2. If the cache is unified, each sequence may use all of n_ctx.
3. Otherwise, each sequence gets n_ctx / n_seq_max, rounded up to a multiple of 256, and n_ctx becomes that times n_seq_max (with a warning if it changed).

tools/server/server.cpp adds one rule: the default -np is -1 (auto), which means 4 slots with a unified cache.

By hand, for -c 5000 -np 3 (split cache): 5000 rounds up to 5120. 5120 / 3 = 1706, which rounds up to 1792. Then n_ctx becomes 1792 * 3 = 5376.

The program reimplements these rules and runs four settings through them. After it, the real server logs show the same four runs.

public class SlotContext {
    static int pad(int x, int n) { return (x + n - 1) / n * n; }        // GGML_PAD

    // returns {n_ctx, n_ctx_seq, n_seq_max}
    static int[] divide(int nCtx, int nParallel, boolean kvUnified) {
        if (nParallel < 0) { nParallel = 4; kvUnified = true; }          // server: -np -1 means auto
        nCtx = pad(nCtx, 256);
        int perSeq;
        if (kvUnified) {
            perSeq = nCtx;
        } else {
            perSeq = pad(nCtx / nParallel, 256);
            nCtx = perSeq * nParallel;
        }
        return new int[] {nCtx, perSeq, nParallel};
    }

    public static void main(String[] args) {
        Object[][] runs = {{"-c 4096 -np 4", 4096, 4, false}, {"-c 4096", 4096, -1, false},
                           {"-c 4096 -np 4 -kvu", 4096, 4, true}, {"-c 5000 -np 3", 5000, 3, false}};
        for (Object[] r : runs) {
            int[] d = divide((int) r[1], (int) r[2], (boolean) r[3]);
            System.out.printf("%-20s n_seq_max = %d, n_ctx = %d, n_ctx_seq = %d%n", r[0], d[2], d[0], d[1]);
        }
    }
}

-c 4096 -np 4        n_seq_max = 4, n_ctx = 4096, n_ctx_seq = 1024
-c 4096              n_seq_max = 4, n_ctx = 4096, n_ctx_seq = 4096
-c 4096 -np 4 -kvu   n_seq_max = 4, n_ctx = 4096, n_ctx_seq = 4096
-c 5000 -np 3        n_seq_max = 3, n_ctx = 5376, n_ctx_seq = 1792

The server logs for the same runs, with the tiny model (llama-server -m tiny-random-qwen2vocab-f32.gguf --port 0 -lv 4 plus the flags in the first column):

Representative output (from the pinned llama.cpp build; selected lines, timestamps removed):

== -c 4096 -np 4
llama_context: n_seq_max             = 4
llama_context: n_ctx                 = 4096
llama_context: n_ctx_seq             = 1024
llama_kv_cache: size =    4.00 MiB (  1024 cells,   4 layers,  4/4 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB
srv    load_model: initializing, n_slots = 4, n_ctx_slot = 1024, kv_unified = 'false'
== -c 4096
srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
srv    load_model: initializing, n_slots = 4, n_ctx_slot = 4096, kv_unified = 'true'
== -c 5000 -np 3
llama_context: n_ctx is not divisible by n_seq_max - rounding down to 5376
llama_context: n_ctx                 = 5376
llama_context: n_ctx_seq             = 1792
srv    load_model: initializing, n_slots = 3, n_ctx_slot = 1792, kv_unified = 'false'

Every value matches. Three points follow from the source and the logs:

• The KV memory does not grow with slots. With -c 4096 -np 4, the cache is still 4 MiB, but each request is limited to 1,024 tokens (4096 / 4). If 4 users each need 32K tokens with a split cache, the server needs -c 131072.
• A unified cache shares one pool. With the default auto setting, each slot may use up to 4,096 tokens, but all slots together share 4,096 cells. One long conversation can use most of the pool, as long as the others are short.
• The warning text says "rounding down" even when the value grew. For -c 5000 -np 3, 5000 rounds up to 5120, 5120 / 3 is 1706, which rounds up to 1792, and n_ctx becomes 5376, larger than requested.

Example 8: compute buffers and where the log reports them

Weights and the KV cache are not the whole bill. For every forward pass, the engine needs scratch space for the intermediate results, such as the output of each layer before the next layer reads it. This scratch space is called the compute buffer.

Its size depends on three things: the micro-batch size, the model width, and the vocabulary. The micro-batch size (-ub, --ubatch-size, default 512 tokens) is how many tokens the engine processes in one pass when reading a prompt. The vocabulary matters because the last step computes logits (one score for every token in the vocabulary) for every position in the micro-batch. Each logit is an FP32 number, 4 bytes.

For the tiny model, which has a very small width but a large vocabulary of 151,936, the logits dominate. The formula is:

logits bytes = vocab * ubatch * 4

By hand: 151,936 * 512 * 4 = 311,164,928 bytes, which is 296.75 MiB. The program computes this for -ub 512 and -ub 128, and the size of the output buffer, which holds one row of logits.

import java.util.Locale;

public class ComputeBuffer {
    public static void main(String[] args) {
        long vocab = 151_936;                         // tiny model uses the Qwen2 vocabulary
        double mib = 1 << 20;
        for (int ubatch : new int[] {512, 128}) {
            double logits = vocab * ubatch * 4.0;     // F32 logits for every position in the micro-batch
            System.out.printf(Locale.ROOT, "ubatch %3d: logits %.2f MiB%n", ubatch, logits / mib);
        }
        System.out.printf(Locale.ROOT, "output buffer (1 row of F32 logits) = %.2f MiB%n", vocab * 4 / mib);
    }
}

ubatch 512: logits 296.75 MiB
ubatch 128: logits 74.19 MiB
output buffer (1 row of F32 logits) = 0.58 MiB

Dividing by 4 gives the expected effect of the smaller micro-batch: 128 is a quarter of 512, so the logits shrink from 296.75 MiB to 74.19 MiB. The output buffer holds only one row of logits, 151,936 * 4 bytes, which is 0.58 MiB.

The logged values, from llama-completion runs with -c 4096 and with -c 4096 -ub 128:

Representative output (from the pinned llama.cpp build):

llama_context:        CPU  output buffer size =     0.58 MiB
sched_reserve:        CPU compute buffer size =   297.75 MiB
...
llama_context: n_ubatch              = 128
sched_reserve:        CPU compute buffer size =    74.44 MiB

The estimate is very close. The logits account for all but about 1 MiB of the tiny model's compute buffer at -ub 512 (296.75 against 297.75) and all but 0.25 MiB at -ub 128 (74.19 against 74.44). The remainder holds the small intermediate tensors of a 256-wide model.

For a real model, the intermediate tensors are larger (an 8B model's FFN is 14,336 wide). With GPU offload, each device that computes layers reserves its own compute buffer, so there is one log line per device. Lowering -ub shrinks the buffer at some cost in prompt speed. For a real model, run it once and use the logged number in place of an estimate.

Example 9: runtime overhead and headroom

Not all of a GPU's memory is available to the model. Three things take some of it before the model starts:

• The display and other programs use some. The memory.used field of nvidia-smi (from lesson 16) reports this.
• The GPU runtime reserves memory for its own context and kernels.
• Memory use moves a little between runs, so a safety margin avoids failures from small changes.

The program starts with a 24 GB card and subtracts each of these to get the budget left for weights, KV cache, and compute buffers. It then checks whether the 8B model at Q4_K with a 32K context fits. The overhead numbers are assumptions that a reader should replace with measured values.

import java.util.Locale;

public class Headroom {
    public static void main(String[] args) {
        double mib = 1 << 20;
        double total = 24_564;                 // MiB, as nvidia-smi reports for a 24 GB card
        double usedByOthers = 1_203;           // MiB, memory.used before the engine starts
        double runtime = 500;                  // MiB, assumed runtime and driver overhead per process
        double headroomShare = 0.05;           // keep 5% free for fragmentation and growth

        double free = total - usedByOthers;
        double budget = free - runtime - total * headroomShare;
        System.out.printf(Locale.ROOT, "total %.0f MiB, free %.0f MiB, budget %.0f MiB (%.2f GiB)%n",
                total, free, budget, budget / 1024);

        double weights = 8.03e9 * 4.5 / 8 / mib, kv32k = 4096, compute = 600;
        double need = weights + kv32k + compute;
        System.out.printf(Locale.ROOT, "8B Q4_K at 32K context: weights %.0f + KV %.0f + compute %.0f = %.0f MiB -> %s%n",
                weights, kv32k, compute, need, need <= budget ? "fits" : "does not fit");
    }
}

total 24564 MiB, free 23361 MiB, budget 21633 MiB (21.13 GiB)
8B Q4_K at 32K context: weights 4308 + KV 4096 + compute 600 = 9004 MiB -> fits

Follow the arithmetic. The card reports 24,564 MiB in total. Other programs use 1,203 MiB, which leaves 23,361 MiB free. The runtime takes another 500 MiB, and the 5% headroom is 0.05 * 24,564 = 1,228 MiB. The budget is 23,361 - 500 - 1,228 = 21,633 MiB, about 21.13 GiB.

The model needs 4,308 MiB of weights, 4,096 MiB of KV cache at 32K (from Example 4), and 600 MiB of compute buffer, a total of 9,004 MiB. That is below the budget, so it fits.

The 500 MiB runtime figure and the 600 MiB compute buffer are assumptions for illustration, not measured values. The actual figures come from nvidia-smi while the engine runs and from the compute buffer size lines in the log.

llama.cpp's --fit option does a similar calculation automatically. It measures the free device memory, projects the use of the requested settings, and keeps a margin per device (--fit-target, in MiB). On a CPU-only machine, the same budget applies to system RAM.

Example 10: how many layers fit on the GPU

When the whole model does not fit on the GPU, llama.cpp can split it. A model is a stack of layers, and each layer can run on either the GPU or the CPU. The option -ngl N (--n-gpu-layers) says how many layers go to the GPU. The GPU is much faster, so more layers on the GPU means faster text.

The rules from the pinned src/llama-model.cpp:

• The model has n_layer + 1 layers that can be offloaded: the repeating blocks plus the output layer.
• The token embedding always stays on the CPU.
• With -ngl N, the last N of the n_layer + 1 go to the GPU. The output layer is offloaded first, then the blocks from the top down.
• The KV cache of an offloaded layer lives on the GPU too.

This last rule is why the context length matters. Each block moved to the GPU brings its weights and its share of the KV cache. The program finds the largest N that fits an 8 GB GPU for the 8B model at Q4_K, once with an 8K context and once with a 32K context. It loops over N and adds up the compute buffer, the output layer, and the blocks with their cache.

import java.util.Locale;

public class GpuLayers {
    public static void main(String[] args) {
        double mib = 1 << 20;
        int layers = 32;
        double bpw = 4.5;
        double layerMiB = 218_112_000 * bpw / 8 / mib;          // weights of one block
        double outputMiB = 525_336_576 * bpw / 8 / mib;          // output projection
        double computeMiB = 600;                                  // assumed GPU compute buffer
        double budget = 8192 - 300 - 500 - 400;                   // 8 GiB card minus display, runtime, headroom
        System.out.printf(Locale.ROOT, "block %.1f MiB, output %.1f MiB, budget %.0f MiB%n", layerMiB, outputMiB, budget);

        for (int ctx : new int[] {8192, 32768}) {
            double kvLayerMiB = 2.0 * 8 * 128 * 2 * ctx / mib;   // one layer's K and V at F16
            int best = 0;
            for (int n = 1; n <= layers + 1; n++) {
                int blocks = n - 1;                               // the first offloaded layer is the output layer
                double used = computeMiB + outputMiB + blocks * (layerMiB + kvLayerMiB);
                if (used <= budget) best = n;
            }
            int blocks = best - 1;
            System.out.printf(Locale.ROOT, "ctx %5d: KV %.0f MiB per layer -> -ngl %d: output + %d of %d blocks on GPU, %d on CPU%n",
                    ctx, kvLayerMiB, best, blocks, layers, layers - blocks);
        }
    }
}

block 117.0 MiB, output 281.8 MiB, budget 6992 MiB
ctx  8192: KV 32 MiB per layer -> -ngl 33: output + 32 of 32 blocks on GPU, 0 on CPU
ctx 32768: KV 128 MiB per layer -> -ngl 25: output + 24 of 32 blocks on GPU, 8 on CPU

Each block costs 117 MiB of weights, plus its KV cache: 32 MiB at 8K tokens and 128 MiB at 32K. Per block, then, the cost is 149 MiB at 8K and 245 MiB at 32K.

At 8K, the whole model fits on an 8 GB GPU. The total is 600 + 282 + 32 * 149, about 5,650 MiB, below the budget of 6,992 MiB. So -ngl 33 (or -ngl all, which the pinned build accepts) offloads all 33 offloadable layers.

At 32K, the KV cache grows by 3 GiB, and the blocks become more expensive. The space left after the compute buffer and the output layer is 6,992 - 600 - 282 = 6,110 MiB. That holds 24 blocks at 245 MiB each. So only the output layer and 24 blocks fit. The other 8 blocks and their KV cache stay in system RAM and run on the CPU.

The pinned build's default is -ngl auto, which works with --fit to pick a value from the measured free memory. An explicit number overrides it.

Because the output layer is offloaded first, -ngl 1 puts only the output projection on the GPU, not the first block. -ngl 32 on a 32-layer model leaves the lowest block on the CPU.

Example 11: a hybrid CPU and GPU split

Some models are too large for one GPU at any quantization. A 70B model at 4.5 bits needs about 40 GB, but a high-end consumer GPU has 24 GB. The remaining layers then run on the CPU from system RAM. This works, but it has a speed cost, and the program shows how large.

The program uses the published Llama 3.1 70B shape: 80 layers, hidden size 8192, 64 query heads, 8 KV heads, FFN size 28,672, and vocabulary 128,256. It places as many blocks as fit on a 21 GiB GPU budget at Q4_K with an 8K context. Then it estimates decode speed, using the idea from lesson 17: generating one token needs every active weight to be read from memory once, so the time per token is the bytes read divided by the memory bandwidth. With a split, the GPU part and the CPU part run one after the other, so their times add up.

The bandwidth figures in the program are 1008 GB/s for the GPU and 89.6 GB/s for the CPU's system memory.

import java.util.Locale;

public class HybridSplit {
    public static void main(String[] args) {
        double gib = 1L << 30, bpw = 4.5;
        long hidden = 8192, layers = 80, kvHeads = 8, headDim = 128, ffn = 28_672, vocab = 128_256;
        long perLayer = hidden * 64 * headDim * 2 + hidden * kvHeads * headDim * 2 + 3 * hidden * ffn + 2 * hidden;
        double layerBytes = perLayer * bpw / 8;
        double outputBytes = vocab * hidden * bpw / 8, embedBytes = outputBytes;
        double kvLayer = 2.0 * kvHeads * headDim * 2 * 8192;              // 8K context, F16
        double budget = 21.0 * gib, compute = 0.8 * gib;                  // GPU budget, assumed compute buffer

        int blocksOnGpu = (int) ((budget - compute - outputBytes) / (layerBytes + kvLayer));
        double gpu = compute + outputBytes + blocksOnGpu * (layerBytes + kvLayer);
        double cpu = embedBytes + (layers - blocksOnGpu) * (layerBytes + kvLayer);
        System.out.printf(Locale.ROOT, "-ngl %d: %d blocks + output on GPU (%.1f GiB), %d blocks + embedding on CPU (%.1f GiB)%n",
                blocksOnGpu + 1, blocksOnGpu, gpu / gib, layers - blocksOnGpu, cpu / gib);

        double gpuRead = outputBytes + blocksOnGpu * (layerBytes + kvLayer);   // bytes read per token
        double cpuRead = (layers - blocksOnGpu) * (layerBytes + kvLayer);
        double ms = gpuRead / 1008e9 * 1000 + cpuRead / 89.6e9 * 1000;
        System.out.printf(Locale.ROOT, "per token: GPU %.1f ms + CPU %.1f ms = %.1f ms -> %.1f tok/s upper bound%n",
                gpuRead / 1008e6, cpuRead / 89.6e6, ms, 1000 / ms);
    }
}

-ngl 41: 40 blocks + output on GPU (20.5 GiB), 40 blocks + embedding on CPU (19.7 GiB)
per token: GPU 21.0 ms + CPU 229.8 ms = 250.9 ms -> 4.0 tok/s upper bound

Half of the 80 blocks plus the output layer fit on the GPU (-ngl 41), using 20.5 GiB. The other 40 blocks and the embedding stay on the CPU, using 19.7 GiB.

The GPU part of each token takes 21 ms. The CPU part reads about the same amount of data, but its memory is more than 11 times slower, so it takes about 230 ms, eleven times as long. The sum, about 251 ms, gives about 4 tokens per second.

The result is set almost entirely by the CPU half. Putting more layers on the GPU helps a lot, and only more VRAM (or a second GPU) can do that. The CPU side needs about 20 GiB of free system RAM, so this setup needs a machine with at least 32 GB of RAM beside the GPU.

Example 12: MoE expert offload, attention on the GPU and experts in RAM

A mixture-of-experts (MoE) model changes the picture. Its feed-forward part is split into many small networks called experts, and for each token a router picks only a few of them. Most of the parameters sit in experts, but each token reads only the few that were chosen (lesson 17, Example 11).

This allows a different split. Keep the experts, which are large but rarely read, in system RAM, and keep everything that every token touches (attention, the output projection, the KV cache) on the GPU. llama.cpp has two options for this:

• -cmoe (--cpu-moe) keeps all MoE weights on the CPU.
• -ncmoe N (--n-cpu-moe) keeps the MoE weights of the first N layers on the CPU.

The program sizes this split for the published Qwen3-30B-A3B shape at Q4_K with a 32K context. That model has 48 layers, 128 experts per layer, and 8 active experts per token. It prints memory on each side, and the bytes read per token from each side.

import java.util.Locale;

public class MoeOffload {
    public static void main(String[] args) {
        double gib = 1L << 30, bpw = 4.5;
        long layers = 48, hidden = 2048, heads = 32, kvHeads = 4, headDim = 128;
        long experts = 128, active = 8, expertFf = 768, vocab = 151_936;
        double attn = (hidden * heads * headDim * 2 + hidden * kvHeads * headDim * 2) * bpw / 8;
        double oneExpert = 3 * hidden * expertFf * bpw / 8;
        double router = hidden * experts * 4.0;                          // small, often kept at F32
        double output = vocab * hidden * bpw / 8;
        double kv = 2.0 * layers * kvHeads * headDim * 2 * 32768;        // 32K context, F16

        double gpu = layers * (attn + router) + output + kv;
        double cpu = layers * experts * oneExpert + output;              // experts + token embedding
        System.out.printf(Locale.ROOT, "GPU: attention %.2f GiB + output %.2f GiB + KV %.2f GiB = %.2f GiB%n",
                layers * (attn + router) / gib, output / gib, kv / gib, gpu / gib);
        System.out.printf(Locale.ROOT, "CPU: experts %.2f GiB + embedding %.2f GiB = %.2f GiB%n",
                layers * experts * oneExpert / gib, output / gib, cpu / gib);

        double cpuRead = layers * active * oneExpert;                    // 8 experts per layer per token
        double gpuRead = layers * (attn + router) + output + kv;
        double ms = gpuRead / 1008e9 * 1000 + cpuRead / 89.6e9 * 1000;
        System.out.printf(Locale.ROOT, "per token: GPU reads %.2f GiB, CPU reads %.2f GiB -> %.1f ms, %.0f tok/s upper bound%n",
                gpuRead / gib, cpuRead / gib, ms, 1000 / ms);
    }
}

GPU: attention 0.52 GiB + output 0.16 GiB + KV 3.00 GiB = 3.68 GiB
CPU: experts 15.19 GiB + embedding 0.16 GiB = 15.35 GiB
per token: GPU reads 3.68 GiB, CPU reads 0.95 GiB -> 15.3 ms, 65 tok/s upper bound

The experts are about 95% of the weights (15.19 GiB of about 16 GiB), and they sit in system RAM. The GPU holds only attention, the output projection, and the KV cache, which come to under 4 GiB at 32K. An 8 GB GPU is therefore enough.

The GPU read figure assumes the full 32K context is in use, so the whole KV cache is read for each token. The CPU reads only the 8 active experts of each layer, about 1 GiB per token. This split is therefore far faster than Example 11's, where half of every layer's weights were read from slow memory: here the estimate is 65 tokens per second, against 4.

The lesson on tuning llama.cpp later in this course shows these flags on a real model, together with -ot (--override-tensor), which places tensors by name pattern.

Example 13: vLLM's memory model, from utilization to KV blocks

vLLM is an inference server built for many users at once, and the vLLM lessons later in this course cover it. It plans memory in a different way from llama.cpp. Instead of the user choosing a context length, vLLM starts from a share of the GPU and gives everything that is left to the KV cache.

Its documentation describes the steps:

1. The engine may use a fraction of the GPU's total memory, set by --gpu-memory-utilization (default 0.92 in vLLM 0.30.0, vllm/config/cache.py).
2. It loads the weights.
3. It runs a profiling forward pass at the maximum batch size to measure the peak activation memory.
4. It gives everything that remains to the KV cache.

The cache is divided into blocks of --block-size tokens (16 by default on CUDA). A request receives blocks as it grows, which avoids reserving a full-length cache in advance. The lesson on paged KV cache later in this course explains blocks in full. The block-size default is stated from vLLM's documentation.

The program computes the block budget for an 8B model in BF16 on a 24 GB GPU. The profiled activation peak and the other overhead are assumed values.

import java.util.Locale;

public class VllmBlocks {
    public static void main(String[] args) {
        double mib = 1 << 20;
        double total = 24_564 * mib;
        double utilization = 0.92;
        double weights = 8.03e9 * 2;                    // BF16
        double activationPeak = 1_500 * mib;            // assumed result of the profiling run
        double otherOverhead = 400 * mib;               // assumed CUDA graphs and runtime buffers
        long kvPerToken = 2L * 32 * 8 * 128 * 2;        // 131072 bytes
        int blockSize = 16;

        double kvBytes = total * utilization - weights - activationPeak - otherOverhead;
        long tokens = (long) (kvBytes / kvPerToken);
        long blocks = tokens / blockSize;
        System.out.printf(Locale.ROOT, "usable %.0f MiB, weights %.0f MiB, KV cache %.0f MiB%n",
                total * utilization / mib, weights / mib, kvBytes / mib);
        System.out.println("KV cache = " + tokens + " tokens = " + blocks + " blocks of " + blockSize);
        System.out.printf(Locale.ROOT, "16K-token requests that fit at once: %.1f%n", tokens / 16384.0);
    }
}

usable 22599 MiB, weights 15316 MiB, KV cache 5383 MiB
KV cache = 43062 tokens = 2691 blocks of 16
16K-token requests that fit at once: 2.6

Follow the steps. The usable memory is 0.92 * 24,564 = 22,599 MiB. The weights take 8.03e9 * 2 bytes = 15,316 MiB. After the activation peak (1,500 MiB) and other overhead (400 MiB), 5,383 MiB remain for the cache. At 128 KiB per token, that is 43,062 tokens, or 2,691 blocks of 16 tokens.

In practice this means about 43,000 tokens of cache shared by all requests, which is room for two 16K requests at once (2.6 in the last line). Raising the 0.92 utilization gives more KV blocks. Lowering it lets two engines share a GPU. Serving the same model with FP8 or 4-bit weights frees gigabytes for the cache, which in vLLM means more concurrent requests. vLLM's startup log reports the resulting KV cache size in tokens.

Example 14: the largest context that fits a memory budget

The formulas so far answer "how much memory does this context need". Turning them around answers a question that is often more useful: "what is the longest context this GPU can hold". After the weights and the compute buffer are paid for, what is left is divided by the cache cost of one token:

max context = (budget - weights - compute) / kvBytesPerToken

The result is rounded down to a multiple of 256. This is because llama.cpp pads n_ctx up to a multiple of 256 (as Example 7 showed), so a rounded-down value is the largest one that stays within budget.

The program computes this for the 8B model at Q4_K on budgets of 6, 10, and 20 GiB, with an F16 cache and a q8_0 cache.

import java.util.Locale;

public class MaxContext {
    public static void main(String[] args) {
        double mib = 1 << 20;
        double weights = 8.03e9 * 4.5 / 8 / mib;            // MiB
        double compute = 600;                               // assumed compute buffer, MiB
        String[] kvTypes = {"f16", "q8_0"};
        double[] bytesPerValue = {2, 34 / 32.0};
        System.out.println("GPU budget   KV type   max context");
        for (double budgetGiB : new double[] {6, 10, 20}) {
            for (int k = 0; k < kvTypes.length; k++) {
                double perToken = 2.0 * 32 * 8 * 128 * bytesPerValue[k] / mib;
                long ctx = (long) ((budgetGiB * 1024 - weights - compute) / perToken);
                ctx = ctx / 256 * 256;
                System.out.printf(Locale.ROOT, "%6.0f GiB   %-7s %12d%n", budgetGiB, kvTypes[k], ctx);
            }
        }
    }
}

GPU budget   KV type   max context
     6 GiB   f16             9728
     6 GiB   q8_0           18432
    10 GiB   f16            42496
    10 GiB   q8_0           80128
    20 GiB   f16           124416
    20 GiB   q8_0          234496

Check one row by hand. For 6 GiB with F16: the budget is 6,144 MiB, minus 4,308 MiB of weights and 600 MiB of compute, leaves 1,236 MiB. One token costs 128 KiB = 0.125 MiB, so 1,236 / 0.125 = 9,888 tokens, which rounds down to 9,728 (a multiple of 256).

On a 6 GiB budget, the 8B model at Q4_K leaves room for about 9K tokens of F16 cache, and a q8_0 cache nearly doubles that. On a 20 GiB budget, the context reaches about 120K with F16 and goes past the model's 128K training length with q8_0.

The training context is a second limit. The model was only trained to handle up to a certain length, and beyond that, quality degrades. llama.cpp warns n_ctx_seq (16384) > n_ctx_train (4096) -- possible training context overflow when this limit is exceeded. The tiny model's -c 16384 runs in Example 6 printed this warning, because its training context is 4,096.

Example 15: Project 4, a memory calculator

Project 4 puts the whole lesson into one program. MemoryCalculator takes a model configuration, a weight type, a KV type, a context length, a slot count, and a GPU budget. It prints a memory table and the -ngl value to use.

It is built in three stages, and each stage reuses an earlier example:

1. A ModelConfig record with parameter counts (Example 2), and a Quant record that gives bytes for a number of values, used for both weights and the KV cache (Examples 1 and 6).
2. A memory table that applies llama.cpp's slot rules (Example 7) and a compute estimate from the vocabulary, the FFN width, and the micro-batch (Example 8).
3. The GPU layer count, using the n_layer + 1 rule (Example 10).

The program runs three reports. The first is the tiny model, which can be checked against the real log. The other two are the 8B model on the 8 GB card of Example 10, once with an F16 cache and once with a q8_0 cache.

import java.util.Locale;

public class MemoryCalculator {
    record ModelConfig(String name, long layers, long hidden, long heads, long kvHeads, long headDim,
                       long ffn, long vocab, boolean tied) {
        long perLayer() { return hidden * heads * headDim * 2 + hidden * kvHeads * headDim * 2 + 3 * hidden * ffn + 2 * hidden; }
        long embedding() { return vocab * hidden; }
        long output() { return tied ? 0 : vocab * hidden; }
        long kvValuesPerTokenPerLayer() { return 2 * kvHeads * headDim; }
    }
    record Quant(String name, double bitsPerValue) {
        double bytes(double values) { return values * bitsPerValue / 8; }
    }

    static final double MIB = 1 << 20;

    static int pad256(int x) { return (x + 255) / 256 * 256; }

    static void report(ModelConfig m, Quant w, Quant kv, int ctx, int slots, boolean unified, double gpuBudgetMiB) {
        int cells = pad256(ctx);
        int perSlot = unified ? cells : pad256(cells / slots);
        if (!unified) cells = perSlot * slots;

        double layerW = w.bytes(m.perLayer()) / MIB;
        double layerKv = kv.bytes((double) m.kvValuesPerTokenPerLayer() * cells) / MIB;
        double embed = w.bytes(m.embedding()) / MIB, out = w.bytes(m.output()) / MIB;
        double compute = (m.vocab() + 2 * m.ffn()) * 512 * 4.0 / MIB;           // estimate at -ub 512
        double weights = m.layers() * layerW + embed + out;
        double kvTotal = m.layers() * layerKv;

        System.out.printf(Locale.ROOT, "%s, %s weights, %s KV, -c %d -np %d%s%n", m.name(), w.name(), kv.name(),
                ctx, slots, unified ? " -kvu" : "");
        System.out.printf(Locale.ROOT, "  weights  %9.0f MiB%n  KV cache %9.0f MiB  (%d cells, %d per slot)%n",
                weights, kvTotal, cells, perSlot);
        System.out.printf(Locale.ROOT, "  compute  %9.0f MiB  (estimate)%n  total    %9.0f MiB%n",
                compute, weights + kvTotal + compute);

        int ngl = 0;
        for (int n = 1; n <= m.layers() + 1; n++) {
            double used = compute + out + (n - 1) * (layerW + layerKv);
            if (used <= gpuBudgetMiB) ngl = n;
        }
        String where = ngl == m.layers() + 1 ? "all layers on GPU" : (ngl - 1) + " blocks + output on GPU";
        System.out.printf(Locale.ROOT, "  GPU budget %.0f MiB -> -ngl %d (%s)%n", gpuBudgetMiB, ngl, where);
    }

    public static void main(String[] args) {
        ModelConfig llama8b = new ModelConfig("Llama-3.1-8B shape", 32, 4096, 32, 8, 128, 14336, 128_256, false);
        ModelConfig tiny = new ModelConfig("tiny-random", 4, 256, 8, 2, 32, 704, 151_936, false);
        Quant q4k = new Quant("Q4_K", 4.5), f16 = new Quant("f16", 16), q8 = new Quant("q8_0", 8.5), f32 = new Quant("F32", 32);

        report(tiny, f32, f16, 4096, 4, false, 1_000);
        report(llama8b, q4k, f16, 32768, 1, false, 6_992);
        report(llama8b, q4k, q8, 32768, 1, false, 6_992);
    }
}

tiny-random, F32 weights, f16 KV, -c 4096 -np 4
  weights        308 MiB
  KV cache         4 MiB  (4096 cells, 1024 per slot)
  compute        300 MiB  (estimate)
  total          611 MiB
  GPU budget 1000 MiB -> -ngl 5 (all layers on GPU)
Llama-3.1-8B shape, Q4_K weights, f16 KV, -c 32768 -np 1
  weights       4308 MiB
  KV cache      4096 MiB  (32768 cells, 32768 per slot)
  compute        307 MiB  (estimate)
  total         8710 MiB
  GPU budget 6992 MiB -> -ngl 27 (26 blocks + output on GPU)
Llama-3.1-8B shape, Q4_K weights, q8_0 KV, -c 32768 -np 1
  weights       4308 MiB
  KV cache      2176 MiB  (32768 cells, 32768 per slot)
  compute        307 MiB  (estimate)
  total         6790 MiB
  GPU budget 6992 MiB -> -ngl 33 (all layers on GPU)

The first report checks the calculator against the tiny model's real run with -c 4096 -np 4. The KV cache is 4 MiB in 4,096 cells, 1,024 per slot, as in Example 7. The weight figure, 308 MiB, matches the 307.51 MiB file. The compute estimate, 300 MiB, is about 2 MiB above the logged 297.75 MiB. The calculator is accurate enough for planning.

The next two reports use the 8 GB card from Example 10 at 32K. With an F16 cache, the total is 8,710 MiB, more than the 6,992 MiB budget. 26 blocks and the output layer fit (-ngl 27). Example 10 found 25 because it assumed a 600 MiB compute buffer instead of this 307 MiB estimate.

With a q8_0 cache, the KV cache drops from 4,096 to 2,176 MiB, the total is 6,790 MiB, and the whole model fits (-ngl 33). A smaller cache type turned a split model into a fully GPU-resident one.

loop fit in section 4 extends this calculator to read the configuration from a GGUF file (The GGUF Format).

Interview points

• Weight memory is parameters times bits per weight divided by 8. Bits per weight includes the block scales, so Q8_0 is 8.5, Q4_K is 4.5, and Q6_K is 6.5625.
• KV cache bytes per token are 2 * layers * kvHeads * headDim * bytesPerValue. For a Llama 3.1 8B shape at F16, that is 128 KiB per token, which is 4 GiB at 32K tokens.
• A q8_0 KV cache is 47% smaller than f16, and q4_0 is 72% smaller. A quantized V cache in llama.cpp needs flash attention.
• In llama.cpp, slots do not add KV memory. A split cache divides n_ctx among the slots (padded to 256), and a unified cache lets all slots share one pool. llama-server's default is 4 slots with a unified cache.
• For small models, the compute buffer is dominated by logits: vocabulary times micro-batch times 4 bytes. Lowering -ub shrinks it.
• llama.cpp offloads the last N of n_layer + 1 layers. The output layer goes first, and the token embedding stays on the CPU.
• With a CPU and GPU split, the time per token is the sum of both parts, and the CPU part dominates. MoE expert offload keeps the CPU part small because only the active experts are read.
• vLLM gives all memory left after the weights and the profiled activations, within --gpu-memory-utilization, to KV blocks of 16 tokens.

Summary

• An 8B model needs 16.06 GB in F16, 8.53 GB in Q8_0, and 4.52 GB in Q4_K or Q4_0. Quantization mixes such as Q4_K_M average more than 4.5 bpw.
• The Llama 3.1 8B shape has 218 million parameters per layer and 525 million each in the embedding and the output projection, 8.03 billion in total.
• The KV cache for the 8B shape is 0.5 GiB at 4K, 4 GiB at 32K, and 16 GiB at 128K tokens.
• -ctk and -ctv accept f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, with a default of f16.
• llama.cpp pads n_ctx to a multiple of 256. With -c 5000 -np 3 it creates 3 slots of 1,792 tokens and n_ctx = 5376.
• The tiny model's compute buffer was 297.75 MiB at -ub 512 and 74.44 MiB at -ub 128, almost all of it logits.
• An 8B Q4_K model fits an 8 GB GPU entirely at 8K context, and with -ngl 25 at 32K.
• -cmoe and -ncmoe N keep MoE expert weights in system RAM while attention and the KV cache stay on the GPU.