codeloopback
Guide contents

Lesson 16 of 73

CPUs, GPUs, and Memory for Inference

A language model is a very large file of numbers called weights. To generate text, the computer has to read those weights from memory and do arithmetic with them. This lesson explains where the weights live while the model runs, how fast each part of a machine can move them, and why that speed decides how many tokens per second you get.

After this lesson you can look at a machine (a laptop, a Mac, a desktop with a graphics card, a cloud GPU) and answer three questions: how large a model fits, which part of the machine limits the speed, and what llama.cpp reports about the hardware when it starts.

Here is the map of the lesson. Each row names an idea and the example that shows it.

Idea What it means for inference Example
Cores and memory size How many threads work at once, and how large a model fits 1
Memory bandwidth Bytes per second from memory to the cores; sets the speed of generating tokens 2, 8
Cache lines Memory moves in fixed chunks, so the order of access matters 3
Matrix-vector product Generating one token reads every weight once 4
Threads More cores read memory faster, up to a ceiling 5
SIMD One instruction works on several numbers at once 6
GPU structure Streaming multiprocessors, warps, tensor cores, tiles 7
Interconnects PCIe and NVLink are much slower than the GPU's own memory 9
Feature flags and logs AVX2, AVX-512, NEON, and the lines llama.cpp prints 10, 11
Choosing hardware Capacity first, bandwidth second, compute third 12

The lesson builds on two facts from section 1 of the course. First, a matrix-vector product (multiplying a grid of numbers by one list of numbers) costs 2 * rows * cols floating-point operations (The Math Behind Transformers, in Java). A floating-point operation is one addition or multiplication on decimal numbers. Second, while the model generates text one token at a time (the "decode" phase), it reads every weight once to produce each token (Generation, Prefill, Decode, and the KV Cache). This lesson asks how fast a machine can do that reading.

Reference machine. Measured numbers in this lesson come from a cloud machine with 4 CPU cores (Intel Xeon, 2.1 GHz), 15 GiB of RAM, and no GPU. They show the shape of the results. A different machine gives different numbers.

Units. Two unit systems appear, and they differ slightly:

  • GB means 10^9 bytes (1,000,000,000) and GB/s means 10^9 bytes per second. Hardware vendors use these units for memory sizes and bandwidth.
  • GiB means 2^30 bytes (1,073,741,824). llama.cpp and the JVM use GiB and MiB when they report memory.

One GiB is about 7% larger than one GB, so a 24 GB graphics card holds a little less than 24 GiB of data in the numbers that llama.cpp prints.

Example 1: machine facts from Java

The first question about any machine is how many cores it has and how much memory a program may use. Cores decide how many threads (independent streams of work) can run at the same time. Memory decides how large a model can be loaded. Java reports both through Runtime and the platform OperatingSystemMXBean.

The program below prints checks (true or false) instead of raw values, so its output is the same on every machine. Look at the last line of each check: they all print true, which confirms that the values are sensible.

Java
import java.lang.management.ManagementFactory;

public class MachineFacts {
    public static void main(String[] args) {
        Runtime rt = Runtime.getRuntime();
        int cpus = rt.availableProcessors();
        long maxHeap = rt.maxMemory();
        long heapNow = rt.totalMemory();

        var os = (com.sun.management.OperatingSystemMXBean)
                ManagementFactory.getOperatingSystemMXBean();
        long physical = os.getTotalMemorySize();

        System.out.println("availableProcessors >= 1: " + (cpus >= 1));
        System.out.println("maxMemory > 0: " + (maxHeap > 0));
        System.out.println("totalMemory <= maxMemory: " + (heapNow <= maxHeap));
        System.out.println("physical memory > max heap: " + (physical > maxHeap));
        System.out.println("os.arch is set: " + !System.getProperty("os.arch").isEmpty());
    }
}
Output
availableProcessors >= 1: true
maxMemory > 0: true
totalMemory <= maxMemory: true
physical memory > max heap: true
os.arch is set: true

Here is what each call means:

  • availableProcessors() is the number of hardware threads the JVM may use. On a machine with hyper-threading (two hardware threads per core), it counts threads, not cores. Inside a container it respects the CPU limit.
  • maxMemory() is the largest heap the JVM will grow to. The heap is the area of memory where Java objects live. Without the -Xmx flag, HotSpot sets the maximum to one quarter of physical memory, which is why physical memory is larger than the maximum heap.
  • getTotalMemorySize() is the physical RAM of the machine (or the container limit).

To see real values, print availableProcessors(), maxMemory(), getTotalMemorySize() (converted to GiB), and the os.arch property. On the reference machine this gives the output below. Values differ by machine.

Representative output (from the reference machine; values differ by machine):

Output
processors  = 4
max heap    = 3.9 GiB
physical    = 15.6 GiB
os.arch     = amd64

The machine has 4 hardware threads, 15.6 GiB of physical memory, and a default maximum heap of 3.9 GiB, which is one quarter of 15.6. The architecture amd64 means an x86 processor.

Two of these numbers matter for local models:

  • Physical memory sets the largest model a CPU-only machine can hold. A model file of 5 GB plus its working memory must fit beside the operating system and other programs.
  • Processor count sets how many threads an engine can use. llama.cpp's -t flag (the number of threads) defaults to a value derived from it.

The Java heap limit matters only for a Java engine that keeps weights on the heap. Section 7 of this course maps weights outside the heap with the FFM (Foreign Function and Memory) API, so the heap limit does not apply there.

Example 2: one pass over a large array

Generating one token reads every weight once. This means the speed at which a machine can read a large block of memory sets an upper limit on tokens per second. Before measuring that speed, the next program fixes the part that does not change between runs: the number of bytes read.

The program sums a 256 MiB array of int values (4 bytes each) and prints the byte count and the sum. Look at the last two lines: the computed checksum equals the value from a closed formula.

Java
public class SequentialPass {
    public static void main(String[] args) {
        int n = 64 * 1024 * 1024;              // 64 Mi ints
        int[] data = new int[n];
        for (int i = 0; i < n; i++) data[i] = i;

        long sum = 0;
        for (int i = 0; i < n; i++) sum += data[i];

        long bytes = (long) n * Integer.BYTES;
        System.out.println("elements     = " + n);
        System.out.println("bytes read   = " + bytes);
        System.out.println("MiB read     = " + bytes / (1024 * 1024));
        System.out.println("checksum     = " + sum);
        System.out.println("expected sum = " + (long) n * (n - 1) / 2);
    }
}
Output
elements     = 67108864
bytes read   = 268435456
MiB read     = 256
checksum     = 2251799780130816
expected sum = 2251799780130816

The array has 64 Mi (67,108,864) elements, and 67,108,864 times 4 bytes is 268,435,456 bytes, which is 256 MiB. The array is much larger than any CPU cache (Example 3 defines caches), so every byte comes from main memory. The checksum matches the formula n * (n - 1) / 2 (the sum of the numbers 0 to n-1), which shows that the loop touched every element exactly once.

Adding a timer turns the byte count into a bandwidth. The program below runs the pass ten times and keeps the fastest run. The checker compiles it but does not run it, because times change from run to run.

Java
import java.util.Locale;

public class SequentialBandwidth {
    public static void main(String[] args) {
        int n = 64 * 1024 * 1024;            // 64 Mi ints = 256 MiB
        int[] data = new int[n];
        for (int i = 0; i < n; i++) data[i] = i;
        long bytes = (long) n * Integer.BYTES;

        long best = Long.MAX_VALUE, sum = 0;
        for (int run = 0; run < 10; run++) {
            long t0 = System.nanoTime();
            long s = 0;
            for (int i = 0; i < n; i++) s += data[i];
            long t = System.nanoTime() - t0;
            best = Math.min(best, t);
            sum = s;
        }
        System.out.println("checksum = " + sum);
        System.out.printf(Locale.ROOT, "best pass = %.1f ms, %.1f GB/s%n",
                best / 1e6, bytes / (double) best);
    }
}

Representative output (from the reference machine; timings differ by machine):

Output
checksum = 2251799780130816
best pass = 29.0 ms, 9.2 GB/s

The fastest pass took 29.0 ms for 268,435,456 bytes. Dividing bytes by nanoseconds gives GB/s directly, because both sides scale by 10^9: 268,435,456 / 29,000,000 is about 9.2.

Memory bandwidth is the number of bytes per second that can move between main memory and the cores. One thread on the reference machine reads about 9 GB/s. A machine's peak bandwidth is higher than what one thread reaches, and Example 5 uses more threads to get closer. The first runs are slower than later ones because the JIT (just-in-time) compiler has not yet turned the loop into machine code. This is why the program keeps the best of ten runs.

Example 3: sequential and strided access in cache lines

Reading from main memory is slow compared with the speed of a core. Memory therefore does not move one value at a time. It moves data in fixed-size chunks and keeps recent chunks in a small, fast memory next to the core.

Two terms describe this:

  • A cache is a small, fast memory next to the cores that keeps recently used data.
  • A cache line is the fixed-size chunk in which data moves between main memory and the caches. It is 64 bytes on current x86 CPUs and 128 bytes on Apple M-series cores.

A shipping analogy fits here. Reading one float is like ordering one item from a warehouse: the warehouse always sends a whole box of 16 floats, whether you use all of them or not. The analogy stops being accurate in one way: real caches also keep recently used boxes close, so reading the same box again is fast.

Text
Memory is cut into 64-byte lines. One float is 4 bytes, so a line holds 16 floats.

line 0: [f0  f1  f2  f3  f4  f5  f6  f7  f8  f9  f10 f11 f12 f13 f14 f15]
line 1: [f16 f17 f18 f19 f20 f21 f22 f23 f24 f25 f26 f27 f28 f29 f30 f31]

stride 1:  read f0 f1 f2 ...   -> 16 useful floats per line loaded
stride 16: read f0 f16 f32 ... -> 1 useful float per line loaded

The next program counts how many 64-byte lines a loop touches when it reads one million float values with different strides. The stride is the distance between consecutive reads, measured in elements. Look at the "useful share" column: it is the fraction of the moved bytes that the loop actually needed.

Java
import java.util.Locale;

public class CacheLines {
    static final int LINE = 64;

    // Reads `count` floats starting at index 0, `stride` elements apart.
    static long linesTouched(int count, int stride) {
        long lines = 0, lastLine = -1;
        for (long k = 0; k < count; k++) {
            long byteOffset = k * stride * Float.BYTES;
            long line = byteOffset / LINE;
            if (line != lastLine) { lines++; lastLine = line; }
        }
        return lines;
    }

    public static void main(String[] args) {
        int count = 1 << 20;                    // 1,048,576 useful floats
        long usefulBytes = (long) count * Float.BYTES;
        System.out.println("stride  lines touched  bytes moved  useful share");
        for (int stride : new int[] {1, 2, 4, 8, 16, 32}) {
            long lines = linesTouched(count, stride);
            long moved = lines * LINE;
            System.out.printf(Locale.ROOT, "%6d  %13d  %11d  %11.2f%%%n",
                    stride, lines, moved, 100.0 * usefulBytes / moved);
        }
    }
}
Output
stride  lines touched  bytes moved  useful share
     1          65536      4194304       100.00%
     2         131072      8388608        50.00%
     4         262144     16777216        25.00%
     8         524288     33554432        12.50%
    16        1048576     67108864         6.25%
    32        1048576     67108864         6.25%

With stride 1, each 64-byte line holds 16 floats and the loop uses all of them, so 4 MiB of useful data costs 4 MiB of memory traffic (65,536 lines times 64 bytes). With stride 2, the loop uses every second float, so half of each line is wasted and the traffic doubles. With stride 16 (16 floats is 64 bytes), each line delivers one useful float and the loop moves 16 times more bytes for the same work. Past stride 16 every read is already on its own line, so the count stops growing at 1,048,576 lines.

This is why inference engines store each weight matrix so that a kernel (the inner loop that does the arithmetic) reads it front to back. A matrix-vector product over a row-major matrix (row after row in memory) reads every byte of every line it loads. Reading the same matrix column by column would waste most of each line.

Hardware prefetchers add a second benefit. A prefetcher is a circuit that detects a sequential access pattern and fetches the next lines before the program asks for them. Strided and random patterns defeat it. The Java Deep Dive lesson CPU Caches and Locality measures these effects with JMH (the Java Microbenchmark Harness).

Example 4: a matrix-vector product reads every weight once

During decode, each layer of the model multiplies a weight matrix by one activation vector. An activation vector is the list of numbers that represents the current token at that point in the model. The matrix holds the learned weights and the vector holds the data flowing through.

The program does a small 4 by 8 product by hand-sized loops and counts the bytes of weights it reads. It then applies the same count to a real model. Look at weight bytes read: it equals rows * cols * 4, because every weight is read once.

Java
import java.util.Locale;

public class MatVecBytes {
    public static void main(String[] args) {
        int rows = 4, cols = 8;
        float[] w = new float[rows * cols];
        float[] x = new float[cols];
        for (int i = 0; i < w.length; i++) w[i] = (i % 5) - 2;    // small fixed values
        for (int j = 0; j < cols; j++) x[j] = j * 0.5f;

        float[] y = new float[rows];
        long weightBytes = 0;
        for (int r = 0; r < rows; r++) {
            float s = 0f;
            for (int c = 0; c < cols; c++) {
                s += w[r * cols + c] * x[c];
                weightBytes += Float.BYTES;
            }
            y[r] = s;
        }
        System.out.println("y = " + java.util.Arrays.toString(y));
        System.out.println("weight bytes read  = " + weightBytes);
        System.out.println("bytes per output   = " + weightBytes / rows);
        System.out.println("vector bytes       = " + cols * Float.BYTES);

        long params = 8_030_000_000L;           // an 8B model
        for (int bytesPerWeight : new int[] {4, 2, 1}) {
            double gb = params * (double) bytesPerWeight / 1e9;
            System.out.printf(Locale.ROOT, "8B model, %d byte(s) per weight: %.2f GB read per token%n",
                    bytesPerWeight, gb);
        }
    }
}
Output
y = [-3.0, -1.0, 1.0, -4.5]
weight bytes read  = 128
bytes per output   = 32
vector bytes       = 32
8B model, 4 byte(s) per weight: 32.12 GB read per token
8B model, 2 byte(s) per weight: 16.06 GB read per token
8B model, 1 byte(s) per weight: 8.03 GB read per token

Walk through the small case first. The matrix has 4 rows and 8 columns, so 32 weights of 4 bytes each: 128 bytes. Each output value is one dot product (multiply matching pairs and add them up), so it reads one full row of 8 weights, which is 32 bytes. The vector has 8 values, which is also 32 bytes. The whole product reads rows * cols * 4 bytes of weights and only cols * 4 bytes of the vector.

The vector is small and stays in the cache after the first read. The weights are large and come from memory. This is the key difference between the two inputs.

The last three lines scale the idea to a real model. A model is a sequence of such products. To generate one token, an 8-billion-parameter model reads all its weights once. The weight count times the bytes per weight gives the traffic:

Text
8,030,000,000 weights x 4 bytes (FP32) = 32.12 GB per token
8,030,000,000 weights x 2 bytes (FP16) = 16.06 GB per token
8,030,000,000 weights x 1 byte  (8-bit) =  8.03 GB per token

FP16 means 16-bit floating-point numbers (2 bytes), and 8-bit quantization stores each weight in 1 byte. Quantization means storing weights with fewer bits to make the file smaller.

At the 9.2 GB/s that one thread measured in Example 2, an FP16 8B model could produce at most about 0.6 tokens per second (16.06 GB divided by 9.2 GB/s is 1.75 seconds per token). This one division, bandwidth over bytes per token, is the most useful estimate in local inference. The next lesson builds the full version.

Example 5: splitting rows across threads

One core cannot use all of a machine's memory bandwidth. To read faster, the work has to be spread over several cores, each running its own thread. A matrix-vector product splits naturally, because each output row is independent of the others. Different threads can compute different rows at the same time.

The program splits a 2048 by 2048 product across 1 and 4 threads. It prints which rows each thread got and checks that the two results are identical, bit for bit. Look at the part lines to see the split, and at the identical line to see that splitting did not change the answer.

Java
import java.util.Arrays;
import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.ArrayList;
import java.util.List;

public class ParallelRows {
    static void rows(float[] w, float[] x, float[] y, int cols, int from, int to) {
        for (int r = from; r < to; r++) {
            float s = 0f;
            for (int c = 0; c < cols; c++) s += w[r * cols + c] * x[c];
            y[r] = s;
        }
    }

    static float[] matVec(float[] w, float[] x, int rows, int cols, int threads) throws Exception {
        float[] y = new float[rows];
        try (ExecutorService pool = Executors.newFixedThreadPool(threads)) {
            List<Future<?>> parts = new ArrayList<>();
            int chunk = rows / threads;
            for (int t = 0; t < threads; t++) {
                int from = t * chunk, to = (t == threads - 1) ? rows : from + chunk;
                parts.add(pool.submit(() -> rows(w, x, y, cols, from, to)));
                System.out.println("threads=" + threads + " part " + t + ": rows " + from + ".." + (to - 1));
            }
            for (Future<?> f : parts) f.get();
        }
        return y;
    }

    public static void main(String[] args) throws Exception {
        int rows = 2048, cols = 2048;
        Random rnd = new Random(42);
        float[] w = new float[rows * cols];
        for (int i = 0; i < w.length; i++) w[i] = rnd.nextFloat() - 0.5f;
        float[] x = new float[cols];
        for (int i = 0; i < cols; i++) x[i] = rnd.nextFloat() - 0.5f;

        float[] one = matVec(w, x, rows, cols, 1);
        float[] four = matVec(w, x, rows, cols, 4);
        System.out.println("identical: " + Arrays.equals(one, four));
        System.out.println("weight MiB: " + (long) w.length * Float.BYTES / (1024 * 1024));
    }
}
Output
threads=1 part 0: rows 0..2047
threads=4 part 0: rows 0..511
threads=4 part 1: rows 512..1023
threads=4 part 2: rows 1024..1535
threads=4 part 3: rows 1536..2047
identical: true
weight MiB: 16

With 4 threads, the 2048 rows split into four chunks of 512 rows. The results are identical because each row is still computed by one thread, in the same order of additions. Splitting rows changes who does the work, not the arithmetic.

Splitting one dot product across threads would be different. It would change the order of additions and could change the last bits of the result, because float addition order matters (lesson 8 showed this).

A timed version uses the same rows method on an 8192 by 8192 matrix. That matrix is 256 MiB, too large for the caches. It runs each thread count eight times on a fixed thread pool, keeps the fastest run, and divides the weight bytes by its time.

Representative output (from the reference machine; timings differ by machine):

Output
threads=1  62.0 ms  4.3 GB/s
threads=2  32.1 ms  8.4 GB/s
threads=4  20.9 ms  12.8 GB/s

Compare the three lines:

  • Two threads are almost twice as fast as one (8.4 against 4.3 GB/s).
  • Four threads are about three times as fast, not four (12.8 against 4.3 GB/s).

One thread here is limited by its own arithmetic. Example 6 explains why a plain float loop is slow, so adding threads helps at first. The 12.8 GB/s at four threads is not this machine's memory limit. The vectorized kernels in Quantized Kernels, the Vector API, and Threads read 39.4 GB/s with four threads on the same machine. The plain loop is still limited by its arithmetic, and the shortfall from a perfect 4 times comes from threads sharing cores and caches with other work.

A loop that does less work per byte reaches the bandwidth ceiling, and from then on each extra thread adds little. On desktop CPUs with two memory channels, decode speed in llama.cpp often stops improving well before all cores are busy, for this reason. Setting -t to the number of physical cores, not hardware threads, is the usual starting point. The lesson on llama.cpp core flags later in this course covers the flag.

Example 6: the SIMD idea with four accumulators

A single core can also do more per instruction. SIMD (single instruction, multiple data) is a CPU feature where one instruction operates on a short vector of values at once. For example, a 256-bit AVX2 register holds 8 floats, so one instruction can multiply 8 pairs.

A plain dot product loop cannot use this well. The loop keeps one running total, and each addition must wait for the previous addition to finish. Think of a cashier who must finish adding one receipt to the running total before starting the next, while three other cashiers stand idle. The analogy stops being accurate in one way: the CPU does not have idle cashiers as separate people, it has idle arithmetic units inside the same core.

The program compares a single-accumulator loop with a loop that keeps four independent partial sums, then combines them at the end. Look at three things in the output: the two float results, the exact double reference, and the longest chain of dependent additions.

Java
import java.util.Locale;
import java.util.Random;

public class FourAccumulators {
    static float scalar(float[] a, float[] b) {
        float s = 0f;
        for (int i = 0; i < a.length; i++) s += a[i] * b[i];
        return s;
    }

    static float four(float[] a, float[] b) {
        float s0 = 0f, s1 = 0f, s2 = 0f, s3 = 0f;
        for (int i = 0; i < a.length; i += 4) {
            s0 += a[i] * b[i];
            s1 += a[i + 1] * b[i + 1];
            s2 += a[i + 2] * b[i + 2];
            s3 += a[i + 3] * b[i + 3];
        }
        return (s0 + s1) + (s2 + s3);
    }

    public static void main(String[] args) {
        int n = 4096;
        Random rnd = new Random(7);
        float[] a = new float[n], b = new float[n];
        for (int i = 0; i < n; i++) { a[i] = rnd.nextFloat(); b[i] = rnd.nextFloat(); }

        float s1 = scalar(a, b), s4 = four(a, b);
        double exact = 0;
        for (int i = 0; i < n; i++) exact += (double) a[i] * b[i];

        System.out.printf(Locale.ROOT, "scalar         = %.6f%n", s1);
        System.out.printf(Locale.ROOT, "4 accumulators = %.6f%n", s4);
        System.out.printf(Locale.ROOT, "double exact   = %.6f%n", exact);
        System.out.println("longest chain of dependent adds: scalar " + n + ", four " + n / 4);
    }
}
Output
scalar         = 1031.151611
4 accumulators = 1031.153564
double exact   = 1031.153396
longest chain of dependent adds: scalar 4096, four 1024

The two float results differ in the third decimal place because they add the same products in different orders, and float addition is not associative (changing the grouping changes the rounding). The double sum is a reference with much smaller rounding error. Here the four-accumulator result is closer to it, because each partial sum stays smaller and loses fewer low bits when a new product is added. Neither order is "the" correct float answer. They are two roundings of the same exact sum.

The speed difference comes from the dependency chain:

Text
one accumulator:   a1 -> a2 -> a3 -> ... -> a4096        (4096 additions in a row)
four accumulators: s0: 1024 additions   s1: 1024   s2: 1024   s3: 1024   (side by side)

A floating-point addition takes several clock cycles to finish. That delay is its latency. The CPU can still start a new independent addition every cycle. With one accumulator, every addition waits for the previous one. With four accumulators, four chains of 1,024 run side by side. The JIT compiler does not reorder float additions on its own, because that would change the result. The programmer has to write the independent sums.

A timed version calls each method 200,000 times on the same 4,096-element arrays. These arrays fit in the CPU cache, so memory speed does not interfere. It keeps the third round and divides 2 * n * reps FLOPs by the elapsed nanoseconds. The unit GFLOP/s means billions of floating-point operations per second.

Representative output (from the reference machine; timings differ by machine):

Output
scalar:        1.84 GFLOP/s
4 accumulators: 6.90 GFLOP/s

The four-accumulator loop is almost four times faster on data that sits in cache (6.90 against 1.84). This explains the single-thread result in Example 5. One thread doing a plain float dot product computes at under 2 GFLOP/s. That equals about 4 GB/s of FP32 weights, because one multiply and one add are done per 4-byte weight (2 FLOPs per 4 bytes is 0.5 FLOP per byte, so 1.84 GFLOP/s needs about 3.7 GB/s). So one thread could not reach the memory ceiling.

Engines such as llama.cpp write their dot products with explicit SIMD instructions (AVX2, AVX-512, NEON), many accumulators, and integer arithmetic for quantized weights. Section 7 does the same in Java with the Vector API, in the lesson on quantized kernels later in this course.

Example 7: how a GPU splits a matrix multiply into tiles

A CPU has a few strong cores. A GPU (graphics processing unit) takes the opposite approach: thousands of simple arithmetic units that all work on different parts of the same problem. Matrix multiplication suits this well, because every output value can be computed independently.

GPU hardware has its own vocabulary. This lesson uses NVIDIA's terms:

  • A streaming multiprocessor (SM) is one of the many identical units that a GPU is built from.
  • A warp is a group of 32 threads that an SM runs together. All 32 threads of a warp execute the same instruction on different data.
  • Tensor cores are units inside each SM that multiply small matrix tiles in one operation. Most of a GPU's matrix FLOP/s come from them. FLOP/s means floating-point operations per second, a rate.
  • VRAM (video RAM) is the GPU's own memory, attached to the GPU chip with a wide, fast bus.

A matrix multiply C = A * B on a GPU is cut into output tiles. A tile is a square block of the result, for example 128 by 128 values of C. One thread block (a group of warps) computes one tile on one SM.

Text
Result matrix C (m rows x n columns), cut into 128 x 128 tiles:

+------+------+------+------+
| tile | tile | tile | tile |   each tile is computed by one thread block
+------+------+------+------+   on one SM
| tile | tile | tile | tile |
+------+------+------+------+

The program counts tiles for prefill and decode shapes. Prefill is the phase that processes the whole prompt at once, so many tokens go through each weight matrix together. Here m is the number of tokens in the batch. The program uses NVIDIA's published figure of 132 SMs for the H100 SXM. Look at three columns: tiles (how many blocks of work exist), waves (tiles divided by SMs), and useful (the share of tile work that is not padding).

Java
import java.util.Locale;

public class GpuTiles {
    static void report(String name, int m, int n, int k, int tile, int sms) {
        int tilesM = (m + tile - 1) / tile, tilesN = (n + tile - 1) / tile;
        int tiles = tilesM * tilesN;
        double waves = (double) tiles / sms;
        double useful = (double) m * n / ((long) tiles * tile * tile);
        System.out.printf(Locale.ROOT, "%-22s tiles %4d x %3d = %5d  waves %5.2f  useful %6.2f%%%n",
                name, tilesM, tilesN, tiles, waves, 100 * useful);
    }

    public static void main(String[] args) {
        int sms = 132;                // NVIDIA H100 SXM, published
        int tile = 128;
        int hidden = 4096;
        // C[m x n] = A[m x k] * W[k x n]: m = tokens in the batch
        report("prefill 2048 tokens", 2048, hidden, hidden, tile, sms);
        report("prefill 512 tokens", 512, hidden, hidden, tile, sms);
        report("decode, batch 1", 1, hidden, hidden, tile, sms);
        report("decode, batch 32", 32, hidden, hidden, tile, sms);

        long threadsPerWarp = 32, warpsPerBlock = 4;
        System.out.println("threads per 128x128 tile block (4 warps): " + threadsPerWarp * warpsPerBlock);
        System.out.println("outputs per thread: " + (tile * tile) / (threadsPerWarp * warpsPerBlock));
    }
}
Output
prefill 2048 tokens    tiles   16 x  32 =   512  waves  3.88  useful 100.00%
prefill 512 tokens     tiles    4 x  32 =   128  waves  0.97  useful 100.00%
decode, batch 1        tiles    1 x  32 =    32  waves  0.24  useful   0.78%
decode, batch 32       tiles    1 x  32 =    32  waves  0.24  useful  25.00%
threads per 128x128 tile block (4 warps): 128
outputs per thread: 128

Work through the first row by hand. With m = 2048 tokens and tile size 128, there are 2048 / 128 = 16 tile rows. The result has n = 4096 columns, so there are 4096 / 128 = 32 tile columns. That gives 16 * 32 = 512 tiles. With 132 SMs, 512 / 132 is about 3.88 waves. A wave is one round in which every SM works on one tile.

For prefill, the batch has hundreds or thousands of tokens, so m is large and every tile is full. There are enough tiles to keep all SMs busy for several waves. This is the shape tensor cores are built for.

For decode with one sequence, m is 1. A 128-row tile computes 1 useful row and 127 rows of padding, so only 1/128 (0.78%) of the tile's work is useful. Only 32 of 132 SMs get a tile at all (0.24 waves). GPU engines therefore use different kernels for decode: matrix-vector kernels that split the weight matrix into many slices so every SM reads part of the weights. Those kernels are limited by how fast the SMs can read VRAM, not by tensor cores.

Batching 32 sequences fills a quarter of each tile with useful rows (32/128 is 25%). It also reads the weights once for all 32 tokens. This is the basis of server throughput. Lesson 17 computes where the crossover lies.

The tile size of 128 is an example value. Real kernels pick tile shapes per GPU and matrix shape.

Example 8: memory types and their bandwidth

The memory bandwidth of a machine is not mysterious. It follows from two published numbers: the width of the memory bus in bits, and the transfer rate per pin. Think of the bus as a highway. The width is the number of lanes, and the transfer rate is how many cars each lane carries per second. The analogy stops being accurate in one way: a real memory bus moves data in both small bursts and channels, but the lanes-times-rate product still gives the peak.

The formula is bandwidth (GB/s) = bus bits / 8 * transfers per second (in G). Dividing by 8 converts bits to bytes. Here is one case worked by hand for a desktop with two channels of DDR5-5600:

Text
bus width      = 2 channels x 64 bits = 128 bits
bytes per step = 128 / 8              = 16 bytes
transfer rate  = 5.6 GT/s (billions of transfers per second)
bandwidth      = 16 x 5.6             = 89.6 GB/s

The program computes it for the main memory types used in inference. Look at the computed and published columns and check that they agree.

Java
import java.util.Locale;

public class MemoryBandwidth {
    record Memory(String system, String type, int busBits, double gigaTransfers, double publishedGBs) {
        double gbPerSecond() { return busBits / 8.0 * gigaTransfers; }
    }

    public static void main(String[] args) {
        Memory[] list = {
            new Memory("Desktop, 2 channels", "DDR5-5600", 128, 5.6, 0),
            new Memory("Apple M4 Max", "LPDDR5X-8533", 512, 8.533, 546),
            new Memory("NVIDIA RTX 4090", "GDDR6X", 384, 21.0, 1008),
            new Memory("NVIDIA RTX 5090", "GDDR7", 512, 28.0, 1792),
            new Memory("NVIDIA H100 SXM", "HBM3", 5120, 5.23, 3350),
        };
        System.out.println("system                type           bus  GT/s   computed  published");
        for (Memory m : list) {
            String pub = m.publishedGBs() == 0 ? "-" : String.format(Locale.ROOT, "%.0f", m.publishedGBs());
            System.out.printf(Locale.ROOT, "%-21s %-13s %5d %5.2f %8.1f GB/s %7s%n",
                    m.system(), m.type(), m.busBits(), m.gigaTransfers(), m.gbPerSecond(), pub);
        }
    }
}
Output
system                type           bus  GT/s   computed  published
Desktop, 2 channels   DDR5-5600       128  5.60     89.6 GB/s       -
Apple M4 Max          LPDDR5X-8533    512  8.53    546.1 GB/s     546
NVIDIA RTX 4090       GDDR6X          384 21.00   1008.0 GB/s    1008
NVIDIA RTX 5090       GDDR7           512 28.00   1792.0 GB/s    1792
NVIDIA H100 SXM       HBM3           5120  5.23   3347.2 GB/s    3350

The published column holds vendor specifications:

  • Apple lists 546 GB/s for the M4 Max with the full memory bus.
  • NVIDIA lists 1,008 GB/s for the RTX 4090, 1,792 GB/s for the RTX 5090, and 3.35 TB/s for the H100 SXM.

The bus widths and transfer rates are also published figures, except the H100 row, which runs the formula backward. 3.35 TB/s over a 5,120-bit bus means each pin runs at about 5.2 Gb/s. The desktop row has no published number (the dash) because its bandwidth depends on which memory modules are installed.

The pattern is clear: a wider bus and faster pins give much higher bandwidth. The table below describes the main memory types.

Memory type Where it is used Character
DDR5 Desktop and server CPUs, on removable modules Large capacity (64 to 256 GB on desktops), narrow bus, about 90 GB/s with two channels
LPDDR5/5X unified memory Apple Silicon, some laptops and mini PCs Soldered next to the chip, wide bus, shared by CPU and GPU
GDDR6X/GDDR7 Consumer GPUs Very high rate per pin, 16 to 32 GB per card
HBM Data-center GPUs Stacked dies beside the GPU, very wide bus, 80 to 192 GB per GPU

A server CPU with more memory channels (8 or 12 channels of DDR5) reaches several hundred GB/s. This is why CPU-only inference of large models is practical on server hardware.

Unified memory means the CPU and the GPU cores on an Apple chip read the same physical memory. A model up to almost the full memory size can therefore run on the GPU without copying it anywhere.

A discrete GPU is a separate card, so it cannot read the computer's main memory at VRAM speed. It connects to the CPU through PCIe (Peripheral Component Interconnect Express), a serial bus made of lanes. A graphics card uses 16 lanes, and each lane carries data in both directions. Data-center GPUs also have NVLink, NVIDIA's direct link between GPUs.

This example checks what happens if the weights had to cross one of these links for every token. The program computes how long it takes to move 4 GB (about the size of a 7B model at 4 bits per weight) over each link. It compares those times with reading the same 4 GB from VRAM. Look at the max tokens/s column: it is the speed limit if the weights cross the link once per token.

Java
import java.util.Locale;

public class LinkTimes {
    public static void main(String[] args) {
        double bytes = 4e9;
        String[] names = {"PCIe 4.0 x16", "PCIe 5.0 x16", "NVLink (H100, total)", "RTX 4090 VRAM read"};
        double[] gbPerSec = {31.5, 63.0, 900.0, 1008.0};
        for (int i = 0; i < names.length; i++) {
            double ms = bytes / (gbPerSec[i] * 1e9) * 1000;
            System.out.printf(Locale.ROOT, "%-22s %7.1f GB/s  %7.1f ms for 4 GB  max %6.1f tokens/s%n",
                    names[i], gbPerSec[i], ms, 1000 / ms);
        }
    }
}
Output
PCIe 4.0 x16              31.5 GB/s    127.0 ms for 4 GB  max    7.9 tokens/s
PCIe 5.0 x16              63.0 GB/s     63.5 ms for 4 GB  max   15.8 tokens/s
NVLink (H100, total)     900.0 GB/s      4.4 ms for 4 GB  max  225.0 tokens/s
RTX 4090 VRAM read      1008.0 GB/s      4.0 ms for 4 GB  max  252.0 tokens/s

PCIe 4.0 runs at 16 GT/s per lane with 128b/130b encoding (130 bits on the wire carry 128 bits of data). For 16 lanes, this gives about 31.5 GB/s per direction. PCIe 5.0 doubles that. NVIDIA lists 900 GB/s of total NVLink bandwidth for the H100 SXM, with both directions together.

Check the first row by hand: 4 GB / 31.5 GB/s = 0.127 s = 127 ms. One token per 127 ms is 1000 / 127, about 7.9 tokens per second. Over PCIe 4.0 that limits a 4 GB model to about 8 tokens per second. The same GPU reads the weights from its own VRAM 32 times faster (1008 / 31.5).

This is why engines load weights into VRAM once at startup and keep them there. When a model does not fit, llama.cpp keeps some layers in system RAM and computes them on the CPU, instead of streaming them over PCIe for every token. Lesson 19 computes such a split.

Example 10: CPU feature flags

CPUs gain new instructions over time. An engine checks which ones the CPU supports and uses the fastest. These instruction sets are named by the vendors, and a few names appear in every llama.cpp log:

  • On x86 (Intel and AMD), AVX2 adds 256-bit SIMD registers (8 floats). AVX-512 adds 512-bit registers (16 floats). VNNI adds instructions that multiply 8-bit integers and add the results into 32-bit sums, which suits quantized dot products.
  • On ARM (Apple Silicon, Graviton, Snapdragon), NEON is the 128-bit SIMD set. Newer cores add dot-product (dotprod) and matrix (i8mm) instructions.

Linux lists the flags of the running CPU in /proc/cpuinfo. llama.cpp prints the ones its build uses in a system_info line at startup. The program below parses both, using text copied from the reference machine. Look at the answers for the five flags, and at the final list of features that llama.cpp reports.

Java
import java.util.Arrays;
import java.util.List;
import java.util.TreeMap;

public class CpuFlags {
    public static void main(String[] args) {
        // Excerpt of the "flags" line from /proc/cpuinfo on the reference machine
        String cpuinfo = "fma sse4_2 avx f16c avx2 avx512f avx512bw avx512vl avx_vnni "
                + "avx512_bf16 avx512_vnni amx_bf16 avx512_fp16 amx_int8";
        List<String> flags = Arrays.asList(cpuinfo.split(" "));
        for (String f : List.of("avx2", "fma", "avx512f", "avx512_vnni", "neon")) {
            System.out.println(f + ": " + (flags.contains(f) ? "yes" : "no"));
        }

        // The system_info line printed by llama.cpp at startup (same machine)
        String systemInfo = "system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CPU : SSE3 = 1 | "
                + "SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | "
                + "AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | "
                + "LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | ";
        TreeMap<String, String> features = new TreeMap<>();
        for (String part : systemInfo.split("\\|")) {
            String p = part.trim().replace("CPU : ", "");
            int eq = p.indexOf(" = ");
            if (eq > 0 && !p.startsWith("system_info")) features.put(p.substring(0, eq), p.substring(eq + 3));
        }
        System.out.println("llama.cpp features: " + features.keySet());
    }
}
Output
avx2: yes
fma: yes
avx512f: yes
avx512_vnni: yes
neon: no
llama.cpp features: [AMX_INT8, AVX, AVX2, AVX512, AVX512_BF16, AVX512_VBMI, AVX512_VNNI, AVX_VNNI, BMI2, F16C, FMA, LLAMAFILE, OPENMP, REPACK, SSE3, SSSE3]

The reference machine is a recent Xeon with AVX-512 and AMX (Intel's matrix extension), so llama.cpp's CPU backend can use its widest kernels. The flag neon is absent because this is an x86 machine.

Three details of the system_info line are worth knowing:

  • REPACK = 1 means the build can rearrange quantized weights at load time into a layout that suits these instructions.
  • n_threads = 4 (n_threads_batch = 4) / 4 reports the threads used for decode, the threads used for batch (prompt) processing, and the hardware threads available.
  • Each NAME = 1 entry is a feature the build was compiled to use.

A build compiled only for AVX-512 does not start on a CPU without it. llama.cpp can instead be built with all CPU variants (the CMake option GGML_CPU_ALL_VARIANTS, which needs GGML_BACKEND_DL). Such a build picks the best variant at runtime, so one binary runs on CPUs with and without AVX-512. With such a build, a missing feature does not stop a model from running. It makes the CPU kernels slower.

Example 11: device lines in llama.cpp logs and nvidia-smi

When llama.cpp starts, it prints where each part of the model went and how large each buffer is. A buffer is a block of memory set aside for one purpose, such as the model weights or the KV cache (key-value cache, the stored keys and values of earlier tokens). Reading these lines tells you whether the model landed on the CPU or the GPU, and how much memory it uses.

On a CPU-only machine the relevant lines come from the command below. The log was captured from the pinned build with the course's tiny model, which has random weights. The log format is the same for any model.

Terminal
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "hello" -n 4 -c 4096 -no-cnv -lv 4

Representative output (from the pinned llama.cpp build; timings and paths removed):

Output
print_info: file size   = 307.51 MiB (32.00 BPW)
load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
load_tensors:   CPU_Mapped model buffer size =   307.51 MiB
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache:        CPU KV buffer size =     4.00 MiB
llama_kv_cache: size =    4.00 MiB (  4096 cells,   4 layers,  1/1 seqs), K (f16):    2.00 MiB, V (f16):    2.00 MiB
sched_reserve:        CPU compute buffer size =   297.75 MiB

Read the lines from top to bottom:

  • file size = 307.51 MiB (32.00 BPW) is the size of the model file, at 32 bits per weight (BPW).
  • CPU_Mapped model buffer is the weights. CPU_Mapped means the weights are memory-mapped from the file: the operating system pages them in from disk as needed instead of copying them up front.
  • CPU KV buffer size = 4.00 MiB is the KV cache for a 4,096-token context.
  • CPU compute buffer size is scratch space for the arithmetic.

Lesson 19 explains the KV cache and compute buffers in detail. A Java program can add up these lines to get the total memory llama.cpp asked for. Look at the final total line.

Java
import java.util.List;
import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class BufferLines {
    public static void main(String[] args) {
        List<String> log = List.of(
            "load_tensors:   CPU_Mapped model buffer size =   307.51 MiB",
            "llama_context:        CPU  output buffer size =     0.58 MiB",
            "llama_kv_cache:        CPU KV buffer size =     4.00 MiB",
            "sched_reserve:        CPU compute buffer size =   297.75 MiB");
        Pattern p = Pattern.compile("(\\S+)\\s+(model|output|KV|compute) buffer size =\\s+([0-9.]+) MiB");
        double total = 0;
        for (String line : log) {
            Matcher m = p.matcher(line);
            if (m.find()) {
                double mib = Double.parseDouble(m.group(3));
                total += mib;
                System.out.printf(Locale.ROOT, "%-11s %-8s %9.2f MiB%n", m.group(1), m.group(2), mib);
            }
        }
        System.out.printf(Locale.ROOT, "total                %9.2f MiB%n", total);
    }
}
Output
CPU_Mapped  model       307.51 MiB
CPU         output        0.58 MiB
CPU         KV            4.00 MiB
CPU         compute     297.75 MiB
total                   609.84 MiB

The total matches the memory breakdown table that the same build prints at exit (609 = 307 + 4 + 297 MiB for model, context, and compute). The first word on each line is the buffer type: where the buffer lives. With a GPU build, the same lines show a device name such as CUDA0 beside CPU, one line per device.

On a CUDA build (NVIDIA's GPU platform), ggml (the tensor library inside llama.cpp) first lists the GPUs it found. The format below comes from ggml/src/ggml-cuda/ggml-cuda.cu at the pinned commit. The device name and sizes are illustrative values, not a capture.

Output
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24080 MiB):
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24080 MiB

Offloading a layer means storing its weights in VRAM so that the GPU computes it. After the device list, load_tensors reports how many layers were offloaded. The lines below use the format from src/llama-model.cpp with illustrative values for a 32-layer model started with -ngl 32. They are not a capture.

Output
load_tensors: offloading output layer to GPU
load_tensors: offloading 31 repeating layers to GPU
load_tensors: offloaded 32/33 layers to GPU

A 32-layer model has 33 offloadable layers in llama.cpp's count: the 32 repeating transformer blocks plus the output layer. The line offloaded 32/33 layers therefore means one layer stayed on the CPU. That layer is the token embedding, which always stays on the CPU. The -ngl flag (--n-gpu-layers) sets how many of these layers go to the GPU. Lesson 19 uses this count to choose its value.

nvidia-smi is NVIDIA's command-line tool for GPU status. Its query form prints chosen fields as CSV (comma-separated values), which a Java program can parse line by line.

Terminal
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

Representative output (values differ by machine and driver):

Output
name, memory.total [MiB], memory.used [MiB]
NVIDIA GeForce RTX 4090, 24564 MiB, 1203 MiB

The first line is the header and the second line holds the values for one GPU. memory.used before an engine starts is memory taken by the desktop, other programs, and the driver. Here 1,203 MiB of the 24,564 MiB is already in use, so the model gets less than the full amount. Lesson 19 subtracts it when it computes how many layers fit.

Example 12: choosing hardware for single-user decode

The previous examples give three checks, made in order:

  1. Capacity. Whether the model fits in fast memory.
  2. Bandwidth. How fast that memory can be read.
  3. Compute. Whether the arithmetic is fast enough for prompts and batches.

For one user generating tokens, the first two decide the result. The program checks three model sizes (8B, 32B, 70B parameters) against six machines. It makes these assumptions:

  • Each weight takes 4.5 bits, the size of the Q4_K format from lesson 22. That is 4.5 / 8 = 0.5625 bytes per weight, so an 8B model is 4.5 GB.
  • The KV cache and buffers add 1.5 GB.
  • Decode speed is bandwidth divided by model bytes.
  • A model "fits" only if it needs at most 90% of the machine's memory.

Look at the cells that say does not fit first, then compare the speeds in the rest of each column.

Java
import java.util.Locale;

public class ChooseHardware {
    record Machine(String name, double memGB, double bandwidthGBs) {}
    record Model(String name, double billions) {
        double weightsGB() { return billions * 4.5 / 8; }       // 4.5 bits per weight
    }

    public static void main(String[] args) {
        Machine[] machines = {
            new Machine("Laptop, 16 GB DDR5", 16, 89.6),
            new Machine("Mac M4 Pro, 48 GB", 48, 273),
            new Machine("Mac M4 Max, 128 GB", 128, 546),
            new Machine("RTX 4090, 24 GB", 24, 1008),
            new Machine("RTX 5090, 32 GB", 32, 1792),
            new Machine("H100 SXM, 80 GB", 80, 3350),
        };
        Model[] models = {new Model("8B", 8), new Model("32B", 32), new Model("70B", 70)};
        double overheadGB = 1.5;

        System.out.printf("%-20s", "machine");
        for (Model m : models) System.out.printf(Locale.ROOT, " %8s (%4.1f GB)", m.name(), m.weightsGB());
        System.out.println();
        for (Machine h : machines) {
            System.out.printf("%-20s", h.name());
            for (Model m : models) {
                double need = m.weightsGB() + overheadGB;
                String cell = need > h.memGB() * 0.9
                        ? "does not fit"
                        : String.format(Locale.ROOT, "%6.0f tok/s", h.bandwidthGBs() / m.weightsGB());
                System.out.printf(" %18s", cell);
            }
            System.out.println();
        }
    }
}
Output
machine                    8B ( 4.5 GB)      32B (18.0 GB)      70B (39.4 GB)
Laptop, 16 GB DDR5             20 tok/s       does not fit       does not fit
Mac M4 Pro, 48 GB              61 tok/s           15 tok/s            7 tok/s
Mac M4 Max, 128 GB            121 tok/s           30 tok/s           14 tok/s
RTX 4090, 24 GB               224 tok/s           56 tok/s       does not fit
RTX 5090, 32 GB               398 tok/s          100 tok/s       does not fit
H100 SXM, 80 GB               744 tok/s          186 tok/s           85 tok/s

Check one cell by hand. The RTX 4090 has 1,008 GB/s and the 8B model has 4.5 GB of weights, so 1008 / 4.5 is about 224 tokens per second. The 70B model needs 39.4 + 1.5 = 40.9 GB, which is more than 90% of 24 GB (21.6 GB), so it does not fit.

The bandwidth figures are the published specifications from Example 8 (Apple lists 273 GB/s for the M4 Pro). The "fits" rule keeps 10% of memory free for the operating system and the GPU driver.

Reading the table:

  • Capacity first. A model that does not fit in fast memory either does not run or runs partly on a slower device. The 70B model at 4.5 bits needs about 41 GB, so it does not fit a 24 or 32 GB GPU even though those GPUs have high bandwidth. Both Macs hold it, and the 48 GB one with little to spare: it would run at about 7 tokens per second with almost no memory left for anything else.
  • Bandwidth second. Among machines where a model fits, decode speed scales with bandwidth. The 8B model runs at about 20 tokens per second on a laptop and over 200 on a 4090.
  • Compute third. Compute decides prompt processing speed (prefill) and how many users a server can batch. A Mac and a GPU with similar bandwidth decode at similar speed, but the GPU processes a long prompt many times faster because it has many more tensor-core FLOP/s. A coding agent sends long prompts full of source files, so prefill speed matters a great deal for it. Lesson 17 puts numbers on this.

These speeds are upper bounds. Real decode speed is usually 50 to 80% of bandwidth divided by model bytes, because engines also read the KV cache, run small operations that do not use the full bandwidth, and synchronize between steps.

Interview points

  • Single-user decode is limited by memory bandwidth. Every weight is read once per token, so tokens per second is at most bandwidth divided by model bytes.
  • An 8B model needs about 16 GB per token at FP16 and about 4.5 GB at 4.5 bits per weight. A 1,000 GB/s GPU therefore tops out near 60 tokens per second at FP16 and 220 at 4.5 bits.
  • Memory moves in cache lines (64 bytes on x86, 128 on Apple M-series). Row-major weights read front to back use every byte of every line, while strided access wastes most of each line.
  • Threads raise bandwidth use until the memory system saturates. More threads after that do not help decode, which is why llama.cpp often runs best with the physical core count.
  • A single float accumulator makes additions wait on each other's latency. Several accumulators or SIMD reductions remove that chain, at the cost of a different addition order and slightly different rounding.
  • GPU matrix multiplies are cut into output tiles, one per streaming multiprocessor. Decode at batch 1 fills less than 1% of a 128-row tile, so decode uses bandwidth-bound matrix-vector kernels instead.
  • PCIe 4.0 x16 moves about 31.5 GB/s per direction, roughly 30 times slower than GPU VRAM. Weights are therefore loaded into VRAM once, and layers that do not fit run on the CPU instead of crossing PCIe every token.
  • For one user, choose hardware by capacity first, bandwidth second, and compute third. Compute matters for prompt processing and batching.
  • llama.cpp counts n_layer + 1 offloadable layers (the repeating blocks plus the output layer) and keeps the token embedding on the CPU.

Summary

  • Runtime.availableProcessors() counts hardware threads, and HotSpot's default maximum heap is one quarter of physical memory.
  • Memory bandwidth (GB/s) is the rate at which bytes move from memory to the cores. Peak bandwidth equals bus width in bytes times transfers per second.
  • One thread on the reference machine read about 9 GB/s, and four threads computing a matrix-vector product reached about 13 GB/s.
  • Strided access wastes cache lines: a stride of 16 floats moves 16 times more bytes than a stride of 1 for the same useful data.
  • A matrix-vector product reads rows * cols * bytesPerWeight bytes of weights and only cols values of the vector.
  • Splitting whole rows across threads gives bit-identical results. Splitting one dot product changes the addition order.
  • Published bandwidths: DDR5-5600 dual channel 89.6 GB/s (computed), Apple M4 Pro 273 GB/s, M4 Max 546 GB/s, RTX 4090 1,008 GB/s, RTX 5090 1,792 GB/s, H100 SXM 3.35 TB/s.
  • NVIDIA GPUs run threads in warps of 32 on streaming multiprocessors. The H100 SXM has 132 SMs, and tensor cores multiply matrix tiles.
  • PCIe 4.0 x16 is about 31.5 GB/s, PCIe 5.0 x16 about 63 GB/s, and H100 NVLink 900 GB/s in total.
  • x86 SIMD sets are AVX2 (256-bit), AVX-512 (512-bit), and VNNI (8-bit integer dot products). ARM uses NEON with dotprod and i8mm extensions.
  • llama.cpp prints the CPU features in system_info, and buffer sizes per device in load_tensors, llama_kv_cache, and sched_reserve lines.
  • nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv prints GPU memory as CSV.
  • Decode estimates from bandwidth are upper bounds. Measured speed is usually lower because of KV cache reads and small operations.