Multi-GPU Inference and Serving at Scale

A large language model (LLM) is stored as billions of numbers called weights, and those weights must sit in GPU memory while the model runs. Sometimes one GPU cannot hold them. Sometimes one GPU holds them but cannot serve all users fast enough. This lesson explains what to do in both cases: split one model across several GPUs, or run several copies of it.

The difficulty is that every split has a price. GPUs that share the work must exchange data, and that exchange can take longer than the work itself. The lesson works through each kind of split in small Java simulations, counts the data each one moves, and shows which kind suits which hardware.

After the lesson, a reader can compute how many GPUs a model needs and why, simulate tensor, pipeline, expert, and data parallelism in Java, estimate the communication each one needs, read the multi-GPU flags of llama.cpp and vLLM, size multi-LoRA serving, and reason about routing and autoscaling for a fleet of engine replicas. The table lists the main terms in plain words, with the examples that teach them.

Earlier lessons supply some background. Each item is repeated here in a half sentence:

• Memory Math showed how to compute the size of the weights and of the KV cache (key-value cache, the memory that stores what the model has already read of each conversation).
• FLOPs, Bandwidth, and the Roofline Model showed how to estimate the time of one step from the bytes read and the arithmetic done.
• CPUs, GPUs, and Memory Systems described PCIe (the general connection between a GPU and the rest of the computer) and NVLink (a much faster direct link between GPUs in one server).
• Serving with vLLM, Example 5, introduced the flags --tensor-parallel-size and --pipeline-parallel-size of vLLM, an engine that serves LLMs.
• Training, Fine-Tuning, and Model Variants introduced mixture of experts (a model made of many small expert sub-networks, of which each token uses only a few) and LoRA (small add-on matrices that change the behavior of a model without changing its weights).

The simulations in this lesson count bytes, tokens, and time slots. They never measure wall-clock time, so their output is the same on every run. The llama.cpp facts come from commit 0c1e57098 and the vLLM facts from the 0.30.0 source.

The hardware numbers are the vendor figures used earlier in the course, for the NVIDIA H100 SXM GPU. Memory bandwidth, the number of bytes per second the GPU can read from its own memory, is 3.35 TB/s. Dense BF16 compute (BF16 is a 16-bit number format) is 989 TFLOP/s, which means about 989 trillion arithmetic operations per second. Total NVLink bandwidth is 900 GB/s, which is 450 GB/s in each direction. A PCIe 4.0 x16 link, for comparison, carries about 31.5 GB/s per direction, roughly fourteen times less than NVLink.

Example 1: why split a model

The word "split" covers three different problems, and each one calls for a different remedy.

1. The weights do not fit on one GPU.
2. The weights fit, but no memory is left for the KV cache of many users at the same time.
3. Everything fits, but one GPU reads the weights too slowly for the token rate that users want. A decode step (producing one new token) must read every weight once, so the memory bandwidth of a GPU sets a ceiling on tokens per second.

The program checks all three for Llama 3.1 models with 8 billion, 70 billion, and 405 billion parameters, each at three precisions (16, 8, and 4.5 bits per weight). For each case it computes the weights, the KV cache for 32 users with 8,192-token contexts, the number of 80 GB GPUs needed when 90% of each card is usable, and the time to read the weights once on one GPU and on that number of GPUs. Look at the GPUs column first, then at the two read-time columns.

import java.util.Locale;

public class WhySplit {
    record Model(String name, double params, int layers, int kvHeads, int headDim) {
        double kvBytesPerToken() { return 2.0 * layers * kvHeads * headDim * 2; }   // K and V, 2 bytes per value
    }

    public static void main(String[] args) {
        Model[] models = {
            new Model("8B", 8.03e9, 32, 8, 128),
            new Model("70B", 70.6e9, 80, 8, 128),
            new Model("405B", 405e9, 126, 8, 128)};
        double usable = 80e9 * 0.9;                                  // one 80 GB GPU, 90% given to the engine
        double bandwidth = 3.35e12;                                  // H100 SXM memory bandwidth, bytes/s
        int users = 32, context = 8192;
        System.out.println("model  bits  weights GB  KV GB  total GB  GPUs  read ms, 1 GPU  read ms, split");
        for (Model m : models) {
            double kv = m.kvBytesPerToken() * users * context;
            for (double bits : new double[] {16, 8, 4.5}) {
                double weights = m.params() * bits / 8, total = weights + kv;
                int gpus = (int) Math.ceil(total / usable);
                System.out.printf(Locale.ROOT, "%-5s  %4.1f  %10.1f  %5.1f  %8.1f  %4d  %13.1f  %14.1f%n", m.name(), bits,
                        weights / 1e9, kv / 1e9, total / 1e9, gpus, weights / bandwidth * 1e3, weights / gpus / bandwidth * 1e3);
            }
        }
        for (Model m : models)
            System.out.printf(Locale.ROOT, "KV cache per token, %s: %.0f KiB%n", m.name(), m.kvBytesPerToken() / 1024);
    }
}

model  bits  weights GB  KV GB  total GB  GPUs  read ms, 1 GPU  read ms, split
8B     16.0        16.1   34.4      50.4     1            4.8             4.8
8B      8.0         8.0   34.4      42.4     1            2.4             2.4
8B      4.5         4.5   34.4      38.9     1            1.3             1.3
70B    16.0       141.2   85.9     227.1     4           42.1            10.5
70B     8.0        70.6   85.9     156.5     3           21.1             7.0
70B     4.5        39.7   85.9     125.6     2           11.9             5.9
405B   16.0       810.0  135.3     945.3    14          241.8            17.3
405B    8.0       405.0  135.3     540.3     8          120.9            15.1
405B    4.5       227.8  135.3     363.1     6           68.0            11.3
KV cache per token, 8B: 128 KiB
KV cache per token, 70B: 320 KiB
KV cache per token, 405B: 504 KiB

Start with the first three lines, the 8B model. Its weights fit on one GPU at every precision. The KV cache for 32 users with 8,192 tokens each is 34.4 GB, twice the size of the BF16 weights, so the cache is a large part of the memory. One 80 GB card still holds both (50.4 GB of the 72 GB usable), so the 8B model does not need a split. When the users outgrow one card, the fix is more copies of the model, which is data parallelism (Example 9).

The 70B lines show the first reason. In BF16 the weights alone are 141.2 GB, more than one card holds, and with the cache the total is 227.1 GB. Dividing by the 72 GB usable per card and rounding up gives 4 GPUs as the minimum. At 8 bits the minimum is 3, and at 4.5 bits it is 2. The KV cache (85.9 GB at 320 KiB per token) is larger than the 4.5-bit weights (39.7 GB). This is why shrinking the weights alone never brings this workload down to one GPU.

The 405B lines need 14, 8, and 6 GPUs. A typical server holds 8 GPUs, so 14 GPUs in BF16 means two servers, and the split must then cross the network between them. A network is much slower than the links inside a server, which later examples show to be a real cost.

The last two columns show the third reason. A decode step reads every weight once. On one GPU the 70B BF16 model would need 42.1 ms per step (if it fit), which is a ceiling of about 24 tokens per second for each sequence (1000 / 42.1). Split over 4 GPUs, each GPU reads a quarter of the weights at the same time, so the step needs 10.5 ms. A split therefore adds memory bandwidth as well as memory capacity. The last column assumes the split is perfect and free. The following examples show what the GPUs must exchange to make it work, and what that costs.

Example 2: tensor parallelism, a column split

The most common way to split one model is tensor parallelism (TP): cut each weight matrix into pieces, give one piece to each GPU, and let the GPUs work on the same layer at the same time. This example shows the simplest cut.

A linear layer computes y = x W. Here x is the input, with one row per token, and W is the weight matrix, with one column per output value. In a column split, each device holds some of the columns of W and computes those output columns for all tokens. The program splits a 4 by 6 matrix between two simulated devices (ordinary Java arrays stand in for GPUs) and compares the result with a single device.

import java.util.Arrays;

public class ColumnSplit {
    static double[][] matmul(double[][] a, double[][] b) {
        double[][] c = new double[a.length][b[0].length];
        for (int i = 0; i < a.length; i++)
            for (int k = 0; k < b.length; k++)
                for (int j = 0; j < b[0].length; j++) c[i][j] += a[i][k] * b[k][j];
        return c;
    }

    static double[][] columns(double[][] w, int from, int to) {
        double[][] part = new double[w.length][];
        for (int i = 0; i < w.length; i++) part[i] = Arrays.copyOfRange(w[i], from, to);
        return part;
    }

    static String show(double[][] m) {
        StringBuilder sb = new StringBuilder();
        for (double[] row : m) {
            sb.append(" [");
            for (double v : row) sb.append(String.format("%4.0f", v));
            sb.append(" ]");
        }
        return sb.toString();
    }

    public static void main(String[] args) {
        double[][] x = {{1, 2, 0, -1}, {0, 1, 3, 2}};               // 2 tokens, hidden size 4
        double[][] w = new double[4][6];                             // 4 inputs, 6 outputs
        for (int i = 0; i < 4; i++) for (int j = 0; j < 6; j++) w[i][j] = (i * 6 + j) % 5 - 2;
        double[][] full = matmul(x, w);
        double[][] dev0 = matmul(x, columns(w, 0, 3));               // device 0 holds output columns 0..2
        double[][] dev1 = matmul(x, columns(w, 3, 6));               // device 1 holds output columns 3..5
        double[][] gathered = new double[2][6];
        for (int i = 0; i < 2; i++) {
            System.arraycopy(dev0[i], 0, gathered[i], 0, 3);
            System.arraycopy(dev1[i], 0, gathered[i], 3, 3);
        }
        System.out.println("one device   :" + show(full));
        System.out.println("device 0     :" + show(dev0));
        System.out.println("device 1     :" + show(dev1));
        System.out.println("concatenated :" + show(gathered));
        System.out.println("equal to one device: " + Arrays.deepEquals(full, gathered));
        System.out.println("weights per device: 12 of 24; both devices need all of x");
    }
}

one device   : [  -5  -3   4   6  -2  -5 ] [   1   7   3  -6  -5   1 ]
device 0     : [  -5  -3   4 ] [   1   7   3 ]
device 1     : [   6  -2  -5 ] [  -6  -5   1 ]
concatenated : [  -5  -3   4   6  -2  -5 ] [   1   7   3  -6  -5   1 ]
equal to one device: true
weights per device: 12 of 24; both devices need all of x

Each output column depends only on x and the matching column of W. Device 0 holds columns 0 to 2 and computes [-5 -3 4] for the first token and [1 7 3] for the second. These are the first three values of each row of the one-device result. Device 1 computes the other three values. Placing the two halves side by side gives the same 2 by 6 result, and equal to one device: true confirms it exactly. The values are small integers, so there is no rounding that could hide a difference.

The last line of the output states the cost and the benefit. Each device stores 12 of the 24 weights, so the weight memory and the bytes each device reads per step are halved. The price is that both devices need the whole input x, and each ends with only half of the output. If the next operation needs the whole output on every device, the halves must be exchanged. That exchange is called an all-gather: every device receives every other device's part. Example 3 shows how transformers avoid it.

Example 3: column then row split, one all-reduce per MLP

A transformer layer contains an MLP (multi-layer perceptron, the feed-forward block). One matrix expands each token's vector, a nonlinearity is applied, and a second matrix shrinks the vector back. In this small version it is y = relu(x A) B. ReLU (rectified linear unit) replaces every negative number with zero. Real models use a variant called SwiGLU, which has the same structure for this purpose.

The problem is to split both matrices so that the devices communicate as little as possible. The Megatron-LM paper (Shoeybi et al., 2019) splits A by columns and B by rows. Each device then computes a full-size but partial result, and the true result is the sum of the partials. The operation that sums a tensor across devices and leaves the sum on every device is an all-reduce. The program runs this split on two devices, then tries a wrong split for comparison. Compare the all-reduce sum line with the one device line, and read the last line for what goes wrong.

import java.util.Arrays;

public class MlpSplit {
    static double[][] matmul(double[][] a, double[][] b) {
        double[][] c = new double[a.length][b[0].length];
        for (int i = 0; i < a.length; i++)
            for (int k = 0; k < b.length; k++)
                for (int j = 0; j < b[0].length; j++) c[i][j] += a[i][k] * b[k][j];
        return c;
    }
    static double[][] columns(double[][] w, int from, int to) {
        double[][] p = new double[w.length][];
        for (int i = 0; i < w.length; i++) p[i] = Arrays.copyOfRange(w[i], from, to);
        return p;
    }
    static double[][] rows(double[][] w, int from, int to) { return Arrays.copyOfRange(w, from, to); }
    static double[][] relu(double[][] m) {
        double[][] r = new double[m.length][];
        for (int i = 0; i < m.length; i++) r[i] = Arrays.stream(m[i]).map(v -> Math.max(0, v)).toArray();
        return r;
    }
    static double[][] add(double[][] a, double[][] b) {
        double[][] r = new double[a.length][a[0].length];
        for (int i = 0; i < a.length; i++) for (int j = 0; j < a[0].length; j++) r[i][j] = a[i][j] + b[i][j];
        return r;
    }
    static String show(double[][] m) {
        StringBuilder sb = new StringBuilder();
        for (double[] row : m) {
            sb.append(" [");
            for (double v : row) sb.append(String.format("%5.0f", v));
            sb.append(" ]");
        }
        return sb.toString();
    }

    public static void main(String[] args) {
        double[][] x = {{1, 2, 0, -1}, {0, 1, 3, 2}};               // 2 tokens, hidden size 4
        double[][] a = new double[4][8], b = new double[8][4];        // up projection 4 -> 8, down projection 8 -> 4
        for (int i = 0; i < 4; i++) for (int j = 0; j < 8; j++) a[i][j] = (i * 8 + j) % 7 - 3;
        for (int i = 0; i < 8; i++) for (int j = 0; j < 4; j++) b[i][j] = (i * 4 + j) % 5 - 2;
        double[][] reference = matmul(relu(matmul(x, a)), b);

        double[][][] partial = new double[2][][];
        for (int d = 0; d < 2; d++) {                                 // device d: columns 4d..4d+3 of A, rows 4d..4d+3 of B
            double[][] h = relu(matmul(x, columns(a, 4 * d, 4 * d + 4)));
            partial[d] = matmul(h, rows(b, 4 * d, 4 * d + 4));
        }
        double[][] sum = add(partial[0], partial[1]);                 // the all-reduce
        System.out.println("device 0 partial :" + show(partial[0]));
        System.out.println("device 1 partial :" + show(partial[1]));
        System.out.println("all-reduce sum   :" + show(sum));
        System.out.println("one device       :" + show(reference));
        System.out.println("equal: " + Arrays.deepEquals(sum, reference));

        // A split by rows instead: each device applies ReLU to a partial sum
        double[][] p0 = relu(matmul(columns(x, 0, 2), rows(a, 0, 2))), p1 = relu(matmul(columns(x, 2, 4), rows(a, 2, 4)));
        double[][] wrong = matmul(add(p0, p1), b);
        System.out.println("A split by rows, ReLU before the sum, equal: " + Arrays.deepEquals(wrong, reference));
    }
}

device 0 partial : [    0    0    0    0 ] [    9   25   11  -33 ]
device 1 partial : [  -28  -10    8   26 ] [   -5    0    5   10 ]
all-reduce sum   : [  -28  -10    8   26 ] [    4   25   16  -23 ]
one device       : [  -28  -10    8   26 ] [    4   25   16  -23 ]
equal: true
A split by rows, ReLU before the sum, equal: false

Device d holds columns 4d to 4d+3 of A, which is half of the 8 hidden units, and the same rows of B. It applies ReLU locally. This is correct because ReLU acts on each value separately, and all the hidden units that the device needs are on that device. Multiplying them by its rows of B gives a 2 by 4 partial result.

For the first token, device 0's partial is all zeros. ReLU set all four of its hidden values to zero, so those units contribute nothing. Device 1's partial [-28 -10 8 26] carries the whole first row.

The all-reduce adds the partials element by element. The sum [-28 -10 8 26] [4 25 16 -23] equals the one-device result, and equal: true confirms it. The 2 by 8 hidden activations never leave their devices. Only the 2 by 4 output crosses between them, once per MLP.

The last line shows why the order of the split matters. If A were split by rows, each device would hold only a partial sum of every hidden unit. ReLU applied to a partial sum is not the same as ReLU applied to the full sum. For example, partial sums of 5 and -7 give 5 after ReLU, but the full sum is -2, which gives 0. The result differs from the reference, so a row split of the first matrix would need an all-reduce before the nonlinearity, which doubles the communication.

Attention uses the same pattern. The query, key, and value projections are split by columns along the heads, so device d holds whole attention heads. Each device runs attention for its heads with no communication. The output projection is split by rows and followed by one all-reduce. A transformer layer under tensor parallelism therefore performs two all-reduces in the forward pass (one trip of the input through the model): one after attention and one after the MLP.

The KV cache is split by heads as well. This is why Serving with vLLM, Example 5, showed the KV cache per GPU halving with each doubling of -tp until there was one KV head per GPU.

Example 4: the ring all-reduce

Example 3 needs an all-reduce, so its cost matters. The simplest all-reduce sends everything to one device, adds, and sends the sum back. That root device then receives and sends (n-1) full tensors (where n is the number of devices) while the others wait, so the link of the root is the bottleneck.

The ring all-reduce (Patarasuk and Yuan, "Bandwidth optimal all-reduce algorithms for clusters of workstations", 2009) avoids this. It arranges the n devices in a ring and splits the tensor into n chunks. It has two phases:

• In the reduce-scatter phase, each device repeatedly sends one chunk to its right neighbour, which adds it to its own copy. After n-1 steps each device holds the complete sum of one chunk.
• In the all-gather phase, the finished chunks travel around the ring for n-1 more steps, until every device has every chunk.

The program runs both phases on 4 devices with 8 values each and counts the values each device sends.

import java.util.Arrays;

public class RingAllReduce {
    static long[][] copy(long[][] v) {
        long[][] c = new long[v.length][];
        for (int i = 0; i < v.length; i++) c[i] = v[i].clone();
        return c;
    }

    public static void main(String[] args) {
        int n = 4, len = 8, chunk = len / n;                          // 4 devices, 8 values each, 4 chunks of 2
        long[][] v = new long[n][len];
        for (int d = 0; d < n; d++) for (int i = 0; i < len; i++) v[d][i] = (d + 1) * 10 + i;
        long[] expected = new long[len];
        for (long[] row : v) for (int i = 0; i < len; i++) expected[i] += row[i];
        int[] sent = new int[n];

        for (int s = 0; s < n - 1; s++) {                             // reduce-scatter: n - 1 steps
            long[][] before = copy(v);                                // all devices send at the same time
            for (int d = 0; d < n; d++) {
                int c = Math.floorMod(d - s, n), to = (d + 1) % n;    // send chunk (d - s) to the right neighbour, which adds it
                for (int i = c * chunk; i < (c + 1) * chunk; i++) v[to][i] += before[d][i];
                sent[d] += chunk;
            }
        }
        for (int d = 0; d < n; d++) {
            int c = (d + 1) % n;
            System.out.println("after reduce-scatter, device " + d + " owns the sum of chunk " + c + ": "
                    + Arrays.toString(Arrays.copyOfRange(v[d], c * chunk, (c + 1) * chunk)));
        }
        for (int s = 0; s < n - 1; s++) {                             // all-gather: n - 1 steps
            long[][] before = copy(v);
            for (int d = 0; d < n; d++) {
                int c = Math.floorMod(d + 1 - s, n), to = (d + 1) % n; // pass on a finished chunk, which the neighbour copies
                for (int i = c * chunk; i < (c + 1) * chunk; i++) v[to][i] = before[d][i];
                sent[d] += chunk;
            }
        }
        for (int d = 0; d < n; d++) System.out.println("device " + d + ": " + Arrays.toString(v[d]));
        System.out.println("all equal to the sum " + Arrays.toString(expected) + ": "
                + Arrays.stream(v).allMatch(row -> Arrays.equals(row, expected)));
        System.out.println("values sent per device: " + Arrays.toString(sent) + "; 2(n-1)/n x len = " + 2 * (n - 1) * len / n);
        System.out.println("through one root device instead: the root receives " + (n - 1) * len + " and sends " + (n - 1) * len);
    }
}

after reduce-scatter, device 0 owns the sum of chunk 1: [108, 112]
after reduce-scatter, device 1 owns the sum of chunk 2: [116, 120]
after reduce-scatter, device 2 owns the sum of chunk 3: [124, 128]
after reduce-scatter, device 3 owns the sum of chunk 0: [100, 104]
device 0: [100, 104, 108, 112, 116, 120, 124, 128]
device 1: [100, 104, 108, 112, 116, 120, 124, 128]
device 2: [100, 104, 108, 112, 116, 120, 124, 128]
device 3: [100, 104, 108, 112, 116, 120, 124, 128]
all equal to the sum [100, 104, 108, 112, 116, 120, 124, 128]: true
values sent per device: [12, 12, 12, 12]; 2(n-1)/n x len = 12
through one root device instead: the root receives 24 and sends 24

Device d starts with the values (d+1)*10 + i, so the sums are 100, 104, ..., 128. After reduce-scatter, device 0 owns the complete sum of chunk 1 ([108, 112]), device 1 owns chunk 2, device 2 owns chunk 3, and device 3 owns chunk 0. Each chunk has travelled three hops and picked up one device's contribution per hop. After all-gather every device holds the full sum, as the four identical lines and true show.

Every device sent 12 values: 2 values per step for 6 steps. The general count is 2(n-1)/n times the tensor size, here 2 x 3/4 x 8 = 12. Two properties follow.

• Traffic does not grow with the number of devices. The traffic per device approaches twice the tensor size as n grows. The root of the naive method handles 24 values in and 24 out here, and that number grows with n.
• Latency does grow. The ring needs 2(n-1) steps that run one after another, and every step has a fixed start-up delay, so more devices mean more waiting.

NVIDIA's NCCL library (NVIDIA Collective Communications Library), which PyTorch and vLLM use for GPU collectives, uses ring and tree algorithms. The nccl-tests documentation converts measured all-reduce bandwidth to "bus bandwidth" with the same 2(n-1)/n factor. Example 5 puts real sizes into both properties.

Example 5: all-reduce cost per layer and per token, NVLink and PCIe

Now the cost can be computed for a real model. A 70B model (hidden size 8,192, 80 layers) with tensor parallelism over 8 GPUs performs 2 x 80 = 160 all-reduces per forward pass. Each one covers a tensor of tokens x 8192 BF16 values of 2 bytes each, so the size of every all-reduce depends on how many tokens are in the pass.

The program computes the traffic per GPU for a decode step with 1 and with 64 sequences, and for a 4,096-token prefill (the first pass, which reads the whole prompt at once). It converts the traffic to time at NVLink and PCIe 4.0 bandwidth and counts the ring steps that must happen one after another. For comparison it prints the time to read each GPU's weights (the decode cost) and the time to do the prefill arithmetic.

import java.util.Locale;

public class AllReduceCost {
    public static void main(String[] args) {
        int hidden = 8192, layers = 80, n = 8;                        // 70B shape, tensor parallel over 8 GPUs
        double nvlink = 450e9, pcie = 31.5e9;                         // bytes per second, one direction
        int allReduces = 2 * layers;                                  // after attention and after the MLP of every layer
        String[] names = {"decode, batch 1", "decode, batch 64", "prefill, 4096 tokens"};
        int[] tokens = {1, 64, 4096};
        System.out.println("forward pass          MB per all-reduce  MB sent per GPU  NVLink ms  PCIe 4.0 ms");
        for (int i = 0; i < names.length; i++) {
            double bytes = tokens[i] * hidden * 2.0;                  // BF16 activations
            double perGpu = 2.0 * (n - 1) / n * bytes * allReduces;   // ring all-reduce traffic per GPU, whole pass
            System.out.printf(Locale.ROOT, "%-20s  %17.3f  %15.1f  %9.2f  %11.2f%n",
                    names[i], bytes / 1e6, perGpu / 1e6, perGpu / nvlink * 1e3, perGpu / pcie * 1e3);
        }
        int steps = allReduces * 2 * (n - 1);
        System.out.printf("ring steps one after another per pass: %d all-reduces x %d steps = %d%n", allReduces, 2 * (n - 1), steps);
        for (double us : new double[] {1, 5})
            System.out.printf(Locale.ROOT, "  at an assumed %.0f us per step: %.2f ms per pass%n", us, steps * us / 1e3);
        double weights = 70.6e9 * 2 / n;
        System.out.printf(Locale.ROOT, "decode reference: reading %.2f GB of weights per GPU at 3.35 TB/s takes %.2f ms%n",
                weights / 1e9, weights / 3.35e12 * 1e3);
        System.out.printf(Locale.ROOT, "prefill reference: 2 x 70.6e9 x 4096 FLOPs over 8 GPUs at 989 TFLOP/s takes %.1f ms%n",
                2 * 70.6e9 * 4096 / n / 989e12 * 1e3);
    }
}

forward pass          MB per all-reduce  MB sent per GPU  NVLink ms  PCIe 4.0 ms
decode, batch 1                   0.016              4.6       0.01         0.15
decode, batch 64                  1.049            293.6       0.65         9.32
prefill, 4096 tokens             67.109          18790.5      41.76       596.52
ring steps one after another per pass: 160 all-reduces x 14 steps = 2240
  at an assumed 1 us per step: 2.24 ms per pass
  at an assumed 5 us per step: 11.20 ms per pass
decode reference: reading 17.65 GB of weights per GPU at 3.35 TB/s takes 5.27 ms
prefill reference: 2 x 70.6e9 x 4096 FLOPs over 8 GPUs at 989 TFLOP/s takes 73.1 ms

Read the table row by row. The traffic grows with the number of tokens in the pass:

• Decode with one sequence moves 16 KB per all-reduce and 4.6 MB per GPU per step. Over NVLink that is 0.01 ms and over PCIe 0.15 ms. Both are small next to the 5.27 ms that each GPU spends reading its 17.65 GB of weights.
• Decode with 64 sequences moves 1 MB per all-reduce and 293.6 MB per GPU. NVLink needs 0.65 ms, which is 12% of the weight-read time. PCIe needs 9.32 ms, which is more than the weight read itself, so the step would take about two and a half times as long as on NVLink (14.6 ms against 5.9 ms).
• A 4,096-token prefill moves 67 MB per all-reduce and 18.8 GB per GPU. NVLink needs 41.8 ms against 73.1 ms of arithmetic. Engines overlap part of the communication with computation. Over PCIe the same traffic takes 596.5 ms, eight times the arithmetic.

The bandwidth term explains why tensor parallelism is used with NVLink. Bandwidth is how much data per second a link carries, and PCIe is too slow for this traffic.

The latency term explains why even NVLink is not free for decoding. Latency is the fixed delay each step pays before any data moves. Each pass contains 2,240 ring steps that cannot overlap with each other (160 all-reduces times 14 steps). With an assumed 1 microsecond per step, a placeholder because the real value depends on the GPUs, the link, and the library, latency adds 2.24 ms per decode step. That is 42% of the 5.27 ms weight-read time. At 5 microseconds it adds 11.2 ms and dominates the step. For small messages, libraries therefore use algorithms with fewer sequential steps than a ring. vLLM ships its own all-reduce kernel for this case. The flag --disable-custom-all-reduce ("Disable the custom all-reduce kernel and fall back to NCCL", ParallelConfig in vllm/config/parallel.py) turns it off.

Two practical rules follow. Tensor parallelism pays off inside one server with NVLink or a similar fast link. Over PCIe or between servers, it moves too much data per token, and pipeline parallelism (next) moves much less.

Example 6: pipeline parallelism, stages and the bubble

Tensor parallelism cuts every layer. Pipeline parallelism (PP) cuts the model between layers instead. GPU 0 holds the first quarter of the layers, GPU 1 the second quarter, and so on. Each range of layers is a stage. A batch passes through the stages in order, like a car moving along an assembly line. Only the activations (the vectors that flow between layers) cross between GPUs: one hidden vector per token, once per stage boundary, instead of 160 all-reduces per pass.

The cost is idle time. While GPU 3 works on a batch, GPUs 0 to 2 have nothing to do. The assembly line analogy holds only when several cars are on the line at once, which is the point of this example. The program draws a timeline for 4 stages, first for one batch, then for one batch cut into 4 micro-batches (smaller groups of sequences that enter the pipeline one after another). Each column is one time slot, the time a stage needs for one micro-batch. A dot means the GPU is idle.

import java.util.Locale;

public class PipelineBubble {
    static void timeline(int stages, int microBatches) {
        int slots = microBatches + stages - 1, busy = 0;
        System.out.printf("%d stages, %d micro-batch(es), %d time slots%n", stages, microBatches, slots);
        for (int s = 0; s < stages; s++) {
            StringBuilder row = new StringBuilder("  GPU " + s + " |");
            for (int t = 0; t < slots; t++) {
                int m = t - s;                                       // micro-batch m reaches stage s in slot m + s
                if (m >= 0 && m < microBatches) { row.append(" m").append(m); busy++; }
                else row.append("  .");
            }
            System.out.println(row);
        }
        System.out.printf(Locale.ROOT, "  busy %d of %d stage-slots (%.1f%%)%n", busy, stages * slots, 100.0 * busy / (stages * slots));
    }

    public static void main(String[] args) {
        timeline(4, 1);
        timeline(4, 4);
    }
}

4 stages, 1 micro-batch(es), 4 time slots
  GPU 0 | m0  .  .  .
  GPU 1 |  . m0  .  .
  GPU 2 |  .  . m0  .
  GPU 3 |  .  .  . m0
  busy 4 of 16 stage-slots (25.0%)
4 stages, 4 micro-batch(es), 7 time slots
  GPU 0 | m0 m1 m2 m3  .  .  .
  GPU 1 |  . m0 m1 m2 m3  .  .
  GPU 2 |  .  . m0 m1 m2 m3  .
  GPU 3 |  .  .  . m0 m1 m2 m3
  busy 16 of 28 stage-slots (57.1%)

With one batch (m0), GPU 0 works in slot 0, GPU 1 in slot 1, and so on. Each GPU is busy one slot in four, which is 25% of the time. The empty cells are the pipeline bubble, the idle time caused by waiting for the pipeline to fill and drain. The batch also takes 4 slots to finish. A single GPU holding all layers would take the same time, because each stage does a quarter of the work in a quarter of the time. Pipeline parallelism lets a model fit. It does not make one batch faster.

With 4 micro-batches, GPU 0 starts m1 while GPU 1 works on m0. After 3 slots of filling, all four GPUs are busy at once (slot 3), and then the pipeline drains. Busy cells rise to 16 of 28, which is 57.1%. The general formula for m micro-batches over p stages is m / (m + p - 1). Check it with the numbers here: 4 / (4 + 4 - 1) = 4/7 = 57.1%. The p - 1 slots of filling and draining are wasted whatever m is, so more micro-batches spread that fixed waste over more useful work.

Example 7: filling the pipeline with micro-batches and batches in flight

Example 6 gave the formula m / (m + p - 1). This example uses it to find how many micro-batches a pipeline needs. The program prints a table of the formula. It then simulates the case that matters for serving, which is decoding. In decoding, a batch must leave the last stage before it can produce its next token, so one batch cannot fill the pipeline by itself. The only way to keep all stages busy is to keep several independent batches in flight at the same time.

import java.util.Locale;

public class PipelineFill {
    public static void main(String[] args) {
        int[] microBatches = {1, 2, 4, 8, 16, 32};
        System.out.println("busy share m / (m + p - 1) for one batch split into m micro-batches");
        System.out.print("stages p");
        for (int m : microBatches) System.out.printf("  %6s", "m=" + m);
        System.out.println();
        for (int p : new int[] {2, 4, 8}) {
            System.out.printf("%8d ", p);
            for (int m : microBatches) System.out.printf(Locale.ROOT, "  %5.1f%%", 100.0 * m / (m + p - 1));
            System.out.println();
        }
        System.out.println("steady decoding on 4 stages, k batches in flight, 8 sequences per batch, 1 slot per stage");
        int p = 4;
        for (int k = 1; k <= p; k++) {
            int tokens = 0;
            for (int t = 100; t < 200; t++)                          // 100 slots after the pipeline has filled
                for (int b = 0; b < k; b++) if ((t - b - p + 1) % p == 0) tokens += 8;   // batch b leaves stage 3 in slots b+3, b+7, ...
            System.out.printf(Locale.ROOT, "  k=%d: GPUs busy %3.0f%%, %3d tokens per 100 slots, each sequence gets a token every %d slots%n",
                    k, 100.0 * k / p, tokens, p);
        }
    }
}

busy share m / (m + p - 1) for one batch split into m micro-batches
stages p     m=1     m=2     m=4     m=8    m=16    m=32
       2    50.0%   66.7%   80.0%   88.9%   94.1%   97.0%
       4    25.0%   40.0%   57.1%   72.7%   84.2%   91.4%
       8    12.5%   22.2%   36.4%   53.3%   69.6%   82.1%
steady decoding on 4 stages, k batches in flight, 8 sequences per batch, 1 slot per stage
  k=1: GPUs busy  25%, 200 tokens per 100 slots, each sequence gets a token every 4 slots
  k=2: GPUs busy  50%, 400 tokens per 100 slots, each sequence gets a token every 4 slots
  k=3: GPUs busy  75%, 600 tokens per 100 slots, each sequence gets a token every 4 slots
  k=4: GPUs busy 100%, 800 tokens per 100 slots, each sequence gets a token every 4 slots

Read the table across. With 4 stages, 4 micro-batches keep the GPUs busy 57.1% of the time, 16 micro-batches 84.2%, and 32 micro-batches 91.4%. With 8 stages, 32 micro-batches reach only 82.1%. Deeper pipelines need more micro-batches for the same efficiency, because there is more filling and draining to hide.

The second part shows decoding on 4 stages with 8 sequences per batch. With one batch in flight (k=1), the batch occupies one stage at a time, the GPUs are busy 25% of the time, and 200 tokens come out per 100 slots. Each additional batch occupies another stage, so throughput grows to 400, 600, and 800 tokens per 100 slots, and at k=4 all GPUs are busy. In every case each sequence receives one token every 4 slots, because its batch must pass all 4 stages. Pipeline parallelism multiplies throughput by keeping p batches in flight. It does not raise the token rate of any single sequence.

vLLM implements this directly. VllmConfig.max_concurrent_batches returns the pipeline size ("PP requires PP-size concurrent batches to fill the pipeline", vllm/config/vllm.py), and the engine core keeps that many scheduled batches in a queue. A server with few concurrent users therefore wastes most of a pipeline-parallel deployment, while a busy server keeps it full.

Example 8: expert parallelism and all-to-all routing

A mixture-of-experts (MoE) layer replaces one large MLP with many small expert MLPs. A router inside the layer sends each token to its top-k experts, for example the best 2 of 8 (Training, Fine-Tuning, and Model Variants, Example 8). Expert parallelism (EP) gives each GPU whole experts instead of a slice of every expert.

This creates a communication problem. A token may sit on GPU 0 while its chosen expert lives on GPU 3. Every token must travel to the GPUs of its chosen experts (dispatch), and the results must travel back (combine). Because every GPU may send to every other GPU, this exchange is called an all-to-all. The program routes 32 tokens (8 starting on each of 4 GPUs) to the top 2 of 8 experts. Experts 0 and 1 are made more popular than the others, as often happens in real models. The program counts the traffic and the work per GPU.

import java.util.Arrays;
import java.util.Locale;
import java.util.Random;

public class ExpertAllToAll {
    public static void main(String[] args) {
        int experts = 8, devices = 4, tokens = 32, topK = 2, hidden = 4096;
        int perDevice = experts / devices;                            // experts 0-1 on device 0, 2-3 on device 1, ...
        Random r = new Random(7);
        int[][] copies = new int[devices][devices];                   // token copies sent from device (row) to device (column)
        int[] expertLoad = new int[experts];
        for (int t = 0; t < tokens; t++) {
            int src = t / (tokens / devices);                         // tokens 0-7 start on device 0, 8-15 on device 1, ...
            double[] score = new double[experts];
            for (int e = 0; e < experts; e++) score[e] = r.nextGaussian() + (e < 2 ? 1.0 : 0.0);   // experts 0 and 1 are popular
            Integer[] order = {0, 1, 2, 3, 4, 5, 6, 7};
            Arrays.sort(order, (a, b) -> Double.compare(score[b], score[a]));
            for (int k = 0; k < topK; k++) {
                copies[src][order[k] / perDevice]++;
                expertLoad[order[k]]++;
            }
        }
        System.out.println("token copies   to dev0  dev1  dev2  dev3");
        for (int s = 0; s < devices; s++) {
            System.out.printf("from device %d", s);
            for (int d = 0; d < devices; d++) System.out.printf("  %4d", copies[s][d]);
            System.out.println();
        }
        System.out.println("tokens per expert: " + Arrays.toString(expertLoad));
        int[] received = new int[devices];
        int remote = 0;
        for (int s = 0; s < devices; s++) for (int d = 0; d < devices; d++) {
            received[d] += copies[s][d];
            if (s != d) remote += copies[s][d];
        }
        double mean = (double) tokens * topK / devices;
        int max = Arrays.stream(received).max().getAsInt();
        System.out.printf(Locale.ROOT, "work per device: %s, max %d vs mean %.0f (imbalance %.2f)%n",
                Arrays.toString(received), max, mean, max / mean);
        int[][] spread = {{0, 7}, {1, 6}, {2, 5}, {3, 4}};             // hot experts on different devices
        int[] spreadLoad = new int[devices];
        for (int d = 0; d < devices; d++) for (int e : spread[d]) spreadLoad[d] += expertLoad[e];
        System.out.println("placement {0,7} {1,6} {2,5} {3,4}: work per device " + Arrays.toString(spreadLoad));
        System.out.printf(Locale.ROOT, "remote copies %d of %d; bytes per layer (dispatch + combine, BF16): %d KiB%n",
                remote, tokens * topK, remote * hidden * 2 * 2 / 1024);
    }
}

token copies   to dev0  dev1  dev2  dev3
from device 0     7     2     3     4
from device 1     9     3     2     2
from device 2     8     2     2     4
from device 3    10     1     3     2
tokens per expert: [16, 18, 4, 4, 6, 4, 5, 7]
work per device: [34, 8, 10, 12], max 34 vs mean 16 (imbalance 2.13)
placement {0,7} {1,6} {2,5} {3,4}: work per device [23, 23, 8, 10]
remote copies 50 of 64; bytes per layer (dispatch + combine, BF16): 800 KiB

The matrix counts token copies. Each row is a source GPU and each column is a destination GPU. Every row sums to 16 (8 tokens times 2 experts). Column dev0 is the largest in every row (7, 9, 8, 10), because device 0 holds the two popular experts.

The line tokens per expert shows the cause. Experts 0 and 1 receive 16 and 18 token copies, while the others receive 4 to 7. The work per device is the sum over its two experts, so device 0 processes 34 copies while the mean is 16. The ratio, 34 / 16 = 2.13, is the imbalance. In an MoE layer all devices wait for the slowest one, so this layer runs at less than half of the speed that a balanced layer would reach.

The placement line puts the two popular experts on different devices, and the maximum work drops from 34 to 23. It cannot drop below 18, the load of expert 1 alone, unless a popular expert exists on two devices. vLLM's expert-parallel load balancer (--enable-eplb) does this. It records expert load over a window (EPLBConfig.window_size, default 1000) and can place extra copies of busy experts (num_redundant_experts, default 0).

The last line is the communication. 50 of the 64 copies leave their GPU. Each carries a 4,096-value BF16 hidden state (8,192 bytes), and it travels twice (dispatch and combine), so the total is 50 x 8,192 x 2 = 800 KiB per layer for 32 tokens. All-to-all traffic grows with the number of tokens and with top-k, and it happens in every MoE layer.

Example 9: data parallelism behind a router

When the model fits on one GPU (or one group of GPUs), the simplest way to serve more users is data parallelism (DP). Run several full copies of the model, called replicas, and put a router in front that sends each request to one of them. Replicas do not communicate during inference, so throughput grows with the number of replicas. What remains is the routing policy, the rule that picks a replica for each request.

The program sends 12 requests, one every 100 time units, to 3 replicas that each serve one request at a time. Requests 0, 3, 6, and 9 are long (700 to 950 units) and the others are short. It compares two policies by mean latency (the average time from arrival to finish) and by maximum latency.

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class DataParallelRouter {
    interface Policy { int pick(int request, long now, List<List<Long>> finishTimes); }

    static void run(String name, Policy policy) {
        int[] work = {900, 50, 60, 800, 40, 70, 950, 30, 55, 700, 65, 45};   // service time of each request
        int replicas = 3, gap = 100;                                          // one arrival every 100 time units
        long[] freeAt = new long[replicas];
        List<List<Long>> finishTimes = new ArrayList<>();
        List<List<Integer>> assigned = new ArrayList<>();
        for (int r = 0; r < replicas; r++) { finishTimes.add(new ArrayList<>()); assigned.add(new ArrayList<>()); }
        long total = 0, worst = 0;
        for (int i = 0; i < work.length; i++) {
            long arrival = (long) i * gap;
            int r = policy.pick(i, arrival, finishTimes);
            long finish = Math.max(arrival, freeAt[r]) + work[i];             // each replica serves one request at a time
            freeAt[r] = finish;
            finishTimes.get(r).add(finish);
            assigned.get(r).add(i);
            total += finish - arrival;
            worst = Math.max(worst, finish - arrival);
        }
        System.out.printf(Locale.ROOT, "%-26s %s  mean latency %6.1f  max %4d%n", name, assigned, (double) total / work.length, worst);
    }

    public static void main(String[] args) {
        run("round-robin", (i, now, f) -> i % 3);
        run("least outstanding requests", (i, now, f) -> {
            int best = 0;
            long[] open = new long[f.size()];
            for (int r = 0; r < f.size(); r++) open[r] = f.get(r).stream().filter(t -> t > now).count();
            for (int r = 1; r < f.size(); r++) if (open[r] < open[best]) best = r;
            return best;
        });
    }
}

round-robin                [[0, 3, 6, 9], [1, 4, 7, 10], [2, 5, 8, 11]]  mean latency  601.3  max 2450
least outstanding requests [[0, 7, 9, 10], [1, 2, 3, 8, 11], [4, 5, 6]]  mean latency  415.0  max  950

Round-robin sends request i to replica i mod 3, which means the replicas take turns. The long requests are at positions 0, 3, 6, and 9, so all four land on replica 0. That replica then has 3,350 units of work while the others have about 200 each. Request 9 waits behind three long requests and finishes 2,450 units after it arrived. The mean latency is 601.3.

Least outstanding requests counts, at each arrival, how many unfinished requests each replica has and picks the replica with the fewest. When request 3 arrives, replica 0 is still busy with request 0, so request 3 goes elsewhere. The long requests spread over all three replicas, the worst latency drops to 950 (the length of the longest request), and the mean drops to 415.0.

The router does not know how long a request will run, because the output length is unknown at arrival. Policies therefore use signals that the replicas report, such as how many requests they hold. vLLM's built-in data-parallel load balancer (DPLBAsyncMPClient.get_core_engine_for_request in vllm/v1/engine/core_client.py) picks the engine with the lowest score. The score is the number of waiting plus running requests, increased for waiting requests when the KV cache of the engine is more than 50% full. For dense models (models without experts), vLLM's data-parallel ranks are independent engines ("Non-MoE DP ranks are completely independent", vllm/v1/engine/core.py). Example 16 adds the second signal a router can use: which replica already caches the prompt.

Example 10: llama.cpp split modes

llama.cpp is a program that runs LLMs on ordinary hardware, including machines with one or more consumer GPUs. It decides how to use several GPUs with the option -sm (--split-mode). The help text of the pinned llama-server lists four modes, plus the related options -ts and -mg:

llama-server --help | grep -A 13 -- '-sm,'

Real output (pinned llama.cpp build):

-sm,   --split-mode {none,layer,row,tensor}
                                        how to split the model across multiple GPUs, one of:
                                        - none: use one GPU only
                                        - layer (default): split layers and KV across GPUs (pipelined)
                                        - row: split weight across GPUs by rows (parallelized)
                                        - tensor: split weights and KV across GPUs (parallelized,
                                        EXPERIMENTAL)
                                        (env: LLAMA_ARG_SPLIT_MODE)
-ts,   --tensor-split N0,N1,N2,...      fraction of the model to offload to each GPU, comma-separated list of
                                        proportions, e.g. 3,1
                                        (env: LLAMA_ARG_TENSOR_SPLIT)
-mg,   --main-gpu INDEX                 the GPU to use for the model (with split-mode = none), or for
                                        intermediate results and KV (with split-mode = row) (default: 0)
                                        (env: LLAMA_ARG_MAIN_GPU)

The four modes map onto the ideas of this lesson. The layer mode is pipeline parallelism, and the row and tensor modes are tensor parallelism.

In layer mode, one micro-batch visits the GPUs one after another, as in Example 6. When all layers are on GPUs and every device supports asynchronous compute, llama.cpp turns on pipelining in its scheduler (llama-context.cpp logs pipeline parallelism enabled). The micro-batches of one large batch (-ub inside -b) then overlap across the GPUs, with up to 4 copies in flight (GGML_SCHED_MAX_COPIES). This speeds up prompt processing.

A decode step for one sequence is a single micro-batch, so it still visits the GPUs in turn. A model that needs two GPUs decodes at about the speed of one GPU reading all the weights. The gain of layer mode is capacity, not speed.

The CPU build used in this course has no GPU. When it is given -sm row -ts 3,1, it prints two real warnings and ignores the flags:

warning: llama.cpp was compiled without support for GPU offload. Setting the split mode has no effect.
warning: llama.cpp was compiled without support for GPU offload. Setting a tensor split has no effect.

These commands show typical use on a machine with GPUs. They are not run in this course. The first gives GPU 0 three quarters of the layers. The second ignores every GPU except the second one.

# two GPUs, layers split 3:1 because GPU 0 has three times the free memory
llama-server -m model.gguf -ngl 99 -sm layer -ts 3,1
# one GPU only, the second one
llama-server -m model.gguf -ngl 99 -sm none -mg 1

Example 11: how llama.cpp assigns layers to GPUs

The option -ts gives proportions such as 3,1, and llama.cpp turns them into a layer range per device. When -ts is not given, the proportions are each device's free memory. The program follows the split-point code of llama_model_base::load_tensors in src/llama-model.cpp. Its steps are:

1. Build normalized running sums of the proportions. For 3,1 the running sums are 3 and 4, and dividing by 4 gives the split points 0.75 and 1.0.
2. Treat the output layer (the final step that turns the last hidden vector into token scores) as layer number n_layer.
3. Place only the last -ngl layers on GPUs. The option -ngl is the number of layers to offload to the GPU, and the other layers stay on the CPU.
4. Give layer il to the first device whose running sum is greater than (il - first_gpu_layer) / gpu_layers. In C++ this is std::upper_bound.

import java.util.ArrayList;
import java.util.List;

public class LayerSplit {
    // The split-point code of llama_model_base::load_tensors (src/llama-model.cpp, commit 0c1e57098)
    static void assign(String label, int nLayer, int ngl, float[] split) {
        int n = split.length;
        float[] points = new float[n];
        float sum = 0;
        for (int i = 0; i < n; i++) { sum += split[i]; points[i] = sum; }    // running sum ...
        for (int i = 0; i < n; i++) points[i] /= sum;                       // ... normalized to end at 1
        int gpuStart = Math.max(nLayer + 1 - ngl, 0);                       // layers before this stay on the CPU
        int gpuLayers = Math.min(ngl, nLayer + 1);                          // the output layer counts as layer nLayer
        List<String> where = new ArrayList<>();
        for (int il = 0; il <= nLayer; il++) {
            if (il < gpuStart || il - gpuStart >= gpuLayers) { where.add("CPU"); continue; }
            float x = (float) (il - gpuStart) / gpuLayers;
            int g = 0;
            while (g < n && points[g] <= x) g++;                            // std::upper_bound: first point above x
            where.add("GPU" + g);
        }
        StringBuilder sb = new StringBuilder(String.format("%-22s:", label));
        int start = 0;
        for (int il = 1; il <= nLayer + 1; il++) {
            if (il <= nLayer && where.get(il).equals(where.get(start))) continue;
            int end = Math.min(il - 1, nLayer - 1);
            if (start <= nLayer - 1) sb.append(String.format(" %s layers %d-%d (%d),", where.get(start), start, end, end - start + 1));
            start = il;
        }
        sb.append(" output on ").append(where.get(nLayer));
        System.out.println(sb);
    }

    public static void main(String[] args) {
        assign("-ngl 99 -ts 1,1", 32, 99, new float[] {1, 1});
        assign("-ngl 99 -ts 3,1", 32, 99, new float[] {3, 1});
        assign("-ngl 99, free 22/10 GB", 32, 99, new float[] {22, 10});
        assign("-ngl 20 -ts 1,1", 32, 20, new float[] {1, 1});
        assign("-ngl 99 -ts 1,1,1", 32, 99, new float[] {1, 1, 1});
    }
}

-ngl 99 -ts 1,1       : GPU0 layers 0-16 (17), GPU1 layers 17-31 (15), output on GPU1
-ngl 99 -ts 3,1       : GPU0 layers 0-24 (25), GPU1 layers 25-31 (7), output on GPU1
-ngl 99, free 22/10 GB: GPU0 layers 0-22 (23), GPU1 layers 23-31 (9), output on GPU1
-ngl 20 -ts 1,1       : CPU layers 0-12 (13), GPU0 layers 13-22 (10), GPU1 layers 23-31 (9), output on GPU1
-ngl 99 -ts 1,1,1     : GPU0 layers 0-10 (11), GPU1 layers 11-21 (11), GPU2 layers 22-31 (10), output on GPU2

The model has 32 repeating layers plus the output layer, 33 units in all. With -ts 1,1 the split points are 0.5 and 1.0. Layer il goes to GPU 0 while il / 33 < 0.5, which holds for layers 0 to 16. GPU 0 therefore gets 17 layers, and GPU 1 gets 15 layers plus the output layer. This is an even split of the 33 units, with the odd one on GPU 0.

-ts 3,1 moves the boundary to 0.75, so GPU 0 takes layers 0 to 24 (25 layers) and GPU 1 takes the rest. Free memory of 22 and 10 GB gives the split points 0.6875 (22 / 32) and 1.0, and 23 layers on GPU 0. This default follows free memory at load time, so another program holding memory on GPU 0 shifts layers to GPU 1. Passing -ts fixes the split.

With -ngl 20, only the last 20 units go to GPUs. Layers 0 to 12 stay on the CPU, and the 20 GPU units split 10 and 10 (9 layers plus the output layer on GPU 1). Partial offload always keeps the first layers on the CPU. The token embedding (the input layer, which turns token ids into vectors) is always on the CPU, because "there is very little benefit to offloading the input layer" (a comment in the same function). With three equal GPUs the 33 units split 11, 11, and 11.

The weights are only part of the memory. In layer mode each GPU also holds the KV cache of its own layers, so an uneven -ts also splits the KV cache unevenly.

Example 12: the RPC backend

Sometimes the GPUs are in different machines. llama.cpp can use GPUs or CPUs on other machines through its RPC backend (RPC means remote procedure call: the main program asks another program over the network to do work). Each remote machine runs ggml-rpc-server (the build target in tools/rpc/CMakeLists.txt), which exposes that machine's devices over TCP. The main machine adds them with --rpc host:port,... and then treats them as additional devices for the split modes above. Both sides need a build with -DGGML_RPC=ON. The pinned CPU build was compiled without it, so the flag fails at once:

llama-server -m tiny-random-qwen2vocab-f32.gguf --rpc 127.0.0.1:50052

Real output (pinned llama.cpp build without RPC support):

error while handling argument "--rpc": RPC not supported in this build

With an RPC build, the commands from tools/rpc/README.md look like this:

# on each remote machine (defaults: host 127.0.0.1, port 50052; -c caches large tensors locally)
ggml-rpc-server -H 192.168.88.10 -p 50052 -c
# on the main machine: local GPUs plus two remote servers
llama-server -m model.gguf -ngl 99 --rpc 192.168.88.10:50052,192.168.88.11:50052

These facts matter for a deployment. They come from the source and README at the pinned commit:

• The server binds to 127.0.0.1 by default (rpc_server_params in rpc-server.cpp). Any other host prints a warning block that says "Never expose the RPC server to an open network!" and "This is an experimental feature and is not secure!". The README calls the backend a proof of concept and "fragile and insecure". It has no authentication, so it belongs on a private network.
• Remote devices are placed at the front of the device list ("add RPC servers at the front of the list to minimize network transfers", src/llama.cpp), so they receive the first layers and the local GPUs the last layers.
• By default the weights and KV cache are split in proportion to each device's free memory; --tensor-split overrides it (Example 11).
• -c enables a local cache of large tensors under $HOME/.cache/llama.cpp/rpc (or LLAMA_CACHE), so later loads do not resend the weights.
• The transport is TCP, or RDMA when both sides support it (GGML_RPC_NO_RDMA=1 forces TCP).

With layer mode, each request crosses the network once per device boundary per step, carrying hidden states. This matches the pipeline traffic of Example 6 and is small. The weights cross the network once, at load time, which is why the cache (-c) exists.

Example 13: vLLM parallelism flags

vLLM exposes each kind of parallelism as a flag (EngineArgs.add_cli_args in vllm/engine/arg_utils.py, which sets fields of ParallelConfig):

The flags combine, so it helps to count what a combination means. In vLLM an engine is one serving process with its own scheduler and its own group of GPUs. The program uses three rules from the source:

1. The world size of one engine (the number of GPUs it uses) is pp x tp (ParallelConfig.__post_init__, without context parallelism).
2. The total GPU count is the world size times dp (world_size_across_dp).
3. MoE layers treat tp x dp GPUs as one group (FusedMoEParallelConfig.make in vllm/model_executor/layers/fused_moe/config.py). Without -ep, every expert is split that many ways. With -ep, whole experts are spread over that many GPUs.

public class VllmLayouts {
    record Layout(String model, int tp, int pp, int dp, boolean ep) {
        int worldSize() { return tp * pp; }                           // ParallelConfig.world_size (no context parallelism)
        int gpus() { return worldSize() * dp; }                       // world_size_across_dp
        String experts() {
            if (model.startsWith("dense")) return "no expert layers";
            int flat = tp * dp;                                       // FusedMoEParallelConfig.make flattens TP across DP
            return ep && flat > 1 ? "EP=" + flat + ": each GPU holds 1/" + flat + " of the experts"
                                  : "TP=" + flat + ": every expert split " + flat + " ways";
        }
        String flags() {
            return "-tp " + tp + (pp > 1 ? " -pp " + pp : "") + (dp > 1 ? " -dp " + dp : "") + (ep ? " --enable-expert-parallel" : "");
        }
    }

    public static void main(String[] args) {
        Layout[] layouts = {
            new Layout("dense 70B", 8, 1, 1, false),
            new Layout("dense 70B", 4, 2, 1, false),
            new Layout("dense 8B", 1, 1, 8, false),
            new Layout("MoE", 8, 1, 1, false),
            new Layout("MoE", 8, 1, 1, true),
            new Layout("MoE", 1, 1, 8, false),
            new Layout("MoE", 1, 1, 8, true),
            new Layout("MoE", 2, 1, 4, true)};
        System.out.println("model      flags                                 GPUs  engines  per engine  expert layers");
        for (Layout l : layouts)
            System.out.printf("%-9s  %-36s  %4d  %7d  %10d  %s%n", l.model(), l.flags(), l.gpus(), l.dp(), l.worldSize(), l.experts());
    }
}

model      flags                                 GPUs  engines  per engine  expert layers
dense 70B  -tp 8                                    8        1           8  no expert layers
dense 70B  -tp 4 -pp 2                              8        1           8  no expert layers
dense 8B   -tp 1 -dp 8                              8        8           1  no expert layers
MoE        -tp 8                                    8        1           8  TP=8: every expert split 8 ways
MoE        -tp 8 --enable-expert-parallel           8        1           8  EP=8: each GPU holds 1/8 of the experts
MoE        -tp 1 -dp 8                              8        8           1  TP=8: every expert split 8 ways
MoE        -tp 1 -dp 8 --enable-expert-parallel     8        8           1  EP=8: each GPU holds 1/8 of the experts
MoE        -tp 2 -dp 4 --enable-expert-parallel     8        4           2  EP=8: each GPU holds 1/8 of the experts

The first three rows are dense models on 8 GPUs. -tp 8 makes one engine of 8 GPUs. -tp 4 -pp 2 also uses 8 GPUs as one engine, with two pipeline stages of 4 tensor-parallel GPUs each. This is the layout for two servers of 4 GPUs. -dp 8 with -tp 1 makes 8 independent engines of one GPU each, for a model that fits on one GPU.

The MoE rows show how the flags combine.

• -tp 8 without -ep splits every expert's matrices 8 ways. Each expert becomes 8 small matrix products plus an all-reduce.
• -tp 1 -dp 8 replicates attention on all 8 GPUs (8 engines), but the MoE layers still form one group of 8, so the experts are split 8 ways across the engines.
• With -ep, the same 8 GPUs hold whole experts instead, one eighth each, and tokens move by all-to-all (Example 8).
• The last row combines both: 4 engines with 2-way tensor-parallel attention, and the experts spread over all 8 GPUs.

Because MoE layers join the data-parallel engines, those engines must run forward passes together. vLLM uses a separate engine class for MoE data parallelism (DPEngineCoreProc, which asserts that the model is MoE), and an idle engine runs dummy batches while its peers are busy. Typical commands follow. The first is for a dense model on one server, the second for an MoE model, and the third spreads a very large model over two servers:

# dense 70B on one 8-GPU server
vllm serve meta-llama/Llama-3.1-70B-Instruct -tp 8
# MoE model: data-parallel attention, expert-parallel MoE layers
vllm serve Qwen/Qwen3-30B-A3B -dp 8 --enable-expert-parallel
# two servers: pipeline stages across them (node 0 shown; node 1 adds --node-rank 1 and --headless)
vllm serve meta-llama/Llama-3.1-405B-Instruct-FP8 -tp 8 -pp 2 --nnodes 2 --node-rank 0 --master-addr 10.0.0.1

--nnodes, --node-rank, --master-addr, and --master-port (default 29501) set up multi-node runs with the default multiprocessing backend. A node is one server. --distributed-executor-backend ray uses Ray (a framework for distributed programs) instead. The --headless flag of vllm serve starts an engine without an API server, which the secondary nodes use.

Example 14: choosing a layout

Which layout should a deployment use? The previous examples give the criteria, collected in this table:

The program writes the table as rules and applies them to six setups. The rules are: keep 40% of each GPU for the KV cache and buffers, round the GPU count up to a power of two (so that attention heads divide evenly), and prefer tensor parallelism inside a node when NVLink is present. Read each output line as the answer to two questions: how many GPUs are needed, and which split to use.

import java.util.Locale;

public class ChooseLayout {
    record Setup(String name, double weightsGB, double gpuGB, int gpusPerNode, int nodes, boolean nvlink, boolean moe) {}

    static String choose(Setup s) {
        int total = s.gpusPerNode() * s.nodes();
        int need = (int) Math.ceil(s.weightsGB() / (0.6 * s.gpuGB()));   // leave 40% of each GPU for KV cache and buffers
        int k = Integer.highestOneBit(need - 1) << 1;                     // round up to a power of two (heads divide evenly)
        if (need == 1) k = 1;
        if (k > total) return "does not fit: " + need + " GPUs needed, " + total + " present";
        String layout;
        if (k == 1) layout = "no split";
        else if (k <= s.gpusPerNode()) layout = s.nvlink() ? "TP=" + k : "PP=" + k + " (PCIe: communicate once per stage)";
        else layout = "TP=" + s.gpusPerNode() + " inside a node, PP=" + k / s.gpusPerNode() + " across nodes";
        int replicas = total / k;
        if (replicas > 1) layout += ", " + replicas + " replicas (DP)";
        if (s.moe() && k > 1) layout += ", expert parallel for the MoE layers";
        return String.format(Locale.ROOT, "needs %d GPU(s) -> %s", need, layout);
    }

    public static void main(String[] args) {
        Setup[] setups = {
            new Setup("8B BF16, 8 x 80 GB NVLink", 16.1, 80, 8, 1, true, false),
            new Setup("70B BF16, 8 x 80 GB NVLink", 141.2, 80, 8, 1, true, false),
            new Setup("32B 4.5-bit, 2 x 24 GB PCIe", 18.5, 24, 2, 1, false, false),
            new Setup("405B FP8, 2 nodes x 8 x 80 GB", 405, 80, 8, 2, true, false),
            new Setup("MoE 235B FP8, 8 x 80 GB NVLink", 235, 80, 8, 1, true, true),
            new Setup("405B FP8, 8 x 80 GB NVLink", 405, 80, 8, 1, true, false)};
        for (Setup s : setups) System.out.printf("%-32s %s%n", s.name(), choose(s));
    }
}

8B BF16, 8 x 80 GB NVLink        needs 1 GPU(s) -> no split, 8 replicas (DP)
70B BF16, 8 x 80 GB NVLink       needs 3 GPU(s) -> TP=4, 2 replicas (DP)
32B 4.5-bit, 2 x 24 GB PCIe      needs 2 GPU(s) -> PP=2 (PCIe: communicate once per stage)
405B FP8, 2 nodes x 8 x 80 GB    needs 9 GPU(s) -> TP=8 inside a node, PP=2 across nodes
MoE 235B FP8, 8 x 80 GB NVLink   needs 5 GPU(s) -> TP=8, expert parallel for the MoE layers
405B FP8, 8 x 80 GB NVLink       does not fit: 9 GPUs needed, 8 present

The 8B model needs one GPU, so the 8-GPU server runs 8 replicas. The 70B BF16 model needs 3 GPUs by the 60% rule (141.2 / 48 = 2.9, rounded up), which rounds up again to a tensor-parallel group of 4, so the server holds 2 replicas of TP=4. A wider group (TP=8, one replica) would give lower latency per token and lower total throughput. The choice depends on whether the service targets latency or throughput.

The 32B model at 4.5 bits needs both 24 GB cards of a PCIe machine, and the rule picks a pipeline (layer) split, which is the llama.cpp default. The 405B FP8 model needs 9 GPUs (405 / 48 = 8.4, rounded up), which rounds to 16. The layout is TP=8 inside each server and PP=2 across the two. The MoE model gets TP=8 with expert parallelism. The last setup does not fit: 405 GB of weights need 9 GPUs at the 60% rule and the server has 8. Such a deployment either lowers the KV reserve (fewer or shorter contexts), quantizes further, or adds a server.

These rules are a starting point. The real choice is made by measuring candidate layouts under the expected load, with the benchmarks of Benchmarking Inference.

Example 15: multi-LoRA serving, memory and flags

Suppose a company has 20 teams, each with its own fine-tuned version of the same base model. Loading 20 full models would need the memory of many GPUs. A LoRA adapter (low-rank adaptation) avoids this. It adds two small matrices to each adapted layer (Training, Fine-Tuning, and Model Variants, Example 10): A with r rows and B with r columns. The number r is the rank, and a small rank keeps the adapter small. A module with in inputs and out outputs gains r x (in + out) parameters.

Multi-LoRA serving keeps one base model in memory and applies a different adapter per request, so many fine-tuned variants share one GPU. The program sizes adapters for the seven projection matrices of a Llama 3 8B shape, the GPU buffers vLLM reserves for them, and the memory for 20 variants.

import java.util.Locale;

public class MultiLoraMemory {
    record Module(String name, int in, int out) {}

    public static void main(String[] args) {
        Module[] modules = {                                          // Llama 3 8B shape, one layer
            new Module("q_proj", 4096, 4096), new Module("k_proj", 4096, 1024), new Module("v_proj", 4096, 1024),
            new Module("o_proj", 4096, 4096), new Module("gate_proj", 4096, 14336), new Module("up_proj", 4096, 14336),
            new Module("down_proj", 14336, 4096)};
        int layers = 32;
        double baseBytes = 8.03e9 * 2;                                // BF16 base model
        long perRankPerLayer = 0;
        for (Module m : modules) perRankPerLayer += m.in() + m.out(); // A is rank x in, B is out x rank
        System.out.println("LoRA parameters per rank per layer: " + perRankPerLayer);
        for (int rank : new int[] {8, 16, 64}) {
            long params = (long) rank * perRankPerLayer * layers;
            System.out.printf(Locale.ROOT, "rank %2d: %,d parameters, %.1f MB in BF16 (%.2f%% of the base model)%n",
                    rank, params, params * 2 / 1e6, 100.0 * params * 2 / baseBytes);
        }
        int maxLoras = 4, maxRank = 64;                               // vLLM sizes its GPU buffers for max_loras x max_lora_rank
        double slots = (double) maxLoras * maxRank * perRankPerLayer * layers * 2;
        System.out.printf(Locale.ROOT, "GPU adapter buffers for --max-loras %d --max-lora-rank %d: %.2f GB%n", maxLoras, maxRank, slots / 1e9);
        int tenants = 20;
        double merged = tenants * baseBytes, shared = baseBytes + tenants * 16L * perRankPerLayer * layers * 2;
        System.out.printf(Locale.ROOT, "%d fine-tuned variants: %.1f GB as merged models, %.1f GB as one base plus rank-16 adapters%n",
                tenants, merged / 1e9, shared / 1e9);
    }
}

LoRA parameters per rank per layer: 81920
rank  8: 20,971,520 parameters, 41.9 MB in BF16 (0.26% of the base model)
rank 16: 41,943,040 parameters, 83.9 MB in BF16 (0.52% of the base model)
rank 64: 167,772,160 parameters, 335.5 MB in BF16 (2.09% of the base model)
GPU adapter buffers for --max-loras 4 --max-lora-rank 64: 1.34 GB
20 fine-tuned variants: 321.2 GB as merged models, 17.7 GB as one base plus rank-16 adapters

One layer has 81,920 adapter parameters per unit of rank. For example, q_proj contributes 4096 + 4096 and gate_proj contributes 4096 + 14336. Over 32 layers, a rank-16 adapter has 41.9 million parameters, which is 83.9 MB in BF16 and 0.52% of the 16 GB base model. Rank 64 is four times larger, at 335.5 MB.

vLLM allocates its adapter weight buffers on the GPU for max_loras adapters at max_lora_rank (create_lora_weights in vllm/lora/layers/base_linear.py). The reservation follows the flags, not the adapters actually loaded. Four slots at rank 64 reserve 1.34 GB, which the KV cache does not get. The last line is the reason multi-LoRA serving exists: 20 merged fine-tuned models need 321.2 GB, while one base model plus 20 rank-16 adapters needs 17.7 GB.

The flags below come from the pinned sources:

vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-lora --max-loras 4 --max-lora-rank 64 \
  --lora-modules sql-expert=/adapters/sql java-expert=/adapters/java

The engines batch differently. vLLM's max_loras is the number of distinct adapters "in a single batch" (LoRAConfig docstring), so requests for different adapters share one forward pass. The llama-server README says that "requests with different LoRA configurations will not be batched together", so mixed traffic splits into smaller batches there.

Example 16: prefix-aware routing

Prefix caching (Paged KV Cache and Prefix Caching) lets an engine skip recomputing the start of a prompt that it has seen before, by reusing the stored KV cache. The cache is local to one replica. A router that ignores this can send turn 2 of a conversation to a replica that never saw turn 1, and that replica must recompute the whole prompt.

The program serves 6 agent sessions for 5 turns on 4 replicas. Every prompt starts with the same 2,000-token system prompt, and each turn appends to the history. Sessions 0 and 4 add 2,400 tokens per turn, and the others add 600. A replica reuses the longest prompt it has cached for the session, or the system prompt if it has served any session. The work per replica counts prefill tokens computed. The program tries four routing policies. Compare the hit rate (the share of prompt tokens reused) and the spread of work across replicas.

import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;

public class PrefixAwareRouting {
    static final int SYSTEM = 2000, SESSIONS = 6, TURNS = 5, REPLICAS = 4;
    static final int[] GROWTH = {2400, 600, 600, 600, 2400, 600};   // tokens each session adds per turn

    interface Policy { int pick(int session, int request, long[] work); }

    static int leastLoaded(long[] work) {
        int best = 0;
        for (int r = 1; r < work.length; r++) if (work[r] < work[best]) best = r;
        return best;
    }

    static void run(String name, Policy policy) {
        @SuppressWarnings("unchecked")
        Map<Integer, Integer>[] cached = new Map[REPLICAS];          // per replica: session -> longest prompt already cached
        for (int r = 0; r < REPLICAS; r++) cached[r] = new HashMap<>();
        long[] work = new long[REPLICAS];                             // prefill tokens computed per replica
        long computed = 0, reused = 0;
        int request = 0;
        for (int turn = 1; turn <= TURNS; turn++) {
            for (int s = 0; s < SESSIONS; s++) {
                int prompt = SYSTEM + GROWTH[s] * turn;                    // append-only history: the old prompt is a prefix
                int r = policy.pick(s, request++, work);
                int hit = cached[r].getOrDefault(s, cached[r].isEmpty() ? 0 : SYSTEM);
                reused += hit;
                computed += prompt - hit;
                work[r] += prompt - hit;
                cached[r].put(s, prompt);
            }
        }
        System.out.printf(Locale.ROOT, "%-28s computed %6d  reused %6d  hit rate %5.1f%%  work per replica %s%n",
                name, computed, reused, 100.0 * reused / (computed + reused), Arrays.toString(work));
    }

    public static void main(String[] args) {
        run("round-robin", (s, i, work) -> i % REPLICAS);
        run("least prefill work", (s, i, work) -> leastLoaded(work));
        run("session affinity", (s, i, work) -> s % REPLICAS);
        run("affinity, bounded load", (s, i, work) -> {
            long mean = Arrays.stream(work).sum() / REPLICAS;
            int home = s % REPLICAS;
            return work[home] <= 1.25 * mean ? home : leastLoaded(work);
        });
    }
}

round-robin                  computed  72800  reused  95200  hit rate  56.7%  work per replica [28400, 10400, 24200, 9800]
least prefill work           computed 104000  reused  64000  hit rate  38.1%  work per replica [25400, 24800, 30800, 23000]
session affinity             computed  44000  reused 124000  hit rate  73.8%  work per replica [26000, 8000, 5000, 5000]
affinity, bounded load       computed  71000  reused  97000  hit rate  57.7%  work per replica [21200, 17600, 15200, 17000]

The four policies trade cache hits against balance:

• Round-robin reuses 56.7% of prompt tokens. Requests rotate, so a session returns to a replica that saw it two turns earlier and recomputes the newer part. Replicas 0 and 2 receive the heavy sessions more often: 28,400 and 24,200 tokens against about 10,000.
• Least prefill work balances the computed tokens but ignores the cache. Sessions keep moving, so the hit rate falls to 38.1%, the total work rises from 72,800 to 104,000 tokens, and the busiest replica does more work (30,800) than under any other policy. Balancing without cache awareness increases the work it is trying to balance.
• Session affinity sends session s to replica s mod 4. Every turn after the first finds its whole previous prompt, the hit rate reaches 73.8%, and total work falls to 44,000. Both heavy sessions map to replica 0, which does 26,000 tokens while replicas 2 and 3 do 5,000 each.
• Affinity with a load bound uses the session's replica unless its work exceeds 1.25 times the mean, and then uses the least-loaded replica. The busiest replica drops to 21,200 tokens, the lowest maximum of the four, at a hit rate of 57.7%.

The general form is called consistent hashing with bounded loads: keep each key on its home server unless that server is overloaded. The SGLang project (another LLM engine) ships a separate router, sglang_router, with a --policy cache_aware option. SGLang 0.5.21's multi-node test scripts start it that way, but the router's own source is not part of the pinned package. The engine landscape lesson described SGLang's cache-aware scheduling inside one engine.

For agents, whose prompts grow by appending to one history, affinity is the difference between recomputing a 30,000-token context each turn and computing only the new part.

Example 17: cold-start arithmetic

Adding a replica is not instant. The new replica must start a container and a process, load the weights into GPU memory, and warm up the engine. vLLM captures CUDA graphs (recorded GPU work that can be replayed faster) and may compile the model at startup. llama-server warms up "with an empty run" unless --no-warmup is given. The time from the decision to add a replica to its first served request is the cold start.

The program adds an assumed 20 s for startup and 40 s for warm-up to the weight-loading time from four sources, for an 8B BF16 model and a 70B FP8 model. It then computes how many requests pile up while a replica starts.

import java.util.Locale;

public class ColdStart {
    public static void main(String[] args) {
        String[] sources = {"Hub download, 1 Gb/s", "network volume, 1 GB/s", "local NVMe, 3 GB/s", "host page cache, 10 GB/s"};
        double[] bytesPerSecond = {0.125e9, 1e9, 3e9, 10e9};          // assumed rates; measure your own
        double start = 20, warmup = 40;                               // assumed: container and process start, engine warm-up (s)
        String[] models = {"8B BF16", "70B FP8"};
        double[] sizes = {16.1e9, 70.6e9};
        System.out.println("source                      model      load s  total cold start s");
        for (int m = 0; m < models.length; m++)
            for (int i = 0; i < sources.length; i++) {
                double load = sizes[m] / bytesPerSecond[i];
                System.out.printf(Locale.ROOT, "%-26s  %-8s  %7.1f  %18.1f%n", sources[i], models[m], load, start + load + warmup);
            }
        double rate = 5, perReplica = 2;                              // requests per second arriving and served per replica
        for (double cold : new double[] {70, 600}) {
            double extra = (rate - perReplica) * cold;                // a load step from 2 to 5 req/s with one replica ready
            System.out.printf(Locale.ROOT, "cold start %3.0f s: %5.0f requests pile up before new capacity arrives%n", cold, extra);
        }
    }
}

source                      model      load s  total cold start s
Hub download, 1 Gb/s        8B BF16     128.8               188.8
network volume, 1 GB/s      8B BF16      16.1                76.1
local NVMe, 3 GB/s          8B BF16       5.4                65.4
host page cache, 10 GB/s    8B BF16       1.6                61.6
Hub download, 1 Gb/s        70B FP8     564.8               624.8
network volume, 1 GB/s      70B FP8      70.6               130.6
local NVMe, 3 GB/s          70B FP8      23.5                83.5
host page cache, 10 GB/s    70B FP8       7.1                67.1
cold start  70 s:   210 requests pile up before new capacity arrives
cold start 600 s:  1800 requests pile up before new capacity arrives

All rates in the program are assumptions to show the arithmetic. Real values are measured on the target system. Weight loading dominates when the source is slow. For example, 16.1 GB at 0.125 GB/s (1 Gb/s) takes 128.8 s. Downloading the 8B model at 1 Gb/s takes 128.8 s and the 70B FP8 model 564.8 s, so a replica that downloads on start is ready after 3 to 10 minutes. From a local NVMe drive the same loads take 5.4 and 23.5 s. From the operating system's page cache (weights read recently on the same host) they take 1.6 and 7.1 s, which leaves the fixed 60 s of startup and warm-up as the largest part.

The last two lines turn the delay into a backlog. Suppose traffic jumps from 2 to 5 requests per second while one replica serves 2. The extra 3 requests per second accumulate until help arrives: 3 x 70 = 210 requests with a 70 s cold start, and 3 x 600 = 1,800 with a 600 s cold start. These requests wait in a queue, and with a timeout they fail. Common ways to shorten cold starts are keeping weights on local disk or in host memory, keeping a warm spare replica, and scaling on a signal that moves before the queue grows.

Example 18: autoscaling signals in a simulation

An autoscaler changes the number of replicas based on metrics. The usual signals are:

• the request rate;
• waiting requests (vllm:num_requests_waiting, llamacpp:requests_deferred);
• running requests;
• KV cache usage (vllm:kv_cache_usage_perc).

GPU utilization is a weak signal for LLM servers. NVIDIA defines the nvidia-smi utilization as the share of time in which at least one kernel runs, so a single decoding request already reports a high value.

The program looks at the situation every 30 s and uses two signals: the arrival rate over the last window, and the waiting requests, which it plans to drain within 60 s. It targets 80% of each replica's capacity of 2 requests per second. Traffic rises from 2 to 8 requests per second between 120 s and 240 s. The same scaler runs with cold starts of 70 s and 190 s. The simulation is a fluid model: requests are counted as fractions per second instead of one by one. Watch the ready, starting, and waiting values.

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class Autoscale {
    static double rate(int t) {                                       // requests per second: 2, ramping to 8 between t=120 and t=240
        return t < 120 ? 2 : t < 240 ? 2 + 6.0 * (t - 120) / 120 : 8;
    }

    static void simulate(int coldStart) {
        double capacity = 2, queue = 0, peakQueue = 0, peakWait = 0, arrivedInWindow = 0;
        List<Integer> readyAt = new ArrayList<>(List.of(0));         // one replica is ready at t=0
        System.out.println("cold start " + coldStart + " s");
        for (int t = 0; t < 600; t++) {
            int now = t, ready = (int) readyAt.stream().filter(r -> r <= now).count();
            queue = Math.max(0, queue + rate(t) - ready * capacity);  // fluid model: arrivals minus what ready replicas serve
            arrivedInWindow += rate(t);
            peakQueue = Math.max(peakQueue, queue);
            peakWait = Math.max(peakWait, queue / (ready * capacity));
            if (t % 30 == 29) {                                       // the autoscaler looks every 30 s
                double observed = arrivedInWindow / 30;               // signal 1: arrival rate; signal 2: waiting requests
                int desired = (int) Math.ceil((observed + queue / 60) / (0.8 * capacity));
                while (readyAt.size() < desired) readyAt.add(t + coldStart);
                arrivedInWindow = 0;
            }
            if (t % 60 == 59) {
                System.out.printf(Locale.ROOT, "  t=%3d s  rate %.1f/s  ready %d  starting %d  waiting %4.0f%n", t + 1, rate(t), ready,
                        readyAt.stream().filter(r -> r > now).count(), queue);
            }
        }
        System.out.printf(Locale.ROOT, "  peak waiting %.0f requests, peak queueing delay %.0f s%n", peakQueue, peakWait);
    }

    public static void main(String[] args) {
        simulate(70);
        simulate(190);
    }
}

cold start 70 s
  t= 60 s  rate 2.0/s  ready 1  starting 1  waiting    0
  t=120 s  rate 2.0/s  ready 2  starting 0  waiting    0
  t=180 s  rate 5.0/s  ready 2  starting 1  waiting   10
  t=240 s  rate 8.0/s  ready 2  starting 5  waiting  158
  t=300 s  rate 8.0/s  ready 5  starting 3  waiting  212
  t=360 s  rate 8.0/s  ready 8  starting 0  waiting    0
  t=420 s  rate 8.0/s  ready 8  starting 0  waiting    0
  t=480 s  rate 8.0/s  ready 8  starting 0  waiting    0
  t=540 s  rate 8.0/s  ready 8  starting 0  waiting    0
  t=600 s  rate 8.0/s  ready 8  starting 0  waiting    0
  peak waiting 254 requests, peak queueing delay 49 s
cold start 190 s
  t= 60 s  rate 2.0/s  ready 1  starting 1  waiting    0
  t=120 s  rate 2.0/s  ready 1  starting 1  waiting    0
  t=180 s  rate 5.0/s  ready 1  starting 3  waiting   89
  t=240 s  rate 8.0/s  ready 2  starting 6  waiting  315
  t=300 s  rate 8.0/s  ready 2  starting 9  waiting  555
  t=360 s  rate 8.0/s  ready 2  starting 12  waiting  795
  t=420 s  rate 8.0/s  ready 6  starting 8  waiting  747
  t=480 s  rate 8.0/s  ready 10  starting 4  waiting  219
  t=540 s  rate 8.0/s  ready 13  starting 1  waiting    0
  t=600 s  rate 8.0/s  ready 14  starting 0  waiting    0
  peak waiting 831 requests, peak queueing delay 208 s

With a 70 s cold start, the scaler starts a second replica in the first window, because 2 requests per second is above 80% of the capacity of one replica (1.6 requests per second). In every window it computes the desired count as ceil((observed rate + waiting / 60) / 1.6). When the ramp begins, each window asks for more replicas, but each takes 70 s to arrive. At 240 s two replicas are ready and five are starting, and 158 requests are waiting. By 360 s eight replicas are ready and the queue is empty. The peak is 254 waiting requests, about 49 s of queueing.

With a 190 s cold start, the same scaler does much worse. The queue reaches 831 requests and 208 s of delay. The queue keeps growing while replicas start, so the queue term keeps adding replicas while the earlier ones are still starting, and the deployment overshoots to 14 replicas where 5 would carry 8 requests per second at 80% (8 / 1.6 = 5). Long cold starts make an autoscaler both late and oversized. Production autoscalers add a stabilization window, count replicas that are starting as capacity in progress, and scale down slowly. The simulation leaves those out to show the effect of the delay alone.

Interview points

• Split a model across GPUs when it does not fit together with its KV cache, or when the bandwidth of one GPU limits the token rate. Add replicas instead when the model fits and only the total throughput is too low.
• Tensor parallelism splits the first MLP matrix by columns and the second by rows, so each layer needs exactly two all-reduces (one after attention and one after the MLP) and no other communication.
• A ring all-reduce sends 2(n-1)/n times the tensor size per device in 2(n-1) sequential steps. The bandwidth cost stays flat as n grows, while the latency grows with n.
• Decode all-reduces are small and limited by latency. Prefill all-reduces are large and limited by bandwidth. Over PCIe, tensor parallelism costs more time than the computation it splits.
• Pipeline parallelism wastes a share (p-1)/(m+p-1) of the time as a bubble, for p stages and m micro-batches. In serving it needs p batches in flight to keep the GPUs busy, and it does not speed up a single sequence.
• Expert parallelism moves tokens with an all-to-all in every MoE layer. The slowest device sets the layer time, so popular experts need balancing or replication.
• Data-parallel replicas need a router. Least-outstanding-requests beats round-robin when request lengths vary, and prefix affinity saves prefill work for conversations and agents.
• llama.cpp defaults to -sm layer, which is pipelined and gives no speed-up for one request. The option -ts sets proportions, the default proportions are free memory, and the input layer stays on the CPU.
• The llama.cpp RPC backend is experimental and has no authentication. It binds to 127.0.0.1 by default and warns on any other host.
• The vLLM world size is tp x pp per engine and x dp overall. MoE layers group tp x dp GPUs, split per expert without -ep and by whole experts with it.
• vLLM multi-LoRA serving batches different adapters together and reserves GPU memory for max_loras x max_lora_rank. llama-server does not batch requests with different LoRA settings.
• Cold start (weight loading plus warm-up) decides how fast a fleet can react. Long cold starts cause both large queues and over-provisioning.

Summary

• A 70B model in BF16 needs 141.2 GB of weights. With 32 users at 8K context the KV cache adds 85.9 GB, so 4 GPUs of 80 GB are the minimum.
• A column split gives each device whole output columns. A row split gives each device a partial sum that an all-reduce completes.
• Megatron-style tensor parallelism performs two all-reduces per layer per forward pass, each over tokens x hidden values.
• Ring all-reduce traffic per device is 2(n-1)/n times the tensor size. Four devices with 8 values each send 12 values per device.
• For a 70B model at TP 8, a 64-sequence decode step moves 293.6 MB per GPU in all-reduces. This takes 0.65 ms on NVLink and 9.32 ms on PCIe 4.0.
• A pipeline with p stages and m micro-batches is busy m/(m+p-1) of the time. vLLM keeps pp batches in flight (max_concurrent_batches).
• In expert parallelism the busiest device sets the layer time. Replicating popular experts (vLLM --enable-eplb, num_redundant_experts) reduces the imbalance.
• The built-in data-parallel balancer of vLLM chooses the engine with the fewest waiting plus running requests, with a penalty for KV cache pressure.
• The llama.cpp split modes are none, layer (default), row, and tensor (experimental). The option -ts sets proportions and -mg sets the main GPU.
• The llama.cpp layer split places layer il on the first device whose normalized running proportion exceeds il / gpu_layers.
• ggml-rpc-server exposes the devices of a machine over TCP on port 50052 by default, and RPC devices come first in the device list.
• The vLLM flags are -tp, -pp, -dp, and -ep, plus --nnodes, --node-rank, and --master-addr for multi-node runs.
• A rank-16 LoRA adapter for an 8B model is 83.9 MB in BF16. Twenty variants need 17.7 GB as adapters and 321.2 GB as merged models.
• Session affinity raised the prefix hit rate from 56.7% (round-robin) to 73.8% but put both heavy sessions on one replica. A load bound gave the lowest maximum work (21,200 tokens) at a 57.7% hit rate.
• A cold start is startup plus weight loading plus warm-up. Loading 70.6 GB takes about 565 s at 1 Gb/s and about 24 s from a 3 GB/s local drive.