Lesson 17 of 73
FLOPs, Bandwidth, and the Roofline Model
The previous lesson, CPUs, GPUs, and Memory for Inference, described the parts of a machine: cores that do arithmetic and memory that holds the weights. This lesson turns those facts into a small model that predicts speed. The model needs only two ideas. A computer must do some amount of arithmetic, and it must move some amount of data. Whichever of the two takes longer decides the speed.
After this lesson you can estimate tokens per second for two different jobs on a given machine. The first job is processing the prompt (prefill). The second is generating the answer one token at a time (decode). You can also explain why generation is limited by memory while prompt processing is limited by arithmetic, and say when batching or a mixture-of-experts model changes the answer.
Here is the vocabulary the lesson builds, with the example that introduces each term.
| Idea | Definition | Example |
|---|---|---|
| FLOP | One floating-point addition or multiplication | 1 |
| Bytes moved | Bytes read from and written to memory by an operation | 2 |
| Arithmetic intensity | FLOPs per byte moved | 3, 4 |
| Peak compute | The most FLOPs per second a device can do | 5 |
| Machine balance | Peak FLOP/s divided by bandwidth | 5 |
| Roofline | Attainable FLOP/s = min(peak, intensity * bandwidth) | 6 |
| Memory bound, compute bound | Which of the two limits applies | 6 to 10 |
| Active parameters | Parameters used per token in a mixture-of-experts model | 11 |
| Offload time | GPU time plus CPU time per token | 12 |
The lesson reuses two earlier results. A matrix-vector product with rows * cols weights costs 2 * rows * cols FLOPs (The Math Behind Transformers, in Java). And a model works in two phases: prefill, which reads the whole prompt, and decode, which produces one new token per step (Generation, Prefill, Decode, and the KV Cache).
Example 1: FLOPs of a matrix-vector and a matrix-matrix product
The first quantity to count is the amount of arithmetic. A FLOP is one floating-point operation: one addition or one multiplication on decimal numbers. (Do not confuse FLOPs, a count, with FLOP/s, a rate: operations per second.)
Multiplying two matrices produces many outputs. Each output is a dot product of length k: multiply k pairs of numbers, then add the k products together. That is k multiplications and about k additions, so 2 FLOPs for each multiply-add pair. A product of an m x k matrix and a k x n matrix has m * n outputs, so it costs 2 * m * n * k FLOPs.
The program counts FLOPs with a loop for small shapes and compares the count with this formula. Then it applies the formula to a real layer size. Look at the first two lines, where the counted and formula values agree.
public class FlopCount {
// C[m x n] = A[m x k] * B[k x n], counting each multiply and each add
static long countFlops(int m, int n, int k) {
long flops = 0;
for (int i = 0; i < m; i++)
for (int j = 0; j < n; j++)
for (int p = 0; p < k; p++) flops += 2; // one multiply, one add
return flops;
}
public static void main(String[] args) {
System.out.println("matrix-vector 4x8 * 8x1: counted " + countFlops(1, 4, 8)
+ ", formula " + 2L * 1 * 4 * 8);
System.out.println("matrix-matrix 4x8 * 8x3: counted " + countFlops(3, 4, 8)
+ ", formula " + 2L * 3 * 4 * 8);
long hidden = 4096;
long perToken = 2 * hidden * hidden;
System.out.println("4096x4096 layer, 1 token: " + perToken + " FLOPs");
System.out.println("4096x4096 layer, 512 tokens: " + 512 * perToken + " FLOPs");
long params = 8_030_000_000L;
System.out.println("8B model, 1 token (2 FLOPs per parameter): " + 2 * params + " FLOPs");
}
}matrix-vector 4x8 * 8x1: counted 64, formula 64
matrix-matrix 4x8 * 8x3: counted 192, formula 192
4096x4096 layer, 1 token: 33554432 FLOPs
4096x4096 layer, 512 tokens: 17179869184 FLOPs
8B model, 1 token (2 FLOPs per parameter): 16060000000 FLOPsCheck the first line by hand. A 4 by 8 matrix times a vector of 8 numbers has 4 outputs, and each output needs 8 multiplications and 8 additions: 4 * 16 = 64. The second line has 3 vectors instead of 1, so 3 times as many outputs: 192.
The matrix-vector case is a matrix-matrix product with n or m equal to 1. In a model, the "vector" is the activation of one token. The activation is the list of numbers that represents that token at the current point in the model. So m is the number of tokens processed together.
One token through a 4096 by 4096 layer costs about 33.5 million FLOPs (2 * 4096 * 4096). A 512-token prompt costs 512 times as much, because every token multiplies the same weights.
Summed over all weight matrices, one token costs about 2 FLOPs per parameter. For an 8B model that is about 16 GFLOPs (16 billion FLOPs). Attention adds FLOPs that grow with context length, because each new token is scored against every cached key. For short contexts they are small next to the weight FLOPs. Example 8 adds them back.
Example 2: bytes moved by the same operations
The second quantity is the amount of data moved. Speed depends on FLOPs and on bytes moved: the bytes an operation reads from memory and writes back. For a product C = A * B, the minimum is reading A once, reading B once, and writing C once.
The program counts bytes for a decode-shaped product (m = 1) and a prefill-shaped product (m = 512). Weights and activations are both FP16, meaning 16-bit decimal numbers that take 2 bytes each. Look at the weight share at the end of each line: it says how much of the traffic is the weight matrix.
public class BytesMoved {
static long bytes(long m, long n, long k, int weightBytes, int actBytes) {
long weights = k * n * weightBytes; // B: the weight matrix
long input = m * k * actBytes; // A: activations for m tokens
long output = m * n * actBytes; // C: results
return weights + input + output;
}
public static void main(String[] args) {
long h = 4096;
for (long m : new long[] {1, 512}) {
long w = h * h * 2, total = bytes(m, h, h, 2, 2);
System.out.printf("m=%-4d weights %d B, activations in+out %d B, total %d B, weight share %.1f%%%n",
m, w, total - w, total, 100.0 * w / total);
}
}
}m=1 weights 33554432 B, activations in+out 16384 B, total 33570816 B, weight share 100.0%
m=512 weights 33554432 B, activations in+out 8388608 B, total 41943040 B, weight share 80.0%For one token, the weights are 99.95% of all bytes moved. The printout shows 100.0% because of rounding. The activation vector is 8 KiB (4096 numbers times 2 bytes) in plus 8 KiB out, which is the 16,384 bytes in the output. The weight matrix is 32 MiB, which is 4096 * 4096 * 2 bytes.
For 512 tokens, the activations grow to 8 MiB in and out. The weights are still read once, because one pass over a weight row serves all 512 tokens. Compare the two growth rates:
m = 1 -> m = 512
FLOPs: 33.5 million -> 17.2 billion (512 times more)
bytes: 33.6 million -> 41.9 million (1.25 times more)The FLOPs grew 512 times while the bytes grew only 1.25 times. That ratio is the subject of the next example.
Example 3: arithmetic intensity of decode
Divide the arithmetic by the data and you get a number that tells how much work each byte pays for. Arithmetic intensity is the number of FLOPs an operation performs per byte it moves. A low number means the operation does little work per byte and mostly waits for memory. A high number means the operation does a lot of work per byte.
An analogy: a cook who needs ten minutes of chopping for each trip to the pantry is limited by chopping, while a cook who needs a trip for every single carrot is limited by walking. The analogy stops being accurate in one way: a real machine can overlap walking and chopping, so the slower of the two sets the speed, not the sum.
The program computes intensity for decode (one token) at several weight sizes. FP32 is 32-bit numbers, and the 8-bit and 4-bit rows are quantized formats. Quantization means storing each weight in fewer bits to shrink the model. The "bpw" label means bits per weight. Look at the intensity column as the weights get smaller.
import java.util.Locale;
public class DecodeIntensity {
public static void main(String[] args) {
long h = 4096;
double flops = 2.0 * h * h; // one token
String[] names = {"FP32", "FP16", "8-bit (8.5 bpw)", "4-bit (4.5 bpw)"};
double[] bitsPerWeight = {32, 16, 8.5, 4.5};
for (int i = 0; i < names.length; i++) {
double bytes = h * h * bitsPerWeight[i] / 8 + 2 * h * 2; // weights + vector in and out
System.out.printf(Locale.ROOT, "%-16s %8.0f KiB intensity %.2f FLOP/byte%n",
names[i], bytes / 1024, flops / bytes);
}
}
}FP32 65552 KiB intensity 0.50 FLOP/byte
FP16 32784 KiB intensity 1.00 FLOP/byte
8-bit (8.5 bpw) 17424 KiB intensity 1.88 FLOP/byte
4-bit (4.5 bpw) 9232 KiB intensity 3.55 FLOP/byteEach weight takes part in exactly one multiply-add (2 FLOPs) per token. So decode intensity is about 2 / bytesPerWeight:
FP16: 2 FLOPs / 2 bytes = 1.0 FLOP per byte
4-bit: 2 FLOPs / 0.5625 B = 3.6 FLOPs per byte (4.5 bits = 0.5625 bytes)The sizes of 8.5 and 4.5 bits per weight are those of the Q8_0 and Q4_K formats, including their scales (the extra numbers that each small group of weights carries). They are built in lesson 21 and lesson 22.
These intensities are tiny. Example 5 shows that a GPU needs hundreds of FLOPs per byte to keep its compute units busy.
Example 4: intensity grows with batch size
Intensity can be raised, and the way to do it is to reuse each weight for more tokens. When m tokens share one pass over the weights, FLOPs grow with m and weight bytes do not. The tokens can be a prompt (prefill), or one token from each of m different sequences (batched decode, where a server answers several users at once).
The program prints intensity for FP16 weights as the batch grows from 1 to 4,096 tokens. Look at how the FLOP/byte column tracks the token count at first, and how it bends later.
import java.util.Locale;
public class BatchIntensity {
public static void main(String[] args) {
long h = 4096;
System.out.println("tokens FLOPs bytes FLOP/byte");
for (long m : new long[] {1, 2, 4, 8, 16, 64, 256, 1024, 4096}) {
double flops = 2.0 * m * h * h;
double bytes = h * h * 2.0 + 2.0 * m * h * 2; // FP16 weights + activations in and out
System.out.printf(Locale.ROOT, "%6d %.3e %.3e %9.1f%n", m, flops, bytes, flops / bytes);
}
}
}tokens FLOPs bytes FLOP/byte
1 3.355e+07 3.357e+07 1.0
2 6.711e+07 3.359e+07 2.0
4 1.342e+08 3.362e+07 4.0
8 2.684e+08 3.369e+07 8.0
16 5.369e+08 3.382e+07 15.9
64 2.147e+09 3.460e+07 62.1
256 8.590e+09 3.775e+07 227.6
1024 3.436e+10 5.033e+07 682.7
4096 1.374e+11 1.007e+08 1365.3For small batches, intensity equals the number of tokens: 8 tokens give 8 FLOPs per byte. The reason is the ratio from Example 2: the FLOPs multiply by m and the bytes stay almost the same.
For large batches, the activations become a noticeable share of the bytes. Intensity then grows more slowly, toward a limit of h / 2 FLOPs per byte for a square layer of width h. For h = 4096 that limit is 2,048, and the last row (1,365) is on its way there.
A 512-token prompt chunk sits in the hundreds. So prefill is high-intensity work, and single-sequence decode is the lowest-intensity work a model does.
Example 5: machine balance for example machines
Now the machine side. Peak compute is the most FLOPs per second a device can perform, using its widest units on ideal data. A machine also has a bandwidth: the bytes per second it can read from memory. Dividing one by the other gives the machine balance: peak compute divided by bandwidth, in FLOPs per byte.
Balance tells how much arithmetic an operation must do per byte before the arithmetic units, not memory, become the bottleneck. It is the intensity at which the compute and the memory finish at the same time. Below that intensity, the device waits for memory.
The program computes the balance for several machines. The CPU peak comes from a calculation shown in the first comment: 8 cores, 4 GHz, 2 FMA units per core, 8 float lanes, 2 FLOPs per FMA. Look at the balance column and compare it with the intensities of 1 to 4 from Example 3.
import java.util.Locale;
public class MachineBalance {
record Machine(String name, double peakTflops, double bandwidthGBs) {
double balance() { return peakTflops * 1e12 / (bandwidthGBs * 1e9); }
}
public static void main(String[] args) {
// 8 cores * 4 GHz * 2 FMA units * 8 float lanes * 2 FLOPs per FMA
double cpuPeak = 8 * 4.0e9 * 2 * 8 * 2 / 1e12;
Machine[] machines = {
new Machine("8-core desktop CPU, DDR5", cpuPeak, 89.6),
new Machine("NVIDIA RTX 4090 (FP16 tensor)", 165.2, 1008),
new Machine("NVIDIA A100 SXM 80GB (BF16)", 312, 2039),
new Machine("NVIDIA H100 SXM (BF16)", 989, 3350),
};
System.out.println("machine peak TFLOP/s GB/s balance FLOP/byte");
for (Machine m : machines) {
System.out.printf(Locale.ROOT, "%-32s %12.2f %6.0f %10.0f%n",
m.name(), m.peakTflops(), m.bandwidthGBs(), m.balance());
}
}
}machine peak TFLOP/s GB/s balance FLOP/byte
8-core desktop CPU, DDR5 1.02 90 11
NVIDIA RTX 4090 (FP16 tensor) 165.20 1008 164
NVIDIA A100 SXM 80GB (BF16) 312.00 2039 153
NVIDIA H100 SXM (BF16) 989.00 3350 295Work out the H100 row by hand: 989 TFLOP/s divided by 3,350 GB/s is 989e12 / 3.35e12, about 295 FLOPs per byte.
The GPU figures are NVIDIA's published dense tensor-core specifications:
- RTX 4090: 165.2 TFLOP/s FP16 with FP32 accumulation.
- A100 SXM 80 GB: 312 TFLOP/s BF16 and 2,039 GB/s.
- H100 SXM: 989 TFLOP/s BF16 and 3.35 TB/s.
NVIDIA also lists figures twice as high "with sparsity". Those apply only to weights with a special 2-of-4 zero pattern, so this lesson does not use them.
The CPU figure is a calculation for a CPU with two 256-bit FMA units per core. An FMA (fused multiply-add) computes a * b + c as one instruction and counts as 2 FLOPs.
The balance numbers explain Example 3. Decode at FP16 has intensity 1. On an H100, the compute could do 295 FLOPs in the time it takes to read one byte. So during decode the tensor cores are idle more than 99% of the time. Even a CPU, with a balance of about 11, is memory bound for decode.
Example 6: the roofline function
Now put the two limits into one formula. The roofline model says that an operation with intensity I on a machine with peak compute P and bandwidth B runs at min(P, I * B) FLOP/s.
There are two cases:
- If
I * Bis smaller thanP, memory cannot feed the arithmetic units fast enough. The operation is memory bound. Doubling the intensity doubles the speed. - If
I * Bis larger thanP, memory keeps up, and the arithmetic units are the limit. The operation is compute bound. Speed stays flat atP.
The point where the two cases meet is the machine balance from Example 5. Drawn as a graph, the speed rises along a slanted line and then flattens, which looks like a roof:
speed
| ____________ <- flat part: peak compute P (compute bound)
| /
| / <- slanted part: I * B (memory bound)
| /
|_________/_________________ intensity (FLOP/byte)
^ machine balanceThe program prints the attainable FLOP/s for a range of intensities on the CPU and the H100 from Example 5. The intensities include 1 (FP16 decode), 3.6 (4-bit decode), and 295 (the H100's balance). Look at the bound columns to see where each machine switches from memory to compute.
import java.util.Locale;
public class Roofline {
static double attainable(double intensity, double peakFlops, double bandwidth) {
return Math.min(peakFlops, intensity * bandwidth);
}
public static void main(String[] args) {
double cpuPeak = 1.024e12, cpuBw = 89.6e9;
double gpuPeak = 989e12, gpuBw = 3350e9;
System.out.println("FLOP/byte CPU GFLOP/s bound H100 TFLOP/s bound");
for (double i : new double[] {0.5, 1, 3.6, 11.4, 64, 295, 1024}) {
double c = attainable(i, cpuPeak, cpuBw), g = attainable(i, gpuPeak, gpuBw);
System.out.printf(Locale.ROOT, "%9.1f %12.1f %-8s %12.1f %s%n", i,
c / 1e9, c < cpuPeak ? "memory" : "compute",
g / 1e12, g < gpuPeak ? "memory" : "compute");
}
}
}FLOP/byte CPU GFLOP/s bound H100 TFLOP/s bound
0.5 44.8 memory 1.7 memory
1.0 89.6 memory 3.4 memory
3.6 322.6 memory 12.1 memory
11.4 1021.4 memory 38.2 memory
64.0 1024.0 compute 214.4 memory
295.0 1024.0 compute 988.3 memory
1024.0 1024.0 compute 989.0 computeCheck one cell by hand. At intensity 1 on the H100, I * B is 1 * 3,350 GB/s = 3.35 TFLOP/s, far below the 989 peak. So the result is 3.4 TFLOP/s and the operation is memory bound.
That is about 0.3% of the H100's peak. At 3.6 (4-bit decode) it attains 12 TFLOP/s. Quantization made decode about 3.6 times faster without any change in compute, only by moving fewer bytes.
The CPU reaches its peak at about 11 FLOPs per byte. The H100 needs about 295. At exactly 295 the H100 row still says "memory" because 295 is below the true balance of 295.2.
The roofline is an upper bound. Real kernels reach a fraction of it because of cache misses, synchronization between steps, and operations that are not matrix products (normalization, softmax, RoPE). Its value is in telling which resource to improve. For a memory-bound operation, fewer bytes help and more FLOP/s do not.
Example 7: decode speed from bandwidth and model bytes
Apply the roofline to generating text. If decode is memory bound, each token costs one full read of the model. So tokens per second is at most bandwidth divided by the bytes read per token, and the bytes read per token are the bytes of the model.
A hand case: a 7B model at FP16 is 7 billion * 2 bytes = 14 GB. On a machine with 90 GB/s, one token takes at least 14 / 90 = 0.156 seconds, which is at most 6.4 tokens per second.
The program does this for a 7B model at FP16 and at 4.5 bits per weight on six machines. Look at how each row changes between the two columns, and how the rows compare with one another.
import java.util.Locale;
public class DecodeEstimate {
public static void main(String[] args) {
double params = 7.0e9;
String[] machines = {"Reference CPU (measured)", "Desktop, DDR5 dual channel", "Apple M4 Pro",
"Apple M4 Max", "RTX 4090", "H100 SXM"};
double[] gbs = {12.8, 89.6, 273, 546, 1008, 3350};
double fp16GB = params * 2 / 1e9, q4GB = params * 4.5 / 8 / 1e9;
System.out.printf(Locale.ROOT, "model bytes: FP16 %.1f GB, 4.5 bpw %.2f GB%n", fp16GB, q4GB);
System.out.println("machine GB/s FP16 tok/s 4.5-bit tok/s");
for (int i = 0; i < machines.length; i++) {
System.out.printf(Locale.ROOT, "%-27s %5.0f %11.1f %14.1f%n",
machines[i], gbs[i], gbs[i] / fp16GB, gbs[i] / q4GB);
}
}
}model bytes: FP16 14.0 GB, 4.5 bpw 3.94 GB
machine GB/s FP16 tok/s 4.5-bit tok/s
Reference CPU (measured) 13 0.9 3.3
Desktop, DDR5 dual channel 90 6.4 22.8
Apple M4 Pro 273 19.5 69.3
Apple M4 Max 546 39.0 138.7
RTX 4090 1008 72.0 256.0
H100 SXM 3350 239.3 850.8The first row uses the 12.8 GB/s that four threads reached on the reference machine in lesson 16. The other rows use published peak bandwidth.
Quantizing from 16 to 4.5 bits per weight multiplies the estimate by 16 / 4.5 = 3.56 on every machine, because the estimate depends only on bytes. For example, 6.4 becomes 22.8 on the desktop.
This estimate predicts the order of magnitude well. The next example explains why measured numbers come in lower.
Example 8: KV cache reads as the context grows
Decode reads more than the weights. Each step also reads the KV cache (key-value cache): the keys and values that attention stored for every earlier token, in every layer (lesson 11). The longer the conversation, the larger this cache.
For a Llama 3.1 8B shaped model, the shape values come from its config.json: 32 layers, 8 KV heads, and head dimension 128. The cache holds 2 * 32 * 8 * 128 = 65,536 values per token. The leading 2 is for keys plus values. At FP16 (2 bytes per value) that is 128 KiB per token. Lesson 19 derives this formula.
The program adds the KV bytes to the weight bytes and recomputes the decode estimate at several context lengths on an RTX 4090. Look at the vs no-KV column: it shows how much speed is left compared with an empty context.
import java.util.Locale;
public class KvReads {
public static void main(String[] args) {
double weights = 8.03e9 * 4.5 / 8; // 4.5 bits per weight
long kvPerToken = 2L * 32 * 8 * 128 * 2; // K and V, 32 layers, 8 heads, dim 128, FP16
double bw = 1008e9; // RTX 4090
System.out.println("KV bytes per token = " + kvPerToken);
System.out.println("context KV read/token total read tok/s vs no-KV");
double base = bw / weights;
for (int ctx : new int[] {0, 1024, 4096, 16384, 32768, 131072}) {
double kv = (double) kvPerToken * ctx;
double total = weights + kv;
System.out.printf(Locale.ROOT, "%7d %10.2f GB %7.2f GB %6.1f %6.1f%%%n",
ctx, kv / 1e9, total / 1e9, bw / total, 100 * (bw / total) / base);
}
}
}KV bytes per token = 131072
context KV read/token total read tok/s vs no-KV
0 0.00 GB 4.52 GB 223.2 100.0%
1024 0.13 GB 4.65 GB 216.7 97.1%
4096 0.54 GB 5.05 GB 199.5 89.4%
16384 2.15 GB 6.66 GB 151.3 67.8%
32768 4.29 GB 8.81 GB 114.4 51.3%
131072 17.18 GB 21.70 GB 46.5 20.8%The first line, KV bytes per token = 131072, is 65,536 values times 2 bytes. At a context of 32,768 tokens, the cache is 131,072 * 32,768 bytes, which is 4.29 GB. The weights are 4.52 GB. So each step reads 8.81 GB instead of 4.52 GB, and the speed falls to 51% of the empty-context speed.
At 32K tokens of context, the KV cache read per token is almost as large as the 4-bit weights, so decode is about half as fast as with an empty context. At 128K tokens the cache (17.2 GB) and the weights (4.5 GB) together take 21.7 GB. That leaves almost nothing of a 24 GB GPU for compute buffers, and decode runs at about a fifth of its empty-context speed.
Coding agents fill long contexts with files and tool output, so this effect is visible in practice: tokens per second drops as a session grows.
The remaining gap between estimate and measurement comes from four sources:
- Kernels reach 60 to 90% of peak bandwidth, not 100%.
- Small operations (normalization, RoPE, softmax, sampling) each launch a kernel and move little data.
- Attention FLOPs grow with context and are done at low intensity.
- Layers must synchronize with each other, and on a CPU, threads finish at different times.
Example 9: prefill speed is compute bound
Prefill is the opposite case from decode. During prefill, all prompt tokens go through each weight matrix together, so intensity is in the hundreds (Example 4). That is above the balance of a CPU and close to or above that of a GPU, so the limit is compute.
The estimate is tokens/s = peak FLOP/s * efficiency / FLOPs per token. Efficiency is the fraction of peak that real code reaches. The program uses 50%, an example of a well-tuned kernel. An 8B model costs 2 FLOPs per parameter per token. A hand case for the RTX 4090: 165.2e12 * 0.5 / 16.06e9 is about 5,143 tokens per second.
The program also converts that into the time to process an 8,000-token prompt, which is a prompt holding several source files. Compare the prefill column with the decode column on each row.
import java.util.Locale;
public class PrefillEstimate {
public static void main(String[] args) {
double flopsPerToken = 2 * 8.03e9;
double efficiency = 0.5;
String[] names = {"8-core desktop CPU", "RTX 4090", "H100 SXM"};
double[] peak = {1.024e12, 165.2e12, 989e12};
double[] bw = {89.6e9, 1008e9, 3350e9};
double weightBytes = 8.03e9 * 4.5 / 8;
int prompt = 8000; // tokens: a prompt with several source files
System.out.println("machine prefill tok/s 8000-token prompt decode tok/s");
for (int i = 0; i < names.length; i++) {
double prefill = peak[i] * efficiency / flopsPerToken;
double decode = bw[i] / weightBytes;
System.out.printf(Locale.ROOT, "%-20s %13.0f %15.1f s %13.1f%n",
names[i], prefill, prompt / prefill, decode);
}
}
}machine prefill tok/s 8000-token prompt decode tok/s
8-core desktop CPU 32 250.9 s 19.8
RTX 4090 5143 1.6 s 223.2
H100 SXM 30791 0.3 s 741.7On the GPU, prefill is about 23 times faster than decode (5,143 against 223 tokens per second). Prefill uses the tensor cores, and decode waits for memory.
On the CPU, prefill and decode are of the same order, and an 8,000-token prompt takes minutes (250.9 seconds). This is the main reason a coding agent feels slow on a CPU-only machine even when decode speed is acceptable. Each turn resends a long prompt. The delay before the first output token, called time to first token (TTFT), is dominated by prefill.
Prompt caching removes most of that cost for repeated prompts. It reuses the KV cache of an unchanged prefix. The lesson on paged KV cache and prefix caching later in this course covers it.
Real CPU prefill in llama.cpp is often faster than this FP32 estimate, because quantized kernels use integer instructions (VNNI, AMX) with higher peak rates than float FMAs.
Example 10: batched decode and the crossover batch size
Example 4 showed that more tokens per weight read means higher intensity. A server can use this. It decodes one token for each of b sequences (b users) in one step. The weights are read once per step for all of them. So the step costs about the same time as for one sequence, until the batch reaches the balance point.
The program computes step time as the larger of memory time and compute time, on an RTX 4090 with a 4.5-bit 8B model. It ignores the KV cache. Look at the memory ms column staying constant and the compute ms column growing with the batch. The step ms column is the larger of the two.
import java.util.Locale;
public class BatchedDecode {
public static void main(String[] args) {
double params = 8.03e9, weightBytes = params * 4.5 / 8;
double peak = 165.2e12, bw = 1008e9;
System.out.println("batch memory ms compute ms step ms total tok/s per-seq tok/s");
for (int b : new int[] {1, 4, 16, 64, 128, 256, 512}) {
double memMs = weightBytes / bw * 1000;
double compMs = 2 * params * b / peak * 1000;
double step = Math.max(memMs, compMs);
System.out.printf(Locale.ROOT, "%5d %9.2f %10.2f %7.2f %12.0f %13.1f%n",
b, memMs, compMs, step, b * 1000 / step, 1000 / step);
}
double crossover = (peak / bw) * (4.5 / 8) / 2; // balance * bytesPerWeight / 2 FLOPs per weight
System.out.printf(Locale.ROOT, "crossover batch = balance * bytesPerWeight / 2 = %.1f%n", crossover);
}
}batch memory ms compute ms step ms total tok/s per-seq tok/s
1 4.48 0.10 4.48 223 223.2
4 4.48 0.39 4.48 893 223.2
16 4.48 1.56 4.48 3571 223.2
64 4.48 6.22 6.22 10286 160.7
128 4.48 12.44 12.44 10286 80.4
256 4.48 24.89 24.89 10286 40.2
512 4.48 49.77 49.77 10286 20.1
crossover batch = balance * bytesPerWeight / 2 = 46.1Follow the rows from the top:
- Batch 1 to 16: the memory time (4.48 ms) is larger than the compute time, so the step takes 4.48 ms. Total tokens per second grows in proportion to the batch (223, 893, 3,571), while each sequence still gets 223 tokens per second.
- Batch 64 and above: compute time (6.22 ms and more) is larger. The step is now compute bound. Total throughput stays at 10,286, and each sequence gets fewer tokens per second.
Up to a batch of about 46, every extra sequence is free. Past the crossover, throughput stops growing. The crossover batch is where 2 * params * b / peak = weightBytes / bw. Solving for b gives b = balance * bytesPerWeight / 2. Here the balance is 164 and a weight takes 0.5625 bytes, so b = 164 * 0.5625 / 2, about 46.
In practice the crossover moves in two directions:
- Each sequence reads its own KV cache, which batching does not share. This adds memory time and moves the crossover to larger batches.
- Kernels that reach less than peak FLOP/s add compute time and move it to smaller batches.
This is the arithmetic behind continuous batching in servers such as vLLM. The lesson on batching and scheduling later in this course covers it.
Example 11: MoE models, active and total parameters
So far, every token used every weight. A mixture-of-experts (MoE) model changes this. It replaces each MLP (the feed-forward part of a layer) with many smaller "expert" MLPs, and a router picks a few of them for each token (lesson 14).
Two numbers describe such a model:
- Total parameters are all the weights in the file. They decide how much memory the model needs.
- Active parameters are the ones used for one token. They decide how many bytes each token reads, and so the speed.
The program uses the published shape of Qwen3-30B-A3B to compute both:
- 48 layers, hidden size 2048.
- 32 query heads and 4 KV heads of dimension 128.
- 128 experts with 8 active per token, expert MLP size 768.
- Vocabulary of 151,936 tokens.
Look at the gap between total params and active params, and then between the two speed estimates at the end.
import java.util.Locale;
public class MoeActive {
public static void main(String[] args) {
long layers = 48, hidden = 2048, heads = 32, kvHeads = 4, headDim = 128;
long experts = 128, active = 8, expertFf = 768, vocab = 151_936;
long attn = hidden * heads * headDim * 2 + hidden * kvHeads * headDim * 2; // q, o + k, v
long oneExpert = 3 * hidden * expertFf; // gate, up, down
long router = hidden * experts;
long embed = vocab * hidden * 2; // input and output
double total = layers * (attn + experts * oneExpert + router) + embed;
double used = layers * (attn + active * oneExpert + router) + embed;
System.out.printf(Locale.ROOT, "total params = %.2f B%n", total / 1e9);
System.out.printf(Locale.ROOT, "active params = %.2f B%n", used / 1e9);
double bpw = 4.5, bw = 89.6e9;
System.out.printf(Locale.ROOT, "file size at 4.5 bpw = %.1f GB (sets memory)%n", total * bpw / 8 / 1e9);
System.out.printf(Locale.ROOT, "bytes read per token = %.2f GB (sets speed)%n", used * bpw / 8 / 1e9);
System.out.printf(Locale.ROOT, "desktop DDR5 decode bound = %.1f tok/s%n", bw / (used * bpw / 8));
System.out.printf(Locale.ROOT, "dense 30B on same machine = %.1f tok/s%n", bw / (total * bpw / 8));
}
}total params = 30.53 B
active params = 3.35 B
file size at 4.5 bpw = 17.2 GB (sets memory)
bytes read per token = 1.89 GB (sets speed)
desktop DDR5 decode bound = 47.5 tok/s
dense 30B on same machine = 5.2 tok/sThe totals match the model's name: about 30.5 billion parameters in all, and about 3.3 billion active ("A3B" means 3 billion active).
The whole 17.2 GB must sit in memory, because any token may pick any expert. But each token reads only 1.89 GB. So a desktop with DDR5 decodes it about 9 times faster than a dense 30B model of the same file size (47.5 against 5.2 tokens per second). This is why MoE models are popular for local use on machines with much RAM and modest bandwidth.
Two cautions apply to this estimate:
- The embedding row read per token is small. Only one row of the input embedding is read, not the whole table. So the estimate overcounts slightly.
- In a batch, different tokens pick different experts. Batched MoE decode therefore reads more experts per step than single-sequence decode.
Example 12: partial offload, GPU time plus CPU time
A model can be larger than the GPU's memory. In that case llama.cpp can put some layers on the GPU and keep the rest in system RAM, where the CPU computes them. Each token passes through all layers in order. So the time per token is the GPU part plus the CPU part, not the larger of the two.
The program estimates a 70B model at 4.5 bits per weight (39.4 GB) on an RTX 4090 with 22 GB available for weights. It tries four splits. Look at the GPU ms and CPU ms columns, and see which one dominates.
import java.util.Locale;
public class OffloadTime {
public static void main(String[] args) {
double modelGB = 70e9 * 4.5 / 8 / 1e9;
double gpuBw = 1008, cpuBw = 89.6; // GB/s
System.out.printf(Locale.ROOT, "model = %.1f GB%n", modelGB);
System.out.println("on GPU on CPU GPU ms CPU ms tok/s");
for (double onGpu : new double[] {0, 10, 22, modelGB}) {
double onCpu = modelGB - onGpu;
double gpuMs = onGpu / gpuBw * 1000, cpuMs = onCpu / cpuBw * 1000;
System.out.printf(Locale.ROOT, "%6.1f %7.1f %7.1f %7.1f %6.2f%n",
onGpu, onCpu, gpuMs, cpuMs, 1000 / (gpuMs + cpuMs));
}
}
}model = 39.4 GB
on GPU on CPU GPU ms CPU ms tok/s
0.0 39.4 0.0 439.5 2.28
10.0 29.4 9.9 327.8 2.96
22.0 17.4 21.8 193.9 4.64
39.4 0.0 39.1 0.0 25.60Check the 22 GB row by hand. The GPU reads 22 GB at 1,008 GB/s, which takes 21.8 ms. The CPU reads the remaining 17.4 GB at 89.6 GB/s, which takes 193.9 ms. The sum is 215.7 ms per token, or 4.64 tokens per second.
That is twice the CPU-only speed (2.28) but far from the 25.6 tokens per second of a GPU that holds everything. The slow device dominates. Moving more layers to the GPU helps in proportion to how much it shortens the CPU part.
Lesson 19 computes how many layers fit. It also shows how to keep MoE experts on the CPU while attention stays on the GPU, which uses this same sum. The llama.cpp flags for it come in the lesson on tuning llama.cpp later in this course.
Example 13: speculative decoding preview
Example 10 showed that checking several tokens in one step costs nothing extra while decode is memory bound. Speculative decoding uses this fact. A small, fast draft model guesses the next k tokens, and then the large model checks all k guesses in one forward pass. The lesson on speculative decoding later in this course covers the method in full.
The check processes k + 1 positions together, like a small batch, so it reads the weights once. The program computes the cost of that check on an RTX 4090 for an 8B model. Look at the multiple of one step in each row.
import java.util.Locale;
public class SpeculativeCost {
public static void main(String[] args) {
double params = 8.03e9, weightBytes = params * 4.5 / 8;
double peak = 165.2e12, bw = 1008e9;
double oneToken = Math.max(weightBytes / bw, 2 * params / peak) * 1000;
System.out.printf(Locale.ROOT, "normal decode step = %.2f ms%n", oneToken);
for (int k : new int[] {1, 4, 8, 16}) {
double verify = Math.max(weightBytes / bw, 2 * params * (k + 1) / peak) * 1000;
System.out.printf(Locale.ROOT, "verify %2d drafts: %.2f ms (%.2fx one step), up to %d tokens%n",
k, verify, verify / oneToken, k + 1);
}
}
}normal decode step = 4.48 ms
verify 1 drafts: 4.48 ms (1.00x one step), up to 2 tokens
verify 4 drafts: 4.48 ms (1.00x one step), up to 5 tokens
verify 8 drafts: 4.48 ms (1.00x one step), up to 9 tokens
verify 16 drafts: 4.48 ms (1.00x one step), up to 17 tokensChecking 16 draft tokens costs the same 4.48 ms as generating one token. The step is memory bound, so the extra FLOPs fit in time the compute units would otherwise spend idle. If all drafts are accepted, one step yields up to 17 tokens (16 drafts plus one more from the large model).
The real speedup depends on how many drafts the large model accepts, and on the cost of running the draft model. The speculative decoding lesson computes both.
Example 14: a roofline estimator for one model on several machines
The previous examples combine into one small class. RooflineEstimator takes a model (parameters, bits per weight, KV bytes per token) and a machine (peak FLOP/s, bandwidth). It prints prefill speed, decode speed at a given context, and the time for a typical coding-agent turn: an 8,000-token prompt followed by 500 output tokens.
The method stepSeconds is the core. It adds the weight bytes and the KV bytes for the context, computes memory time and compute time, and returns the larger one. That is the roofline in three lines. Look at the turn seconds column in the output, which adds prefill time and decode time.
import java.util.Locale;
public class RooflineEstimator {
record Model(String name, double params, double bitsPerWeight, long kvBytesPerToken) {
double weightBytes() { return params * bitsPerWeight / 8; }
}
record Machine(String name, double peakFlops, double bandwidth) {}
static double stepSeconds(Model m, Machine h, int tokens, int context) {
double bytes = m.weightBytes() + (double) m.kvBytesPerToken() * context;
double flops = 2 * m.params() * tokens;
return Math.max(bytes / h.bandwidth(), flops / h.peakFlops());
}
public static void main(String[] args) {
Model m = new Model("8B, 4.5 bpw", 8.03e9, 4.5, 131_072);
Machine[] machines = {
new Machine("Desktop CPU, DDR5", 1.024e12, 89.6e9),
new Machine("Apple M4 Max", 30e12, 546e9),
new Machine("RTX 4090", 165.2e12, 1008e9),
new Machine("H100 SXM", 989e12, 3350e9),
};
int prompt = 8000, output = 500, ctx = 8000;
double eff = 0.5;
System.out.println("model: " + m.name() + ", prompt " + prompt + ", output " + output);
System.out.println("machine prefill tok/s decode tok/s turn seconds");
for (Machine h : machines) {
Machine real = new Machine(h.name(), h.peakFlops() * eff, h.bandwidth() * 0.7);
double prefillTps = prompt / stepSeconds(m, real, prompt, 0);
double decodeTps = 1 / stepSeconds(m, real, 1, ctx);
double turn = prompt / prefillTps + output / decodeTps;
System.out.printf(Locale.ROOT, "%-19s %13.0f %13.1f %13.1f%n", h.name(), prefillTps, decodeTps, turn);
}
}
}model: 8B, 4.5 bpw, prompt 8000, output 500
machine prefill tok/s decode tok/s turn seconds
Desktop CPU, DDR5 32 11.3 295.3
Apple M4 Max 934 68.7 15.8
RTX 4090 5143 126.8 5.5
H100 SXM 30791 421.3 1.4The estimator applies 50% efficiency to compute and 70% to bandwidth. These are two assumptions that you can adjust after a measurement. The Apple M4 Max line uses 30 TFLOP/s as an illustrative figure. Apple does not publish a tensor FLOP/s number in the same form as NVIDIA, so this row is a rough example, not a specification.
The turn time shows the trade-off for a coding agent:
- On the CPU, the 8,000-token prompt dominates: 251 of 295 seconds.
- On the Mac, prefill takes 8.6 seconds and decode 7.3 seconds.
- On the GPUs, both parts are short.
Measuring prefill and decode separately on a real machine gives the two efficiency factors that make this estimator match that machine. A tool such as llama-bench does the measuring. The lesson on benchmarking inference later in this course covers it.
Interview points
- A matrix product of
m x kbyk x ncosts2 * m * n * kFLOPs, and a forward pass costs about 2 FLOPs per parameter per token. - Arithmetic intensity is FLOPs per byte moved. Decode at batch 1 has an intensity of about
2 / bytesPerWeight, which is 1 at FP16. - Machine balance is peak FLOP/s divided by bandwidth: about 11 for an 8-core desktop CPU, about 164 for an RTX 4090, and about 295 for an H100.
- The roofline says attainable FLOP/s = min(peak, intensity * bandwidth). Decode is memory bound on every common device, and prefill is compute bound.
- Decode tokens per second is at most bandwidth / (weight bytes + KV bytes read per token). KV reads grow linearly with context.
- Quantization speeds up decode by reducing the bytes read, without needing more compute.
- Batched decode is nearly free up to a crossover batch of about
balance * bytesPerWeight / 2. After that, total throughput stops growing. - MoE models need memory for their total parameters, but each token reads only the active parameters.
- With partial offload, time per token is GPU time plus CPU time, and the slower CPU part dominates.
- Verifying
kspeculative tokens costs about one decode step while decode is memory bound.
Summary
- One token through a 4096 by 4096 layer costs 33.5 MFLOPs, and an 8B model costs about 16 GFLOPs per token.
- For one token, weights are almost all of the bytes moved. For a 512-token batch, the same weights serve every token.
- Decode intensity is 0.5 FLOP/byte at FP32, 1.0 at FP16, 1.88 at 8.5 bits, and 3.55 at 4.5 bits per weight.
- For small batches, intensity equals the number of tokens in the batch.
- Published dense peaks: RTX 4090 165.2 TFLOP/s FP16, A100 SXM 312 TFLOP/s BF16, H100 SXM 989 TFLOP/s BF16.
- On an H100, FP16 decode attains about 0.3% of peak compute.
- A 7B model at 4.5 bits per weight is about 3.9 GB. Its decode bound is about 23 tokens/s on dual-channel DDR5 and 256 on an RTX 4090.
- An 8B model at 32K context reads about 4.3 GB of KV cache per token at FP16, which halves decode speed on a 4090.
- Prefill on a GPU is tens of times faster than decode. On a CPU, long prompts take minutes and dominate time to first token.
- Qwen3-30B-A3B has about 30.5B total and 3.35B active parameters.
- Partial offload of a 70B model with 22 GB on an RTX 4090 gives an estimate of about 4.6 tokens/s, against 2.3 on the CPU alone.
- Estimates are upper bounds. Efficiency factors from a measurement make them match a real machine.