Finding the Bottleneck: Compute or Memory

A GPU has two limited resources. It can do only so many arithmetic operations per second, and it can move only so many bytes per second from memory. Any program uses both, and one of them runs out first. The one that runs out first is the bottleneck, and improving the other resource gains nothing. This lesson teaches a calculation that tells which resource limits a given step of inference, and applies it to LLM prefill, LLM decode, and image and video generation.

Afterwards you can compute a GPU's ops:byte ratio, compute the arithmetic intensity of an operation, place the operation on a roofline, and say why LLM decode is memory-bound while prefill and image generation are compute-bound. You can also explain why batching helps decode.

The lesson uses attention as its example and the prefill and decode phases from How an LLM Generates Text. All calculations are done by hand in text blocks.

Example 1: two resources and what a bottleneck is

In a perfectly optimized system every resource is fully used all the time. A GPU has two main resources:

Ideally compute never idles while waiting for data, and bandwidth is never unused while waiting for compute to finish. Real systems have bottlenecks: imbalances in which one resource sits idle while the other is saturated. Finding them is the first step in improving performance. If an operation is limited by memory bandwidth, no amount of compute optimization makes it faster, and the reverse is also true.

An analogy: a kitchen with a fast cook and a slow delivery of ingredients. The cook stands idle, and hiring a faster cook changes nothing. The analogy stops at one point: a GPU can often change how much work it does per byte it fetches (for example by batching), which a kitchen cannot do.

In most inference systems the bottlenecks are:

The aim of optimization for each workload is to make the bottleneck limit the whole system less. For example, batching several requests together makes LLM decode less memory-bound, because a batch of requests does more arithmetic for the same memory traffic.

flowchart LR
    W["Operation"] --> Q{"Arithmetic intensity above the GPU's ops:byte ratio?"}
    Q -->|"yes"| C["Compute-bound: arithmetic is the limit"]
    Q -->|"no"| M["Memory-bound: memory traffic is the limit"]

Example 2: the ops:byte ratio of a GPU

Every GPU has a compute speed (operations per second) and a memory bandwidth (gigabytes or terabytes per second). Dividing one by the other gives its ops:byte ratio: how many operations it can do in the time it takes to move one byte.

Take the H100 GPU from NVIDIA with FP16 numbers (FP16 is a 16-bit floating-point format, two bytes per value). It performs 989 teraFLOPS of dense computation (989 x 10^12 operations per second) against 3.35 TB/s of memory bandwidth (3.35 x 10^12 bytes per second).

ops:byte = compute / bandwidth
         = 989 x 10^12 ops/s  /  3.35 x 10^12 bytes/s
         = 295.2 ops per byte    (about 295)

In words: for FP16 inference on an H100 to be perfectly balanced, the work must do about 295 operations for every byte it reads or writes in memory. Less than that means the arithmetic units wait for data. More than that means the memory has spare capacity.

Example 3: arithmetic intensity and the roofline

To compare a piece of work against the hardware ratio, compute the arithmetic intensity of the work, also called operational intensity: the ratio of work to memory traffic for the calculation at hand.

arithmetic intensity = total operations / total bytes moved

The ops:byte ratio is a property of the GPU measured per second. Arithmetic intensity is a property of one function or algorithm, measured over one run of it.

A roofline model draws this on a chart. The x axis is arithmetic intensity, and the y axis is achieved performance. Two lines limit performance: a diagonal line for the bandwidth ceiling and a flat horizontal line for the compute ceiling. They meet at the GPU's ops:byte ratio.

flowchart LR
    A["Low intensity: below the ratio"] -->|"performance = intensity x bandwidth (diagonal ceiling)"| B["Memory-bound"]
    C["High intensity: above the ratio"] -->|"performance = peak compute (flat ceiling)"| D["Compute-bound"]

The rule of the chart:

• Compute-bound: intensity above the hardware's ops:byte ratio. The work reaches the horizontal ceiling.
• Memory-bound: intensity below the hardware's ops:byte ratio. The work reaches the diagonal ceiling.

The achievable speed is the smaller of two numbers: peak compute, and intensity times bandwidth. For an operation with intensity 62 on the H100:

bandwidth ceiling = 62 ops/byte x 3.35 x 10^12 bytes/s = 208 x 10^12 = 208 teraFLOPS
compute ceiling   = 989 teraFLOPS
achievable        = min(208, 989) = 208 teraFLOPS   (21 percent of peak)

To find a bottleneck, look at the arithmetic intensity of the most expensive calculations in the system. In inference, one of them is attention.

Example 4: why prefill is compute-bound and decode is memory-bound

LLM inference has two phases:

For each phase the bottleneck can be shown by comparing the arithmetic intensity of the most important operation with the hardware's ops:byte ratio. In both phases the most expensive operation is attention, and its exact intensity depends on the model architecture (dimensions and heads), the input length, and the implementation of attention.

The key difference is that prefill processes the whole input sequence in parallel, while decode produces tokens one at a time.

• Prefill: the weights are loaded once, and then many large matrix-by-matrix multiplications between the input matrix and the weight matrices follow. That is a large amount of arithmetic for one read from memory, so arithmetic intensity is high.
• Decode: the weights are loaded again for every token, and each token needs only a much cheaper vector-by-matrix multiplication. Few operations are done for the amount of data loaded, so arithmetic intensity is low.

A simplified count shows the difference. Take one weight matrix of size d x d in FP16, with d = 4,096. It is read from memory once per pass, and moving it costs 2 x d x d bytes.

weight bytes = 2 x 4,096 x 4,096 = 33,554,432 bytes

Decode, 1 token (vector x matrix):
  operations = 2 x d x d = 33,554,432   (one multiply and one add per weight)
  intensity  = 33,554,432 / 33,554,432 = 1 op per byte

Prefill, 1,000 prompt tokens (matrix x matrix):
  operations = 2 x 1,000 x d x d = 33,554,432,000
  intensity  = 33,554,432,000 / 33,554,432 = 1,000 ops per byte

H100 ops:byte = 295
decode  (1)     << 295   -> memory-bound
prefill (1,000)  > 295   -> compute-bound

This count ignores the traffic for activations and the KV cache, which only makes the result for decode worse. It shows the main point: the weights are read once per pass whatever the number of tokens, so the more tokens share a pass, the higher the intensity.

That is also why batching helps decode. A batch of B requests shares one read of the weights, and the intensity of the matrix part is about B. On the H100 the weights part of decode reaches balance at a batch of about 295 tokens per pass, and below that it stays memory-bound.

flowchart TB
    subgraph PF["Prefill"]
        P1["1,000 prompt tokens"] --> P2["Weights read once, large matrix x matrix: intensity about 1,000"]
    end
    subgraph DC["Decode, one request"]
        D1["1 token"] --> D2["Weights read for every token, vector x matrix: intensity about 1"]
    end

Example 5: the full attention calculation, 62 against 295

The next calculation computes the exact intensity of the standard (unoptimized) attention algorithm on a sequence of 4,096 tokens. It is often presented as a decode example. One caveat applies and is easier to state now than to discover later: the matrix shapes used below are full N x N matrices, which is the shape of attention when a whole sequence is processed at once. A real single decode step with a KV cache has a query of one token, so its shapes are smaller and its intensity is even lower (the end of this example computes it). The two cases agree on the conclusion, which is why the example still teaches the right lesson: unoptimized attention is memory-bound on an H100.

The goal is practical: see that the argument holds with real numbers. The count illustrates the method, and the exact intensity depends on the implementation. Engineers do not do this routinely, and seeing it once builds intuition.

The setup: an attention head with dimension d = 128 on a sequence of N = 4,096 tokens. The standard algorithm has three lines. Each line loads data from memory, computes, and stores its result back to memory:

1. Load Q and K, compute S = Q K^T, write S.
2. Read S, compute P = softmax(S), write P.
3. Load P and V, compute O = P V, write O.

The matrices are:

FP16 is assumed, so each value takes 2 bytes. For scale, a 4,096 x 4,096 matrix is 2 x 4,096 x 4,096 = 33,554,432 bytes, which is 32 MiB, about the size of one high-resolution RAW photo from a DSLR camera. (MiB is a mebibyte, 1,048,576 bytes.)

The table lists the reads, the compute, and the writes of each line. Bytes are 2 x the number of values. The compute counts are the source's: 2d operations per entry of S, 3 operations per entry of P, and 2N operations per entry of O.

Total memory movement is the sum of all reads and writes:

reads:   line 1: 2Nd + 2Nd = 4Nd
         line 2: 2NN
         line 3: 2NN + 2Nd
writes:  line 1: 2NN
         line 2: 2NN
         line 3: 2Nd

total = 4Nd + 2NN + 2NN + 2Nd + 2NN + 2NN + 2Nd
      = 8 N^2 + 8 N d bytes

N^2 = 16,777,216       N d = 524,288
8 N^2 = 134,217,728    8 N d = 4,194,304
total = 138,412,032 bytes (about 138 MB)

Total compute is the sum of the compute column:

total = (2d) N N + 3 N N + (2N) N d
      = 2 N^2 d + 3 N^2 + 2 N^2 d
      = 4 N^2 d + 3 N^2 operations

4 N^2 d = 4 x 16,777,216 x 128 = 8,589,934,592
3 N^2   = 50,331,648
total   = 8,640,266,240 operations (about 8.6 billion)

Arithmetic intensity is the quotient:

intensity = 8,640,266,240 ops / 138,412,032 bytes = 62.4 ops per byte (about 62)
H100 ops:byte ratio = 295
62 < 295  ->  the operation is memory-bound on an H100

Intensity 62 is much lower than 295. The exact numbers change with model, sequence length, and hardware, but the example illustrates the general rule that unoptimized attention is memory-bound. With the roofline arithmetic of Example 3, this step reaches only 21 percent of the H100's peak compute.

Two checks make the caveat concrete. First, the result does not depend much on N. Dividing the two totals gives intensity = (4 N^2 d + 3 N^2) / (8 N^2 + 8 N d), which is 515 N / (8 (N + d)) for d = 128. This stays below 515 / 8 = 64.4 for every N, so longer sequences never rescue the unfused algorithm. Second, here is a real decode step, where the query is one token (1 x d) and S and P are 1 x N:

operations = 2Nd (S) + 3N (P) + 2Nd (O)          = 2,109,440
bytes      = Q 2d + K 2Nd + S write 2N + S read 2N + P write 2N
             + P read 2N + V 2Nd + O 2d            = 2,130,432
intensity  = 2,109,440 / 2,130,432 = 0.99 ops per byte

A true decode step does about one operation per byte, far below 295, because every cached key and value is read once and used for only a couple of operations. So decode attention is memory-bound by a wide margin, and the N x N example above is a different situation (a whole sequence at once) that is memory-bound for another reason: the S and P matrices are written to memory and read right back. This also explains why prefill is called compute-bound even though unfused full-sequence attention scores 62. The prefill claim in Example 4 rests on the large weight multiplications, and FlashAttention lifts the attention part by removing the S and P traffic. The second calculation uses the same counting method.

Most of the traffic in the full-matrix case, 8 N^2 bytes, is the S and P matrices. The attention optimization lesson shows how to avoid that traffic.

Example 6: image and video generation bottlenecks

Image and video models are relatively small. They have about a tenth as many parameters as frontier language models. Their attention is just as demanding, though, for the reasons from the previous lesson:

• They use iterative denoising and not autoregressive token generation.
• Like LLM prefill, which processes the whole input sequence at once, the attention for media considers the whole image or video object as represented in latent space.
• Like LLM prefill, image and video inference is therefore bottlenecked on compute.

A rough count for an image latent of 128 x 128 = 16,384 positions: each denoising pass compares every position with every other one, which is 16,384^2 = 268,435,456 pairs, the way prefill compares all prompt tokens, and every step does it for the whole latent. Plenty of arithmetic is done against each weight read, which is the compute-bound pattern. Techniques for these modalities are covered in the serving lesson.

Interview points

• The two GPU resources are compute (FLOPS) and memory bandwidth (bytes per second). A bottleneck is a case where one is saturated and the other idle. Improving the non-limiting one gains nothing.
• ops:byte ratio = compute / bandwidth. For an H100 in FP16, 989 teraFLOPS / 3.35 TB/s is about 295.
• Arithmetic intensity (operational intensity) = total operations / total memory traffic for one algorithm.
• On a roofline, intensity above the hardware ratio is compute-bound (flat ceiling), and below it is memory-bound (diagonal ceiling).
• Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPS. Image and video generation are compute-bound.
• Prefill loads the weights once for large matrix-matrix products. Decode reloads weights for every token with cheap vector-matrix products.
• Batching makes decode less memory-bound because more arithmetic is done for the same memory traffic.
• Worked example: standard unfused attention with full N x N matrices, N = 4,096 and d = 128 in FP16, moves 8N^2 + 8Nd bytes and computes 4N^2d + 3N^2 operations, giving intensity about 62 against 295, so it is memory-bound. It is often labeled a decode example. A single decode step with a one-token query is lower still, about 1 op per byte.
• Image and video models have about a tenth of the parameters of frontier LLMs but attention that is just as demanding.

Summary

• Find the bottleneck before optimizing: compare an operation's arithmetic intensity with the GPU's ops:byte ratio.
• H100 in FP16: 989 teraFLOPS, 3.35 TB/s, ratio about 295.
• The standard attention example (full N x N matrices) gives about 62 ops per byte, below 295, so it is memory-bound. One-token decode attention is about 1 op per byte.
• LLM prefill and image and video generation are compute-bound. LLM decode is memory-bound.
• Batching raises the arithmetic done per byte of weights read, which moves decode toward compute-bound.