Inside a GPU: Compute, Memory, and Caches

A language model is a very large pile of arithmetic. Serving it means doing that arithmetic millions of times per second for many users at once. An ordinary CPU (central processing unit, the main chip of a computer) is built to run complicated instructions one after another very quickly. It is a poor fit for this job. An accelerator is a chip built for exactly this kind of work: it loads huge amounts of data and performs trillions of simple operations per second. The most common accelerator for inference is the GPU (graphics processing unit). The market leader for inference GPUs is NVIDIA, so this course focuses on NVIDIA GPUs in the datacenter. GPU generations covers the product line. Instances, multi-GPU, MIG, and other accelerators covers other vendors, and local inference covers laptops, desktops, and phones.

This lesson explains what is inside one GPU. You will see how it computes, how it stores data, and which two numbers on a spec sheet matter most for inference: FLOPS (floating point operations per second, the speed of arithmetic) and memory bandwidth (how many bytes per second can move from memory to the chips that compute). Afterwards you will be able to read a GPU spec sheet, check whether a model fits in a GPU's memory, and decide whether a workload needs more FLOPS or more bandwidth.

Example 1: three kinds of GPU, three ways to run them

Across vendors, GPUs come in three types.

Inference at scale uses datacenter GPUs mounted in racks. A rack is a tall metal frame that holds servers and provides standardized power, networking, and cooling, and a full one is about the size of a refrigerator. A datacenter GPU such as the B200 has the highest performance of a single GPU. More importantly, it has fast links to the other GPUs next to it, it is installed in highly standardized configurations, and millions of them exist in datacenters around the world.

Few teams keep a B200 under a desk, so datacenter GPUs are used in one of three modes.

Most inference engineers use cloud GPUs. Large enterprises and governments run on-premise and air-gapped deployments. The cloud wins for fast-growing AI products because it gives flexibility and quick access to more GPUs. Choosing hardware is still complicated, because cloud providers differ from each other and NVIDIA's own product names have their own rules, which GPU generations explains.

Example 2: why a GPU suits model inference

Think of a CPU as a small team of skilled chefs. Each chef can cook any dish, however complicated, but there are only a few of them. A GPU is a kitchen with thousands of cooks who each know one simple move, and all of them make the same move at the same moment on different ingredients. The analogy fails at one point: GPU workers are not independent. They are grouped, and a whole group executes the same instruction together. This is why a GPU is great at one uniform operation on thousands of independent pieces of data, and poor at long chains of different, dependent steps.

Model inference is a series of vector and matrix multiplications. A vector is a list of numbers, and a matrix is a grid of numbers. Multiplying them means doing the same multiply-and-add on thousands of independent numbers, which matches the GPU's strength. The worked case below doubles every element of an array of 1,000,000 numbers in the CPU style (one after another) and in the GPU style (all at once). Look at the last line: the result is the same, and only the order of the work changes.

Task: double every element of an array of 1,000,000 numbers.
CPU style:  element 0, then element 1, then element 2, ... (1,000,000 steps in order)
GPU style:  element i is computed by its own thread, all at once, in any order
Result:     element i is data[i] x 2 in both styles, so the two arrays are identical
Why it works: element i needs only data[i], never the result of another element

Each element's result depends only on that element. No element waits for another, so they can all be computed at the same time. A GPU applies the same idea at a far larger scale. It has tens to hundreds of thousands of threads (small independent streams of instructions) that work concurrently, switch tasks in a single clock cycle, and run simple instructions in parallel. A CPU has dozens to hundreds of threads. The idea of a thread is similar on both chips, but the idea of a "core" is not, as the next example shows.

GPUs themselves are extraordinarily complicated machines. Inference engineers work at a comfortable level above the hardware, but a clear mental picture of what is inside the box is needed to build fast systems.

Example 3: cores, streaming multiprocessors, and the three kinds of compute

On a CPU, a core is one complete processor, as in an 8-core Intel i9 in a gaming computer. On a GPU the word means something smaller. A GPU is built from many Streaming Multiprocessors (SMs). An SM is a block of the chip that has its own compute units, its own small memory, and its own scheduler. Each SM contains several kinds of compute units, and there are three kinds.

flowchart TD
  GPU["GPU"] --> SM1["Streaming Multiprocessor 1"]
  GPU --> SM2["Streaming Multiprocessor 2"]
  GPU --> SMN["... more SMs"]
  SM1 --> CC["CUDA cores (scalars)"]
  SM1 --> TC["Tensor cores (matrices)"]
  SM1 --> SFU["Special function units (sin, cos, log)"]
  SM1 --> L1["L1 cache / shared memory"]

Inference compute is measured in Tensor core compute. SFUs matter, because softmax (the step that turns scores into probabilities) uses exponentials, but the Tensor cores carry the load. They run Matrix Multiply and Accumulate (MMA) instructions. An MMA multiplies matrix A by matrix B and then adds a matrix C, producing matrix D. The "accumulate" step is the addition of C. A worked case with two 2 by 2 matrices makes the arithmetic concrete. Each cell of D is a row of A times a column of B, plus the matching cell of C.

A = [[1, 2],    B = [[5, 6],    C = [[1, 1],
     [3, 4]]         [7, 8]]         [1, 1]]

D[0][0] = 1x5 + 2x7 + C[0][0] = 5 + 14 + 1 = 20
D[0][1] = 1x6 + 2x8 + C[0][1] = 6 + 16 + 1 = 23
D[1][0] = 3x5 + 4x7 + C[1][0] = 15 + 28 + 1 = 44
D[1][1] = 3x6 + 4x8 + C[1][1] = 18 + 32 + 1 = 51

D = [[20, 23],
     [44, 51]]
multiply-adds: 2 rows x 2 columns x 2 terms = 8

The first cell of D is 1 x 5 + 2 x 7 + 1 = 20. The last line counts the work: a 2 by 2 times 2 by 2 product needs 2 x 2 x 2 = 8 multiply-adds. A Tensor core performs a block of such multiply-adds as one instruction, instead of one at a time as written above. That is why Tensor cores, not CUDA cores, set the speed of inference.

Example 4: reading FLOPS on a spec sheet

Compute is measured in FLOPS. One FLOPS is one floating point operation (an add or a multiply on a number with a decimal point) per second. Datacenter GPUs reach trillions (teraFLOPS) or quadrillions (petaFLOPS). A spec sheet gives two Tensor core numbers for each precision, and two details change how to read them.

Dense versus sparse. The dense number is the raw rate when every element of the tensor takes part. A tensor is a block of numbers with any number of dimensions. The sparse number applies to tensors with 2:4 structured sparsity, a pattern in which two of every four values are 0. Half of all values are 0, and Tensor cores can skip the multiplication by 0. A GPU's sparse FLOPS at a given precision are often, but not always, double the dense FLOPS at that precision. Inference is dense by default, so the dense number is the one to read.

Precision. Precision is the number of bits used to store each number. FLOPS generally double with each halving of precision. A GPU that does one petaFLOPS on 16-bit numbers does two petaFLOPS on 8-bit numbers. Two GPUs can be compared only at identical precision.

The first part of the next calculation shows why a sparse tensor needs half the multiplications. The second part is the precision ladder for a GPU with 1 petaFLOPS at 16 bits.

weights = [0, 3, 0, 5,  2, 0, 7, 0]     two zeros in each group of four (2:4 pattern)
inputs  = [1, 2, 3, 4,  5, 6, 7, 8]

dense  dot product: 0x1 + 3x2 + 0x3 + 5x4 + 2x5 + 0x6 + 7x7 + 0x8
                  = 6 + 20 + 10 + 49 = 85, using 8 multiplications

sparse dot product: skip every weight that is 0
                  = 3x2 + 5x4 + 2x5 + 7x7 = 6 + 20 + 10 + 49 = 85, using 4 multiplications

Precision ladder for a GPU with 1 petaFLOPS dense on 16-bit numbers:
  16-bit: 1 petaFLOPS
   8-bit: 1 x (16 / 8) = 2 petaFLOPS
   4-bit: 1 x (16 / 4) = 4 petaFLOPS

Both dot products give 85, because multiplying by zero adds nothing, but the sparse one used 4 multiplications instead of 8. This is the reason sparse FLOPS can be double the dense FLOPS. The ladder follows the rule of doubling for each halving of precision. The 4-bit row continues that pattern for hardware that supports 4-bit formats (Blackwell does, as GPU generations shows). A spec sheet that lists 3,958 teraFLOPS with sparsity at some precision, with sparse double the dense, lists 1,979 teraFLOPS dense. Compare dense with dense and the same precision with the same precision.

Compute is the bottleneck for prefill (the first phase of LLM inference, when the whole prompt is processed at once, see How an LLM generates text) and for image and video generation. To serve those workloads, pick the accelerator with more FLOPS.

Example 5: DRAM, SRAM, and the three cache levels

Arithmetic is useless without data to work on, and data must travel from memory to the compute units. There are two types of memory on any chip, CPU or GPU.

The big, slow, cheap memory next to the GPU is the VRAM (video RAM). The "V" is a callback to the original purpose of these chips, graphics, just as the "G" in GPU is. VRAM is a type of DRAM. Today it is built as HBM (high-bandwidth memory) in generations called HBM3, HBM3e, and HBM4. A GPU has dozens or hundreds of gigabytes of VRAM, and high-end GPUs have terabytes per second of memory bandwidth.

The GPU also has SRAM on the chip itself, arranged as three levels of cache. A cache is a small, fast memory that keeps copies of data that the compute units are likely to need again soon.

flowchart TD
  VRAM["VRAM (HBM): gigabytes, off the compute chip"] --> L2["L2 cache: shared by all SMs"]
  L2 --> L1a["SM 1: L1 cache"]
  L2 --> L1b["SM 2: L1 cache"]
  L1a --> TCa["Tensor cores with L0 instruction caches"]
  L1b --> TCb["Tensor cores with L0 instruction caches"]

An H100 has 256 KB of L1 cache per Streaming Multiprocessor and 50 MB of L2 cache in total. The calculation below adds these up for an SXM H100 (which has 132 SMs) and compares them to the weights of a model, to show what the caches can and cannot hold.

L1 total  = 132 SMs x 256 KB = 33,792 KB = 33,792 / 1,024 = 33.0 MB
L2 total  = 50 MB
weights   = 8B parameters x 2 bytes = 16 GB = 16 x 1,024 = 16,384 MB
weights / L2 = 16,384 / 50 = about 328

All the caches together hold tens of megabytes, while the weights of even a small model hold gigabytes. So the weights cannot live in the caches. Every pass through the model streams them from VRAM, and this is why VRAM speed matters so much. VRAM bandwidth is the peak transfer rate between the GPU's compute cores and VRAM over the memory bus. In practice it decides how fast VRAM can feed data into the cache hierarchy.

Example 6: does the model fit in VRAM

The amount of VRAM limits which model can be loaded. The weights are the model's learned numbers, and their size is the number of parameters times the bytes for each parameter. At 2 bytes per parameter (16-bit), a 70-billion-parameter model has 70 x 2 = 140 GB of weights. Number formats are the topic of Quantization.

Weights are not the whole story. During generation the model also stores a KV cache (key-value cache), which is the saved attention data of earlier tokens, so it does not need to be recomputed. KV caching explains it. The rule of thumb is that VRAM should hold the weights plus at least 50 percent headroom for the KV cache, and more headroom for long context, high batch sizes, or video generation models. Two failures follow:

• If the weights do not fit, loading the model fails with an OOM (out of memory) error.
• If there is no headroom, inference is slow or crashes with an OOM.

The calculation below applies the rule (weights x 1.5) to a 70B model at 16-bit and 8-bit, against GPUs whose memory sizes appear in GPU generations.

rule: needed = weights x 1.5   (weights plus 50 percent headroom)

70B at 2 bytes: weights 140 GB, needed 210 GB
  L4   24 GB: does not fit          H200 141 GB: weights fit, no headroom (141 < 210)
  L40  48 GB: does not fit          B200 192 GB: weights fit, no headroom (192 < 210)
  H100 80 GB: does not fit          B300 288 GB: fits (288 >= 210)

70B at 1 byte: weights 70 GB, needed 105 GB
  L4   24 GB: does not fit          H200 141 GB: fits (141 >= 105)
  L40  48 GB: does not fit          B200 192 GB: fits
  H100 80 GB: weights fit, no headroom (70 <= 80 < 105)     B300 288 GB: fits

At 16 bits, only the B300 holds both the weights and the headroom, and an H200 can load the model but would be slow or crash under load. Halving the bytes per parameter moves each GPU one step along: at 8 bits an H200 fits. A 70 GB model on an 80 GB H100 loads but has no room for a KV cache. This is why people use several GPUs for one model (see Instances, multi-GPU, MIG, and other accelerators) or smaller number formats.

Example 7: memory bandwidth sets the speed of decode

Decode is the second phase of LLM inference: the model writes its answer one token at a time. Each new token needs a pass over all the weights, which are read from VRAM. At low to medium batch sizes (few requests at once) the arithmetic units finish quickly and then wait for the next weights to arrive. The speed is limited by memory bandwidth, which is why decode is called memory-bound. Finding the bottleneck: compute or memory develops this carefully with the ops:byte ratio.

A rough ceiling follows from one division. If a pass reads W bytes and the memory moves B bytes per second, one token takes at least W / B seconds, so the ceiling is B / W tokens per second for one request. This ignores the KV cache, other overheads, and batching, so real speeds are lower. Applied to an 8B model at 8 bits (8 GB of weights) on an H100 (3.35 TB/s) and an H200 (4.8 TB/s), which have the same compute and different bandwidth.

ceiling = bandwidth / weight bytes
8B model at 8 bits: weights = 8 GB
H100: 3,350 GB/s / 8 GB = 418.75, about 419 tokens per second
H200: 4,800 GB/s / 8 GB = 600 tokens per second
H200 / H100 = 600 / 418.75 = 1.43, the same as the bandwidth ratio 4.8 / 3.35 = 1.43

The ceiling grows exactly with bandwidth, because the formula divides bandwidth by the same weight size. The H200 does not have more compute than the H100, and it still decodes about 43 percent faster. This is the reason the selection rule is:

Interview points

• A GPU is a throughput machine: it applies one simple operation to thousands of independent data items. A CPU is built for complex sequential execution. Model inference is vector and matrix multiplication, so it fits the GPU.
• A GPU "core" is smaller than a CPU core. A GPU has Streaming Multiprocessors, and each contains CUDA cores (scalars), Tensor cores (vectors and matrices), and Special Function Units (sin, cos, log).
• Inference compute is measured in Tensor core FLOPS, because Tensor cores run the Matrix Multiply and Accumulate (MMA) instructions: D = A x B + C.
• Spec sheets list dense and sparse FLOPS. Sparse needs 2:4 structured sparsity, where half the values are zero, and is often double dense. Inference is dense by default, so use the dense number.
• FLOPS double with each halving of precision, so compare GPUs only at identical precision.
• Compute limits LLM prefill and image and video generation. Memory bandwidth limits LLM decode at low to medium batch sizes.
• VRAM is DRAM, built as HBM3, HBM3e, or HBM4. SRAM on the chip forms the L0 (per Tensor core instruction cache), L1 (per SM shared memory), and L2 (global) caches. An H100 has 256 KB of L1 per SM and 50 MB of L2.
• VRAM must hold the weights plus at least 50 percent headroom for the KV cache. Too little VRAM causes an OOM error at load or slow or crashing inference.
• Datacenter GPUs run in the cloud, on-premise, or air-gapped.

Summary

• GPU types: datacenter (B200), workstation (RTX Pro 6000), and personal computing (GeForce RTX 5090). Scaled inference uses datacenter GPUs in racks, rented from hyperscalers (AWS, GCP) or neoclouds (Coreweave, Nebius), or run on-premise or air-gapped.
• GPUs perform uniform operations on thousands of independent items, with tens to hundreds of thousands of threads.
• A Streaming Multiprocessor holds CUDA cores, Tensor cores, and SFUs. Tensor cores do MMA and set inference speed.
• FLOPS are dense or sparse (2:4 structured sparsity) and double with each halving of precision.
• DRAM is off-chip and in gigabytes. SRAM is on-chip and in kilobytes or megabytes. VRAM is DRAM, built as HBM.
• The caches are L0, L1, and L2. An H100 has 256 KB of L1 per SM and 50 MB of L2.
• VRAM bandwidth is the peak transfer rate between the GPU cores and VRAM.
• Choose by workload: more FLOPS for prefill and image or video generation, more bandwidth for decode, and enough VRAM for weights plus at least 50 percent headroom.