CUDA, Kernels, and Kernel Fusion

Hardware sets the limits of what is possible, and software decides how close a deployment gets. NVIDIA's lead in inference comes partly from the software around its chips. Hardware changes slowly: the best companies, such as Apple and NVIDIA, release new architectures at most once a year, and every two years is more common. Software changes quickly. To run a newly released open model on its first day, you often have to install a nightly build or another prerelease version of every software dependency, because each one needs support for the new model.

This lesson opens the software part of the course. It lists the main organizations and the layers of the stack, then goes to the lowest layer, CUDA, where code talks to the GPU directly. Afterwards you can explain what a kernel is, name the libraries that supply kernels, say why a kernel written for one GPU can fail on the next one, and compute how much memory traffic kernel fusion removes. Terms such as VRAM, bandwidth, and decode come from Inside a GPU.

Example 1: who builds the software and how it is layered

Hardware centers on NVIDIA and a few competitors. Software has a lower barrier to entry, so countless companies build parts of the inference stack. These are the key players for inference engineers.

Thousands more companies, universities, and research institutions contribute open-source work. The field is too big and too fast to document completely, so this chapter of the course presents foundational technologies that stay relevant. They are arranged from the lowest to the highest level of abstraction. An abstraction hides detail: each layer offers a simpler interface over the one below it, at the cost of control.

flowchart BT
  GPU["GPU hardware"] --> CUDA["CUDA: direct control of computation and memory (this lesson)"]
  CUDA --> FW["Deep learning frameworks: PyTorch and others (next lesson)"]
  FW --> ENG["Inference engines: vLLM, SGLang, TensorRT-LLM"]
  ENG --> DYN["NVIDIA Dynamo: large-scale orchestration"]

Read the diagram from the bottom. CUDA gives explicit control over computation and memory. Frameworks wrap CUDA for training, exporting, and running neural networks in Python (PyTorch, model formats, ONNX, TensorRT, Transformers). Inference engines are highly configurable inference programs for common model architectures, backed by PyTorch (Inference engines). Dynamo sits on top of the engines to run large deployments.

Most inference engineering today happens at the higher levels: configuring and deploying engines and organizing inference across several GPUs. Whatever level you work at, you need a working picture of the levels next to it, because problems cross the boundaries.

Example 2: what CUDA is

CUDA is how you write code that runs on NVIDIA GPUs. More precisely, it is NVIDIA's proprietary computing platform and programming model for running parallel tasks on GPUs. The name stands for Compute Unified Device Architecture, although nobody spells it out any more. It is the foundation of the whole generative AI ecosystem on NVIDIA GPUs. "Platform" and "programming model" are broad terms, so it is simpler to list its four parts.

A directed acyclic graph is a set of steps with arrows showing which step must finish before another starts, and no arrow loops back. A CUDA graph records such a set of GPU operations once, so a workflow that repeats can be launched again as a whole instead of being assembled step by step every time.

flowchart LR
  A["Copy input to GPU"] --> B["Kernel 1"]
  B --> C["Kernel 2"]
  B --> D["Kernel 3"]
  C --> E["Kernel 4"]
  D --> E
  E --> F["Copy output back"]

Here Kernels 2 and 3 both depend only on Kernel 1, so the graph shows that they may run side by side. Kernel 4 waits for both.

CUDA is not a programming language. CUDA programs are written in a language, most often C++, and then a compiler such as nvcc splits them into separate CPU code and GPU code. A kernel is a function that does some parallel computation. Whenever you read "CUDA kernel", you can substitute "a piece of code written for NVIDIA GPUs".

Example 3: a first kernel, doubling an array

The standard "Hello, World" kernel takes an array of length n and doubles every element. It has about six lines. On a CPU the work would take linear time, because each element is doubled one after another. On a GPU, thousands of elements are processed at once.

The way this works is that the GPU starts many threads (see Inside a GPU) that all run the same kernel, and each thread computes which element it owns from its position. The threads are grouped into blocks. Each thread learns its block number and its position within the block, and the kernel combines them into an array index:

index = (block number) x (threads per block) + (thread number inside the block)

A worked case uses an array of n = 10 elements and 4 threads per block (a tiny number chosen to make the table short).

blocks needed = ceiling(10 / 4) = 3, so 3 x 4 = 12 threads start
block 0: threads 0..3  -> indexes 0 x 4 + 0..3 = 0, 1, 2, 3
block 1: threads 0..3  -> indexes 1 x 4 + 0..3 = 4, 5, 6, 7
block 2: threads 0..3  -> indexes 2 x 4 + 0..3 = 8, 9, 10, 11
indexes 10 and 11 are not less than n = 10, so those two threads do nothing

The kernel therefore starts with a bounds check ("only work if the index is less than n"), because the number of threads is rounded up to whole blocks. Every thread with a valid index reads one element, multiplies it by 2, and writes it back.

Writing kernels changes how an inference engineer thinks: from algorithms to implementations. The traditional attention algorithm, central to generative AI, can be written in a few dozen lines. FlashAttention computes the same mathematical operation and takes tens of thousands of lines of code, because it implements the algorithm in a more memory-efficient way for a specific GPU (Optimizing attention explains why).

Example 4: kernels you do not have to write

Writing CUDA kernels does not mean starting from nothing. The prior art is decades older than CUDA. BLAS (Basic Linear Algebra Subprograms) is a specification of common linear algebra operations, from dot products to matrix multiplication. It was first implemented for Fortran in the 1970s.

With CUTLASS and CuTe, kernels can be written at a higher level of abstraction while keeping strong performance.

Within BLAS, the most used operation in inference is GEMM (General Matrix-Matrix Multiplication). Every linear layer in a model is a matrix multiplication, and cuBLAS is a strong starting point. GEMM is so central that you might still want finer control, for example different GEMM kernels for matrices of different shapes or tuned for one GPU architecture.

A worked count shows how GEMM scales. Multiplying a matrix of M rows by K columns with a matrix of K rows by N columns needs 2 x M x N x K floating point operations (one multiply and one add per pair).

Large case (prefill, 4,096 tokens at once), M = N = K = 4,096:
  FLOPs = 2 x 4,096 x 4,096 x 4,096 = 137.4 billion
  On an H100 at 1,979 teraFLOPS (FP8 dense): 137.4e9 / 1,979e12 = 0.07 ms

Small case (decode, 1 token), M = 1, N = K = 4,096:
  FLOPs = 2 x 1 x 4,096 x 4,096 = 33.6 million
  weights read = 4,096 x 4,096 = 16.8 million numbers
  operations per weight = 33.6 million / 16.8 million = 2
  at 2 bytes per weight: 2 operations per 2 bytes = 1 operation per byte

The H100's ratio of arithmetic to memory speed is about 591 operations per byte (GPU generations). The decode case does 1 operation per byte, far below 591, so it waits on memory. The large case reuses each weight for 4,096 tokens and keeps the arithmetic units busy. This is the same split between memory-bound decode and compute-bound prefill from Finding the bottleneck: compute or memory, and it is why one GEMM kernel is rarely best for all shapes.

Example 5: kernel selection and its hardware ties

Most inference engineers never write a kernel, but they do select kernels, which means choosing the best one from a range of options. This is an important part of optimization.

Kernel implementations are highly specialized. CUDA exposes low-level interfaces for memory management and parallel computing, and kernel engineers make decisions from the exact details of the hardware they target. Kernels often contain hard-coded values based on a GPU's memory bandwidth or the number and layout of its Tensor cores. As a result:

• A kernel written for an H100 will likely not exploit the architecture and larger memory of a B200.
• A kernel written for a B200 may not work on the previous Hopper architecture.
• Each new GPU generation requires substantial engineering work to port handwritten kernels so that they run well.

Most selection is automatic. Deep learning frameworks and inference engines ship pre-configured kernels for various architectures, and PyTorch and TensorRT-LLM choose kernels automatically during their compilation steps. Still, you may manually pick a few kernels for the key algorithms.

An example is DeepGEMM. Most production GEMM kernels come from cuBLAS. When the DeepSeek lab released an updated version of DeepSeek-V3, it also released DeepGEMM, with more efficient GEMM kernels for FP8 matrix multiplication on Hopper. Manual selection lets you insert a DeepGEMM kernel as a plugin to speed up one step, such as multiplying two matrices of specific dimensions. Compatibility needs care: after an upgrade to a B200, you would have to swap the kernel back, wait for Blackwell support in DeepGEMM (which exists at the time of writing), or port the kernel yourself.

Example 6: kernel fusion

Two kernels that run back to back on the same data waste reads and writes to memory. Take two kernels, multiply_by_2 and multiply_by_3, run in sequence on the vector [1, 2, 3].

Unfused:
 1. read  [1, 2, 3] from memory
 2. run multiply_by_2
 3. write [2, 4, 6] to memory
 4. read  [2, 4, 6] from memory      (steps 3 and 4 are an unnecessary round trip)
 5. run multiply_by_3
 6. write [6, 12, 18] to memory

Fused (one kernel, multiply_by_6):
 1. read  [1, 2, 3] from memory
 2. run multiply_by_6
 3. write [6, 12, 18] to memory

Decode is the bandwidth-bound phase of LLM inference, so an engine cannot afford unnecessary reads and writes. Kernel fusion takes two or more kernels and re-implements them as a single kernel that does both jobs. In this toy case the fused kernel is multiply_by_6. The cost of the round trip grows with the size of the data.

flowchart LR
  subgraph Unfused
    M1["memory"] --> K1["kernel 1"] --> M2["memory"] --> K2["kernel 2"] --> M3["memory"]
  end
  subgraph Fused
    N1["memory"] --> KF["fused kernel"] --> N2["memory"]
  end

A worked case with a vector of one billion 4-byte numbers, which is 4 GB, on an H100 with 3.35 TB/s of bandwidth:

Unfused: read 4 GB + write 4 GB + read 4 GB + write 4 GB = 16 GB of traffic
         16 GB / 3,350 GB/s = 4.78 ms
Fused:   read 4 GB + write 4 GB = 8 GB of traffic
         8 GB / 3,350 GB/s = 2.39 ms
Saved:   half of the memory traffic and half of the time, for the same result

Real fusion is more complicated, since functions are more complex and the data overlap is less clean. There are common patterns for inference, such as combining a matrix multiplication, a bias addition, and an activation function (a bias is a number added to each output, and an activation is a function such as ReLU applied to each value afterwards). Written separately, the matrix result of size O is written once by the multiplication, read and written again by the bias step, and read and written again by the activation: O + 2O + 2O = 5O of traffic on that result. Fused, it is written once: O. Fusion cuts that traffic to one fifth.

Fusion can be automatic or manual. Compilers can find simple opportunities and generate fused kernels themselves. More sophisticated algorithms such as FlashAttention need handwritten fused kernels, which are plugged in during inference.

Interview points

• Hardware iterates yearly or every two years, and software iterates in weeks. Day-zero support for a new model often needs nightly builds of every dependency.
• Key software players: NVIDIA (CUDA up to Dynamo), Hugging Face (registry, transformers, diffusers), the Linux Foundation (PyTorch, vLLM), and LMSYS Org (SGLang).
• The stack from low to high abstraction is CUDA, deep learning frameworks, inference engines, and NVIDIA Dynamo. Most inference engineering happens at the higher levels.
• CUDA is a proprietary platform and programming model, not a language. Programs are usually C++ compiled by nvcc into CPU and GPU code. Its parts are the kernel, graph (a DAG for repeated workflows), driver (low-level), and runtime (developer API).
• A kernel is a user-defined function that runs in parallel on the GPU. FlashAttention is the same math as attention in tens of thousands of lines because it is tuned to one GPU.
• BLAS (1970s, Fortran) is a specification, cuBLAS is its CUDA implementation, cuDNN supplies neural-network primitives, CUTLASS and CuTe are template libraries, and FlashInfer holds LLM inference kernels. GEMM is the most used BLAS operation in inference.
• Kernels hard-code hardware details, so a kernel for an H100 may not suit a B200 and the reverse. Porting each generation takes substantial work.
• PyTorch and TensorRT-LLM select kernels automatically. DeepGEMM is a manual plugin for FP8 GEMM on Hopper.
• Kernel fusion merges kernels to avoid memory round trips, which matters most in bandwidth-bound decode. Compilers find simple fusions, and FlashAttention needs handwritten fusion.

Summary

• CUDA components: kernel, graph, driver, runtime. Programs are written in C++ and compiled by nvcc.
• A kernel gives each GPU thread an index from its block and thread numbers and processes many elements at once.
• GEMM costs 2 x M x N x K operations. Single-token decode does about one operation per byte read and is memory-bound.
• Libraries: cuBLAS, cuDNN, CUTLASS, CuTe, FlashInfer, DeepGEMM.
• Kernel selection is mostly automatic and the manual choices are tied to a GPU architecture.
• Kernel fusion removes round trips to memory, halving the traffic for two back-to-back kernels on the same data.