Serving Image and Video Generation Models

Image and video generators work differently from language models, so serving them works differently too. A language model writes tokens one after another and spends most of its time waiting for memory. An image model starts from noise and cleans it up over a fixed number of steps, and it spends most of its time on arithmetic. The tools, the bottleneck, and the quality trade-offs all change.

This lesson covers the architecture and tooling of image generation, the kernels and libraries that make it fast, a trick that saves a fifth of the work, and then video generation, the most demanding modality: attention cost, caching, quantization, and splitting a video across eight GPUs with Context Parallelism. Afterwards you can pick a serving stack for a diffusion model, estimate the saving from skipping guidance, and explain why video models run on whole nodes with a batch size of one.

Terms from earlier lessons. Iterative denoising is generation by starting from random noise and refining it over many steps (How Image and Video Models Generate). Latent space is a compressed internal representation in which the model works, smaller than the pixels. A kernel is a routine that runs on the GPU, and kernel fusion merges several small kernels into one so that data makes fewer trips to memory (CUDA, Kernels, and Kernel Fusion). Compute-bound means limited by arithmetic, not by memory bandwidth (Finding the Bottleneck).

Example 1: what is different about image generation

Image generation models may accept text and also reference images to create new images. Compared with LLMs they differ on several axes:

Judging image quality automatically is hard. Pipelines that use a vision language model as a judge give a directional signal at best and can disagree with human preference. Most image-quality evaluations ask people to choose among thousands of images, which gathers taste and preference into a benchmark.

Example 2: kernels and libraries for image generation

The model card for an image model usually shows examples with the diffusers library and very few optimizations. Image generation is compute-bound in theory, but in practice you often need memory-efficient kernels and kernel fusion to even reach that limit. High-performance image inference uses one of three libraries:

For something that works well now, use the SGLang Diffusion or TensorRT version of the model. PyTorch leaves room for deep customization by advanced inference engineers.

The kernels that matter most are these.

• Attention is the most important kernel. Many image models use FlashAttention 2 out of the box, but FlashAttention 3 and 4 perform better on Hopper and Blackwell GPUs, respectively.
• Small kernels, especially normalization functions such as RMSNorm, are good candidates for fusion to use memory efficiently.
• GEMM kernels (general matrix multiplication, used by linear layers) matter once the model is compute-bound. They are generally safe to quantize into 8-bit floating point, which gives twice the FLOPS on Tensor Cores (Quantization). Kernels from CuTe, CUTLASS, or DeepGEMM may be the most efficient, depending on the model.
• Torch compilation includes automatic kernel fusion with a plugin system for adding hand-picked kernels. The compiled engine can be cached for faster loading when a node starts, which matters because compilation takes several minutes.

Like most high-performance engines, Torch compilation targets the exact GPU model and architecture that does the compiling. To run on a B200, compile on a B200.

graph LR
    M["PyTorch model"] --> C["Torch compile: fuse kernels, add custom kernels"]
    C --> E["Compiled engine for this GPU type"]
    E --> K["Cached for fast node startup"]

Example 3: skipping guidance, a faster image trick

Kernel selection and compilation are standard optimizations. Here is a smaller trick that saves work without losing steps.

Diffusion is a step-by-step process, and the general outline of the image appears in the early steps. Generation time is linear in the step count, which is why few-step models and latent consistency models are much faster than full 50-step models. But cutting steps can reduce quality below what is acceptable.

Each step does two passes through the denoising model, so each step runs at a batch size of two: one pass with the prompt and one without. This is classifier-free guidance. The guidance parameter sets how strongly the prompt-guided result is weighted when the two are combined. With guidance zero, the prompt-guided pass is not needed at all.

After the first few steps the outline is fixed, and later steps only fill in details. Prompt adherence matters mostly in the early steps that set the broad shapes. The model will not decide to draw a dog when it is already in the middle of drawing a cat. So guidance can be switched off for the last steps, which removes passes without removing steps.

50 steps with guidance on every step: 50 steps x 2 passes = 100 passes
guidance skipped for the last 20 steps:
    30 steps x 2 passes + 20 steps x 1 pass = 60 + 20 = 80 passes
saving: (100 - 80) / 100 = 20 percent fewer passes through the model, quality generally stays high

graph LR
    A["Steps 1 to 30: with and without prompt, 2 passes each"] --> B["Steps 31 to 50: one pass each"]

Example 4: why video generation is the hardest

Video generation is the most demanding modality. Whenever possible, run it on Blackwell GPUs (or Rubin when available). They offer large memory for Context Parallelism, fast Tensor Cores for attention, and microscaling data formats for more precise quantization (GPU Generations).

Architecturally, video generation resembles image generation, except that it renders a whole video from latent space instead of one frame. Following the chapter's principle that greater scale allows more techniques, video models use all the image techniques and more. Like image generation, video generation is compute-bound and works by iterative denoising in latent space. The step count is about the same (around 50), but each step handles much more data.

Because video generation is compute-bound, batching does not help the way it does for text. Video models usually run on a full node of eight GPUs with a batch size of one: all eight GPUs work on one video at a time. With batched workloads, changing the batch size trades latency for throughput. Here that trade does not exist, so the only way to improve throughput and cost is to make the model itself faster.

Early video models were framewise: they generated one frame at a time, which hurt quality and consistency. Now models denoise the whole video at once in latent space. Image latent space has two dimensions (width, height). Video latent space has three (width, height, time). Every attention calculation then handles a huge amount of data, and attention is 70 to 80 percent of compute time in video models, which makes it the most important target.

A worked case of why attention dominates (Amdahl-style reasoning): if attention takes 75 percent of the time and a change makes attention twice as fast, the new time is 0.25 + 0.75 / 2 = 0.625 of the old time, a speedup of 1 / 0.625 = 1.6 times. Making a part that takes only 5 percent twice as fast gives 1 / (0.95 + 0.025) = 1.03 times. The big part is the one worth optimizing.

Example 5: attention optimization, caching, and quantization for video

Attention work starts with kernel selection. Try FlashAttention, DeepGEMM, CuTe, and CUTLASS kernels and see which perform best for the model.

Language models cache keys and values to speed up attention. Video models use other caching patterns that try to reuse earlier outputs. Reusing parts of the attention computation can make video generation 30 to 40 percent faster in practice. Methods change as research moves, but there are two basic approaches:

Algorithms and implementations range from negligible quality loss to unusable output, so test these strategies carefully before production.

The main other tool for faster attention is quantization. For a memory-bound language model, quantization means less data to load from memory. For a video model it means access to twice the FLOPS on lower-precision Tensor Cores. Language model quantization focuses on weights: large linear layers where the effect of quantization is negligible. In video models, weights still help, but those layers, which use most of the memory bandwidth that limits language models, are a small share of compute time. So quantization for video targets attention.

Attention is the riskiest part of any model to quantize because errors build up across inference. In video models the risk is slightly lower because there are about 50 steps, not thousands of autoregressive iterations, but it still matters. Three approaches limit the damage:

1. Blockwise quantization with a microscaling format (MXFP8), available on Hopper and Blackwell. Microscaling formats keep outlier values better, and outliers strongly affect attention accuracy.
2. Selective quantization by step: keep early steps in FP16 and quantize later steps.
3. Selective quantization by layer: keep the first and last layers and quantize the hidden layers.

Quantizing by step follows the same insight as the guidance trick: early steps set the outline and matter more for prompt adherence and accuracy. By layer, the first and last layers take the input and produce the final output, so they matter more. Hidden layers do intermediate work that tolerates approximation. Quantizing only the less important parts keeps quality. These tactics appear in kernels such as SageAttention, an 8-bit attention kernel for quality low-precision attention on video models.

50 steps: keep steps 1-10 in FP16, quantize steps 11-50 (illustration of the idea)
layers  : keep first and last layer in FP16, quantize all hidden layers

Example 6: Context Parallelism

Video models usually run on a full eight-GPU node, but with Context Parallelism and not Tensor Parallelism (Model Parallelism). Context Parallelism copies the weights onto every GPU. Video models are small enough that eight copies take a meaningful amount of memory, but it is feasible on a B200 (for illustration, a 30 GB model replicated eight times uses 8 x 30 = 240 GB of total VRAM, though each GPU still holds only 30 GB).

Instead of splitting the model, it splits the attention calculation across the GPUs. A mechanism such as ring attention coordinates this: each GPU holds a piece of the context and passes intermediate results to the next GPU in a ring. Attention in transformers is multi-head, usually with eight or more heads. The heads are independent, so they can run separately and be combined afterwards.

graph LR
    G1["GPU 1: full weights, piece 1 of the latent"] --> G2["GPU 2: piece 2"]
    G2 --> G3["GPU 3: piece 3"]
    G3 --> G4["GPU 4: piece 4"]
    G4 --> G1

Attention is not the only part that can be split. The latent decoding step, which uses a variational autoencoder (VAE) to turn the latent video into pixels, takes three to five percent of total inference time and can also run across GPUs.

These techniques make AI video feasible. As videos get longer and models bigger, parallelism will remain the most important technique for video generation inference.

Interview points

• Image and video generators are mostly iterative denoisers built as pipelines of small models in latent space, about ten to twenty times smaller than frontier LLMs, and compute-bound instead of bandwidth-bound.
• Tooling is newer: SGLang Diffusion and vLLM Omni are new, and many deployments use PyTorch or TensorRT directly. High-performance options are SGLang Diffusion, TensorRT, and PyTorch.
• Model-card diffusers examples are barely optimized. Memory-efficient kernels and fusion are needed to even become compute-bound. Key kernels: attention (FlashAttention 3 on Hopper, 4 on Blackwell), fused normalization such as RMSNorm, and GEMM kernels (CuTe, CUTLASS, DeepGEMM) with FP8 for twice the Tensor Core FLOPS.
• Torch compilation fuses kernels, takes minutes, and is specific to the GPU type, so compile on the target GPU and cache the result.
• Image time is linear in steps. Skipping guidance for the last 20 of 50 steps cuts 100 passes to 80, because each guided step runs two passes. Early steps set the outline, so guidance matters there.
• Image quality evaluation relies on human preference. VLM judges are only directional.
• Video generation: Blackwell (or Rubin), batch size one on eight GPUs, about 50 steps, latent space with width, height, and time, and attention at 70 to 80 percent of compute.
• Video caching (timestep-based or transformer-based) gives 30 to 40 percent speedups but can ruin quality, so test it. Quantize attention carefully: MXFP8, FP16 for early steps, and keep first and last layers in high precision. SageAttention is an 8-bit attention kernel.
• Context Parallelism replicates weights and splits attention across GPUs with ring attention. The VAE decoding step (3 to 5 percent of time) can also be parallelized.

Summary

• Image models denoise in latent space as pipelines of small models and are compute-bound, so kernel choice and fusion decide speed.
• Pick SGLang Diffusion or TensorRT for a quick result and PyTorch with Torch compilation for deep control. Compile for the GPU type that will run the model.
• Switching guidance off in late steps saves passes (100 to 80 in the 50-step example) without cutting steps.
• Video models are the most demanding: eight GPUs, batch size one, 3D latent space, and attention taking 70 to 80 percent of compute.
• Caching timesteps or hidden states gives 30 to 40 percent speedups with a quality risk. Attention quantization needs microscaling formats and selective step or layer choices.
• Context Parallelism copies the weights and shares the attention work through ring attention, and the VAE decode can be parallelized.