How Image and Video Models Generate

An LLM writes text one token after another. An image model works in a different way: it starts with a canvas of random noise and cleans it up, step by step, until a picture that matches the prompt appears. Video models do the same for a whole clip at once. These models run on the same GPUs, but they stress them in a different way, so serving them needs different techniques.

Afterwards you can name the components of an image generation pipeline, explain latent space and why it is needed, count the forward passes behind one image, compare SDXL with Qwen Image, explain few-step models, and describe why video generation needs a whole eight-GPU node for a single request.

The lesson builds on How an LLM Generates Text and on the attention lesson. Image models use transformers too, which means the same attention mechanism.

Example 1: the image pipeline and its parts

Image generation models take a text prompt and create an image from it. They slightly predate the public rise of LLMs: closed models from Midjourney and the open Stable Diffusion models were first released in the summer of 2022.

An image model is not one monolithic model like an LLM. It is a pipeline of several models. A foundation model for image generation has three essential components:

flowchart LR
    P["Text prompt"] --> TE["Text encoder"]
    TE --> DN["Denoising model, repeated for 30 to 50 steps"]
    N["Random noise in latent space"] --> DN
    DN --> V["VAE decoder"]
    V --> IMG["Image in pixels"]

The pipeline idea reaches into the surrounding ecosystem. Beyond the base model, image inference often adds:

• LoRAs: lightweight fine-tunes that change style and improve quality.
• ControlNets: outlines and edges that steer the output image to match broad shapes and colors.

The open-source ecosystem around image generation is large. Tools such as ComfyUI build complex pipelines of foundation models and add-ons, and swap components to produce unique outputs.

Example 2: latent space

The whole pipeline works in latent space. An ordinary HD image of 1024 x 1024 pixels has well over a million pixels. The denoising model needs attention over the whole image in parallel, and doing that over every pixel is not feasible. Latent space is a low-dimensional representation of the image. A latent matrix for such an image can be 128 x 128.

pixel space:   1024 x 1024 = 1,048,576 values (per channel)
latent space:    128 x  128 =    16,384 values
share          = 16,384 / 1,048,576 = 0.0156  (about 1.6 percent, roughly one percent)
reduction      = 1,048,576 / 16,384 = 64 times fewer values

Attention cost grows with the square of the number of items it compares (see the previous lesson). Comparing 16,384 latent positions instead of about a million pixels cuts the number of pairs from about 1.1 x 10^12 to about 2.7 x 10^8, which is 4,096 times fewer. The numbers show why latent space makes attention affordable.

The VAE is the part that translates between the two spaces. During training, images enter in overlapping patches of 2 x 2 or 4 x 4 pixels, which are embedded into latent space. During inference the direction is reversed, and the latent space is turned back into pixels when the generation is finished.

Example 3: denoising steps and two forward passes per step

The latent space starts as random values, which is noise. The denoising model refines that noise into an image, over a series of steps, using the text prompt. Every step updates the entire latent space at once. That is different from an LLM, which handles one token at a time. Most image models take 30 to 50 steps to create a high-quality image.

flowchart LR
    S0["Step 0: pure noise"] --> S1["Step 1"] --> S2["Step 2"] --> SD["..."] --> SN["Step 30 to 50: clean latent"]

Each step runs two forward passes: one with conditioning, which means with the text prompt, and one without it. The two results are blended using a guidance scale. So the cost of one image is:

forward passes = steps x 2
50 steps       = 100 forward passes
30 steps       =  60 forward passes

A single image therefore costs up to 100 passes through the denoiser, while an LLM answer of 100 tokens costs 100 decode passes that are much lighter each.

Image requests are controlled by arguments set per request:

Example 4: diffusion transformers and the SDXL versus Qwen Image comparison

The architecture of these models has changed a lot in a few years. Today they are built on transformers, specifically diffusion transformers. A diffusion transformer is much like the transformer in an LLM, but it processes image data (patches) instead of embeddings of discrete tokens.

A clean example of the pipeline is Stable Diffusion XL (SDXL). It is an old model, and its architecture is still useful. SDXL's pipeline has two diffusion models for denoising, a base and a refiner, trained for separate tasks:

• The base model goes from pure noise to a coherent image.
• The refiner model adds detail and checks prompt adherence.

flowchart LR
    T["Prompt"] --> TE["CLIP-based text encoder"]
    TE --> B["Base denoiser: noise to coherent image"]
    B --> R["Refiner denoiser: adds detail"]
    R --> V["VAE: latent to pixels"]

Modern models do much better than SDXL on prompt adherence, accurate faces, hands, and details, legible text, and image-to-image inference (starting from an existing image). Models such as the Qwen Image family follow a similar pipeline, with larger and more capable models at every step.

The improvement comes from larger component models and a more complex pipeline. Legible text and photorealistic faces came from replacing a small NLP (natural language processing) model with a full LLM as the text encoder, and from raising the denoiser size by a factor of five (20B against under 4B). Larger models need more resources. GPUs have also grown more powerful, but engineers cannot count on hardware gains alone to run these models efficiently.

The newest direction. Research now blends the diffusion transformer with LLM architecture. Anything that can be tokenized can be modeled as an LLM. LLMs have text understanding built in, and they already serve as one part of image pipelines. They also solve some problems that are inherent to diffusion models:

HunyuanImage-3.0 is an example of a model with this LLM-style architecture.

Example 5: few-step image generation

The slowest part of image generation is the 30 to 50 denoising steps. Few-step models avoid the problem by using fewer steps instead of making each step faster. They are trained to produce high-resolution images with eight or fewer denoising steps.

standard model:  50 steps
few-step model:   8 steps
steps removed  = (50 - 8) / 50 = 0.84   (84 percent fewer steps)

These models are 80 to 90 percent faster out of the box than traditional image models, and that their output quality is noticeably lower. The numbers agree: 84 percent fewer steps is inside that range.

Two methods create few-step models:

Distillation is more common than latent consistency today. When models such as FLUX and Qwen Image are released, the open image community makes distillations, as well as LoRAs for quality and style. (The model selection lesson defines distillation.) Few-step models fit use cases where latency matters and quality matters less, such as real-time generative filters.

Example 6: video generation

Video generation models are architecturally similar to image models, only bigger. They have three to five times more parameters and encode ten to one hundred times more information in latent space.

image latent       = 128 x 128             = 16,384 values
video latent       = 10 to 100 times that  = 163,840 to 1,638,400 values

The numbers are illustrative multiples of the image example above.

Why not frame by frame? Early video generation worked framewise: make a first frame, use it to make the next frame, and so on. The problem is error accumulation. Small flaws in early frames carry into later ones, and the video goes off course quickly. As a feel for the effect, if each frame inherited and grew its predecessor's error by 2 percent, then after 120 frames (5 seconds at 24 frames per second) the error would be 1.02^120 = 10.8 times its starting size. The 2 percent is illustrative, and the compounding is the point.

Modern video models keep the whole video in latent space and change it at each denoising step. Every frame attends to every other frame and is updated on every forward pass. If the latent space of an image model has two dimensions, X and Y, the latent space of a video model has three: X, Y, and T (time).

flowchart LR
    subgraph F["Framewise (early)"]
        F1["Frame 1"] --> F2["Frame 2"] --> F3["Frame 3: errors have compounded"]
    end
    subgraph W["Whole-video latent (modern)"]
        L["All frames in one latent, every frame attends to every other, updated each step"]
    end

The limits of this approach:

• Videos have a fixed number of frames, just as image models have fixed aspect ratios. Modern models generate sequences of a few seconds.
• The limit on length is compute. Even on the latest GPUs, attention over the huge latent space is extremely expensive, taking several seconds of inference per second of video.
• Video models are so compute-heavy that they typically run with batch size one, so a full node of eight GPUs works on a single request.
• Video models have about the same number of steps as image models, generally around 50. Each step is much more expensive.

Video is a recent modality, and its limits resemble those of LLMs two years earlier:

(TTFT is time to first token and TPS is tokens per second, from the metrics lesson. Ampere and Blackwell are NVIDIA GPU generations, covered in the GPU generations lesson.) Most of these limits are gone for LLMs today. Removing them for cinematic video generation is an active research area, as it is for related fields such as world models and 3D object generation (X, Y, Z dimensions in place of X, Y, T) and other generative rendering models.

One research direction returns to autoregressive generation, in the same way that LLM architecture is being blended into image models. Pure framewise generation has unusable error accumulation, so new techniques such as Self Forcing combine a global view of quality with an iterative way of generating. Adding autoregressive parts to video models can partly address the attention bottleneck. Attention remains the most important and most expensive component. The serving lesson covers the optimization techniques that follow from it.

Interview points

• Image generation is a pipeline: text encoder, denoising model, and VAE. LoRAs and ControlNets extend it, and ComfyUI builds such pipelines.
• The pipeline works in latent space, for example 128 x 128 for a 1024 x 1024 image (about one percent of the pixel values), because attention over every pixel is not feasible.
• The latent starts as noise and is refined over 30 to 50 steps. Each step updates the whole latent space. Each step has two forward passes (with and without conditioning), so 50 steps are 100 forward passes.
• Request arguments: prompt, negative prompt, number of steps, guidance scale (around 4), and image size from a fixed menu.
• Image models use diffusion transformers over patches of 2 x 2 or 4 x 4 pixels.
• SDXL has a base and a refiner. Qwen Image (2025) uses Qwen 2.5 VL (7B) as text encoder, a 20B denoiser (about five times SDXL's, which is under 4B), and dual encoding in the VAE.
• LLM-style image models such as HunyuanImage-3.0 give variable-length output and need fewer forward passes.
• Few-step models use eight or fewer steps, are 80 to 90 percent faster, and have noticeably lower quality. Methods: latent consistency and distillation, with distillation more common. FLUX and Qwen Image have community distillations.
• Video models have three to five times more parameters and ten to one hundred times more latent information. They keep the whole video in X, Y, T latent space to avoid framewise error accumulation.
• Video generation runs at batch size one on an eight-GPU node, at several seconds of inference per second of video, with around 50 steps. Self Forcing mixes global quality with iterative generation.

Summary

• Image models denoise a latent over 30 to 50 steps, with two passes per step. The VAE converts the result to pixels.
• Latent space is far smaller than pixel space, which makes attention affordable.
• Modern image models gain quality from larger denoisers and from LLM text encoders. Their architecture is moving toward LLM-style generation.
• Few-step models trade some quality for speed gains of 80 to 90 percent.
• Video models treat the whole clip as a 3D latent. They are limited by attention compute, run at batch size one, and are an active research area.