Prefill-Decode Disaggregation

Serving a prompt has two phases that behave in opposite ways. Prefill reads the whole prompt at once and builds the KV cache (the stored keys and values of every token). It needs a lot of arithmetic. Decode then writes the answer one token at a time, and it spends most of its time reading memory. Because the phases need different things, running both on the same GPUs makes them get in each other's way under heavy traffic.

Disaggregation (also called disaggregated serving) puts prefill and decode on separate engines, on separate GPUs or even separate nodes. This lesson explains the three ideas it combines, the step-by-step flow, how to decide whether it is worth the extra hardware, and how NVIDIA Dynamo runs it with changing traffic. Afterwards you can say when to use it, when not to, and what the new bottlenecks are.

Terms from earlier lessons. Compute-bound means limited by arithmetic speed. Memory-bound means limited by memory bandwidth (Finding the Bottleneck). TTFT (time to first token) is the wait before the first output token, mostly set by prefill. TPS (tokens per second) is the decode speed. TP is Tensor Parallelism, splitting each layer over several GPUs (Model Parallelism). ISL is input sequence length, the number of prompt tokens.

Example 1: the three ideas behind disaggregation

Disaggregation combines three ideas:

1. Prefill is compute-bound and determines TTFT. Decode is memory-bound and determines TPS.
2. Specialization improves performance everywhere, from kernel selection (choosing the GPU routines) to tuning engine parameters.
3. A model can be served in parallel over several GPUs or nodes, as long as slower links between them do not become bottlenecks.

Ideally prefill uses mostly compute and decode uses mostly memory, so the two co-exist on one node without conflict. But with larger batches and more compute-heavy optimizations, they begin to compete for the same resources. A request that arrives with a long prompt takes compute that the decode of other requests needed, and those users see their tokens slow down.

An analogy: a restaurant kitchen where one cook both preps ingredients (heavy, bursty work) and plates dishes at the table (steady, time-sensitive work). Separating the prep station from the plating station lets each be organized for its own job. The analogy stops at one point: in the kitchen, ingredients are carried by hand, while in disaggregation the thing handed over is the KV cache, which is large and must move over the hardware interconnect.

Example 2: how disaggregation works

Disaggregation turns inference into a three-step process:

1. The prefill engine takes the input sequence, builds the KV cache, and computes the first token.
2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine.
3. The decode engine computes all later tokens.

graph LR
    R["Request"] --> P["Prefill worker: builds KV cache, first token"]
    P -->|KV cache over interconnect| D["Decode worker: all later tokens"]
    D --> O["Response"]

In conditional disaggregation, the request goes first to the decode engine. That engine checks whether the input is already cached or short enough to handle locally:

1. If yes, the decode engine does the prefill itself and skips disaggregation.
2. If no, it hands the request to the prefill engine, which then follows the three-step flow.

Conditional disaggregation is better for real traffic, because many requests are short or hit the prefix cache (KV Caching), and moving those would cost more than doing them on the spot.

A worked case: suppose the threshold is 1,000 uncached tokens. A chat message with a 3,000-token system prompt that is already cached has 40 uncached tokens, so the decode engine handles it locally. A first request with 20,000 new tokens of code exceeds the threshold, so it goes to the prefill engine.

request                         uncached tokens   threshold 1,000   route
chat turn, prompt cached        40                below             decode engine, local prefill
new code review, no cache       20,000            above             prefill engine, then KV transfer
short question, no cache        300               below             decode engine, local prefill

Another benefit is that each engine can be tuned alone, and the system as a whole too. For example, the compute-bound prefill engine needs a lower TP than the memory-bound decode engine.

Example 3: when to use disaggregation

Disaggregation is very effective, but it needs multiple GPUs and extra engineering. Use it only when all of these hold:

1. Volume: a large amount of traffic, starting around one hundred million to one billion tokens per day, depending on model size.
2. Model size: a large model, at least a hundred billion parameters.
3. Traffic shape: prefill-heavy traffic with long input sequences.

If either volume or size is missing, the extra hardware likely wastes money for small gains. If the traffic is not prefill-heavy, the extra GPUs are probably better spent on more replicas (horizontal scaling), because decode engines work efficiently for short sequences or prefix cache hits.

A good use case is serving a frontier LLM inside a code editor, where many developers at once send large and varied chunks of code as context. Huge token counts, mostly prefill, on a trillion-parameter model are the textbook workload for disaggregation.

Example 4: dynamic disaggregation with NVIDIA Dynamo

NVIDIA Dynamo provides production-ready support for disaggregation and can handle mixed real-world traffic. Its tools and pre-built optimizations include:

• A prefill queue that holds requests when all prefill engines are busy.
• Support for conditional disaggregation, with prefill routing decided by configurable thresholds on the ISL that remains after the prefix cache, and on the prefill queue size.
• Efficient KV transfer from prefill to decode engines using NIXL (NVIDIA's transfer library), including a kernel that transposes KV blocks between memory layouts when the two engines use different TP settings.

Together these allow dynamic disaggregation: the number of prefill and decode engines is set at run time and can be changed as traffic changes.

The ratio does not have to be one to one. Real systems have several of each. The count is written xPyD: for example 5P3D means five prefill engines and three decode engines serving one model deployment. If the traffic has long prompts and short answers, the system can shift toward more P. For short prompts and long answers it can shift toward more D.

graph LR
    Q["Prefill queue"] --> P1["P1"]
    Q --> P2["P2"]
    Q --> P3["P3"]
    P1 --> D1["D1"]
    P2 --> D2["D2"]
    P3 --> D1

More parts bring more places to get stuck. With disaggregation there are two new bottlenecks:

• Prefill queue size. The queue must not grow too long. Two controls help: set a reasonable threshold for local prefill on the decode engine, and change the xPyD ratio at run time to give prefill more resources.
• KV cache on the decode engines. Under high load the decode engines can run out of cache space. Quantization and KV cache offloading increase the availability of cache (see Quantization and KV Caching).

This ties back to the general principles of the chapter: more constraints (an engine that only does one phase) and more traffic (enough to keep many engines busy) allow more optimization.

Interview points

• Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPS. On one node under heavy load they compete, which is what disaggregation removes.
• The flow has three steps: prefill engine builds the KV cache and first token, the cache moves over the interconnect, the decode engine produces the rest. The KV transfer is the added cost.
• Conditional disaggregation checks at the decode engine whether the prompt is cached or short, and prefills locally if so. It suits real traffic better than always disaggregating.
• Use it only with large volume (about 100 million to 1 billion tokens per day), a large model (100B parameters or more), and prefill-heavy traffic. Otherwise add replicas.
• Each engine can be tuned separately, for example prefill with a lower TP than decode.
• Dynamo supplies a prefill queue, threshold-based conditional routing, and NIXL-based KV transfer with a layout-transposing kernel for mismatched TP. The xPyD notation (5P3D) describes the engine counts, which can change at run time.
• New bottlenecks: prefill queue length (fix with the local-prefill threshold and xPyD changes) and decode KV cache space (fix with quantization and offloading).

Summary

• Disaggregation runs prefill and decode on separate engines so each can use hardware and settings suited to its bottleneck.
• Prefill builds the KV cache and first token, the cache is sent to a decode engine, and decode writes the rest.
• Conditional disaggregation keeps short or cached requests on the decode engine.
• It pays off for large models, large volumes, and prefill-heavy traffic such as code editors, and wastes money otherwise.
• Dynamo adds a prefill queue, configurable routing thresholds, NIXL KV transfer, and runtime xPyD ratios.
• Watch the prefill queue size and the decode engines' KV cache space.