KV Caching, Prefix Reuse, and Long Context

A language model reads a prompt and then writes an answer token by token. To write each new token, attention needs information about every earlier token. Recomputing that information from scratch for every new token would be extremely slow, so the engine stores it. The stored data is the KV cache (key-value cache): for each token, the key and value vectors that attention computed. Every inference engine uses it by default within a single request.

This lesson is about getting more value from that cache. The first idea is reuse between requests, so that a prompt that starts like an earlier prompt skips most of its work. The second is where to keep a cache that is far larger than GPU memory allows. The third is how to send requests to the machine that already holds the right cache. The last is what to do when a single prompt is so long that its cache causes trouble by itself. Afterwards you can estimate the size of a cache, structure prompts so that they hit the cache, and choose where the cache lives.

Terms from earlier lessons. Prefill is the phase where the model reads the whole input at once and builds the cache. Decode is the phase where it generates tokens one at a time, adding one entry to the cache per token. TTFT (time to first token) is dominated by prefill. VRAM is the memory on the GPU itself. Attention and the KV cache are explained in Attention and Mixture of Experts and Optimizing Attention.

Without the KV cache, LLM inference would be unbearably slow, because every earlier value in the whole sequence would be recalculated for each new token. Since each new token's value depends on every previous token, the cache is the natural thing to keep. The question this lesson asks is what else can be done with it.

Example 1: how big is a KV cache

The cache is big, and the size drives every decision below. For each token and each layer, the model stores one key vector and one value vector for each KV head. The size per token is:

bytes per token = 2 (key and value) x layers x KV heads x head dimension x bytes per number

A small case by hand with a Llama-style 70B shape (80 layers, 8 KV heads, head dimension 128) at 16 bits (2 bytes): 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, about 0.31 MB. These parameters are an illustration chosen for the calculation. Real models differ, and the figures for a given model come from its configuration. In words: every token of context costs about a third of a megabyte, so a long conversation costs gigabytes. The table below scales this up (tokens x bytes per token) and shows the effect of storing the cache in 8 bits (see Quantization).

bytes per token, 16-bit cache: 327680
bytes per token, 8-bit cache : 163840
  4,000 tokens:    1.3 GB (16-bit)     0.7 GB (8-bit)
 32,000 tokens:   10.5 GB (16-bit)     5.2 GB (8-bit)
128,000 tokens:   41.9 GB (16-bit)    21.0 GB (8-bit)
tokens that fit in 64 GB: 195,312 (16-bit), 390,625 (8-bit)

One 128,000-token conversation holds tens of gigabytes of cache at 16 bits. Halving the bytes per value doubles the number of tokens that fit in the same space. This is the benefit of KV cache quantization: the engine can keep more of the cache in memory and read it faster, which also gives extra gains for prefix caching and disaggregation (see Prefill-Decode Disaggregation).

Example 2: prefix caching

Take two prompts that each have four tokens on most tokenizers:

By default, the engine runs prefill on all four tokens of each prompt. But "Weather in" is the same in both. A cache entry depends only on the tokens before and including it, so the entries for "Weather" and "in" computed for prompt A are exactly right for prompt B too. With prefix caching, the engine reads those two entries from the stored cache and skips prefill for them. This improves TTFT for the second request.

This is also the reason pay-per-token APIs charge less for "cache hit" input tokens than for "cache miss" tokens: reusing cached tokens takes little compute and little time. The same principle lowers latency and cost on your own deployments.

Saving two tokens is not much. But prefix caching can skip prefill on thousands of tokens in these domains:

• Complex system prompts: agents, customer-facing chatbots, RAG scaffolds (RAG means retrieval-augmented generation, where retrieved documents are put in the prompt), and tool calls often have long system prompts on every call.
• Code completion: code completion and code generation pass the same thousands of lines of code as shared context.
• Documents and retrieval: summarization, question answering, and retrieval add repeated context before the user's question.
• Multi-turn conversations: each turn repeats all earlier messages in the chat template, so savings grow with every turn.

The reusable part is a prefix: it starts at the first token and ends at the first token that is not repeated. In the weather example, the fourth token (a question mark) is shared, but the prefix already ended at "SF" versus "NYC", so the fourth token is not read from the cache.

Because the prefix ends at the first unique token, the order of your prompt decides how much you save. Compare a second version of the same two prompts:

Here nothing is saved. The first token differs, so the prefix has length zero, even though the last three tokens are identical. The practical rule: put novel tokens as late in the context as possible, and stable content (system prompt, instructions, documents) first. The result below counts the shared prefix for both versions, and for a realistic agent prompt (a 3,000-token system prompt followed by a 40-token question, against the question first).

novel token in the middle  prefill 2 of 4 tokens, reuse 2
novel token first          prefill 4 of 4 tokens, reuse 0
system prompt first: reuse 3000 of 3040 tokens (98.7%)
question first     : reuse 0 of 3040 tokens (0.0%)

The first two lines have the same tokens in a different order, with a very different result: 2 of 4 tokens are reused, then none. In the realistic case with a long system prompt first, about 98 percent of the prompt is reused.

Prefix caching is the dominant form of KV reuse because LLMs are autoregressive. Each token influences every later token, so one different token changes the internal representation of everything after it, even when the later text looks identical to a human reader. Research on other kinds of reuse exists. Caching arbitrary pieces from the middle of a prompt requires fixing the positional embeddings (the numbers that tell the model where each token sits) and selectively recomputing some KV entries to keep quality. Tools such as CacheBlend and LMCache support non-prefix sequences and widen what can be reused.

Example 3: where to store the KV cache

The cache is valuable, but it takes lots of memory and a GPU has limited VRAM. Engines let you set how much memory goes to the cache. In TensorRT-LLM (see Inference Engines) the setting is a fraction of the free GPU memory, written as a configuration line (an illustration of the setting, not a runnable example):

kv_cache_config.free_gpu_memory_fraction = 0.8
# or any value 0.0 < n < 1.0

A worked case: a B200 GPU has 180 GB of VRAM. Model weights and buffers use 100 GB, so 80 GB is free. Setting the fraction to 0.8 gives the cache 0.8 x 80 = 64 GB. When this space fills up, and it fills quickly, saved caches must be deleted, so later requests are more likely to miss.

To get more room, move cache entries to other nearby storage (offloading). There are four levels, listed from highest to lowest bandwidth to the GPU:

Some machine types, such as the GB200, have CPUs and interconnects that make G2 much faster, so they are well suited to KV cache offloading. NVIDIA Dynamo supports offloading through KVBM (KV Block Manager), which provides APIs for moving KV cache blocks between memory levels. The general rule is to keep the most frequently used blocks in higher-bandwidth memory and leave the less-used blocks in slower storage until they are needed.

The next calculation shows what the levels mean in time. Formula: time = size / bandwidth. It uses one illustrative speed per level taken from inside the ranges in the table (the real values depend on the hardware), for a 4 GB cache. By hand for G2: 4 GB / 100 GB per second = 0.04 s = 40 ms.

time to load a 4.0 GB cache (illustrative speeds)
G1 GPU VRAM      4000.0 GB/s ->       1.0 ms
G2 CPU RAM        100.0 GB/s ->      40.0 ms
G3 local SSD        7.5 GB/s ->     533.3 ms
G4 network SSD      2.0 GB/s ->    2000.0 ms

The G1 load takes about a millisecond and the G4 load takes seconds. A load from G3 or G4 can still be much cheaper than recomputing the prefill of a very long prompt, but it is slower than a hit in VRAM, which is why the most-used blocks stay near the GPU.

Example 4: cache-aware routing

A production system runs many copies of the inference server, called replicas, and splits incoming traffic across them. The usual rule sends each request to the least busy replica. If the server relies on prefix caching, that rule wastes the cache. A user in a long chat, or a user asking many questions about one codebase, should reach the same replica again, so that the earlier turns are a cache hit and the request is faster and cheaper.

graph LR
    U1["User 1 turn 1"] --> R1["Replica 1: caches user 1"]
    U1b["User 1 turn 2"] --> R1
    U2["User 2 turn 1"] --> R2["Replica 2: caches user 2"]
    U2b["User 2 turn 2"] --> R2

Cache-aware routing assigns traffic based on what each replica has in its KV cache, instead of spreading requests evenly. The worked comparison below covers two policies on 4 users with 4 turns each, 3 replicas, and a prompt that grows by 200 tokens per turn from a 500-token start. Every turn includes the whole earlier conversation. Round robin sends request number i to replica i mod 3, and cache-aware routing sends user u always to replica u mod 3. A turn is a hit only if the replica handling it already served that user.

prefill tokens, round robin : 10800
prefill tokens, cache-aware : 4400
saved: 59%

Round robin sends the turns of one user to different replicas, so most turns miss and prefill nearly the whole prompt again. Cache-aware routing keeps each user on one replica, so every turn after the first only prefills the 200 new tokens. The simulation ignores the load on each replica, which a real router also has to balance.

Another option is the G4 networked storage as a global KV cache shared across replicas. Routing still matters, since a replica with a hot G1 cache serves the request faster than one reading from G4. But a global cache means any replica can eventually reach any computed sequence, and cached sequences are not lost when nodes are replaced or shut down during autoscaling (see Autoscaling, Batching, Cold Starts, and Routing).

Example 5: long context

"Long context" is a circular definition: a sequence is long context once its KV cache is big enough to cause problems during inference. Depending on the model, hardware, engine, and traffic, the problems can start past common cutoffs such as 32K, 64K, or 128K tokens. A good benchmark sends very large inputs on purpose to test the service against long-context requests (see Benchmarking, Load Testing, and Profiling).

Foundation model labs use scaling techniques such as RoPE (rotary position embeddings, a way of encoding token position) to give models longer and more accurate context windows. Supporting those windows creates new inference challenges. The cost of attention, once the KV cache is accounted for, grows linearly with sequence length. With long sequences, attention can become the main user of VRAM, which is the very resource that decode is limited by.

Some model designs address this per model: sliding window attention (each token looks only at a recent window), compressed attention, and sparse attention (each token looks at a chosen subset). Beyond those, three general techniques improve standard attention:

The count below shows the idea of chunked prefill. Formula: passes = ceiling(prompt tokens / chunk size) = ceiling(100,000 / 8,192) = 13. A 100,000-token prompt arrives while an engine is also decoding for other users. Without chunking, one pass handles all 100,000 tokens. With chunks of 8,192 tokens, each pass handles one chunk, leaving room for decode work between chunks.

one block: 1 pass of 100000 tokens (decode waits for all of it)
chunked  : 13 passes, each at most 8192 tokens
  pass 1: 8192 tokens, 91808 left
  pass 2: 8192 tokens, 83616 left
  ...
  pass 13: 1696 tokens, 0 left

If, after these techniques, the cache still needs more VRAM than one GPU has, the next step is to spread inference across several GPUs, covered in Model Parallelism.

Interview points

• Every engine caches keys and values within a request. Prefix caching extends that across requests: the shared prefix is read from cache, so prefill and TTFT shrink. Pay-per-token APIs price cache hits lower for this reason.
• A prefix ends at the first token that differs. Put stable content first and novel tokens late. A different first token means no reuse at all.
• Autoregression means one different token changes everything after it, so prefix reuse dominates. CacheBlend and LMCache handle non-prefix reuse by correcting positions and selectively recomputing entries.
• The cache can be configured as a share of free VRAM (for example 0.8 of what remains after weights). When full, entries are evicted and misses rise. Four levels: G1 VRAM, G2 CPU RAM, G3 local SSD, G4 networked SSD, with falling bandwidth and rising size. NVIDIA Dynamo KVBM moves blocks between levels. GB200-class machines have fast G2.
• Cache-aware routing sends requests to the replica that holds their prefix. A global cache on G4 prevents loss during autoscaling but is slower than a G1 hit.
• Long context means a cache large enough to cause problems, often past 32K, 64K, or 128K tokens. Use FlashAttention, PagedAttention, and chunked prefill, then parallelism if a single GPU is not enough.
• KV cache quantization stores more of the cache and reads it faster, but its errors compound across tokens.

Summary

• The KV cache holds each token's keys and values, so attention does not recompute them. Its size per token is 2 x layers x KV heads x head dimension x bytes per value.
• Prefix caching reuses the cache of the longest shared prefix between requests and skips its prefill. It helps system prompts, code, documents, and multi-turn chats.
• Prompt order decides savings, because the prefix stops at the first non-repeated token.
• The cache can live on four levels: GPU VRAM (G1), CPU RAM (G2), local SSD (G3), and networked SSD (G4). Dynamo KVBM moves blocks between them.
• Cache-aware routing keeps repeat users on the replica that holds their cache, and a global G4 cache survives replica changes.
• Long contexts are handled with sliding window, compressed, or sparse attention in some models, and with FlashAttention, PagedAttention, and chunked prefill in general, before multi-GPU parallelism.