Caching

A support chatbot has a 14-page returns policy pasted into every request. A customer asks a question and then asks a second one. For the second question, the model receives the same 14 pages again, plus the first question, plus the first answer, plus one new sentence.

A model does not look answers up in a table. It computes every answer from the beginning, and the computing is the expensive part of inference, the stage where a finished model is used to answer requests. So reading the same 14 pages twice means paying for the same work twice.

Caching is the technique that avoids this. In computing, a cache is a store of results that were already worked out, kept so that the same work does not have to be done again. This lesson shows what one request costs the machine, what is stored, how a follow-up uses the store, and what the term means for AI models. After this lesson you can explain why a long chat with caching is cheaper and quicker than the same chat without it.

1. What one request costs the machine

To see what caching saves, first look at what a single request asks the machine to do. There are three costs.

The first cost is fixed. A model has billions of numbers, and all of them must be loaded before the first request arrives. They stay loaded between requests.

The second cost grows with the length of what you send. A layer is one stage of the neural network, and a model has many stages stacked one after another. Every token of your input goes through all of them. A 14-page document is thousands of tokens, so it is thousands of trips through the stack.

The third cost grows with the length of the answer. The model writes one token, then runs again to write the next, and so on.

Caching targets the second cost. The input is the part that repeats from one request to the next.

2. Caching makes the model do less, not work faster

A common misreading is that caching speeds the model up. It does not. The model computes each token at the same speed as before. Caching reduces how many tokens it has to compute.

An everyday comparison: a cook who makes a large pot of stock on Monday does not chop faster on Tuesday. On Tuesday the cook starts from the stock that is already made and only prepares what is new. In this comparison the stock is the stored work, and the new ingredients are the new text. The comparison stops being accurate at one point: stock can go into many different dishes, while cached work can be reused only by a request that begins with exactly the same text.

When people talk about "optimising inference" through caching, this is what they mean: the same model, the same machine, less repeated work.

3. What goes into the cache

After a request is finished, two things are kept on the provider's server, and they are kept as a pair.

The input, exactly as it arrived:

system   Answer questions about our returns policy.

policy   [ returns-policy.pdf, 14 pages, pasted in full ]

user     My parcel arrived torn.

The output that was generated for it:

assistant   Sorry about that. A replacement can go out
            to the same address. Shall I start that?

The input here has three parts: a system prompt with the rule for the chatbot, the full policy document, and the customer's message. The output is the model's reply. The output is stored beside the input that produced it, because the two belong together. The reply only makes sense as the continuation of that exact input.

One detail matters for understanding how the saving works. What the server keeps is not only the words. It keeps the results of the work the model did on those words: the internal numbers the network produced while reading each token. Those numbers are what took the time to compute, so those numbers are what is worth keeping.

The phrase "exactly as it arrived" is strict. The cache matches text piece by piece from the start. If one word near the beginning changes, everything after that word no longer matches.

4. A follow-up reuses the stored work

Now the customer sends a second message, for example "Can it go to a different address". As the lesson on context explained, the app sends the whole conversation again with every message, because the model keeps no memory between requests. So the second request is mostly the same text as the first.

flowchart TB
    subgraph R1["First request"]
        direction LR
        A["System prompt, policy,<br/>first question<br/>(computed in full)"] --> B["First answer<br/>(generated a token at a time)"]
    end
    R1 --> S["Kept on the server:<br/>the input and the answer"]
    S --> R2
    subgraph R2["Follow-up request"]
        direction LR
        C["The same text again<br/>(loaded from the cache,<br/>not run through the model)"] --> D["New question<br/>(computed)"] --> E["Updated answer<br/>(generated)"]
    end

The top box is the first request. Nothing is in the cache yet, so the system prompt, the policy, and the question are computed in full, and the answer is generated one token at a time. Both are then kept on the server.

The bottom box is the follow-up. Its first part is exactly the same text as before. The server recognises it and loads the stored work back. That part is not run through the model again. Only the new question at the end goes through the network, and then the model generates the updated answer.

A worked case with made-up numbers shows the size of the saving:

first request
  system prompt + policy + question:   9,000 tokens   computed
  answer:                                  40 tokens   generated

follow-up request
  same text as before:                  9,040 tokens   loaded from the cache
  new question:                            12 tokens   computed
  updated answer:                          45 tokens   generated

Without caching, the follow-up would push 9,052 input tokens through the network. With caching, it pushes 12. The long unchanged beginning is the work that was skipped, and the longer that beginning is, the more is saved.

When the stored work can be reused, this is called a cache hit. When nothing stored matches, it is a cache miss, and the request is computed in full as if caching did not exist.

Two results follow from a cache hit. The reply starts sooner, because the model has less to read. The request also costs less, because providers usually charge a lower price for input tokens that were served from the cache. Cached tokens are not free: they are still counted and billed, at the reduced price. Some providers also charge a little extra on the request that first writes an entry into the cache. The answer itself is still generated token by token at the normal price.

5. Deciding what to keep and for how long

The principle is short: work that has already been paid for should not be paid for a second time. Doing this well in a real service is harder, and the difficulty is in two decisions.

What to keep. Storing the work for a request takes memory on the server. Not every request is worth storing. A one-off question that nobody will follow up on gains nothing from being cached. A long system prompt or document that starts thousands of requests gains a lot.

For how long. A cache cannot grow forever. Entries are removed after some time without use. If an entry is removed too early, the next follow-up is a cache miss and is computed in full. If entries are kept too long, the server fills with work nobody will reuse. At the time of writing, a cached prompt is commonly kept for a few minutes after its last use, and some services offer longer lifetimes. The details differ between providers.

These decisions lead to a practical rule for anyone who builds on a model. Put the parts that do not change, such as the system prompt, the tool descriptions, and reference documents, at the beginning of the request. Put the parts that change, such as the newest message, at the end. The cache matches from the start, so a stable beginning gives the longest reusable part. Placing something that changes every time, such as the current time, at the top of the prompt breaks the match for everything after it.

Some providers apply caching automatically. Others ask the developer to mark which part of the prompt should be cached. Check the provider's current documentation, because this changes.

6. What caching means for AI models

With the engineering details removed, caching comes down to three actions.

One clarification helps avoid confusion. The caching in this lesson reuses work for the unchanged beginning of a request and still generates a fresh answer. This is often called prompt caching. A different and simpler kind, often called response caching, stores a complete answer and returns it again when exactly the same request arrives, without running the model at all. That kind helps only when requests repeat word for word, which is rare in a conversation. It is usually built by the application's developer, not by the model provider.

Interview points

• A model computes each answer from scratch. Inference cost has three parts: holding the parameters in memory, passing every input token through every layer, and running the network once per output token.
• Caching does not change the model, the hardware, or the cost of each token that is computed. It reduces the number of tokens that have to be computed.
• After a request, the provider keeps the input exactly as it arrived together with the output generated for it, including the internal results of processing them.
• A follow-up request repeats the earlier conversation. With caching, the repeated beginning is loaded from the store and only the new tail goes through the network.
• Cache matching starts at the first token and stops at the first difference, so stable content belongs at the start of a prompt and changing content at the end.
• A cache hit lowers latency and cost, and providers usually bill cached input tokens at a reduced rate. Output tokens are still generated and billed normally.
• The hard part is the policy: which requests to store and how long to keep them. Entries expire, and an expired entry means the next request is computed in full.
• Prompt caching reuses work for a shared prefix and still generates a new answer. Response caching returns a stored answer for an identical request. They solve different problems.

Summary

• Answers are computed, not looked up, and the computing is the expensive part.
• A cache stores work that was already done.
• The input and its output are stored together on the server.
• A follow-up reuses the stored work for the unchanged beginning and computes only the new part.
• Caching makes replies quicker and cheaper by making the model do less.
• What to keep and for how long are the difficult decisions.