Compaction

A long chat with an AI tool sometimes shows a short notice such as "Compacting earlier messages..." and then carries on. Sometimes there is no notice, and the assistant has lost track of a detail from the start of the conversation. Both come from the same cause. The conversation grew too large for the space the model has.

The context window is the fixed limit on how much text a model can take in one request. A conversation that keeps growing will reach that limit sooner or later. Compaction is what has to happen when a conversation outgrows the window it must fit inside: something is removed or shortened to make room.

This lesson shows why conversations grow, the two ways of making room, who carries out the work, and what is lost. After this lesson you can explain what a tool is doing when it compacts, predict which kind of detail may go missing afterwards, and protect the details that matter.

1. What fills the context besides your prompt

The sentence you type is a small part of what the model receives. In a typical request, the context also holds:

The lesson on context listed these parts in full. The point here is their size. Two attached PDF files can be tens of thousands of tokens. A typed question is perhaps twenty.

2. In a chat, the context keeps growing

In a single request, the context has a fixed size. In a chat, it changes with every message. Take a conversation about a rental lease:

[you]    What's the break clause in this lease?

[model]  Clause 14: either party may end the term at 36 months,
         on three months' written notice.

[you]    And what happens to the deposit if we break early?

         (added automatically: deposit-policy.pdf and two pages
          of the tenancy handbook)

A break clause is a part of a lease that lets either side end it early. The person asked two short questions. After the second one, the tool pulled in a policy document and two handbook pages by itself. The context now holds the lease, both questions, the first answer, and the new material.

Each exchange of a message and its reply is called a turn. Every turn adds text, and some turns add far more than what was typed.

3. Every turn sends the whole conversation again

The model keeps nothing between requests, so the tool sends the entire conversation with every new message. The context therefore grows like a stack.

flowchart TB
    L["The limit on what the model can take as input"]
    S["Space that is left"]
    T5["Turn 5"]
    T4["Turn 4, plus a database row"]
    T3["Turn 3, plus two PDFs"]
    T2["Turn 2: question and answer"]
    T1["Turn 1: question and answer"]
    L --- S --- T5 --- T4 --- T3 --- T2 --- T1

Read the stack from the bottom. Turn 1 is a question and an answer. Turn 2 adds another pair. Turn 3 adds two PDF files. Turn 4 adds a row from a database. Each new turn carries all the earlier ones with it, because everything in the stack is sent to the model on every turn. The space left under the limit shrinks each time.

A worked case with made-up numbers:

context window:                    100,000 tokens

after turn 1:                        2,000
after turn 2:                        4,500
after turn 3 (two PDFs added):      61,000
after turn 4 (database row):        78,000
after turn 5:                       96,000

room left:                           4,000

After turn 5, only 4,000 tokens remain, and that space also has to hold the model's next reply. A turn 6 that brings in any new material will not fit. The tool has to do something before it can send the next request.

4. What compaction means

The window cannot be made larger. It is a fixed property of the model. So the only way to continue is to make the content smaller.

That is what the word names: making room by throwing something away. Nothing is added. Something has to leave. Two matters remain open: what goes, and who decides.

There are two main methods. The first cuts from the front. The second writes a summary. In everyday use, "compaction" most often refers to the second, but both make room in the same window.

5. The blunt method: cut the oldest turns

The simplest method is called truncation. To truncate means to cut something short. The tool drops the oldest turns until the rest fits.

Five turns no longer fit. The oldest go first, in the order they arrived. Turns 1 and 2 are removed, and with them go the break clause and the deposit rules.

The weakness is in how the choice is made. The tool drops turns because they are old, not because they are unimportant. Whether they still mattered was never part of the decision. In this example the person is drafting a letter in turn 4, and the letter depends on the break clause from turn 1. The model can no longer see that clause. It may ask for it again, or it may fill the gap with a guess.

Truncation has two advantages: it is instant, and it costs nothing extra.

6. The common method: replace the conversation with a summary

The more common method does not cut. It hands the conversation to a model and asks for a shorter version. Shortening a text while keeping its main points is something language models do well.

What gets sent:

[ turns 1 - 5, in full ]

"Create a detailed summary
 of this conversation and
 all the key insights we
 gained."

What comes back:

<h2 id="summary">Summary</h2>

- Break clause: cl. 14, at
  36 months, 3 months notice
- Deposit returned in full
  if notice is served right
- Drafting a tenant letter;
  tone agreed: formal, brief

The summary is a few lines, and the five turns were many thousands of tokens. Unlike truncation, the summary keeps something from every turn, including the oldest ones: the break clause from turn 1 and the deposit rule from turn 2 are both there.

The summary can be written by the same model that has been answering all along, or by a different one, for example a smaller and cheaper model. In both cases it is one more prompt, sent like any other. This means summarising takes a moment and costs tokens. There is no special mechanism behind it.

7. Who runs it

Compaction is usually not a command the person types. It is housekeeping that the tool does while the conversation continues.

The tool, which earlier lessons called the harness, does three things without being asked. It watches how full the context is. When the context gets close to the limit, it sends the earlier turns off for a summary. Then it swaps the summary in where the turns used to be.

The model that answers your questions does not know this happened. On its next request it receives a summary followed by the recent messages, and it works from that.

Details differ between tools. Some show a notice and some do not. Some also let the person start compaction by hand, which is useful before a new phase of a long task.

8. What the swap achieves

flowchart LR
    subgraph B["Before"]
        direction TB
        b5["Turn 5"] --- b4["Turn 4"] --- b3["Turn 3"] --- b2["Turn 2"] --- b1["Turn 1"]
    end
    subgraph A["After"]
        direction TB
        a2["Room again<br/>for everything still to come"] --- a1["The summary"]
    end
    B -- "compaction" --> A

Before compaction, five turns fill the window. After it, a short summary sits at the bottom and most of the window is free for the turns still to come.

The freed space fills up again as the conversation continues. When it does, the same thing happens again: the earlier summary and the newer turns are summarised together into a new summary. In a very long session this can happen many times, and each round is a summary of material that already included a summary.

9. What a summary loses

A summary is shorter than the original, so some of the original is gone. The technical word for this is lossy. Look closely at turn 2 of the example.

The original answer gave four conditions. The summary gives one sentence. That sentence is true, and it is far shorter. The four conditions underneath it are not coming back.

From this point on, the model no longer has the conditions. If the person asks "does the notice have to go to a specific address", the model sees only "served correctly". It may say it cannot tell, or it may produce a plausible answer that is not based on the document, which is a hallucination.

Repeated compaction makes this worse. A summary of a summary loses detail at each step.

Some practical consequences follow:

• Exact details such as numbers, names, conditions, and exact wording are the most likely to be lost. Decisions and conclusions are the most likely to survive.
• After a compaction, important specifics may need to be stated again or the source file attached again.
• Details that must survive a long task belong somewhere more durable than the conversation, such as a memory file or a document in the project.
• Starting a fresh session for a new task avoids carrying a long, compacted history.

10. The trade-off between the two methods

Both methods forget. They differ in how the forgetting is chosen.

Choosing what to forget usually keeps more of what matters than forgetting by position. Truncation drops whatever happens to be oldest. Summarisation is asked to keep what matters: the decisions, the results, and the conclusions. This is why many chat tools and agents use summaries as their main method, sometimes with truncation as a fallback.

Interview points

• Compaction makes room in a full context window by removing or shortening earlier content. The window is fixed, so content has to shrink.
• A chat context grows because every turn is added and the whole history is sent again on each request, together with attached files, database rows, and retrieved sources.
• Truncation drops the oldest turns. It is free and immediate, but it chooses by age and not by importance.
• Summarisation sends the earlier turns to a model with an instruction to summarise and replaces them with the result. It preserves the main points of all turns in a shorter form.
• The summary can come from the same model or a different one. It is an ordinary extra request that costs time and tokens.
• The harness runs compaction automatically when the context nears its limit. The answering model only sees the result.
• Summaries are lossy. Specific conditions, numbers, and wording are dropped while general conclusions survive, and repeated compaction compounds the loss.
• After compaction a model may answer from a vague summary line and hallucinate the missing detail.
• Details that must persist are safer in memory files or project documents than in conversation history.
• Summarisation is often preferred over truncation because the choice of what to forget is deliberate.

Summary

• A conversation grows with every turn until it reaches the context window limit.
• Compaction makes room by removing or shortening earlier content.
• Truncation cuts the oldest turns, whether or not they still matter.
• Summarisation replaces earlier turns with a short summary written by a model.
• The tool does this automatically, often with only a brief notice.
• A summary keeps conclusions and loses detail.
• Important specifics should be restated or stored outside the conversation.