Context Rot

A long working session with an AI tool often goes well for the first twenty minutes and then gets worse. The assistant brings back an idea that was rejected an hour ago. It mixes up two documents. Its answers become vague. Nothing was changed in the model, and the context window is not full.

The cause is called context rot. A context window has a size, and the previous lesson on compaction dealt with what happens when the size runs out. What sits inside the window also has a quality, and quality is the harder of the two problems. A context can be well under its limit and still be in poor condition.

This lesson shows how a context goes bad during normal work, explains the two separate causes, and gives three habits that prevent it. After this lesson you can recognise context rot, explain why adding more material often lowers answer quality, and decide what to put into a prompt and what to leave out.

1. Twenty minutes into one session

Here is a short working session. A few terms first. A spec, short for specification, is a document that describes what should be built. A sprint is a fixed working period of a week or two. An importer is a program that loads data from files. CSV is a common file format for tables. A vendor is an outside company that supplies a product or service.

[you]    Here's the spec. Can we ship the importer this sprint?

[model]  Yes, if you drop the CSV branch.

[you]    Dropping CSV. Attaching vendor-integration.pdf.

[model]  That PDF is for the previous vendor. None of it applies here.

[you]    Fine. What's left to do?

That is five turns. Look at what they contain:

• Two turns are settled business. The CSV part was discussed and dropped. Nothing about it needs to be read again.
• One turn carries a document that nobody needs. The PDF belongs to a different vendor.

All five turns will be sent to the model again on turn six, because the whole conversation is sent with every request. The dropped idea and the wrong PDF go along each time.

2. What a later turn sends

Nothing falls out of a conversation between turns. The request is rebuilt from the whole history every time. Suppose the session continues to turn twelve. This is what the request contains:

A retry idea is a plan to try a failed action again. An endpoint is an address that a program sends requests to, and v1 means its first version.

Two rows are doing work: the spec and the current question. The other four are old or wrong, and the model has now read each of them on every turn since they appeared.

These four rows are of three kinds. Stale content was useful earlier and is not any more, such as the dropped CSV branch and the dropped retry idea. Irrelevant content never applied, such as the PDF for the wrong vendor. Wrong content is a mistake that is still on the page, such as the earlier answer about the v1 endpoint. Wrong content does the most harm, because the model may repeat the mistake as if it were a fact.

3. How old content pushes the model the wrong way

The model has no way of knowing which lines are dead. Nothing in the context is marked "ignore this". The model reads every line and gives every line some weight.

On turn twelve you ask:

What's left to finish
this sprint?

What comes back:

The CSV branch still needs
a test, and the vendor PDF
mentions a retry window you
have not handled yet.

Both items in this answer were dropped long ago. The CSV branch was cancelled on turn 2. The vendor PDF was for the wrong vendor. The model was not being careless. It found CSV work and a retry window in its context, both looked like unfinished tasks, and the question asked for unfinished tasks.

flowchart LR
    A["Still needed<br/>the spec<br/>the current question"] --> M["The model<br/>weighs every line"]
    B["Stale or wrong<br/>dropped CSV branch<br/>dropped retry idea<br/>wrong vendor PDF<br/>wrong endpoint answer"] --> M
    M --> O["An answer shaped<br/>by both groups"]

The diagram shows both groups of content flowing into the model with nothing to separate them. The answer is shaped by both. There is a second cost as well: the dead content takes up room in a window that has a limit, and it is paid for as input tokens on every turn.

This is context rot in its first form. The word "rot" fits because the content was fine when it arrived and went bad while sitting there.

4. Two different causes

Clearing out the dead content is part of the fix, and it is not the whole fix. Two different failures share the name context rot.

Cause one is clutter. Stale, irrelevant, and wrong content misleads the model, as section 3 showed.

Cause two is length. Even when every line in the context is relevant, a very long context gives less precise answers than a short one. The model has to pick out the few lines that answer the question from a much larger body of text, and it does this less reliably as that body grows. Details from the middle of a long context are more likely to be overlooked or blended together.

The second cause surprises people, because the window may be nowhere near full.

5. The same question in a short and a long context

Here is one question asked two ways. In both cases the context is well under the advertised limit.

In a short, focused context:

Q  Which vendor did we
   settle on?

A  Northwind. It is in the
   decision note you gave me.

The same question, buried in a long context:

Q  Which vendor did we
   settle on?

A  Northwind, or possibly
   Calder - the notes point
   both ways. Contract ref
   ends 0447.

The names are made up for the example. Three things changed in the second answer:

1. It is vaguer. One clear answer became two candidates.
2. It is hedged. "The notes point both ways" reports confusion that the long context created. Somewhere in the material, Calder was mentioned as an option, and the model could not tell which mention was the final decision.
3. It contains an invented detail. The contract reference ending in 0447 was in none of the material the model was given. It is a hallucination.

The decision note was present in both contexts. In the long one it was surrounded by so much other text that the model no longer used it cleanly.

6. Being under the limit does not mean being safe

It is tempting to treat the context window like a fuel gauge: as long as there is room left, everything is fine. Room is not the only thing that runs out. Quality runs out earlier.

In general, the more of the window is filled, the less reliable the answers become, and the more likely the model is to invent details. The drop is gradual. It does not begin at the limit. How early it begins and how steep it is differ from model to model and from task to task, and newer models tend to hold up better than older ones.

This also explains why the limit exists at all. There are two reasons:

• Cost. Processing a long context takes a lot of computation, and the provider has to do that work on every turn.
• Quality. Past a certain length, results get much worse. A limit keeps requests out of the range where the model performs poorly.

The limit also reflects the input lengths the model was built and trained to handle.

So an advertised window of, say, 200,000 tokens states how much the model accepts. It does not promise that the model works as well at 190,000 tokens as at 10,000.

7. Three habits that prevent it

Three working habits keep a context in good condition.

Start the session again

When a session has become long and cluttered, open a new one and carry over only the conclusions. A first message for the new session can look like this:

New session. Where we are: vendor is Northwind,
CSV branch dropped, retries off. What is left for
the sprint?

Three short facts replace an hour of discussion. The rule is to carry the conclusions across and leave behind the route taken to reach them. The model does not need to know that CSV was debated. It needs to know that CSV is out.

Compact on purpose

Compaction replaces earlier turns with a summary. Many tools do it automatically when the window is nearly full, which is the latest possible moment. Where the tool allows it, start compaction by hand at a natural break in the work, before the limit forces it.

Compacting early means the summary is written while the context is still in good condition, and the following turns start from a short, clean base.

Decide before pasting

The third habit applies before anything enters the context. For each item you are about to attach or paste, ask one thing: would leaving this out change the answer.

If leaving an item out would not change the answer, it does not belong in the context. A wiki export is a copy of a team's entire internal documentation site, and nearly all of it is unrelated to any one question.

8. Two ways to hand over the same job

The instinct when giving a task to a model is to include whatever might help. That instinct is the problem.

In the left column, the question is a small item buried in a large pile. The model has to find the relevant parts by itself, and every unrelated page is a chance to be misled. In the right column, the model gets three things, and all three are needed.

The conclusion needs careful wording. Context that the task needs does improve answers: a model cannot summarise a contract it has not been given. Beyond what the task needs, more context does not produce a better answer. It usually produces a worse one, and worse again when most of the added material was never needed.

This is the same goal that context engineering and RAG pursue from the software side: give the model the right material, and only that.

Interview points

• Context rot is the decline in answer quality as a context becomes longer and more cluttered. It is a quality problem, separate from running out of window space.
• A conversation is sent in full on every turn, so dropped ideas, wrong documents, and corrected mistakes keep being read by the model.
• A model cannot tell which lines are outdated. It gives weight to everything in its context, so stale content can reappear in answers as if it were current.
• There are two causes. Clutter misleads the model. Length alone reduces precision even when all content is relevant.
• Degradation is gradual and begins well below the advertised context limit. The amount varies by model and task.
• Symptoms include vaguer answers, hedging between options, revived rejected ideas, and invented details.
• Context limits exist partly because long inputs are costly to process and partly because quality falls at long lengths.
• Prevention has three habits: start a fresh session carrying only conclusions, trigger compaction deliberately before the limit, and filter what goes in by asking whether leaving it out would change the answer.
• Needed context improves answers. Extra context beyond that tends to lower quality and raises cost.

Summary

• A context can go bad long before it is full.
• Old, irrelevant, and wrong content stays in the conversation and is sent again on every turn.
• The model weighs all of it, so dead content leaks into answers.
• Length alone also lowers precision, even with only relevant content.
• A fresh session with a short summary of conclusions resets the problem.
• Compacting early is better than waiting for the limit.
• Include what the task needs, and leave out what is there only in case it helps.