The Context Window

A long chat with an AI assistant sometimes ends with a message such as "this conversation is too long, please start a new one". In other cases the assistant seems to forget something you told it at the start. Both effects have the same cause: the context window.

The term comes up constantly once you work with models, and even more with AI agents, which are AI systems that carry out long tasks in many steps. An agent collects a lot of text as it works, and all of it competes for the same limited space.

This lesson explains what the context window measures, what fills it, why it has a limit, and what is in it before you have typed anything. After this lesson you can explain why long chats get slower and more expensive, and why an assistant forgets.

1. What the context window measures

The context window is how much text a model can take in and work with in one go. It works like the model's short-term memory. It is measured in tokens.

flowchart LR
    W["The context window<br/>the most tokens the model<br/>can handle at once"] -- "input" --> M["The model<br/>one request"]
    M -- "output" --> O["New tokens"]

1. You send your input: your question, files, and any earlier conversation, all turned into tokens.
2. The model reads all of it at once.
3. It produces new tokens as its answer.

Everything you send has to fit inside the window before the model can answer. For most models, the window has to hold both the input and the answer being written. So a very long input leaves less room for the reply, and anything beyond the limit cannot be taken in at all.

2. A summary request sends the whole document

Suppose you attach a file and ask for a summary.

The model receives something like this, for all 38 pages:

Quar | terly | re | sults | for | the | per | iod | end | ing | ...

The model cannot skim a document or read only the important parts. To summarise it, the whole document has to be sent in first.

• Every page becomes tokens. A 38-page report might come to roughly 20,000 to 30,000 tokens, depending on how dense the text is.
• All of it fills the context window. Those tokens take up space before the model writes a single word of the summary.
• All of it counts towards cost. Every one of those tokens is billed as input.

So a short request such as "Summarise this" can use a large part of the context window, because of what is attached to it.

3. A follow-up question sends everything again

The conversation continues:

You:    Summarise this document for me.    (38-page report attached)

Model:  Here are the main points: revenue grew, costs held flat,
        and two regions fell behind...

You:    And which region was worst?

The model has no memory of the previous turn. Each request to the model starts from nothing. The model only appears to remember the conversation because the app collects the whole chat history and sends it again every time.

So to answer "And which region was worst?", the next request contains:

flowchart TB
    subgraph R1["Request 1"]
        A1["Document + first question"]
    end
    subgraph R2["Request 2"]
        B1["Document + first question"]
        B2["The model's summary"]
        B3["New question"]
    end
    subgraph R3["Request 3"]
        C1["Document + first question"]
        C2["The model's summary"]
        C3["Second question + second answer"]
        C4["New question"]
    end
    R1 --> R2 --> R3

Three things follow from this:

• Your new question looks small, but the request also carries the full document and all earlier messages.
• Each turn makes the request bigger, because the history keeps growing.
• Context and cost add up. The whole bundle fills the context window and counts as input tokens, every time you send a message.

Some providers lower the cost of sending the same content again. They reuse work on the parts they have already processed. This is called caching. Cached text is cheaper, but it still takes up space in the context window.

4. What "context" means

All of it together is the context. Context is everything the model is given in one request. It is not only your latest sentence. It is the whole bundle the model receives in order to write its next answer.

Context usually includes:

• your latest message
• every earlier message in the conversation, both yours and the model's
• any files or documents you attached
• hidden instructions the app adds to guide the model (section 8)

The model reads this whole bundle each time before it answers. The context window is the maximum size that bundle can be.

5. The window has a hard limit

A model cannot process endless text. There is a fixed maximum for each request.

======================= the limit =======================

  +---------------------+
  |  your follow-ups    |       +-----------------------+
  +---------------------+       | the next thing you    |
  |  its answers        |       | send: does not fit    |
  +---------------------+       +-----------------------+
  |                     |
  |  the document       |
  |                     |
  +---------------------+
   one request, filling up

Think of the context window as a container with a fixed size.

1. The document goes in first and takes a large share of the space.
2. Each answer and each follow-up question is added on top, so the container keeps filling.
3. Once it is full, anything new does not fit. The request can fail with an error, or the extra text cannot be taken in.

In practice, many chat apps handle this for you. They quietly drop or summarise older parts of the conversation to make room. The cost of that is that the model may then "forget" details from earlier in the chat, because those details are no longer in what it receives.

6. Why there is a limit

The limit exists for two main reasons.

Quality. The more text a model has to hold at once, the harder it is to give attention to every part.

So the limit is not only about capacity. It is also about quality. Providers set a cap before quality slips too far. They prefer to refuse an oversized request over returning an answer you cannot rely on. A limit you can see is more useful than a bad answer you cannot explain.

Cost and hardware. Processing very long inputs takes much more computing power and memory. Models are also trained on text up to a certain length, and they do not work well beyond it.

A practical consequence: even inside the limit, sending only what is relevant usually gives better results.

7. "Context window" has two meanings

People use the same phrase in two ways.

One way to picture it is a bag. Sometimes people mean what is in the bag, and sometimes they mean how big the bag is. The sentence usually makes clear which one is meant.

8. The model sees more than your screen shows

The conversation on your screen is only part of the context. This is what the model is sent:

[hidden: not shown to you, but sent with every single message]

You:    Summarise this document for me.
Model:  Here are the main points: revenue grew, costs held flat...
You:    And which region was worst?
Model:  The northern region, down 11% on the quarter.

Each time you send a message, the app sends the model a bundle: some hidden text at the top, then the whole conversation so far, then your new message.

An analogy: in a stage play, the audience sees the actors. Offstage there is a prompter who quietly gives them lines. The audience never sees the prompter, but the prompter shapes what happens. The analogy stops at one point. A prompter speaks during the play, while the hidden text is written once in advance and sent again with every message.

9. The system prompt

The hidden part usually starts with an instruction that the company running the assistant places in front of everything you write. It is called the system instruction or, more often, the system prompt.

More about the system prompt:

• The model reads it before your first message, every time.
• It sets the ground rules: the tone, what the assistant should or should not do, what tools it can use, and what its name is.
• You normally cannot see it, but it takes up space in the context window and uses tokens, as your messages do.
• It is written by whoever built the assistant. If a bank builds a support chatbot on top of another company's model, the bank writes the system prompt.

An everyday comparison: a new shop assistant gets a briefing from the manager before the shop opens ("Be polite, no refunds over 50 pounds without approval"). Customers never hear the briefing, but it shapes every conversation.

The hidden part often holds more than the system prompt. It can also include descriptions of tools, saved memories, today's date, and the text of files you uploaded. The next lesson lists all the parts.

10. The window is rarely empty

You never see all of the context. The context window is rarely empty, even before you have typed a single word.

• Before you type anything, the window already holds hidden text: at least the system prompt, and often more.
• So the model's answer depends on more than your message. Two apps that use the same model can behave very differently because their hidden instructions differ.
• Some of the window's tokens are used up before you start, which leaves a little less room for your conversation.
• If an assistant behaves in an unexpected way, such as refusing something or insisting on a certain style, it may be following instructions you cannot see. It may not have misunderstood you.

Interview points

• The context window is the maximum number of tokens a model can handle in one request. For most models it covers the input and the output together.
• Context is everything sent to the model in one request: the latest message, the conversation history, attached files, and hidden instructions.
• A model has no memory between requests. The application sends the whole conversation again with every message.
• Because history is sent again each turn, every request in a long chat is larger than the one before, and input-token cost grows with it.
• A whole attached document is tokenized and placed in the window. The model cannot read only parts of it.
• Caching lowers the price of repeated content. It does not free space in the window.
• When the window is full, the request fails or the application drops or summarises older messages, which is why assistants forget earlier details.
• The limit exists because answer quality falls as the input grows, because long inputs need more compute and memory, and because models are trained up to a certain length.
• The term has two meanings: the content currently being sent, and the size limit.
• A system prompt is a hidden instruction written by the company or developer behind the assistant. It is sent with every request and uses tokens.
• Two products built on the same model can behave differently because their system prompts differ.

Summary

• The context window is a size limit in tokens for one request.
• Everything has to fit: documents, the whole conversation, hidden instructions, and the reply.
• The model does not remember. The app sends the full history every time.
• Long conversations fill the window and cost more with each turn.
• The limit protects answer quality and reflects hardware cost.
• "Context window" can mean the content or the limit.
• A hidden system prompt is in the window before you type anything.