Memory

Yesterday you told an AI coding tool that your team always uses British spelling. Today you open a new session, and it writes "color". You explain again. Tomorrow you will explain a third time, unless the tool has a way to carry things over from one session to the next.

That ability is called memory. It is a common term around agents, and the name suggests that the model remembers. It does not. A language model keeps nothing between one request and the next. Memory is built outside the model, by the tool that talks to it.

This lesson shows what memory is made of, which is usually a few plain text files. It shows how a memory is written, how it gets back in front of the model, and where the files are kept. After this lesson you can explain how any tool's memory feature works, find and read its memory files, and say why memory uses up space in the context window.

1. Two ways of bringing outside material to a model, so far

Earlier lessons covered two ways of putting outside material in front of a model.

flowchart LR
    W["A web page<br/>handed over in the prompt"] --> M["The model<br/>sees only what is in the prompt"]
    R["report.pdf<br/>handed over in the prompt"] --> M
    K["A knowledge database<br/>documents, chunked and indexed"] -- "the relevant ones,<br/>found for you" --> M

The first way is grounding by hand: a web page or a file such as report.pdf is handed over in the prompt. The second way is RAG: a knowledge database holds documents that were cut into chunks and indexed, and the software finds the relevant ones for each question.

Both end at the same place. The model sees only what is in the prompt. Memory is a third way of bringing material to the model, and it ends at that same place too.

2. What is worth keeping between sessions

Between one session and the next, some things ought to survive. Three kinds come up most.

Preferences are about how you want the work done. Decisions are things that were settled and should not be reopened. Past results are facts about work that is already finished, so the tool does not redo it or contradict it.

Without memory, each of these has to be repeated in every new session, or the tool acts as if the earlier session never happened.

3. The model keeps nothing

By default a large language model keeps nothing between one request and the next. The technical word for this is stateless.

flowchart LR
    I["Text in"] --> M["The model<br/>no brain that learns from the chat<br/>no memory<br/>no database"]
    M --> O["Text out"]

People tend to imagine that there is a brain inside the model that learns from each chat, or a memory, or a database of past conversations. None of these is in there. The model is a fixed set of weights. Text goes in, text comes out, and the weights are the same afterwards as they were before. Nothing from the exchange is stored in the model.

This is why a chat app sends the whole conversation again with every message, as the lesson on context explained. It is also why a new session starts blank.

4. Everything else is added from outside

Since the model brings only "text in, text out", every other ability is attached from the outside.

The third row is the important one. The tool, which earlier lessons called the harness, assembles every prompt. So any ability that looks like the model "knowing" something from before must come from the tool putting that information into the prompt. Memory is one more thing the tool adds.

5. Memory is usually a few text files

The word "memory" covers two ways of building it. In practice one of them is far more common.

A team could store memories in a vector database and search them by meaning. That works, and it requires running a database.

Quite often, memory is a few plain text files. They are usually written in Markdown, a way of writing text files in which a few plain symbols mark headings and lists. Such files have names ending in .md. Any text editor can open them, and no extra software has to run.

This has a practical benefit. A person can open the memory file, read exactly what the tool "remembers", correct a wrong line, or delete one.

6. Where the files are kept

The files usually sit in the project folder that the person and the agent are already working in.

release-tool/              the project you and the agent are working in
  memory/                  written by the tool, not typed by you
    decisions.md           the key pieces kept back
    preferences.md         for the next session

In this example, release-tool/ is the project. Inside it is a memory/ folder. The files in that folder are written by the tool that drives the model. The person does not type them. They hold the key pieces that should be available in the next session, here split into decisions and preferences.

Folder and file names differ from tool to tool. The layout shown is one example of the pattern.

7. How a memory is written

Here is one way a memory can be saved. Tools differ, and this is only one of several possibilities.

[you]    Remember that releases always go out on a Thursday.

[model]  Remembering: releases go out on Thursdays.

         (the tool appends that line to memory/decisions.md)

The model only produced text. It cannot write a file. The tool turned that text into a line in memory/decisions.md. In many tools this goes through the mechanism from the lesson on tools: the tool offers the model something like a save_memory tool, the model asks for it, and the tool writes the line to disk.

Other ways exist. Some tools decide on their own which facts from a conversation are worth saving. Some let the person edit the memory file directly. In every case, the writing is done by the software around the model.

8. How a memory gets back to the model

Writing the note down is half of the mechanism. The other half happens on every later prompt.

flowchart LR
    F["memory/decisions.md<br/>Releases go out on Thursdays.<br/>British spelling, always."] --> T["The tool<br/>puts the two together"]
    P["What you typed<br/>Draft the release note."] --> T
    T --> M["The model<br/>receives your prompt<br/>with the stored memory"]

Suppose that in a new session you type "Draft the release note." The tool reads memory/decisions.md, joins its contents to your message, and sends both to the model. The model then writes a release note dated for a Thursday, with British spelling, although you mentioned neither today.

The idea in one sentence: the model never remembers, and the prompt does. The model sees those two lines as part of its input, exactly as if you had typed them yourself a moment ago. Because the model keeps nothing, the file has to be sent again with every prompt. If the tool skipped it once, the model would have no trace of it.

An everyday comparison is a colleague with no long-term memory who keeps a notebook. Every morning the colleague reads the notebook from the first page before starting work. The knowledge is in the notebook, and reading it again each day is what makes it usable. The comparison stops being accurate in one place: this colleague reads the notebook before every single task, not once a day.

9. Two places a memory file can live

Some of what should be kept belongs to one project. Some of it belongs to the person, whatever they work on.

Beside the project:

release-tool/
  memory/
    decisions.md

only true inside this
one piece of work

Somewhere global on the computer:

~/agent/
  memory/
    preferences.md

true in every project
you ever open

The ~ sign is a common short way to write "the user's home folder". The folder name agent is an example.

"Releases go out on Thursdays" is true for the release tool and may be false for another project, so it belongs in project memory. "British spelling, always" is true wherever this person works, so it belongs in global memory. A tool that supports both reads the global file and the current project's file and sends both.

A related pair of terms divides memory by time instead of by place. Short-term memory usually means what the model can see in the current session, which is the conversation in its context window. It is gone when the session ends. Long-term memory means what is stored outside and survives across sessions, such as the files in this lesson. Both project files and global files are long-term in this sense.

Some people use the pair more loosely: short-term for notes that matter only during the current piece of work, and long-term for notes that stay true for good. Used that way, the pair lines up with the split between project memory and global memory, drawn along time instead of place.

10. Memory and knowledge bases overlap

Memory and a knowledge base sound like different things.

The line between them is not sharp. A long memory can be stored in a vector database and searched like a knowledge base. A small knowledge base can be a few text files sent with every prompt, like memory. A vector database serves either one when efficient retrieval is needed.

Both follow the same principle from section 1: saved text is put into the prompt.

11. Two ways to load memory into the prompt

There are two strategies for getting stored memory in front of the model.

The selective way works like RAG. For each prompt, the tool searches the memory store, pulls the closest matches, and sends only those. This keeps prompts short when the memory is large, and it needs search machinery.

The common way has no search. The whole file is placed in front of what you typed. This is what reaches the model:

--- memory/decisions.md ---
Releases go out on Thursdays.
British spelling, always.
The hosted database was settled on.
---------------------------

Draft the release note.

Every prompt from then on looks like this. The approach is blunt and cheap, and a great many projects use it.

The trade-off between the two can be measured in tokens. Sending the whole file costs its full size on every prompt. A worked case with made-up numbers: a memory file of 400 tokens, sent with 50 prompts in a day, uses 20,000 input tokens. That is small. A memory file that has grown to 20,000 tokens uses 1,000,000 input tokens over the same 50 prompts and takes a noticeable share of the context window. So sending everything fits a short file, and selective loading fits a large store.

This also gives a reason to keep memory files tidy. Old or wrong lines are sent every time too, and the model treats them as current.

Interview points

• A language model is stateless. It keeps nothing between requests, and using it does not change its weights.
• Memory is a feature of the tool around the model. The tool saves information and inserts it into later prompts.
• Typical memory content is preferences, decisions, and results of past work.
• Memory can be built on a vector database with embeddings and similarity search, but it is very often a few plain Markdown files that need no running system.
• The model cannot write files. It produces text or a tool call, and the tool writes the line to the memory file.
• Stored memory has to be sent again with every prompt, because the model has no other access to it.
• Project memory lives beside the project and applies only there. Global memory lives in a user-wide location and applies to every project.
• Short-term memory is the current session's context. Long-term memory is anything stored outside the model that survives across sessions.
• Memory and knowledge bases overlap. Both put saved text into the prompt, and either can use a vector database.
• Loading can be selective, by searching and sending only matching items, or complete, by sending the whole file. Complete loading is common and costs the file's full token count on every prompt.

Summary

• The model remembers nothing. The tool remembers for it.
• Memory is saved text that the tool adds to later prompts.
• Most memory is plain text files that a person can open and edit.
• The tool writes memories, often after the model asks for them to be saved.
• The memory is sent again with every prompt.
• Memory can belong to one project or to the person across all projects.
• Sending the whole file is the common method, and it uses tokens every time.