Lesson 60 of 73
Context Engineering for Coding Agents
A coding agent works by repeating a cycle: it sends the conversation to the model, the model asks to run a tool (read a file, run a command), the agent runs it, and the result is added to the conversation for the next request. The important detail is that the model has no memory between requests. The agent must send the whole conversation again every time. Every file the agent reads and every command it runs makes the next request longer.
A model can only look at a limited number of tokens at once. A token is a small piece of text, often a word or part of a word, and the limit is called the context window. A session on a local model with an 8K to 32K window (8,192 to 32,768 tokens) fills the window after a few steps. When it is full, the next request fails or the model loses the start of the task.
Context engineering is the work of deciding which tokens go into each request, so the window is never wasted. This lesson measures where the tokens go, shows the techniques real coding agents use to stay inside the window, and ends with a ContextManager class that loop agent uses. After the lesson you can calculate how much history fits in a given window, truncate and trim a conversation safely, summarize old messages, and arrange a prompt so that the engine can reuse work from the previous request.
| Idea | What it does, in plain words |
|---|---|
| Context budget | Splits the window into fixed parts, space for the reply, and space for history |
| Token counting | Compares real token counts from llama-tokenize with a quick estimate from character counts |
| Truncation policy | Decides which part of a long tool output the model gets to see |
| Pair-safe trimming | Removes old messages without leaving a tool result whose tool call was removed |
| Compaction | Replaces old messages with a short summary written by a model |
| Repository map | A short list of files and their declarations, so the model does not read whole files to find things |
| Instruction files | AGENTS.md files that hold project rules, loaded from the repository root down to the working directory |
| Todo tool | A plan that lives in a tool result instead of in the model's memory |
| Subagent | A second agent with its own history that sends back only a short answer |
| Prompt-cache layout | A message order that lets the engine reuse earlier work, measured with cache_n |
The examples build on the agent from The Agent Loop and a Tool Framework (Project 21). Where an example needs the message types, it restates the small sealed hierarchy (a fixed set of record types) from that lesson: SystemMessage (the standing instructions), UserMessage, AssistantMessage with a list of ToolCalls (the requests to run tools), and ToolMessage (a tool result). Facts about other harnesses come from their source code at the versions in the course plan: Pi 0.73.1, Crush v0.97.1, OpenHands SDK 1.51.0, mini-swe-agent 2.4.6, and Aider 0.86.2. A harness is the program around the model that runs the loop and the tools.
Example 1: context budget arithmetic
Before any technique makes sense, the agent needs to know how much room it really has. The context window is the maximum number of tokens the engine can hold for one sequence, and that sequence is the prompt plus the reply. With llama-server it is the -c value divided among the slots, as llama-server: Endpoints, Slots, and Prompt Cache showed.
Some parts of a request are the same on every step: the system prompt, the tool definitions, the project instructions, and (optionally) a repository map. These fixed parts are paid again with every request. Part of the window must also stay free for the model's answer, which is called the reply reserve. Whatever is left is the space for history.
history = window - (system + tools + instructions + repo map) - reserveThe program below computes this for three common window sizes. Look at the history% column: it shows how much of the window is left for the conversation itself.
import java.util.Locale;
public class ContextBudget {
// Sizes in tokens. The fixed parts are sent with every request.
record Budget(int window, int system, int tools, int instructions, int repoMap, int reserve) {
int fixed() { return system + tools + instructions + repoMap; }
int history() { return window - fixed() - reserve; }
}
public static void main(String[] args) {
int system = 900, tools = 1400, instructions = 600;
int stepCost = 60 + 1200; // one assistant tool call plus one tool result
System.out.println("window fixed reserve history history% steps");
for (int window : new int[] {8192, 16384, 32768}) {
int repoMap = Math.min(1024, window / 16);
int reserve = Math.max(1024, window / 8);
Budget b = new Budget(window, system, tools, instructions, repoMap, reserve);
System.out.println(String.format(Locale.ROOT, "%6d %5d %7d %7d %7.1f%% %5d",
window, b.fixed(), reserve, b.history(),
100.0 * b.history() / window, b.history() / stepCost));
}
}
}window fixed reserve history history% steps
8192 3412 1024 3756 45.8% 2
16384 3924 2048 10412 63.5% 8
32768 3924 4096 24748 75.5% 19The program uses assumed sizes for the parts of a request: a 900-token system prompt, 1,400 tokens of tool definitions (six tools with JSON Schemas), and 600 tokens of project instructions. Example 2 shows how to measure such numbers instead of assuming them. The repository map gets a sixteenth of the window, at most 1,024 tokens, and the reply reserve gets an eighth of the window, at least 1,024 tokens.
The reply reserve is the number of tokens kept free for the model's answer. The engine fails or cuts the reply when prompt plus reply would pass the window, so the agent subtracts the reserve before it fills the window with history.
Each row of the output reads as follows. Take the 8,192-token row and compute it by hand:
repo map = min(1024, 8192 / 16) = 512
reserve = max(1024, 8192 / 8) = 1024
fixed = 900 + 1400 + 600 + 512 = 3412
history = 8192 - 3412 - 1024 = 3756 (45.8% of 8192)
one step = 60 + 1200 = 1260 tokens -> 3756 / 1260 = 2 steps (rounded down)- 8,192 tokens. The fixed parts take 3,412 tokens (900 + 1,400 + 600 + 512), the reserve takes 1,024, and 3,756 tokens remain for history, 45.8 percent of the window. One step of a coding agent (an assistant message with a tool call of about 60 tokens, then a tool result of about 1,200 tokens) costs about 1,260 tokens, so two full steps fit. The third file read already needs trimming or compaction.
- 16,384 tokens. The fixed parts grow by 512 tokens because the repository map reaches its 1,024-token cap, and 10,412 tokens remain: eight steps.
- 32,768 tokens. The fixed parts stay at 3,924 tokens while the window doubles, so the share for history rises to 75.5 percent and 19 steps fit.
The fixed parts do not shrink with the window, which is why small windows suffer most. On an 8K window more than half the room is gone before the user types anything. The rest of the lesson works on both sides of this equation: smaller fixed parts (fewer tools, shorter prompts, a smaller map) and cheaper steps (truncated tool results, compaction, subagents).
Example 2: counting tokens, real counts against an estimate
A budget is only as good as its token counts, and the agent usually cannot afford to run a tokenizer on every message. A tokenizer splits text into tokens from its vocabulary, as Tokens, Vocabularies, and Special Tokens explained, so the exact count depends on the model. The common shortcut is to estimate: divide the number of characters by four. This example first measures real counts and then checks how good the shortcut is.
llama-tokenize from the pinned llama.cpp build prints the real tokens for a vocabulary file. Here it counts a short JSON tool result with the Llama 3 tokenizer. Each output line is a token id, an arrow, and the text piece it stands for.
printf '%s' '{"path":"src/main/java/app/OrderService.java","line":42,"status":"modified","sha":"9f2c4e1a7b"}' > json.txt
llama-tokenize -m models/ggml-vocab-llama-bpe.gguf -f json.txt --no-bos --show-count --log-disableReal output (pinned llama.cpp build, ggml-vocab-llama-bpe.gguf). Read the pieces on the right and count how many characters each one covers:
5018 -> '{"'
2398 -> 'path'
3332 -> '":"'
3632 -> 'src'
15711 -> '/main'
48300 -> '/java'
10867 -> '/app'
14 -> '/'
4531 -> 'Order'
1898 -> 'Service'
11085 -> '.java'
2247 -> '","'
1074 -> 'line'
794 -> '":'
2983 -> '42'
1359 -> ',"'
2899 -> 'status'
3332 -> '":"'
28261 -> 'modified'
2247 -> '","'
15605 -> 'sha'
3332 -> '":"'
24 -> '9'
69 -> 'f'
17 -> '2'
66 -> 'c'
19 -> '4'
68 -> 'e'
16 -> '1'
64 -> 'a'
22 -> '7'
65 -> 'b'
9388 -> '"}'
Total number of tokens: 33--no-bos leaves out the beginning-of-sequence token (a special token the model normally gets at the start of every sequence) so that the count covers only the text. The 95-character JSON line became 33 tokens. Words and path parts such as path, /main, and .java are single tokens, but the 10-character hexadecimal hash costs 10 tokens, one per character, because random letter and digit mixes are rare in training text and the vocabulary has no merged pieces for them.
The same command ran on five samples with two vocabularies: ggml-vocab-llama-bpe.gguf (Llama 3) and ggml-vocab-qwen2.gguf (Qwen2, the vocabulary of the Qwen 2.5 Coder models). The program below holds those measured counts and compares them with the estimate of one token per four characters. The error column is positive when the estimate is too high and negative when it is too low.
import java.util.List;
import java.util.Locale;
public class TokenEstimate {
// chars and token counts measured with llama-tokenize --no-bos --show-count
record Sample(String kind, int chars, int llama3, int qwen2) {}
static int estimate(int chars) { return (chars + 3) / 4; } // ceil(chars / 4)
public static void main(String[] args) {
List<Sample> samples = List.of(
new Sample("prose", 96, 21, 21),
new Sample("java", 203, 42, 42),
new Sample("json", 95, 33, 34),
new Sample("maven log", 228, 66, 69),
new Sample("git hash", 40, 38, 40));
System.out.println("kind chars est llama3 qwen2 error chars/token");
int totalEst = 0, totalReal = 0;
for (Sample s : samples) {
int est = estimate(s.chars());
int real = Math.max(s.llama3(), s.qwen2());
totalEst += est;
totalReal += real;
System.out.println(String.format(Locale.ROOT, "%-10s %5d %3d %6d %5d %+4d%% %11.2f",
s.kind(), s.chars(), est, s.llama3(), s.qwen2(),
Math.round(100.0 * (est - real) / real), (double) s.chars() / real));
}
System.out.println(String.format(Locale.ROOT, "total: estimate %d, real %d (%+d%%)",
totalEst, totalReal, Math.round(100.0 * (totalEst - totalReal) / totalReal)));
}
}kind chars est llama3 qwen2 error chars/token
prose 96 24 21 21 +14% 4.57
java 203 51 42 42 +21% 4.83
json 95 24 33 34 -29% 2.79
maven log 228 57 66 69 -17% 3.30
git hash 40 10 38 40 -75% 1.00
total: estimate 166, real 206 (-19%)The samples were an English sentence about fixing a test, a seven-line Java class with four-space indentation, the JSON line above, five lines of Maven error output, and a 40-character commit hash. The error column compares the estimate with the larger of the two real counts, because an agent that must stay inside the window cares about the worst case. For the JSON row the estimate is ceil(95 / 4) = 24 and the real count is 34, so the error is (24 - 34) / 34, about -29 percent.
- Prose and Java have more than four characters per token (4.57 and 4.83), so the estimate is too high by 14 and 21 percent. Java does well because keywords such as
finalandclassand the four-space indent are single tokens. - JSON and logs have fewer characters per token (2.79 and 3.30): quotes, colons, brackets, line numbers, and paths split into many short pieces. The estimate is too low by 29 and 17 percent.
- The hash is one token per character in Qwen2 (40 tokens), because the Qwen2 pre-tokenizer splits every digit into its own token, and close to that in Llama 3 (38 tokens), which groups digits in runs of up to three. The estimate is 75 percent too low.
- The total is 19 percent too low. Tool results in a coding session are mostly JSON, logs, stack traces, and paths, the kinds of text where the estimate errs low.
Pi estimates tokens as Math.ceil(chars / 4) (dist/core/compaction/compaction.js), and a comment there calls the heuristic conservative, meaning it overestimates. The table shows that this holds for prose and code but not for logs and JSON. Two consequences follow for loop:
- The estimate needs a safety margin, or a lower characters-per-token value (three instead of four) for tool output.
- The real count is available for free after every request: the server returns
usage.prompt_tokens, which an agent can use to correct its estimate for the messages already sent and estimate only the new ones. Pi does the same thing: its context-size function takes the usage of the last assistant reply and estimates only the messages added after it.
Example 3: tool output truncation policies
Most of the context is spent on tool results. One cat of a large file or one noisy build can add thousands of tokens that the model barely needs. Truncation means cutting a long result down before it goes into the history. A truncation policy is the rule for which part to keep.
This example applies four policies to the same 43-line build log and checks whether each one keeps the line that matters, the compiler error. The log has 40 boring Compiling lines in the middle, a header at the start, and the error and the verdict at the end. Look at the keeps error line and keeps first line flags in the output.
import java.util.ArrayList;
import java.util.List;
public class TruncationPolicies {
static String head(String s, int max) {
return s.length() <= max ? s : s.substring(0, max) + "\n[" + (s.length() - max) + " chars cut]";
}
static String tail(String s, int max) {
return s.length() <= max ? s : "[" + (s.length() - max) + " chars cut]\n" + s.substring(s.length() - max);
}
static String headTail(String s, int max) {
if (s.length() <= max) return s;
int half = max / 2;
return s.substring(0, half) + "\n[" + (s.length() - 2 * half) + " chars cut]\n"
+ s.substring(s.length() - half);
}
static String headTailLines(String s, int keepHead, int keepTail) {
String[] lines = s.split("\n");
if (lines.length <= keepHead + keepTail) return s;
List<String> out = new ArrayList<>(List.of(lines).subList(0, keepHead));
out.add("[" + (lines.length - keepHead - keepTail) + " lines cut]");
out.addAll(List.of(lines).subList(lines.length - keepTail, lines.length));
return String.join("\n", out);
}
public static void main(String[] args) {
StringBuilder log = new StringBuilder("[INFO] Building app 1.0\n");
for (int i = 1; i <= 40; i++) log.append("[INFO] Compiling Source").append(i).append(".java\n");
log.append("[ERROR] OrderServiceTest.java:[31,9] cannot find symbol\n");
log.append("[INFO] BUILD FAILURE");
String out = log.toString();
System.out.println("original: " + out.length() + " chars, " + out.split("\n").length + " lines");
String[] names = {"head", "tail", "head+tail", "lines 2+2"};
String[] results = {head(out, 160), tail(out, 160), headTail(out, 160), headTailLines(out, 2, 2)};
for (int i = 0; i < names.length; i++) {
String r = results[i];
System.out.println("== " + names[i] + ": " + r.length() + " chars, keeps error line: "
+ r.contains("[ERROR]") + ", keeps first line: " + r.contains("Building app"));
}
System.out.println("== lines 2+2 text:");
System.out.println(results[3]);
}
}original: 1331 chars, 43 lines
== head: 177 chars, keeps error line: false, keeps first line: true
== tail: 177 chars, keeps error line: true, keeps first line: false
== head+tail: 178 chars, keeps error line: true, keeps first line: true
== lines 2+2: 145 chars, keeps error line: true, keeps first line: true
== lines 2+2 text:
[INFO] Building app 1.0
[INFO] Compiling Source1.java
[39 lines cut]
[ERROR] OrderServiceTest.java:[31,9] cannot find symbol
[INFO] BUILD FAILUREThe log is 1,331 characters. Each character policy keeps 160 characters plus a marker that says how much was cut, which is why the results are 177 or 178 characters long.
- Head keeps the start. The error line is at the end of the log, so the model sees forty lines of
Compilingand never learns why the build failed. Head-only truncation suits outputs where the start matters most, such as the first lines of a large file. - Tail keeps the end, including the error, but loses the first line, which names the module being built. Build tools and test runners print the verdict last, so tail-only is better than head-only for commands.
- Head and tail keep both ends and cut the middle. This is the policy of mini-swe-agent: its
observation_templateinconfig/mini.yamlpasses outputs under 10,000 characters through unchanged and otherwise sendsoutput_head(the first 5,000 characters),output_tail(the last 5,000),elided_chars, and the warningOutput too long. - Line-based head and tail keeps whole lines, two from each end. Cutting in the middle of a line can split a path or a number and mislead the model, so whole lines are safer. Project 21's
ToolRegistry.truncateuses this policy with a character budget.
The marker is part of the policy. It tells the model that content is missing and how much, so the model can ask for a range of lines with a read offset or rerun a command piped through grep instead of guessing.
Example 4: trimming history while keeping call and result pairs valid
When the history is too long, the simplest fix is to drop the oldest messages. The question is where the cut may fall. In the message format, a tool call and its result belong together: the assistant message lists calls with ids, and each tool message carries the id of the call it answers. Chat templates and servers expect every tool result to follow the assistant message that made the call. An orphan result (one whose call was dropped) makes many templates fail or confuses the model, because the model sees an answer to a question it never asked.
The program below keeps the system message plus the last six messages, in three variants, and runs a validator on each. The validator is the one from Project 21, reduced to the pairing rule. In the printed lines, S is the system message, U a user message, A[c1] an assistant message that calls c1, and T(c1) the result of c1.
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
public class PairSafeTrim {
sealed interface Message permits SystemMessage, UserMessage, AssistantMessage, ToolMessage {}
record SystemMessage(String text) implements Message {}
record UserMessage(String text) implements Message {}
record ToolCall(String id, String name, String arguments) {}
record AssistantMessage(String text, List<ToolCall> toolCalls) implements Message {}
record ToolMessage(String callId, String name, String content, boolean isError) implements Message {}
// Every result needs an earlier call with its id; every call needs a result.
static String validate(List<Message> history) {
Set<String> open = new HashSet<>();
for (Message m : history) {
if (m instanceof AssistantMessage a) a.toolCalls().forEach(c -> open.add(c.id()));
if (m instanceof ToolMessage t && !open.remove(t.callId())) return "orphan result " + t.callId();
}
return open.isEmpty() ? "valid" : "calls without results " + open;
}
static List<Message> keepLast(List<Message> h, int n, boolean safe, boolean userOnly) {
int start = Math.max(1, h.size() - n);
while (safe && start < h.size() && !(h.get(start) instanceof UserMessage
|| (!userOnly && h.get(start) instanceof AssistantMessage))) start++;
List<Message> kept = new ArrayList<>(List.of(h.get(0))); // the system message stays
kept.addAll(h.subList(start, h.size()));
return kept;
}
static String names(List<Message> h) {
List<String> out = new ArrayList<>();
for (Message m : h) out.add(switch (m) {
case SystemMessage s -> "S";
case UserMessage u -> "U";
case AssistantMessage a -> a.toolCalls().isEmpty() ? "A" : "A" + a.toolCalls().stream().map(ToolCall::id).toList();
case ToolMessage t -> "T(" + t.callId() + ")";
});
return String.join(" ", out);
}
static AssistantMessage calls(String... ids) {
return new AssistantMessage("", List.of(ids).stream().map(id -> new ToolCall(id, "read", "{}")).toList());
}
public static void main(String[] args) {
List<Message> h = List.of(new SystemMessage("You are loop."), new UserMessage("Fix the test"),
calls("c1"), new ToolMessage("c1", "read", "test source", false),
calls("c2", "c3"), new ToolMessage("c2", "read", "service source", false),
new ToolMessage("c3", "read", "pom.xml", false), new AssistantMessage("Found it.", List.of()),
new UserMessage("Now run it"), calls("c4"), new ToolMessage("c4", "bash", "BUILD SUCCESS", false));
System.out.println("full: " + names(h) + " -> " + validate(h));
for (String mode : List.of("naive", "pi-style", "user-only")) {
List<Message> k = keepLast(h, 6, !mode.equals("naive"), mode.equals("user-only"));
System.out.printf("%-10s %s -> %s%n", mode + ":", names(k), validate(k));
}
}
}full: S U A[c1] T(c1) A[c2, c3] T(c2) T(c3) A U A[c4] T(c4) -> valid
naive: S T(c2) T(c3) A U A[c4] T(c4) -> orphan result c2
pi-style: S A U A[c4] T(c4) -> valid
user-only: S U A[c4] T(c4) -> validThe history has eleven messages: the system prompt (S), two user turns (U), assistant messages (A, with the ids of the calls they make), and tool results (T). The second assistant message makes two parallel calls, c2 and c3.
Every mode keeps the system message and the last six messages, then adjusts the start:
- Naive starts at the sixth message from the end, which is the result of
c2. The call that produced it was dropped, so the validator reportsorphan result c2. Sent to a server, this history either fails in the template or reaches the model as a result with no question. - Pi-style moves the start forward until it reaches a user or assistant message. Pi's compaction documentation (
docs/compaction.md) lists the valid cut points as user, assistant, bash-execution, and custom messages, and states "Never cut at tool results (they must stay with their tool call)." Here the start moves past both orphan results to the assistant textFound it., and the history is valid. - User-only moves the start to the next user message, so every kept turn is complete from the user's request to the last result. Some chat templates reject a conversation whose first non-system message comes from the assistant, so cutting at a user message is the safer choice for local models whose templates are strict.
Both safe modes keep fewer than six messages. That is the cost of pair safety: the cut can only fall at certain positions, so the kept part is a little smaller than the target. OpenHands SDK states the same rule in a general form: its view keeps "manipulation indices" (context/view/manipulation_indices.py), the positions where events may be inserted or deleted, and properties such as ToolCallMatchingProperty and BatchAtomicityProperty (all calls from one model response stay together) decide which positions are allowed.
Example 5: compaction by summarization
Trimming throws information away: once a message is dropped, the model no longer knows that the file was read or the edit was made. Compaction replaces the old messages with a summary that a model writes, so the facts survive in fewer tokens. It is the same idea as meeting notes: the minutes keep the decisions and drop the conversation.
The summarizer here is scripted: a function that returns a fixed text, so the output is deterministic. In loop the same function calls the model with a summary prompt. The program prints what the summarizer would receive, then the size of the history before and after.
import java.util.ArrayList;
import java.util.List;
import java.util.function.Function;
public class Compaction {
sealed interface Message permits SystemMessage, UserMessage, AssistantMessage, ToolMessage {}
record SystemMessage(String text) implements Message {}
record UserMessage(String text) implements Message {}
record ToolCall(String id, String name, String arguments) {}
record AssistantMessage(String text, List<ToolCall> toolCalls) implements Message {}
record ToolMessage(String callId, String name, String content, boolean isError) implements Message {}
static int tokens(List<Message> h) { return (serialize(h, 100_000).length() + 3) / 4; }
// Plain-text transcript, so the summarizer reads it as data, not as a chat to continue.
static String serialize(List<Message> h, int maxResult) {
StringBuilder sb = new StringBuilder();
for (Message m : h) switch (m) {
case SystemMessage s -> sb.append("[System]: ").append(s.text()).append('\n');
case UserMessage u -> sb.append("[User]: ").append(u.text()).append('\n');
case AssistantMessage a when !a.toolCalls().isEmpty() -> sb.append("[Assistant tool calls]: ")
.append(String.join("; ", a.toolCalls().stream().map(c -> c.name() + "(" + c.arguments() + ")").toList())).append('\n');
case AssistantMessage a -> sb.append("[Assistant]: ").append(a.text()).append('\n');
case ToolMessage t -> {
String c = t.content();
if (c.length() > maxResult) c = c.substring(0, maxResult) + " [" + (c.length() - maxResult) + " chars cut]";
sb.append("[Tool result]: ").append(c).append('\n');
}
}
return sb.toString();
}
static AssistantMessage call(String id, String name, String args) {
return new AssistantMessage("", List.of(new ToolCall(id, name, args)));
}
public static void main(String[] args) {
String source = "public class OrderService { ... }".repeat(20);
List<Message> h = List.of(new SystemMessage("You are loop."),
new UserMessage("Add a findAll method to OrderService and make the tests pass."),
call("c1", "read", "{\"path\":\"src/OrderService.java\"}"), new ToolMessage("c1", "read", source, false),
call("c2", "edit", "{\"path\":\"src/OrderService.java\"}"), new ToolMessage("c2", "edit", "ok", false),
new UserMessage("Now run the tests."),
call("c3", "bash", "{\"command\":\"mvn -q test\"}"), new ToolMessage("c3", "bash", "BUILD SUCCESS", false));
int cut = 6; // index of the last user message: keep that turn
List<Message> old = h.subList(1, cut), kept = h.subList(cut, h.size());
String transcript = serialize(old, 60);
System.out.print("to summarizer:\n" + transcript);
// Scripted summarizer: returns a canned summary instead of calling a model.
Function<String, String> summarizer = text -> "Goal: add findAll to OrderService. Done: findAll added in "
+ "src/OrderService.java. Files read: src/OrderService.java. Files modified: src/OrderService.java.";
List<Message> compacted = new ArrayList<>(List.of(h.get(0)));
compacted.add(new UserMessage("The conversation before this point was compacted into this summary:\n<summary>\n"
+ summarizer.apply(transcript) + "\n</summary>"));
compacted.addAll(kept);
System.out.println("before: " + h.size() + " messages, ~" + tokens(h) + " tokens");
System.out.println("after: " + compacted.size() + " messages, ~" + tokens(compacted) + " tokens");
}
}to summarizer:
[User]: Add a findAll method to OrderService and make the tests pass.
[Assistant tool calls]: read({"path":"src/OrderService.java"})
[Tool result]: public class OrderService { ... }public class OrderService { [600 chars cut]
[Assistant tool calls]: edit({"path":"src/OrderService.java"})
[Tool result]: ok
before: 9 messages, ~257 tokens
after: 5 messages, ~97 tokensThe output has two parts: the transcript that goes to the summarizer, and the size of the history before and after.
The transcript sent to the summarizer covers messages 1 to 5: the first user request, the read call and its result, and the edit call and its result. Each message becomes one line with a role label. Pi serializes in the same way ([User]:, [Assistant]:, [Assistant tool calls]:, [Tool result]:, in dist/core/compaction/utils.js) and gives the reason in its documentation: a plain transcript "prevents the model from treating it as a conversation to continue." The long read result is cut to 60 characters with a marker. Pi cuts tool results to 2,000 characters at this step (TOOL_RESULT_MAX_CHARS in the same file), because tool results are the largest messages and the summarizer needs to know what was done, not every line that was read.
The compacted history has five messages: the system prompt, a user message that carries the summary, and the last turn unchanged (the user's request to run the tests, the bash call, and its result). The estimate falls from about 257 to about 97 tokens, roughly 62 percent fewer. Pi sends its summary as a user message that starts with "The conversation history before this point was compacted into the following summary:" and wraps the text in <summary> tags (COMPACTION_SUMMARY_PREFIX in dist/core/messages.js); the program follows that layout.
The summary in this example is a fixed string. A real summary is only as good as the prompt that asks for it, which is the subject of the next example.
Example 6: when to compact and what to keep
Compaction costs a model call and loses detail, so an agent compacts only when the history is close to the limit. The rule that decides "close to the limit" is the trigger. Each harness has its own trigger rule, and the rules were written for different window sizes.
This program evaluates three rules from source code, plus one scaled to the window, for four window sizes. Each cell is a number of used tokens. When the used tokens go above that number, compaction starts. Watch what happens to the Pi column on the 8,192-token window.
import java.util.Locale;
public class CompactionTriggers {
// Pi 0.73.1: compact when used > window - reserveTokens (reserveTokens = 16384).
static long pi(long window) { return window - 16_384; }
// Crush v0.97.1: compact when window - used <= threshold;
// threshold = 20,000 for windows above 200,000, otherwise 20% of the window.
static long crush(long window) {
long threshold = window > 200_000 ? 20_000 : (long) (window * 0.2);
return window - threshold;
}
// A rule scaled to the window: keep max(1024, window/8) for the reply, compact at 80% of the rest.
static long scaled(long window) {
long reserve = Math.max(1024, window / 8);
return (long) ((window - reserve) * 0.8);
}
static String show(long trigger) { return trigger <= 0 ? "always" : String.valueOf(trigger); }
public static void main(String[] args) {
System.out.println("window pi crush scaled");
for (long w : new long[] {8_192, 32_768, 131_072, 262_144}) {
System.out.println(String.format(Locale.ROOT, "%-8d %-8s %-8s %s", w,
show(pi(w)), show(crush(w)), show(scaled(w))));
}
// OpenHands SDK 1.51.0 LLMSummarizingCondenser defaults
int maxSize = 240, keepFirst = 2;
int target = maxSize / 2;
int tail = target - keepFirst - 1; // one slot is taken by the summary event
System.out.println("openhands: condense above " + maxSize + " events, keep first " + keepFirst
+ ", summary 1, keep last " + tail + " -> " + (keepFirst + 1 + tail) + " events");
}
}window pi crush scaled
8192 always 6554 5734
32768 16384 26215 22937
131072 114688 104858 91750
262144 245760 242144 183500
openhands: condense above 240 events, keep first 2, summary 1, keep last 117 -> 120 eventsEach column is the number of used tokens above which the rule starts compaction.
- Pi compacts when
contextTokens > contextWindow - reserveTokens(shouldCompactindist/core/compaction/compaction.js), with a defaultreserveTokensof 16,384. On a 131,072-token window it compacts above 114,688 tokens. On an 8,192-token window the threshold is negative (8,192 - 16,384 = -8,192), so the rule fires on every request, and the defaultkeepRecentTokensof 20,000 is larger than the whole window. Pi's defaults assume hosted models with large windows; its documentation shows both values insettings.json, so a local setup lowers them. - Crush compacts when the remaining tokens fall to 20 percent of the window, or to 20,000 tokens for windows above 200,000 (
largeContextWindowThreshold,largeContextWindowBuffer, andsmallContextWindowRatioininternal/agent/agent.go). Its used-token count is the prompt plus completion tokens that the provider reported for the session. When the model's context window is unknown (zero), Crush skips automatic summarization; the source comment says this avoids "immediately truncating custom/local models." - Scaled is the rule
loopuses: subtract a reply reserve of an eighth of the window, then compact at 80 percent of what remains. For 8,192 tokens that is (8,192 - 1,024) x 0.8 = 5,734. It gives a positive threshold for every window and leaves room for one more large tool result after the check. - OpenHands counts events, not tokens, by default:
LLMSummarizingCondenserincontext/condenser/llm_summarizing_condenser.pyhasmax_size = 240andkeep_first = 2, and condenses to half ofmax_size. The last line of the output computes the result: the first 2 events, 1 summary event, and the last 117 events, 120 in total. The condenser also accepts a token limit (max_tokens) and treats token pressure as a hard requirement and event count as a soft one.
What to keep is the other half of the decision. The summary prompts of Pi (docs/compaction.md) and Crush (internal/agent/templates/summary.md) ask for the same kinds of facts:
- the user's goal and constraints,
- what is done and what is in progress,
- files read and files modified,
- decisions with reasons,
- commands that worked and commands that failed,
- exact next steps.
Pi also carries the lists of read and modified files from one compaction to the next in a details field, so that the list survives repeated compactions. Three parts never go into the summary: the system prompt, the instruction files, and the most recent turn, which stays verbatim because the model is in the middle of it. OpenHands keeps the first two events (keep_first) for the same reason: the system prompt and the original task.
Example 7: a repository map
A model that does not know which files exist reads directories and whole files to find out, and each read costs a tool result. A repository map is a short listing of files and their declarations (classes, interfaces, methods) that the agent puts into the prompt instead. It works like the table of contents of a book: enough to know where to look, without the full text.
This version extracts Java types and non-private method signatures with regular expressions. A regular expression is a text pattern that finds matching pieces of a string. The program builds a map for three small files and prints its size next to the size of the sources.
import java.util.Map;
import java.util.TreeMap;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class RepoMap {
// A type declaration; group 3 holds record components.
static final Pattern TYPE = Pattern.compile(
"^\\s*(?:(?:public|protected|private|static|final|abstract|sealed)\\s+)*"
+ "(class|interface|record|enum)\\s+(\\w+)(\\([^)]*\\))?", Pattern.MULTILINE);
// A non-private method header ending in { or ; (interface methods end in ;).
static final Pattern METHOD = Pattern.compile(
"^\\s*(?:(?:public|protected|static|default|abstract|final)\\s+)*+"
+ "(?!(?:private|class|interface|record|enum|return|new)\\b)"
+ "([\\w.]+(?:<[^()]*>)?(?:\\[\\])*)\\s+(\\w+)\\(([^)]*)\\)[^;{]*[;{]", Pattern.MULTILINE);
public static void main(String[] args) {
Map<String, String> files = new TreeMap<>(Map.of(
"src/main/java/app/Order.java",
"package app;\n\npublic record Order(String id, long cents) {\n}\n",
"src/main/java/app/OrderService.java", """
package app;
import java.util.*;
public final class OrderService {
private final Map<String, Order> orders = new HashMap<>();
public Optional<Order> find(String id) {
return Optional.ofNullable(orders.get(id));
}
public List<Order> findAll() {
return new ArrayList<>(orders.values());
}
private void audit(String msg) {
System.out.println(msg);
}
}
""",
"src/main/java/app/OrderRepository.java", """
package app;
public interface OrderRepository {
Order load(String id) throws java.io.IOException;
}
"""));
StringBuilder map = new StringBuilder();
int sourceChars = 0;
for (var e : files.entrySet()) {
String src = e.getValue();
sourceChars += src.length();
map.append(e.getKey()).append('\n');
Matcher t = TYPE.matcher(src);
while (t.find()) map.append(" ").append(t.group(1)).append(' ').append(t.group(2))
.append(t.group(3) == null ? "" : t.group(3)).append('\n');
Matcher m = METHOD.matcher(src);
while (m.find()) map.append(" ").append(m.group(1).trim()).append(' ')
.append(m.group(2)).append('(').append(m.group(3).trim()).append(")\n");
}
System.out.print(map);
System.out.println("map: " + map.length() + " chars (~" + (map.length() + 3) / 4 + " tokens), sources: "
+ sourceChars + " chars (~" + (sourceChars + 3) / 4 + " tokens)");
}
}src/main/java/app/Order.java
record Order(String id, long cents)
src/main/java/app/OrderRepository.java
interface OrderRepository
Order load(String id)
src/main/java/app/OrderService.java
class OrderService
Optional<Order> find(String id)
List<Order> findAll()
map: 279 chars (~70 tokens), sources: 572 chars (~143 tokens)The map lists the three files in path order, because a TreeMap sorts its keys. Under each file it shows the declared types and the methods another class can call:
Order.javaholds a record (a short Java class that is only data), and the map shows its components(String id, long cents), which are the record's fields and constructor in one line.OrderRepository.javaholds an interface with one method. The method pattern accepts a header that ends in;, which is how interface methods without a body end, and thethrowsclause is matched by[^;{]*and left out of the map.OrderService.javashowsfindandfindAll. The privateauditmethod is missing on purpose: the negative lookahead(?!(?:private|...)\b)rejects it, and the possessive quantifier*+on the modifier group stops the regular expression engine from backing up and treatingprivateas a return type. (A lookahead is a check that the text ahead does not start with certain words. A possessive quantifier is a repeat that never gives characters back once it has taken them.)
The map is 279 characters against 572 characters of source, about half. In a real project the ratio is far smaller, because method bodies are most of the code and the map skips them all. The regular expressions do not handle every Java construct (annotations on their own line, generic methods with <T> before the return type, constructors), and a parser such as the JDK's com.sun.source tree API or tree-sitter handles all of them. A parser reads the code into a tree of declarations, so it does not depend on how the code is laid out.
Aider builds its map that way. aider/repomap.py extracts definitions and references with tree-sitter queries, builds a graph of which files refer to which, ranks files with PageRank (networkx), and fills a token budget with the highest-ranked declarations. PageRank is the algorithm that web search used to rank pages by how many important pages link to them; here a file ranks high when many files refer to it. The default budget is map_tokens=1024; Model.get_repo_map_tokens in aider/models.py sets it to an eighth of the model's input limit, clamped between 1,024 and 4,096 tokens. Ranking matters more than extraction once the repository is large: the map cannot list everything, so it lists what the current files refer to.
Example 8: loading instruction files from the repository root to the working directory
Every project has rules that a new teammate must learn: the build command, the Java version, naming conventions. Instruction files are Markdown files in the repository that tell an agent these rules. AGENTS.md is the most common name; CLAUDE.md and others also exist. A large repository can have one at the root and more in subdirectories. The agent loads every file on the path from the root to its working directory, root first, so that the more specific file comes last and reads as the refinement.
The program creates a temporary directory tree with four instruction files, loads them, and prints the prompt text it built. Check which of the four files appear in the output and which do not.
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
public class InstructionFiles {
static final List<String> CANDIDATES = List.of("AGENTS.md", "CLAUDE.md");
// Walks from cwd up to the workspace root; returns files root-first, one per directory.
static List<Path> find(Path workspace, Path cwd) {
List<Path> found = new ArrayList<>();
for (Path dir = cwd; dir != null && dir.startsWith(workspace); dir = dir.getParent()) {
for (String name : CANDIDATES) {
Path f = dir.resolve(name);
if (Files.isRegularFile(f)) { found.add(0, f); break; } // first candidate wins
}
}
return found;
}
public static void main(String[] args) throws IOException {
Path outer = Files.createTempDirectory("home").toRealPath();
Path ws = Files.createDirectories(outer.resolve("ws"));
Path orders = Files.createDirectories(ws.resolve("services/orders"));
Files.writeString(outer.resolve("AGENTS.md"), "Outside the workspace: not loaded.");
Files.writeString(ws.resolve("AGENTS.md"), "Build with mvn -q verify. Java 21.");
Files.writeString(orders.resolve("AGENTS.md"), "Money is in cents (long). Never use double.");
Files.writeString(orders.resolve("CLAUDE.md"), "Ignored: AGENTS.md exists in this directory.");
List<Path> files = find(ws, orders);
StringBuilder prompt = new StringBuilder();
for (Path f : files) {
String rel = ws.relativize(f).toString().replace('\\', '/');
prompt.append("## ").append(rel).append('\n').append(Files.readString(f)).append("\n\n");
}
System.out.print(prompt);
System.out.println("loaded " + files.size() + " files, " + prompt.length() + " chars");
try (var paths = Files.walk(outer)) {
paths.sorted(Comparator.reverseOrder()).forEach(p -> p.toFile().delete());
}
}
}<h2 id="agents-md">AGENTS.md</h2>
Build with mvn -q verify. Java 21.
<h2 id="services-orders-agents-md">services/orders/AGENTS.md</h2>
Money is in cents (long). Never use double.
loaded 2 files, 123 charsThe temporary tree has four instruction files. Two were loaded:
AGENTS.mdat the workspace root comes first, because the walk inserts each file at the front of the list while it moves up from the working directory.services/orders/AGENTS.mdcomes second. Its directory also has aCLAUDE.md, which was skipped because the candidates are tried in order and the first one found wins.- The
AGENTS.mdabove the workspace was not loaded, because the loop stops when the directory no longer starts with the workspace path.
Each file gets a heading with its relative path, so the model can tell where an instruction came from, and the program prints the size so that the budget from Example 1 can account for it.
Real harnesses differ in exactly these choices. Pi (loadProjectContextFiles in dist/core/resource-loader.js) first loads a global file from its agent directory (~/.pi/agent), then walks from the working directory up to the file-system root, not the repository root, and tries AGENTS.md, AGENTS.MD, CLAUDE.md, CLAUDE.MD in each directory, keeping the first match. Its README says that all matching files are concatenated and that --no-context-files turns loading off. Crush has a longer list of default context paths in internal/config/config.go, including .github/copilot-instructions.md, .cursorrules, CLAUDE.md, GEMINI.md, CRUSH.md, and AGENTS.md, plus global paths ~/.config/crush/CRUSH.md and ~/.config/AGENTS.md. Stopping at the workspace root, as loop does, means a file in a home directory never reaches a project's prompt unless the user configures it as a global file.
Instruction files are part of the fixed cost from Example 1, sent on every request. A 600-token file costs 600 tokens times the number of steps in a session, so short files with commands and rules, not prose, fit small windows best.
Example 9: a todo-list tool
On a long task a model forgets steps, and the history that recorded them may be compacted away. A todo tool keeps the plan in the conversation in a fixed form: the model sends the complete list on every call, the tool validates it, stores it, and returns it rendered. The latest tool result always holds the whole plan, so the plan survives even when older results are trimmed.
The program below plays four scripted model calls against the tool: a valid plan, an invalid status, two tasks in progress, and a valid update. The tool answers each one with a result string. Watch which calls are accepted and which are rejected with an error message the model can read.
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
public class TodoTool {
record Todo(String content, String status) {}
static final Set<String> STATUSES = Set.of("pending", "in_progress", "completed");
static final ObjectMapper JSON = new ObjectMapper();
static List<Todo> current = List.of();
// The model always sends the whole list; the tool replaces the stored list and renders it.
static String call(String argumentsJson) throws Exception {
JsonNode todos = JSON.readTree(argumentsJson).path("todos");
List<Todo> next = new ArrayList<>();
int inProgress = 0;
for (JsonNode t : todos) {
Todo todo = new Todo(t.path("content").asText(), t.path("status").asText());
if (!STATUSES.contains(todo.status()))
return "error: invalid status \"" + todo.status() + "\" for \"" + todo.content()
+ "\"; use pending, in_progress, or completed";
if (todo.status().equals("in_progress")) inProgress++;
next.add(todo);
}
if (inProgress > 1) return "error: " + inProgress + " tasks are in_progress; keep exactly one";
current = next;
StringBuilder sb = new StringBuilder();
long done = next.stream().filter(t -> t.status().equals("completed")).count();
sb.append("todos ").append(done).append('/').append(next.size()).append(" done\n");
for (Todo t : next) sb.append(switch (t.status()) {
case "completed" -> "[x] ";
case "in_progress" -> "[>] ";
default -> "[ ] ";
}).append(t.content()).append('\n');
return sb.toString().strip();
}
public static void main(String[] args) throws Exception {
System.out.println(call("""
{"todos":[{"content":"Read OrderServiceTest","status":"in_progress"},
{"content":"Add findAll to OrderService","status":"pending"},
{"content":"Run mvn test","status":"pending"}]}"""));
System.out.println(call("""
{"todos":[{"content":"Read OrderServiceTest","status":"done"}]}"""));
System.out.println(call("""
{"todos":[{"content":"Read OrderServiceTest","status":"completed"},
{"content":"Add findAll to OrderService","status":"in_progress"},
{"content":"Run mvn test","status":"in_progress"}]}"""));
System.out.println(call("""
{"todos":[{"content":"Read OrderServiceTest","status":"completed"},
{"content":"Add findAll to OrderService","status":"completed"},
{"content":"Run mvn test","status":"in_progress"}]}"""));
System.out.println("stored: " + current.size() + " items");
}
}todos 0/3 done
[>] Read OrderServiceTest
[ ] Add findAll to OrderService
[ ] Run mvn test
error: invalid status "done" for "Read OrderServiceTest"; use pending, in_progress, or completed
error: 2 tasks are in_progress; keep exactly one
todos 2/3 done
[x] Read OrderServiceTest
[x] Add findAll to OrderService
[>] Run mvn test
stored: 3 itemsThe four calls are four scripted model outputs:
- The first call creates a plan of three items with the first one in progress. The tool renders it with
[>]for in progress and[ ]for pending, and a count of done items. - The second call uses the status
done, which is not one of the three allowed values. The tool returns an error that names the bad value and lists the allowed ones, and it does not replace the stored list. A small model that invents a status reads the error and retries with a valid one. - The third call marks two items as in progress. The tool rejects it, because the plan should name one current step.
- The fourth call is valid: two items completed, one in progress. The stored list has three items.
Crush has the same tool, named todos (internal/agent/tools/todos.go). Its items have content, status (pending, in_progress, or completed), and active_form (a present-tense label such as "Running tests" for the user interface), and its description tells the model to keep exactly one task in progress and to skip the tool for simple tasks. Crush also passes the current todo list into the summary prompt when it compacts a session (buildSummaryPrompt(currentSession.Todos) in internal/agent/agent.go), which shows the connection between the two techniques: the plan is the part of the history that a summary must not lose.
The tool costs tokens too: a full list on every call. For an 8K window a plan of three to seven short items is the useful size.
Example 10: subagents with separate contexts
Searching a codebase produces large tool results that matter only until the answer is found. After the model has located OrderController.java:41, the 40 grep lines and 50 lines of source that led there have no further use, yet they stay in the history and are resent on every later step.
A subagent is a second agent loop, started by a tool call, with its own history. The parent sends a task, the subagent does the searching in its own context, and only its final answer returns to the parent as one tool result. The program below runs a parent and a subagent with scripted steps and prints the size of each history when it finishes.
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
public class Subagent {
record Msg(String role, String text) {}
record Step(String tool, String arg, String answer) {} // tool == null means final answer
// Runs one agent with its own history; the model is a script of steps.
static String run(String name, String task, Deque<Step> script, Map<String, Function<String, String>> tools) {
List<Msg> history = new ArrayList<>(List.of(new Msg("system", "You are " + name + "."), new Msg("user", task)));
while (true) {
Step s = script.poll();
if (s.tool() == null) {
history.add(new Msg("assistant", s.answer()));
System.out.println(name + ": " + history.size() + " messages, ~" + tokens(history) + " tokens");
return s.answer();
}
history.add(new Msg("assistant", s.tool() + "(" + s.arg() + ")"));
history.add(new Msg("tool", tools.get(s.tool()).apply(s.arg())));
}
}
static int tokens(List<Msg> h) { return h.stream().mapToInt(m -> (m.text().length() + 3) / 4).sum(); }
public static void main(String[] args) {
Map<String, Function<String, String>> readOnly = Map.of(
"grep", q -> "src/app/OrderController.java:41: Order.parse(id)\n".repeat(40),
"read", p -> "class OrderController { /* 90 lines */ }\n".repeat(50));
Deque<Step> childScript = new ArrayDeque<>(List.of(
new Step("grep", "Order.parse", null),
new Step("read", "src/app/OrderController.java", null),
new Step(null, null, "Order ids are parsed only in OrderController.java line 41 via Order.parse(id).")));
Map<String, Function<String, String>> parentTools = Map.of(
"task", prompt -> {
String result = run("subagent", prompt, childScript, readOnly);
System.out.println("task result for parent: " + result.length() + " chars");
return result;
});
Deque<Step> parentScript = new ArrayDeque<>(List.of(
new Step("task", "Find where Order ids are parsed. Report file and line.", null),
new Step(null, null, "The parsing happens in OrderController.java:41.")));
run("parent", "Where are order ids parsed?", parentScript, parentTools);
}
}subagent: 7 messages, ~1056 tokens
task result for parent: 78 chars
parent: 5 messages, ~58 tokensThe order of the lines follows the order of events. The parent's first step calls task, which runs the whole subagent before the parent continues:
- The subagent's history grew to 7 messages and about 1,056 tokens: its system prompt, the task, a
grepcall with 40 matching lines, and areadcall with 50 lines of source. - Only the subagent's 78-character final answer went back to the parent.
- The parent finished with 5 messages and about 58 tokens. Its history holds the
taskcall and the short result instead of the search output.
The saving is the difference between the two histories, about a thousand tokens here, and it repeats on every later request of the parent, because the parent's history is resent on each step. The cost is extra model calls (the subagent's three steps) and the risk that the subagent's answer leaves out a detail the parent needed. A precise task with a required answer format ("Report file and line") lowers that risk.
Crush implements this as the agent tool (internal/agent/agent_tool.go): it starts a task agent in a new session with the prompt from the tool call, and its description (internal/agent/templates/agent_tool.md) says the agent has the tools glob, grep, ls, and view, all read-only. Read-only tools mean a subagent can run without the user approving each step, and several subagents can run at once without editing the same files.
Example 11: a prompt-cache-friendly layout, measured with cache_n
Each request is processed in two phases: the engine first reads the whole prompt (prefill), then generates the reply. Prefill is expensive for long prompts. To avoid repeating it, llama-server keeps the KV cache (key-value cache) of the last prompt in each slot and reuses the longest common prefix of the next prompt, as llama-server: Endpoints, Slots, and Prompt Cache showed. The KV cache holds the keys and values that attention computed for each token (Generation, Prefill, Decode, and the KV Cache). In plain words: if the new prompt starts with exactly the same tokens as the old one, the engine skips the work for those tokens.
Reuse stops at the first token that differs, so the order of the parts of a prompt decides how much of each request must be computed again. A timestamp near the top of the prompt changes every request and spoils the whole cache; the same timestamp at the end spoils almost nothing. This measurement compares three layouts of the same two-step conversation.
The server runs the course's tiny model (random weights; its text is meaningless, but token counts and timings fields are real) with the Qwen 2.5 template, one slot, and --cache-ram 0, which turns off the extra RAM prompt cache so that only the slot's own cache can be reused:
llama-server -m tiny-random-qwen2vocab-f32.gguf --port 18572 -np 1 -c 4096 --cache-ram 0 \
--jinja --chat-template-file models/templates/Qwen-Qwen2.5-7B-Instruct.jinjaEach layout sends two requests: the first with the system prompt and the user's task, the second with the same messages plus an assistant tool call and its result, as the agent loop does after one step. A shell function sends a request with one tool definition and max_tokens: 1 (only one reply token, since only the prompt processing is measured) and prints usage.prompt_tokens and two fields of timings:
req() { # $1 = system prompt, $2 = the messages after it
curl -s http://127.0.0.1:18572/v1/chat/completions -d "{\"messages\":[{\"role\":\"system\",\"content\":\"$1\"},$2],
\"tools\":$TOOLS, \"max_tokens\":1, \"temperature\":0}" |
jq -c '{prompt: .usage.prompt_tokens, cache_n: .timings.cache_n, prompt_n: .timings.prompt_n}'
}
SYS='You are loop, a coding agent. Use the tools to read, edit, and run code in the workspace. Keep answers short.'
# TOOLS holds one "read" tool; U1, A1, T1 hold the user task, the assistant tool call, and the tool result as JSON
# stable: the system prompt never changes
req "$SYS" "$U1"; req "$SYS" "$U1,$A1,$T1"
# volatile: the current time and a todo count at the end of the system prompt
req "$SYS Current time: 10:41:07. Open todos: 1." "$U1"
req "$SYS Current time: 10:41:19. Open todos: 0." "$U1,$A1,$T1"
# volatile data at the end: the time goes into the newest message instead
req "$SYS" "$U1_WITH_TIME"; req "$SYS" "$U1_WITH_TIME,$A1,$T1_WITH_TIME"Real output (pinned llama.cpp build, tiny model, requests in this order on a fresh server). Compare the cache_n value of each second request:
stable layout
{"prompt":188,"cache_n":0,"prompt_n":188}
{"prompt":248,"cache_n":188,"prompt_n":60}
volatile layout
{"prompt":207,"cache_n":29,"prompt_n":178}
{"prompt":267,"cache_n":40,"prompt_n":227}
volatile data at the end
{"prompt":200,"cache_n":29,"prompt_n":171}
{"prompt":272,"cache_n":200,"prompt_n":72}cache_n is the number of prompt tokens taken from the slot's cache and prompt_n the number computed for this request; their sum is prompt_tokens. The lines read in pairs:
- Stable. The first request computes all 188 tokens, because the server is new. The second request starts with the same 188 tokens (the rendered first prompt ends with
<|im_start|>assistant\n, and the second continues from there with the tool call), so all 188 come from the cache and only the 60 new tokens are computed. - Volatile. The time and the todo count are appended to the system prompt. In the Qwen 2.5 template the system text comes first and the tool definitions follow it inside the same system block, so a change in the system text also forces the tool definitions to be computed again. The first request reuses 29 tokens from the stable layout's prompt (the system text up to
short.). The second request reuses 40 tokens, up toCurrent time: 10:41:, where the seconds differ, and computes 227 tokens, almost four times as many as the stable layout's 60. - Volatile data at the end. The time moves into the user message and the tool result, the newest parts of the prompt. The first request reuses only 29 tokens because the slot held the volatile layout's prompt, but the second request reuses all 200 tokens of the first and computes 72.
Prefill time grows with prompt_n, so in a session whose prompt has grown to 20,000 tokens the difference between computing the newest few hundred tokens and computing the whole prompt again is the difference between a short pause and a wait for a full prefill on every step. The rules for a cache-friendly layout follow from the measurement:
- Fixed parts first, in a fixed order: system prompt, tool definitions, instruction files, repository map.
- No timestamps, counters, or other changing values in the fixed parts. Data that changes goes into the newest message.
- Append-only history. Editing or deleting an old message (trimming or compaction) changes the prompt from that point on, so every compaction costs one full recomputation of everything after the system prompt. Compacting rarely and in large steps is cheaper than trimming one message per request.
Example 12: predicting cache reuse from the rendered prompt
The previous example needed a running server. An agent can predict the same effect offline by rendering two consecutive prompts and measuring their common prefix, the longest start that both texts share. This is useful in a unit test, because no model is needed.
This program renders ChatML, the format the Qwen 2.5 template produces (each message is wrapped as <|im_start|>role, the text, and <|im_end|>), and compares three layouts. In that template a tool result is rendered inside a user turn as <tool_response>. For each layout it prints how many characters the two prompts share and how many are new.
import java.util.List;
import java.util.Locale;
public class PrefixReuse {
record Msg(String role, String text) {}
// ChatML, the format of the Qwen 2.5 template: <|im_start|>role\ntext<|im_end|>\n
static String render(List<Msg> msgs) {
StringBuilder sb = new StringBuilder();
for (Msg m : msgs) sb.append("<|im_start|>").append(m.role()).append('\n').append(m.text()).append("<|im_end|>\n");
return sb.append("<|im_start|>assistant\n").toString();
}
static int commonPrefix(String a, String b) {
int n = Math.min(a.length(), b.length()), i = 0;
while (i < n && a.charAt(i) == b.charAt(i)) i++;
return i;
}
static void compare(String name, List<Msg> first, List<Msg> second) {
String a = render(first), b = render(second);
int p = commonPrefix(a, b);
System.out.println(String.format(Locale.ROOT, "%-14s req1 %d chars, req2 %d chars, "
+ "shared %d (%.0f%% of req1), new %d",
name, a.length(), b.length(), p, 100.0 * p / a.length(), b.length() - p));
}
public static void main(String[] args) {
String sys = "You are loop, a coding agent. Tools: read, edit, bash.";
Msg user = new Msg("user", "Fix the failing test.");
Msg call = new Msg("assistant", "<tool_call>{\"name\":\"read\",\"arguments\":{\"path\":\"Test.java\"}}</tool_call>");
Msg result = new Msg("user", "<tool_response>class Test {}</tool_response>");
compare("stable", List.of(new Msg("system", sys), user),
List.of(new Msg("system", sys), user, call, result));
compare("time on top", List.of(new Msg("system", "Time 10:41:07. " + sys), user),
List.of(new Msg("system", "Time 10:41:19. " + sys), user, call, result));
compare("edited history", List.of(new Msg("system", sys), user),
List.of(new Msg("system", sys), new Msg("user", "Fix the failing test (retry 2)."), call, result));
}
}stable req1 155 chars, req2 331 chars, shared 155 (100% of req1), new 176
time on top req1 170 chars, req2 346 chars, shared 30 (18% of req1), new 316
edited history req1 155 chars, req2 341 chars, shared 121 (78% of req1), new 220The three lines match the measurements of Example 11 in characters instead of tokens:
- Stable. The whole first prompt (155 characters) is a prefix of the second, so only the 176 new characters need computing.
- Time on top. The time is the first thing in the system message, so the prompts agree only up to
Time 10:41:, 30 characters, and 316 of 346 characters are new. Putting volatile data at the very top is the worst position: it invalidates everything. - Edited history. The user message was rewritten between the requests, so reuse stops inside it at character 121, and everything after that point is recomputed even though the system prompt was unchanged.
The engine compares tokens, not characters, and a change in one character can change how the neighbouring characters are tokenized, so the character count is an approximation of the token count. It is good enough for a test in loop's test suite: render two consecutive requests and assert that the first is a prefix of the second whenever no compaction happened between them.
Example 13: strategies for 8K to 32K windows
The techniques above have settings: how large a tool result may be, how much history stays after compaction, whether to build a repository map. The right values depend on the window. This program derives one set of settings per window size from a few rules, so the numbers can be compared side by side. Read each column together with the rule that follows the output.
import java.util.Locale;
public class WindowStrategies {
static final double CHARS_PER_TOKEN = 3.0; // conservative for code, JSON, and logs
public static void main(String[] args) {
System.out.println("window reserve tool-cap read-lines repo-map keep-recent compact-at search");
for (int w : new int[] {8_192, 16_384, 32_768}) {
int reserve = Math.max(1024, w / 8);
int toolCapTokens = w / 16;
int toolCapChars = (int) (toolCapTokens * CHARS_PER_TOKEN);
int readLines = toolCapChars / 40; // about 40 characters per source line
int repoMap = w < 16_384 ? 0 : Math.min(1024, w / 16);
int keepRecent = w / 4;
int compactAt = (int) ((w - reserve) * 0.8);
String search = w <= 16_384 ? "subagent" : "direct";
System.out.println(String.format(Locale.ROOT, "%6d %7d %8d %10d %8d %11d %10d %s",
w, reserve, toolCapChars, readLines, repoMap, keepRecent, compactAt, search));
}
}
}window reserve tool-cap read-lines repo-map keep-recent compact-at search
8192 1024 1536 38 0 2048 5734 subagent
16384 2048 3072 76 1024 4096 11468 subagent
32768 4096 6144 153 1024 8192 22937 directEach column comes from one rule, and the rule states the reason:
- reserve is an eighth of the window, at least 1,024 tokens: room for a reply that contains a tool call with a code edit.
- tool-cap is the largest tool result in characters: a sixteenth of the window in tokens, times three characters per token (the conservative ratio from Example 2). On 8K that is 1,536 characters, so a single result can never take more than about a tenth of the window.
- read-lines is the default line limit of the
readtool, derived from the cap at about 40 characters per line: 38 lines on 8K, 153 on 32K. The model asks for more with an offset, as File Tools describes. - repo-map is off below 16K, because 512 tokens of map on an 8K window cost more than the
globandgrepcalls they would save on small tasks. From 16K it is capped at 1,024 tokens, Aider's default. - keep-recent is a quarter of the window: the verbatim part kept after compaction.
- compact-at is the scaled trigger from Example 6.
- search chooses subagents for code search on windows up to 16K, so search results never enter the main history, and direct
grepcalls on larger windows, where a few search results fit and the extra model calls of a subagent cost more than they save.
These rules are starting points for measurement, not constants. Evaluating Models and Agents shows how to compare two settings by running the same tasks with each.
Other strategies apply to every window size and need no table:
- Fewer tools mean shorter fixed parts (the later lesson Making Small Local Models Reliable counts this).
- Tool descriptions are part of the fixed cost, so they say what the tool does in one or two sentences.
- A
readtool that returns numbered lines lets the model refer to lines without quoting them. - A failing command's output is cut to the error lines before it reaches the model, which keeps the useful part of a 300-line Maven log inside the cap.
Example 14: a ContextManager for loop agent
The last example combines the techniques into one class that loop agent uses between the agent loop and the model client. The agent loop adds every message to the manager and asks it for the list to send. The manager owns the history and does three jobs: it truncates large tool results when they are added, it estimates tokens with the conservative ratio, and it compacts at user-message boundaries when the estimate passes the trigger. The message types are those of Project 21; the summarizer is a function so that the test can script it.
The main method plays six turns on an 8,192-token window. Each turn adds a user message and two tool calls. In the output, look for the truncated lines, the steady growth of the token estimate, and the single compacted line.
import java.util.ArrayList;
import java.util.List;
import java.util.function.Function;
public class ContextManagerDemo {
sealed interface Message permits SystemMessage, UserMessage, AssistantMessage, ToolMessage {}
record SystemMessage(String text) implements Message {}
record UserMessage(String text) implements Message {}
record ToolCall(String id, String name, String arguments) {}
record AssistantMessage(String text, List<ToolCall> toolCalls) implements Message {}
record ToolMessage(String callId, String name, String content, boolean isError) implements Message {}
static final class ContextManager {
private final List<Message> history = new ArrayList<>();
private final int toolCapChars, compactAt, keepRecent;
private final Function<String, String> summarizer;
ContextManager(int window, SystemMessage system, Function<String, String> summarizer) {
int reserve = Math.max(1024, window / 8);
this.toolCapChars = window / 16 * 3;
this.compactAt = (int) ((window - reserve) * 0.8);
this.keepRecent = window / 4;
this.summarizer = summarizer;
history.add(system);
}
static int tokens(Message m) { // about 3 characters per token, rounded up
int chars = switch (m) {
case SystemMessage s -> s.text().length();
case UserMessage u -> u.text().length();
case AssistantMessage a -> a.text().length()
+ a.toolCalls().stream().mapToInt(c -> c.name().length() + c.arguments().length()).sum();
case ToolMessage t -> t.content().length();
};
return (chars + 2) / 3 + 4; // plus 4 tokens of template markers per message
}
int total() { return history.stream().mapToInt(ContextManager::tokens).sum(); }
void add(Message m) {
if (m instanceof ToolMessage t && t.content().length() > toolCapChars) {
int half = toolCapChars / 2;
String c = t.content();
m = new ToolMessage(t.callId(), t.name(), c.substring(0, half) + "\n[" + (c.length() - 2 * half)
+ " chars cut]\n" + c.substring(c.length() - half), t.isError());
System.out.println("truncated " + t.callId() + ": " + c.length() + " -> "
+ ((ToolMessage) m).content().length() + " chars");
}
history.add(m);
}
// Called before every model request.
List<Message> request() {
if (total() > compactAt) compact();
return List.copyOf(history);
}
private void compact() {
int before = total(), kept = 0, cut = history.size();
for (int i = history.size() - 1; i > 1; i--) { // walk back, cut only at a user message
kept += tokens(history.get(i));
if (history.get(i) instanceof UserMessage) cut = i;
if (kept >= keepRecent && cut < history.size()) break;
}
List<Message> old = history.subList(1, cut);
String summary = summarizer.apply(old.size() + " messages");
int removed = old.size();
old.clear();
history.add(1, new UserMessage("<summary>\n" + summary + "\n</summary>"));
System.out.println("compacted: removed " + removed + " messages, " + before + " -> " + total() + " tokens");
}
}
public static void main(String[] args) {
ContextManager cm = new ContextManager(8192, new SystemMessage("You are loop. ".repeat(40)),
text -> "Task: fix OrderService. Done: read 3 files, edited OrderService.java. Next: run tests.");
for (int turn = 1; turn <= 6; turn++) {
cm.add(new UserMessage("Turn " + turn + ": continue the task."));
for (int call = 1; call <= 2; call++) {
String id = "c" + turn + call;
cm.add(new AssistantMessage("", List.of(new ToolCall(id, "read", "{\"path\":\"src/F" + id + ".java\"}"))));
cm.add(new ToolMessage(id, "read", "x".repeat(1200 * call), false));
}
List<Message> sent = cm.request();
System.out.println("turn " + turn + ": sent " + sent.size() + " messages, ~" + cm.total() + " tokens");
}
}
}truncated c12: 2400 -> 1553 chars
turn 1: sent 6 messages, ~1158 tokens
truncated c22: 2400 -> 1553 chars
turn 2: sent 11 messages, ~2125 tokens
truncated c32: 2400 -> 1553 chars
turn 3: sent 16 messages, ~3092 tokens
truncated c42: 2400 -> 1553 chars
turn 4: sent 21 messages, ~4059 tokens
truncated c52: 2400 -> 1553 chars
turn 5: sent 26 messages, ~5026 tokens
truncated c62: 2400 -> 1553 chars
compacted: removed 20 messages, 5993 -> 2165 tokens
turn 6: sent 12 messages, ~2165 tokensThe scripted session has six turns on an 8,192-token window. Each turn adds a user message and two tool calls: a 1,200-character result and a 2,400-character result. The constructor derives the settings of Example 13: a 1,536-character tool cap, compaction above 5,734 tokens, and 2,048 tokens kept verbatim.
- Truncation. Every second result (2,400 characters) is above the cap and is cut to 768 characters from each end plus the marker, 1,553 characters in total. The first result of each turn (1,200 characters) passes unchanged.
- Growth. Each turn adds five messages and about 967 tokens: from about 1,158 tokens after turn 1 to about 5,026 after turn 5. The total is checked in
request(), before each model call. - Compaction. After turn 6 the estimate is 5,993 tokens, above 5,734.
compact()walks back from the newest message, adding up tokens, and remembers the last user message it passed. It stops once it has passed 2,048 tokens and a user message, so the kept part is turns 5 and 6 (ten messages, about 1,934 tokens, a little underkeepRecentbecause the cut can only fall at a user message). The 20 messages of turns 1 to 4 become one summary message, and the history drops to 12 messages and about 2,165 tokens (191 for the system prompt, 40 for the summary, 1,934 for the kept turns).
The design choices are the ones the earlier examples measured:
- Truncation happens on
add, so an oversized result never enters the history. - The cut falls only at user messages (Example 4).
- Compaction happens only above the trigger, so between compactions the history is append-only and the prompt cache reuses it (Example 11).
- The summarizer receives only what it needs; in
loopit receives the serialized transcript of Example 5 and a summary prompt with the sections of Example 6.
Workshop: loop agent connects this class to the Agent of Project 21: the agent calls context.add for every message and sends context.request() to the model instead of its own list.
Interview points
- A coding agent resends the whole history on every step, because the model keeps no memory between requests. Tool results therefore dominate the context, and the fixed parts (system prompt, tool schemas, instruction files, map) cost the same on every request and matter most on small windows.
- Characters divided by four overestimates tokens for prose and Java but underestimates JSON, logs, and hashes by 17 to 75 percent in measurements with real tokenizers. Use a lower ratio for tool output and correct the estimate with the server's
usage.prompt_tokens. - Truncate tool output at both ends with a marker that says how much was cut. Build tools print the verdict last, so head-only truncation hides errors.
- History may be cut only where no tool result loses its call. Pi never cuts at a tool result, OpenHands restricts cuts to manipulation indices, and cutting at a user message keeps every turn complete.
- Compaction summarizes old messages into one message. Serialize them as a plain transcript so the summarizer does not continue the conversation, and keep the system prompt, the instructions, and the last turn verbatim.
- Trigger rules differ: Pi compacts above window minus 16,384 tokens, Crush at 20 percent remaining (20,000 tokens above 200K windows), and OpenHands above 240 events. Defaults built for large hosted windows need scaling for an 8K local window.
- A repository map lists files and declarations within a token budget. Aider ranks files with PageRank over tree-sitter references and spends 1,024 to 4,096 tokens on it.
- Instruction files load from the root to the working directory, root first, one file per directory. The stop point (repository root or file-system root) decides whether files outside the project leak in.
- Subagents keep search output out of the parent's history. The parent pays only for the final answer, at the cost of extra model calls.
- The KV cache is reused up to the first differing token. Keep fixed parts first, put changing data in the newest message, and compact rarely, because every compaction recomputes the prompt after the system part.
Summary
- The usable history is the window minus the fixed parts and a reply reserve. On 8,192 tokens with 3,412 tokens of fixed parts and a 1,024-token reserve, 3,756 tokens remain, which is about two steps with 1,200-token tool results.
- Real token counts from
llama-tokenizefor the five samples were 206 tokens against an estimate of 166 from characters divided by four (19 percent low). A 40-character hash took 38 to 40 tokens. - Pi estimates tokens as
ceil(chars / 4)and uses the provider's usage numbers for messages already sent. - mini-swe-agent passes outputs under 10,000 characters unchanged and otherwise sends the first and last 5,000 characters with the number of elided characters.
- A trimmed history must start at a user or assistant message. Starting at a tool result leaves an orphan result.
- Pi serializes messages to a
[User]:/[Assistant]:transcript for summarization, cuts tool results to 2,000 characters there, and sends the summary as a user message in<summary>tags. - Pi's defaults are
reserveTokens16,384 andkeepRecentTokens20,000. Crush summarizes at 20 percent remaining, or 20,000 tokens for windows above 200,000, and skips automatic summarization when the window is unknown. OpenHands condenses above 240 events withkeep_first2. - Aider's repository map uses tree-sitter and PageRank with a default of 1,024 tokens, scaled to an eighth of the input limit within 1,024 to 4,096.
- Pi loads a global
AGENTS.md, then one ofAGENTS.md,AGENTS.MD,CLAUDE.md,CLAUDE.MDper directory from the file-system root down to the working directory. Crush has a longer default list includingCRUSH.mdand.cursorrules. - Crush's
todostool takes the whole list with statusespending,in_progress,completed, and its summary prompt includes the current todos. - Crush's
agenttool runs a subagent in a new session with the read-only toolsglob,grep,ls, andview. - With a stable layout, the second request of a conversation took all 188 tokens of the first from the cache (
cache_n188). With a timestamp in the system prompt it reused 40 tokens and computed 227. ContextManagertruncates on add, compacts inrequest()above 80 percent of the window minus the reserve, and cuts only at user messages.