Making Small Local Models Reliable

A small local model is a language model with about 1 billion to 14 billion parameters (the learned numbers inside it) that runs on your own computer. When such a model works as a coding agent, it has to write tool calls. A tool call is a short piece of structured text that asks the program to do something, for example "run the tool read_file with the path pom.xml". The program reads that text and runs the tool. If one bracket is missing or one name is wrong, the program cannot run the call, and the agent stops making progress.

Large hosted models rarely make these mistakes. Small models make them often. They write the call in the wrong format, break the JSON (the text format that carries the call), invent a tool or an argument name, stop in the middle of a call, or repeat the same call again and again. These failures are predictable, so each one has a known remedy.

This lesson takes the failures one at a time. For each one it shows a technique that prevents the failure or recovers from it, and it measures the effect with numbers. Afterwards you can configure an agent so that a 7B model on a laptop finishes tasks that it would otherwise abandon, and you can measure which technique helped instead of guessing.

The real outputs come from llama-server at the pinned llama.cpp commit 0c1e57098, running the course's tiny model with random weights. A model with random weights has learned nothing, so it writes meaningless text. That is useful here. A random model cannot "understand" a request, so any good result comes from the technique and not from the model's intelligence. The Java examples run against scripted model replies (replies written in advance), so their output is the same on every run.

The message types (ToolCall, AssistantMessage, ToolMessage) are the ones from Project 21 in The Agent Loop and a Tool Framework. The tool names read_file, edit_file, and shell are the ones the earlier tool lessons use.

Example 1: a catalog of small-model failures

Before fixing failures, an agent needs to recognize them. This example builds a classifier: a function that looks at one raw model reply and names what is wrong with it. Naming the failure matters because each failure has a different remedy, and the catalog below links each name to the example that handles it.

The replies use the Hermes format that Qwen models write: a <tool_call> tag, then a JSON object with a name and arguments, then a closing tag. Function Calling from First Principles described this format. The program checks eleven replies, and each reply shows one failure. It also reads a field called finish_reason, which tells why the model stopped writing: stop means the model ended its reply by itself, and length means the server cut the reply at the max_tokens limit.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class FailureCatalog {
    static final ObjectMapper JSON = new ObjectMapper();
    static final Map<String, List<String>> TOOLS = Map.of(          // tool name -> required string arguments
        "read_file", List.of("path"), "edit_file", List.of("path", "old_string", "new_string"), "shell", List.of("command"));
    static final Set<String> FILES = Set.of("pom.xml", "src/main/java/app/OrderService.java");
    static final Pattern CALL = Pattern.compile("<tool_call>\\s*(.*?)\\s*(</tool_call>|$)", Pattern.DOTALL);
    static final Map<String, String> REMEDY = Map.ofEntries(
        Map.entry("ok", "run it"),
        Map.entry("call outside the call format", "grammar (Example 2) or a fallback parser"),
        Map.entry("invalid JSON", "repair (Example 4)"),
        Map.entry("unknown tool", "error with the tool list, retry (Example 5)"),
        Map.entry("missing argument", "error with the schema, retry (Example 5)"),
        Map.entry("wrong argument type", "error with the schema, retry (Example 5)"),
        Map.entry("cut off by max_tokens", "never run it; retry with more tokens"),
        Map.entry("valid call, path does not exist", "the tool's own error result"),
        Map.entry("same call as the previous step", "loop detection (Example 11)"),
        Map.entry("text only, no call", "accept, or check that work was verified"));
    record Reply(String text, String finishReason) {}

    static String classify(Reply r, String previousCall) {
        Matcher m = CALL.matcher(r.text());
        if (!m.find()) return r.text().contains("\"name\"") ? "call outside the call format" : "text only, no call";
        if (r.finishReason().equals("length")) return "cut off by max_tokens";
        JsonNode call;
        try { call = JSON.readTree(m.group(1)); } catch (Exception e) { return "invalid JSON"; }
        String name = call.path("name").asText();
        if (!TOOLS.containsKey(name)) return "unknown tool";
        JsonNode args = call.path("arguments");
        for (String required : TOOLS.get(name)) {
            if (!args.has(required)) return "missing argument";
            if (!args.get(required).isTextual()) return "wrong argument type";
        }
        if (call.toString().equals(previousCall)) return "same call as the previous step";
        if (name.equals("read_file") && !FILES.contains(args.get("path").asText())) return "valid call, path does not exist";
        return "ok";
    }

    public static void main(String[] args) {
        String fence = "`".repeat(3);
        List<Reply> replies = List.of(
            new Reply("<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>", "tool_calls"),
            new Reply("I will read it.\n" + fence + "json\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n" + fence, "stop"),
            new Reply("<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\",}}\n</tool_call>", "stop"),
            new Reply("<tool_call>\n{'name': 'read_file', 'arguments': {'path': 'pom.xml'}}\n</tool_call>", "stop"),
            new Reply("<tool_call>\n{\"name\": \"cat\", \"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>", "stop"),
            new Reply("<tool_call>\n{\"name\": \"shell\", \"arguments\": {\"cmd\": \"mvn test\"}}\n</tool_call>", "stop"),
            new Reply("<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": [\"pom.xml\"]}}\n</tool_call>", "stop"),
            new Reply("<tool_call>\n{\"name\": \"edit_file\", \"arguments\": {\"path\": \"src/main/ja", "length"),
            new Reply("<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"src/OrderService.java\"}}\n</tool_call>", "stop"),
            new Reply("<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"src/OrderService.java\"}}\n</tool_call>", "stop"),
            new Reply("The tests pass now.", "stop"));
        String previous = "";
        for (int i = 0; i < replies.size(); i++) {
            String kind = classify(replies.get(i), previous);
            System.out.printf("%2d. %-31s -> %s%n", i + 1, kind, REMEDY.get(kind));
            Matcher m = CALL.matcher(replies.get(i).text());
            try { previous = m.find() ? JSON.readTree(m.group(1)).toString() : ""; } catch (Exception e) { previous = ""; }
        }
    }
}

 1. ok                              -> run it
 2. call outside the call format    -> grammar (Example 2) or a fallback parser
 3. invalid JSON                    -> repair (Example 4)
 4. invalid JSON                    -> repair (Example 4)
 5. unknown tool                    -> error with the tool list, retry (Example 5)
 6. missing argument                -> error with the schema, retry (Example 5)
 7. wrong argument type             -> error with the schema, retry (Example 5)
 8. cut off by max_tokens           -> never run it; retry with more tokens
 9. valid call, path does not exist -> the tool's own error result
10. same call as the previous step  -> loop detection (Example 11)
11. text only, no call              -> accept, or check that work was verified

Each output line is one reply, the failure kind that the classifier found, and the remedy that the rest of this lesson builds. The classifier checks in a fixed order: is a call present, was it cut by the token limit, does the JSON parse, does the tool exist, are the arguments right, is it a repeat. The order matters, because a later check is meaningless when an earlier one fails. You cannot check the argument names of a call that does not parse.

• Reply 1 is a correct call. It is the only reply an agent can run without help.
• Reply 2 contains a correct call, but inside a Markdown code fence after a sentence, and without the <tool_call> tags. A server that looks for tool calls does not recognize it, so it arrives as plain text. The classifier detects it because the text contains "name".
• Replies 3 and 4 are invalid JSON. Reply 3 has a trailing comma (a comma after the last item), reply 4 uses single quotes. Both are legal in Python and JavaScript literals and illegal in JSON. They are mechanical errors, and a repair step can fix them without asking the model again.
• Reply 5 calls cat, a tool that does not exist. The model wrote a familiar shell command name instead of read_file.
• Reply 6 calls shell with cmd instead of command. The argument name is plausible but wrong, so the required argument is missing.
• Reply 7 passes path as an array. The schema (the description of allowed arguments) says string.
• Reply 8 stopped at max_tokens in the middle of a path. finish_reason is length, so the call is incomplete and must never run.
• Reply 9 is a valid call to a path that does not exist (src/OrderService.java instead of src/main/java/app/OrderService.java). Nothing about the format is wrong. The tool itself reports the error, and the model reads it and can try again.
• Reply 10 repeats reply 9 exactly. One repeat can be a retry. Several in a row is a loop.
• Reply 11 has no call. Whether this is a failure depends on the history. Here the model claims the tests pass without having run them.

The remedies fall into three groups:

• Prevention stops the failure from happening: a grammar (Example 2), sampling settings (Example 6), and fewer and clearer tools (Example 7).
• Recovery fixes a failure after it happened: repair (Example 4), validation and retry (Example 5), and loop detection (Example 11).
• Measurement shows which technique helps: the harness of Example 13. It also shows that no format technique fixes an unverified final answer.

Example 2: grammar-constrained tool calls in llama-server

The first remedy tries to make most failures impossible. A grammar is a set of rules that describes exactly which text is allowed. Grammar-constrained decoding applies those rules while the model writes. At every step the model scores all possible next tokens (a token is a small piece of text, often a word or part of a word). The server removes every token that the grammar does not allow, so the output always matches the grammar. Constrained Decoding and Structured Output explains the mechanism in detail.

llama-server builds a grammar from the tool definitions whenever a request contains tools. This example asks what that grammar does on a real server and where its guarantee ends. The tool_choice request field has three values that matter here:

• auto lets the model decide whether to call a tool.
• required demands a call.
• none forbids calls.

One more term: a prefill is a final assistant message that the server continues instead of starting a new message. It lets the client put the first words in the model's mouth.

The script sends the same question 24 times: four configurations with six seeds each, at temperature 0.8. A seed makes the random sampling repeatable, so the same seed and the same settings give the same text. The server runs on a free port with the Qwen 2.5 template, which describes tools in its system turn. --jinja is the default at this commit and is written out for clarity.

PORT=38351   # any free port
llama-server -m tiny-random-qwen2vocab-f32.gguf --host 127.0.0.1 --port $PORT \
  --jinja --chat-template-file models/templates/Qwen-Qwen2.5-7B-Instruct.jinja -c 4096 -np 1 -t 2 &

TOOL='{"type": "function", "function": {"name": "read_file", "description": "Read a text file from the workspace",
  "parameters": {"type": "object", "properties": {"path": {"type": "string", "enum": ["pom.xml", "src/Main.java"]}}, "required": ["path"]}}}'
ask() {  # $1 = tool_choice, $2 = prefill (yes/no), $3 = max_tokens, $4 = seed, $5 = extra request fields
  local msgs='[{"role": "user", "content": "What is in pom.xml?"}]'
  [ "$2" = yes ] && msgs='[{"role": "user", "content": "What is in pom.xml?"}, {"role": "assistant", "content": "<tool_call>\n"}]'
  curl -s 127.0.0.1:$PORT/v1/chat/completions -d '{"messages": '"$msgs"', "tools": ['"$TOOL"'],
    "tool_choice": "'"$1"'", "parallel_tool_calls": false'"$5"',
    "temperature": 0.8, "seed": '"$4"', "max_tokens": '"$3"'}' \
  | jq -c --arg cfg "$1 prefill=$2" --arg seed "$4" '.choices[0] | {cfg: $cfg, seed: ($seed|tonumber), finish: .finish_reason,
      calls: [(.message.tool_calls // [])[] | .function.arguments | (try fromjson.path catch "not JSON")], content: (.message.content // "" | .[0:30])}'
}
for seed in 1 2 3 4 5 6; do ask auto no 40 $seed ""; done
for seed in 1 2 3 4 5 6; do ask none yes 40 $seed ""; done
for seed in 1 2 3 4 5 6; do ask required no 40 $seed ""; done
for seed in 1 2 3 4 5 6; do ask required yes 200 $seed ', "logit_bias": {"151645": 100, "522": 100}'; done
kill %1

Real output (from the pinned build; the model has random weights):

{"cfg":"auto prefill=no","seed":1,"finish":"length","calls":[],"content":"_initializer旃匿ropount.chain.ge"}
{"cfg":"auto prefill=no","seed":2,"finish":"length","calls":[],"content":"CreateMap밨תחרshelfinsn拿到了Ped C"}
{"cfg":"auto prefill=no","seed":3,"finish":"length","calls":[],"content":"(stepことがあるshelfapestdetalleins"}
{"cfg":"auto prefill=no","seed":4,"finish":"length","calls":[],"content":"villa PodsDummy wikipedia_gene"}
{"cfg":"auto prefill=no","seed":5,"finish":"length","calls":[],"content":"ことがある allaldb вос;jįHar纪委 Jeספ"}
{"cfg":"auto prefill=no","seed":6,"finish":"length","calls":[],"content":"=function磕.HtmlControls UNESCO"}
{"cfg":"none prefill=yes","seed":1,"finish":"length","calls":[],"content":"<tool_call>\nочно Distribinsnaz"}
{"cfg":"none prefill=yes","seed":2,"finish":"length","calls":[],"content":"<tool_call>\nsterreich_genes拿到了"}
{"cfg":"none prefill=yes","seed":3,"finish":"length","calls":[],"content":"<tool_call>\nmanaged rearr匿alar"}
{"cfg":"none prefill=yes","seed":4,"finish":"length","calls":[],"content":"<tool_call>\nолов想办法万个managedst"}
{"cfg":"none prefill=yes","seed":5,"finish":"length","calls":[],"content":"<tool_call>\nshelf--*/\nstoppedA"}
{"cfg":"none prefill=yes","seed":6,"finish":"length","calls":[],"content":"<tool_call>\n走得굳.getPasswordcas"}
{"cfg":"required prefill=no","seed":1,"finish":"length","calls":[],"content":"_initializer旃匿ropount.chain.ge"}
{"cfg":"required prefill=no","seed":2,"finish":"length","calls":[],"content":"CreateMap밨תחרshelfinsn拿到了Ped C"}
{"cfg":"required prefill=no","seed":3,"finish":"length","calls":[],"content":"(stepことがあるshelfapestdetalleins"}
{"cfg":"required prefill=no","seed":4,"finish":"length","calls":[],"content":"villa PodsDummy wikipedia_gene"}
{"cfg":"required prefill=no","seed":5,"finish":"length","calls":[],"content":"ことがある allaldb вос;jįHar纪委 Jeספ"}
{"cfg":"required prefill=no","seed":6,"finish":"length","calls":[],"content":"=function磕.HtmlControls UNESCO"}
{"cfg":"required prefill=yes","seed":1,"finish":"length","calls":["src/Main.java","src/Main.java","src/Main.java","not JSON"],"content":""}
{"cfg":"required prefill=yes","seed":2,"finish":"tool_calls","calls":["pom.xml"],"content":""}
{"cfg":"required prefill=yes","seed":3,"finish":"tool_calls","calls":["pom.xml","src/Main.java"],"content":""}
{"cfg":"required prefill=yes","seed":4,"finish":"tool_calls","calls":["src/Main.java","src/Main.java"],"content":""}
{"cfg":"required prefill=yes","seed":5,"finish":"tool_calls","calls":["pom.xml","src/Main.java"],"content":""}
{"cfg":"required prefill=yes","seed":6,"finish":"tool_calls","calls":["pom.xml"],"content":""}

Each output line is one request: the configuration, the seed, finish_reason, the path of every parsed call (or not JSON when the arguments string does not parse), and the first 30 characters of content. Read the four configurations in turn.

Configuration 1, auto without help. All six requests end with length after 40 tokens of random text and no call. With tool_choice: "auto" the server builds a lazy grammar: one that stays inactive until a trigger text appears, here <tool_call> followed by a newline. A trained model writes the trigger when it decides to call a tool. A random model never writes it, so the grammar never switches on.

Configuration 2, none with a prefilled call opening. The final assistant message <tool_call>\n is a prefill, so the output starts inside a call. With none the server adds no grammar and parses no calls. The content after <tool_call>\n is random text, so a client parser finds a call opening followed by garbage. The result is 0 of 6 replies with a call that parses.

Configuration 3, required without a prefill. The content is identical, character for character, to configuration 1 with the same seed, and every request ends with length. This is a good check of what a grammar does. required makes the grammar active from the first token, but the grammar allows any text before the first <tool_call> (Example 3 shows the rule). The random text never contained the marker, so the grammar removed no token that the model chose, and the same seed produced the same text. Such a grammar does not force a call within any token limit.

Configuration 4, required with the prefill and a logit bias. All six requests produced calls, and every complete call has a path from the enum (the fixed list of allowed values). Two request fields help the random model. The prefill puts the output inside a call from the first token. A logit bias is a number added to a token's score before sampling. A bias of +100 on </ (id 522) and <|im_end|> (id 151645) makes the model close the call and the turn as soon as the grammar allows it. A trained model needs neither field. The results:

• Seeds 2 and 6 produced one call each and ended with tool_calls.
• Seeds 3, 4, and 5 produced two calls each, although the request said "parallel_tool_calls": false. Example 3 shows why: at this commit the grammar's call list is "one or more calls".
• Seed 1 produced three complete calls and a fourth that the token limit cut after its first character. finish_reason is length, and the fourth call's arguments string is {, which is not JSON. The grammar guarantees that every prefix of the output is valid. It cannot guarantee that the output is complete when max_tokens runs out.

The tally for configuration 4 is a short sum:

seed 1: 3 complete calls (+1 cut off)   3
seed 2: 1                               1
seed 3: 2                               2
seed 4: 2                               2
seed 5: 2                               2
seed 6: 1                               1
                                       --
complete calls                         11

Across the 24 requests, the grammar produced 11 complete calls that all parse and all match the schema, and 1 cut-off call. Without a grammar, 0 of 6 attempts parsed. The grammar guarantees the shape, not the choice. The question asked about pom.xml, and 7 of the 11 calls chose src/Main.java, the other value the enum allowed. A random model does not read the question, and the grammar cannot read it either.

For an agent, three rules follow:

1. Use the grammar (the server builds it from tools).
2. Check finish_reason and never run a call that came with length.
3. Check the number of calls when the agent expects one.

Example 3: the grammar llama-server builds for a tool call

Example 2 showed that a grammar changes what the model can write. This example opens the grammar and reads it, so that its guarantees and its gaps are clear. A request with "verbose": true returns the server's internal settings under __verbose, including the grammar in GBNF, the grammar format of llama.cpp. The request below is configuration 4 of Example 2 with max_tokens set to 1, since only the settings matter and no real text is needed.

curl -s 127.0.0.1:$PORT/v1/chat/completions -d '{"messages": [{"role": "user", "content": "What is in pom.xml?"},
    {"role": "assistant", "content": "<tool_call>\n"}], "tools": ['"$TOOL"'], "tool_choice": "required",
  "parallel_tool_calls": false, "temperature": 0.8, "seed": 1, "max_tokens": 1, "verbose": true}' \
| jq -r '.__verbose.generation_settings | .grammar, "lazy: \(.grammar_lazy)", "generation_prompt: \(.generation_prompt | @json)"'

Real output (the generic json-* rules and 9 of the 12 until-161 rules removed):

root ::= "<|im_start|>assistant\n" space until-161? tool-calls
space ::= | " " | "\n"{1,2} [ \t]{0,20}
tool-call ::= "<tool_call>\n" space (tool-read-file) space "</tool_call>"
tool-calls ::= (tool-call space)+
tool-read-file ::= "{" space "\"name\"" space ":" space ("\"" "read_file" "\"") space "," space "\"arguments\"" space ":" space tool-read-file-schema space "}"
tool-read-file-schema ::= "{" space tool-read-file-schema-path-kv space "}"
tool-read-file-schema-path ::= ("\"pom.xml\"" | "\"src/Main.java\"")
tool-read-file-schema-path-kv ::= "\"path\"" space ":" space tool-read-file-schema-path
until-161 ::= | [<] until-161-01 | [^<] until-161
until-161-01 ::= | [<] until-161-01 | [t] until-161-02 | [^<t] until-161
until-161-11 ::= | [<] until-161-01 | [^\n<] until-161
lazy: false
generation_prompt: "<|im_start|>assistant\n<tool_call>\n"

A grammar in GBNF is a list of rules. Each rule has a name, then ::= (read it as "is defined as"), then what the name may expand to. A few symbols are enough to read this output:

Reading the rules from the top:

• root is the whole output: the start of the assistant turn, optional whitespace, optional free text (until-161?), then tool-calls. The question mark makes the free text optional. This rule explains configuration 3 of Example 2: the model may write anything before the call.
• until-161 and its numbered helpers form a small automaton (a machine that tracks its position through a pattern). It accepts any text that does not contain <tool_call>\n. Each helper remembers how much of the marker has been seen: until-161-01 after <, until-161-02 after <t. When the marker is complete, the free text ends and a call must follow.
• tool-calls ::= (tool-call space)+ accepts one or more calls. This rule is the reason for the two-call replies in Example 2: at this commit the generated grammar does not limit the count when parallel_tool_calls is false. The agent therefore checks the count itself.
• tool-call is the marker, the JSON object, and </tool_call>. tool-read-file fixes the key order (name first, then arguments) and the literal tool name. With several tools, this position holds an alternation with one rule per tool.
• tool-read-file-schema-path is the enum turned into an alternation of two literal strings. This is how a schema restricts values: a token that would start any other string has no path through the grammar, so the model cannot choose it.
• space allows nothing, one space, or one or two newlines followed by up to 20 spaces or tabs. The random model's calls in Example 2 are full of tabs because this rule permits them.

Two more output lines matter. lazy: false confirms that required builds a grammar that applies from the first token. The auto mode builds similar call rules with lazy: true and the trigger word <tool_call>\n. generation_prompt is the text that the chat template already placed at the end of the prompt, including the prefill. The server code in common/sampling.cpp feeds these tokens to the grammar before sampling begins. The prefilled <tool_call>\n therefore matches the start of tool-call, and the first token the model generates is already inside the JSON object.

The grammar removes these failures from Example 1 at the source: the call outside the format, invalid JSON, an unknown tool, a missing argument, a wrong type, and a value outside an enum. It leaves three: a cut-off call, more calls than expected, and a valid call with the wrong values.

Example 4: JSON repair

A grammar is not always available. Some engines and hosted endpoints do not constrain tool calls. A model can write a call as plain text (reply 2 of Example 1). Prompt-based tool calling parses the reply in the client, with no server help. In these cases the client receives text that is almost JSON. JSON repair is a parser that accepts the common mistakes and writes valid JSON, without another model call. This example answers which mistakes a repair step can fix safely and which it must refuse. The program repairs eight inputs and reports which repairs it made.

import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;

public class JsonRepair {
    record Result(String json, List<String> repairs) {}

    /** Repairs the common syntax errors of model-written JSON objects. Returns null when there is no object. */
    static Result repair(String text) {
        List<String> repairs = new ArrayList<>();
        int start = text.indexOf('{');
        if (start < 0) return null;
        if (start > 0 || !text.strip().endsWith("}")) repairs.add("text around the object");
        StringBuilder out = new StringBuilder();
        Deque<Character> open = new ArrayDeque<>();
        char quote = 0;                                           // 0 = outside a string, else ' or "
        int end = text.length();
        for (int i = start; i < end; i++) {
            char c = text.charAt(i);
            if (quote != 0) {                                     // inside a string
                if (c == '\\' && i + 1 < end) { out.append(c).append(text.charAt(++i)); continue; }
                if (c == quote) { out.append('"'); quote = 0; continue; }
                if (c == '"') { out.append("\\\""); continue; }   // a double quote inside a single-quoted string
                if (c == '\n') { out.append("\\n"); add(repairs, "raw newline in a string"); continue; }
                if (c == '\t') { out.append("\\t"); add(repairs, "raw tab in a string"); continue; }
                out.append(c);
                continue;
            }
            if (c == '"' || c == '\'') {
                if (c == '\'') add(repairs, "single quotes");
                quote = c; out.append('"');
            } else if (c == '{' || c == '[') {
                open.push(c); out.append(c);
            } else if (c == '}' || c == ']') {
                if (dropTrailingComma(out)) add(repairs, "trailing comma");
                open.pop(); out.append(c);
                if (open.isEmpty()) { end = i + 1; break; }       // the object is complete; ignore what follows
            } else if (text.startsWith("True", i) || text.startsWith("False", i) || text.startsWith("None", i)) {
                String word = text.startsWith("True", i) ? "True" : text.startsWith("False", i) ? "False" : "None";
                out.append(switch (word) { case "True" -> "true"; case "False" -> "false"; default -> "null"; });
                add(repairs, "Python literal " + word);
                i += word.length() - 1;
            } else out.append(c);
        }
        if (quote != 0 || !open.isEmpty()) {                      // the text ended inside the object
            add(repairs, "truncated");
            if (quote != 0) out.append('"');
            dropTrailingComma(out);
            if (out.toString().stripTrailing().endsWith(":")) out.append("null");
            while (!open.isEmpty()) out.append(open.pop() == '{' ? '}' : ']');
        }
        return new Result(out.toString(), repairs);
    }

    static boolean dropTrailingComma(StringBuilder out) {
        int i = out.length() - 1;
        while (i >= 0 && Character.isWhitespace(out.charAt(i))) i--;
        if (i >= 0 && out.charAt(i) == ',') { out.setLength(i); return true; }
        return false;
    }

    static void add(List<String> list, String repair) { if (!list.contains(repair)) list.add(repair); }

    public static void main(String[] args) {
        ObjectMapper json = new ObjectMapper();
        String[] inputs = {
            "{\"path\": \"pom.xml\",}",
            "{'path': 'src/Main.java'}",
            "{\"path\": \"A.java\", \"new_string\": \"int a;\n\tint b;\"}",
            "{\"command\": \"mvn -q test\", \"verbose\": True}",
            "Sure: {\"path\": \"pom.xml\"} I will read it now.",
            "{\"path\": \"src/Ma",
            "{\"path\": \"A.java\", \"old_string\": \"int a;\", \"new_string\":",
            "path=pom.xml"};
        for (String in : inputs) {
            Result r = repair(in);
            System.out.println("in:      " + in.replace("\n", "\\n").replace("\t", "\\t"));
            if (r == null) { System.out.println("result:  no JSON object, nothing to repair\n"); continue; }
            String parses;
            try { json.readTree(r.json()); parses = "parses"; } catch (Exception e) { parses = "still invalid"; }
            System.out.println("out:     " + r.json() + "  (" + parses + ")");
            System.out.println("repairs: " + r.repairs() + "\n");
        }
    }
}

in:      {"path": "pom.xml",}
out:     {"path": "pom.xml"}  (parses)
repairs: [trailing comma]

in:      {'path': 'src/Main.java'}
out:     {"path": "src/Main.java"}  (parses)
repairs: [single quotes]

in:      {"path": "A.java", "new_string": "int a;\n\tint b;"}
out:     {"path": "A.java", "new_string": "int a;\n\tint b;"}  (parses)
repairs: [raw newline in a string, raw tab in a string]

in:      {"command": "mvn -q test", "verbose": True}
out:     {"command": "mvn -q test", "verbose": true}  (parses)
repairs: [Python literal True]

in:      Sure: {"path": "pom.xml"} I will read it now.
out:     {"path": "pom.xml"}  (parses)
repairs: [text around the object]

in:      {"path": "src/Ma
out:     {"path": "src/Ma"}  (parses)
repairs: [text around the object, truncated]

in:      {"path": "A.java", "old_string": "int a;", "new_string":
out:     {"path": "A.java", "old_string": "int a;", "new_string":null}  (parses)
repairs: [text around the object, truncated]

in:      path=pom.xml
result:  no JSON object, nothing to repair

The repair is a single scan over the characters of the input. It keeps three pieces of state: whether the scan is inside a string and which quote opened it, a stack of open brackets (so it knows what to close), and the list of repairs made. Each block of output is one input:

• Trailing comma. {"path": "pom.xml",} becomes valid when the comma before } is dropped. dropTrailingComma runs before every closing bracket and removes a comma that is followed only by whitespace.
• Single quotes. Every ' that opens or closes a string is written as ". A double quote inside a single-quoted string is escaped, so 'say "hi"' becomes "say \"hi\"".
• Raw newline and tab. JSON forbids raw control characters inside strings. The third input has a real newline and a real tab inside new_string. The input line shows them as \n and \t only because the program prints them escaped. The repair writes the two-character escapes instead. Code in edit_file calls is the usual source of this error, since a model writes code with real line breaks.
• Python literal. True becomes true. The repair also handles False and None, but only outside strings, so a string such as "None of the tests" is left alone.
• Text around the object. The scan starts at the first { and stops when the bracket stack is empty, so Sure: and I will read it now. are dropped.
• Truncation in a string. {"path": "src/Ma ended inside a string. The repair closes the string and the object, and the result parses: {"path": "src/Ma"}. The JSON is now valid and the value is wrong. A file named src/Ma does not exist, and on an edit_file call the same repair would write half a line of code.
• Truncation after a key. The repair adds null after the dangling "new_string": and closes the object. The result parses and would replace int a; with nothing.
• No object. path=pom.xml has no {, so there is nothing to repair. This reply goes to the retry step of Example 5.

The two truncation cases show the limit of repair. The other repairs change only syntax, so the repaired text means what the model meant. A truncation repair invents an ending, so the result looks valid but may do the wrong thing. The policy that follows has two parts. Apply syntax repairs silently. Treat truncated as a failure that goes back to the model with a higher max_tokens. The repairs list tells the agent which case it has. Example 13 uses this split.

One more point for measurements: a parse rate measured after repair is higher than the model's own rate. A report must state which of the two it shows.

Example 5: validation and retry with the error message

Repair fixes syntax. It cannot fix a call that parses but means the wrong thing, such as the right tool with a wrong argument name. For that case the agent validates the arguments against the tool's schema. If they do not match, it sends a precise error back to the model as the tool result and asks again, up to a retry limit. The model reads the error and usually corrects itself. This example shows the loop. The program uses the Project 21 message records and replays scripted replies in place of the model, so the output is the same on every run.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

public class ValidateAndRetry {
    static final ObjectMapper JSON = new ObjectMapper();
    sealed interface Message permits UserMessage, AssistantMessage, ToolMessage {}
    record UserMessage(String text) implements Message {}
    record ToolCall(String id, String name, String arguments) {}
    record AssistantMessage(String text, List<ToolCall> toolCalls) implements Message {}
    record ToolMessage(String callId, String name, String content, boolean isError) implements Message {}

    static final Map<String, String> PARAMS = Map.of("path", "string");      // read_file: one required string

    static List<String> validate(String arguments) {
        List<String> problems = new ArrayList<>();
        JsonNode args;
        try { args = JSON.readTree(arguments); } catch (Exception e) { return List.of("arguments are not valid JSON"); }
        if (!args.isObject()) return List.of("arguments must be a JSON object");
        for (String p : PARAMS.keySet()) if (!args.has(p)) problems.add("missing required argument '" + p + "'");
        for (var e : args.properties()) {
            if (!PARAMS.containsKey(e.getKey())) problems.add("unknown argument '" + e.getKey() + "'");
            else if (!e.getValue().isTextual()) problems.add("'" + e.getKey() + "' must be a string, got " + e.getValue());
        }
        return problems;
    }

    static String errorText(List<String> problems) {
        return "Invalid arguments for read_file: " + String.join("; ", problems)
            + ". Expected {\"path\": \"<file path>\"}. Call read_file again with corrected arguments.";
    }

    /** Asks until a call validates or maxRetries retries are used; failed attempts and errors go into the history. */
    static ToolCall callWithRetry(Iterator<AssistantMessage> model, List<Message> history, int maxRetries) {
        for (int attempt = 1; attempt <= maxRetries + 1; attempt++) {
            AssistantMessage reply = model.next();                 // stands in for a request with the whole history
            history.add(reply);
            ToolCall call = reply.toolCalls().get(0);
            List<String> problems = validate(call.arguments());
            System.out.println("  attempt " + attempt + ": " + call.arguments() + " -> " + (problems.isEmpty() ? "valid" : problems));
            if (problems.isEmpty()) return call;
            history.add(new ToolMessage(call.id(), call.name(), errorText(problems), true));
        }
        return null;
    }

    static AssistantMessage call(String id, String arguments) {
        return new AssistantMessage("", List.of(new ToolCall(id, "read_file", arguments)));
    }

    public static void main(String[] args) {
        System.out.println("scenario 1: wrong argument name, fixed on the first retry");
        List<Message> history = new ArrayList<>(List.of(new UserMessage("What is in pom.xml?")));
        Iterator<AssistantMessage> model = List.of(call("c1", "{\"file\": \"pom.xml\"}"), call("c2", "{\"path\": \"pom.xml\"}")).iterator();
        ToolCall ok = callWithRetry(model, history, 2);
        System.out.println("  result: run " + ok.id() + ", history has " + history.size() + " messages");
        System.out.println("  error sent back: " + ((ToolMessage) history.get(2)).content());
        List<Message> pruned = new ArrayList<>(List.of(history.get(0), history.get(history.size() - 1)));
        System.out.println("  pruned after success: " + pruned.size() + " messages (failed attempt and its error removed)");

        System.out.println("scenario 2: three different mistakes, retry budget 2");
        history = new ArrayList<>(List.of(new UserMessage("What is in pom.xml?")));
        model = List.of(call("c1", "{\"file\": \"pom.xml\"}"), call("c2", "{\"paths\": [\"pom.xml\"]}"),
                        call("c3", "{\"path\": 7}")).iterator();
        ToolCall failed = callWithRetry(model, history, 2);
        System.out.println("  result: " + (failed == null ? "give up after 3 attempts, report to the user" : "run")
            + ", history has " + history.size() + " messages");
    }
}

scenario 1: wrong argument name, fixed on the first retry
  attempt 1: {"file": "pom.xml"} -> [missing required argument 'path', unknown argument 'file']
  attempt 2: {"path": "pom.xml"} -> valid
  result: run c2, history has 4 messages
  error sent back: Invalid arguments for read_file: missing required argument 'path'; unknown argument 'file'. Expected {"path": "<file path>"}. Call read_file again with corrected arguments.
  pruned after success: 2 messages (failed attempt and its error removed)
scenario 2: three different mistakes, retry budget 2
  attempt 1: {"file": "pom.xml"} -> [missing required argument 'path', unknown argument 'file']
  attempt 2: {"paths": ["pom.xml"]} -> [missing required argument 'path', unknown argument 'paths']
  attempt 3: {"path": 7} -> ['path' must be a string, got 7]
  result: give up after 3 attempts, report to the user, history has 7 messages

Scenario 1. The first attempt sends {"file": "pom.xml"}. The validator reports two problems: the required path is missing, and file is not a known argument. The agent appends the assistant message and a ToolMessage with isError set to true (a flag that tells the model the tool failed). The full error text is printed on its own line. It has three parts: what is wrong, the shape that is expected ({"path": "<file path>"}), and what to do next. Together they contain everything the corrected call needs. A bare "invalid arguments" does not. The second attempt is valid, and the agent runs call c2.

The history now has 4 messages: the user question, the failed attempt, its error, and the valid attempt. The failed pair stays valid for the chat API, because every call has a result. The line pruned after success: 2 messages shows an alternative. Once the corrected call has run, the agent can remove the failed attempt and its error and keep the question and the valid call (its result follows when the tool runs). The pruned history is shorter, and the model never sees its own mistake again, which removes a pattern that a small model may copy. The cost is that the beginning of the prompt changes, so the server's prompt cache cannot reuse the tokens after the removed messages. The prompt cache is the server's memory of already processed prompt tokens, and Context Engineering for Coding Agents measures its reuse with cache_n.

Scenario 2. The model makes a different mistake each time: the wrong name, then an array under another wrong name (paths), then a number. The validator catches each one, and the third message is 'path' must be a string, got 7. After the retry budget of 2 is used, the agent stops asking and reports the failure. The history has 7 messages: the question and three pairs of attempt and error.

A retry limit is needed because each retry costs a full model request with the whole history. When the budget is used, the remaining options are outside the loop: report to the user, or simplify the tool (Example 7).

Example 6: sampling settings for tool use

Even with a grammar, the model still chooses among the tokens that the grammar allows. Sampling settings decide how that choice is made. This example asks how temperature and the sampler chain change the chance of a wrong token inside a tool call. A sampler is the step that picks the next token from the model's scores. Temperature controls how much randomness the pick has: low values favor the top-scoring token, high values spread the choice. Sampling: Greedy, Temperature, Top-k, Top-p, and Min-p and Penalties, Logit Bias, and Sampler Chains define each sampler.

The program computes, for one decision inside a call, the probability of each kind of token under several settings. It then shows the effect of a repeat penalty on the " token.

import java.util.Locale;

public class SamplingForTools {
    /** Probability of each token after min-p filtering and temperature, in llama.cpp's order (min_p before temperature). */
    static double[] probs(double[] logits, double minP, double temp) {
        double max = Double.NEGATIVE_INFINITY;
        for (double l : logits) max = Math.max(max, l);
        if (temp <= 0) {                                          // greedy: all mass on the top token
            double[] p = new double[logits.length];
            for (int i = 0; i < logits.length; i++) if (logits[i] == max) { p[i] = 1; break; }
            return p;
        }
        double cutoff = max + Math.log(minP);                     // keep p_i >= minP * p_max
        double[] p = new double[logits.length];
        double sum = 0;
        for (int i = 0; i < logits.length; i++)
            if (minP == 0 || logits[i] >= cutoff) { p[i] = Math.exp((logits[i] - max) / temp); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        // Next token after {"name": "  : the right tool, two plausible wrong ones, and 20 unrelated tokens.
        double[] logits = new double[23];
        logits[0] = 4.0; logits[1] = 2.5; logits[2] = 1.5;          // read_file, read, list_files
        for (int i = 3; i < logits.length; i++) logits[i] = -0.5;  // tail
        System.out.println("temp  min_p  P(read_file)  P(wrong tool)  P(tail)  P(5 such steps all right)");
        for (double temp : new double[] {0.0, 0.2, 0.6, 1.0})
            for (double minP : new double[] {0.0, 0.05}) {
                if (temp == 0 && minP > 0) continue;
                double[] p = probs(logits, minP, temp);
                double tail = 0;
                for (int i = 3; i < p.length; i++) tail += p[i];
                System.out.printf(Locale.ROOT, "%4.1f  %5.2f  %12.4f  %13.4f  %7.4f  %25.4f%n",
                    temp, minP, p[0], p[1] + p[2], tail, Math.pow(p[0], 5));
            }

        // Repeat penalty on the token " (it appeared in the last 64 tokens of a JSON call).
        double[] json = {3.0, 1.0, 0.5};                          // ", ', [
        System.out.println("repeat_penalty  logit(\")  P(\")");
        for (double penalty : new double[] {1.0, 1.1, 1.3, 1.5}) {
            double[] l = json.clone();
            l[0] = l[0] > 0 ? l[0] / penalty : l[0] * penalty;    // llama.cpp: divide positive logits, multiply negative
            double[] p = probs(l, 0.0, 1.0);
            System.out.printf(Locale.ROOT, "%14.1f  %9.3f  %.4f%n", penalty, l[0], p[0]);
        }
    }
}

temp  min_p  P(read_file)  P(wrong tool)  P(tail)  P(5 such steps all right)
 0.0   0.00        1.0000         0.0000   0.0000                     1.0000
 0.2   0.00        0.9994         0.0006   0.0000                     0.9972
 0.2   0.05        0.9994         0.0006   0.0000                     0.9972
 0.6   0.00        0.9020         0.0880   0.0100                     0.5971
 0.6   0.05        0.9111         0.0889   0.0000                     0.6278
 1.0   0.00        0.6547         0.1998   0.1455                     0.1203
 1.0   0.05        0.7662         0.2338   0.0000                     0.2640
repeat_penalty  logit(")  P(")
           1.0      3.000  0.8214
           1.1      2.727  0.7779
           1.3      2.308  0.6971
           1.5      2.000  0.6285

The first table describes one position: the token after {"name": ". The model's scores (logits) favor read_file (4.0), with two plausible wrong tools (read 2.5, list_files 1.5) and 20 unrelated tokens at -0.5. Each row is one setting:

• Temperature 0 is greedy decoding: the top token always wins, so read_file has probability 1.
• Temperature 0.2 sharpens the distribution: read_file gets 0.9994, and the wrong tools share 0.0006.
• Temperature 0.6 gives read_file 0.902 without min_p. The wrong tools get 0.088 and the tail (the 20 unrelated tokens) 0.010. With min_p 0.05 the tail is removed, and its share moves to the remaining tokens.
• Temperature 1.0 gives read_file only 0.655, and 0.146 goes to unrelated tokens. min_p 0.05 removes the tail and raises read_file to 0.766.

The probabilities come from the softmax formula: divide each score by the temperature, take e to that power, and divide by the sum. The hand calculation for temperature 1.0 without min_p shows every step:

e^4.0  = 54.598   (read_file)
e^2.5  = 12.183   (read)
e^1.5  =  4.482   (list_files)
20 * e^-0.5 = 20 * 0.6065 = 12.131   (the tail)
sum = 83.394

P(read_file) = 54.598 / 83.394 = 0.6547
P(tail)      = 12.131 / 83.394 = 0.1455

Both results match the table. Lowering the temperature divides the scores by a smaller number, which widens the gaps between them before the exponential, so the top token takes almost all the probability.

min_p keeps only tokens whose probability is at least min_p times the top token's probability. The unrelated tokens have exp(-4.5), about 0.011, of the top token's probability, which is below 0.05, so they are removed. With min_p 0.05 and temperature 1.0 the sum becomes 54.598 + 12.183 + 4.482 = 71.263, and 54.598 / 71.263 gives the 0.766. The program applies min_p before the temperature, which is the order of llama.cpp's default chain. The server reports penalties, dry, top_n_sigma, top_k, typ_p, top_p, min_p, xtc, temperature in the samplers field of its generation settings at this commit.

The last column shows why small differences matter. A call needs many decisions, and all of them must go right. For five decisions like this one, the probability is the single-step probability multiplied five times. At temperature 0.6 it is 0.902 to the fifth power, which is 0.597. At temperature 1.0 it is 0.120. The real numbers depend on the model, but the multiplication does not: per-token error rates compound over the length of a call. Low temperature (0 to 0.3) keeps calls on the most likely path. Greedy decoding can repeat the same output after a failure, and some model publishers state recommended settings. A setting is therefore measured with the harness of Example 13 instead of being fixed for every model.

The second table shows a sampler that works against tool calls. A repeat penalty lowers the score of every token that appeared in the last penalty_last_n tokens (64 by default). llama.cpp divides positive scores by the penalty and multiplies negative ones (llama_sampler_penalties_apply in src/llama-sampler.cpp). For a penalty of 1.1, the score of " becomes 3.0 / 1.1 = 2.727, which is the value in the table. JSON repeats ", :, and , constantly, so a penalty meant to reduce repetition in prose also lowers the characters that every call needs. " drops from 0.821 to 0.629 at a penalty of 1.5, which gives the competing tokens ' and [ more room.

The server defaults at this commit (common/common.h) are temperature 0.8, top_k 40, top_p 0.95, min_p 0.05, and repeat_penalty 1.0 (off). Each can be set per request, so an agent can send a low temperature and no penalty with tool-call steps, and other settings for long text answers.

Example 7: fewer, simpler tools with precise descriptions

The tool list itself can cause failures. Every tool definition is written into the prompt, so each tool costs tokens on every request, and each pair of similar tools is a chance for the model to pick the wrong one. This example asks how to find such problems before the model does. A linter is a program that checks something against a list of rules and reports findings. The program below lints two tool sets, one large and one small, and reports their size and findings.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;

public class ToolLint {
    static final ObjectMapper JSON = new ObjectMapper();

    static String tool(String name, String description, String properties, String required) {
        return "{\"type\":\"function\",\"function\":{\"name\":\"" + name + "\",\"description\":\"" + description
            + "\",\"parameters\":{\"type\":\"object\",\"properties\":{" + properties + "},\"required\":[" + required + "]}}}";
    }

    static List<String> lint(List<JsonNode> tools) {
        List<String> findings = new ArrayList<>();
        for (int i = 0; i < tools.size(); i++) {
            JsonNode f = tools.get(i).get("function");
            String name = f.get("name").asText();
            if (f.path("description").asText().length() < 25) findings.add(name + ": description shorter than 25 characters");
            int optional = 0;
            for (var p : f.at("/parameters/properties").properties()) {
                if (!p.getValue().has("description")) findings.add(name + "." + p.getKey() + ": no description");
                if (p.getValue().path("type").asText().equals("object")) findings.add(name + "." + p.getKey() + ": nested object");
                boolean req = false;
                for (JsonNode r : f.at("/parameters/required")) req |= r.asText().equals(p.getKey());
                if (!req) optional++;
            }
            if (optional > 2) findings.add(name + ": " + optional + " optional parameters");
            for (int j = 0; j < i; j++) {
                String other = tools.get(j).at("/function/name").asText();
                String a = name.split("_")[0], b = other.split("_")[0];
                if (a.equals(b)) findings.add(name + ": same verb as " + other);
            }
        }
        return findings;
    }

    public static void main(String[] args) throws Exception {
        String P = "\"path\":{\"type\":\"string\",\"description\":\"File path relative to the workspace root\"}";
        List<String> large = List.of(
            tool("read_file", "Read a file", "\"path\":{\"type\":\"string\"}", "\"path\""),
            tool("read_files", "Read several files at once", "\"paths\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}", "\"paths\""),
            tool("view_file", "Show a file with line numbers", P + ",\"start\":{\"type\":\"integer\"},\"end\":{\"type\":\"integer\"},\"context\":{\"type\":\"integer\"}", "\"path\""),
            tool("edit_file", "Edit a file", P + ",\"edit\":{\"type\":\"object\"}", "\"path\",\"edit\""),
            tool("write_file", "Create or overwrite a file with the given content", P + ",\"content\":{\"type\":\"string\"}", "\"path\",\"content\""),
            tool("run_tests", "Run tests", "\"pattern\":{\"type\":\"string\"}", ""),
            tool("run_command", "Run a shell command in the workspace", "\"command\":{\"type\":\"string\"}", "\"command\""),
            tool("search_code", "Search code", "\"query\":{\"type\":\"string\"}", "\"query\""));
        List<String> small = List.of(
            tool("read_file", "Read a text file and return numbered lines", P, "\"path\""),
            tool("edit_file", "Replace old_string with new_string in one file; old_string must occur exactly once",
                 P + ",\"old_string\":{\"type\":\"string\",\"description\":\"Exact text to replace\"},\"new_string\":{\"type\":\"string\",\"description\":\"Replacement text\"}",
                 "\"path\",\"old_string\",\"new_string\""),
            tool("shell", "Run a bash command in the workspace root and return exit code and output",
                 "\"command\":{\"type\":\"string\",\"description\":\"The command line, for example mvn -q test\"}", "\"command\""),
            tool("grep", "Search files for a regular expression and return matching lines with paths",
                 "\"pattern\":{\"type\":\"string\",\"description\":\"Java regular expression\"}", "\"pattern\""));
        for (var set : List.of(List.of("large", large), List.of("small", small))) {
            @SuppressWarnings("unchecked") List<String> defs = (List<String>) set.get(1);
            List<JsonNode> tools = new ArrayList<>();
            for (String d : defs) tools.add(JSON.readTree(d));
            int chars = String.join(",", defs).length();
            List<String> findings = lint(tools);
            System.out.printf("%s set: %d tools, %d characters (about %d tokens at 4 characters per token), %d findings%n",
                set.get(0), tools.size(), chars, chars / 4, findings.size());
            findings.forEach(f -> System.out.println("  " + f));
        }
    }
}

large set: 8 tools, 1821 characters (about 455 tokens at 4 characters per token), 18 findings
  read_file: description shorter than 25 characters
  read_file.path: no description
  read_files.paths: no description
  read_files: same verb as read_file
  view_file.start: no description
  view_file.end: no description
  view_file.context: no description
  view_file: 3 optional parameters
  edit_file: description shorter than 25 characters
  edit_file.edit: no description
  edit_file.edit: nested object
  write_file.content: no description
  run_tests: description shorter than 25 characters
  run_tests.pattern: no description
  run_command.command: no description
  run_command: same verb as run_tests
  search_code: description shorter than 25 characters
  search_code.query: no description
small set: 4 tools, 1283 characters (about 320 tokens at 4 characters per token), 0 findings

The large set has 8 tools and 18 findings. They fall into five kinds:

• Short descriptions. read_file: Read a file says nothing about what the tool returns (numbered lines or raw text, the whole file or a part). The model learns a tool only from its name, description, and schema, so a description shorter than a sentence leaves it guessing.
• Parameters without descriptions. view_file.start could be a line number or a byte offset, and could start at 0 or 1. Each undocumented parameter is a likely wrong argument.
• Overlapping tools. read_file, read_files, and view_file all read files. The lint finds read_files and read_file by their shared verb. view_file overlaps with both under a different verb, which a name check cannot see. Near-duplicates force a choice that the descriptions do not inform, and each choice has different arguments to get wrong.
• Nested objects. edit_file.edit is an object whose fields the schema does not describe. Nested arguments give the model more structure to get wrong. Flat string arguments are the simplest shape to produce.
• Many optional parameters. view_file has three. Optional parameters invite the model to invent values.

The small set has 4 tools and 0 findings:

• read_file returns numbered lines.
• edit_file uses the exact-replacement arguments from Edit Formats.
• shell covers tests and builds.
• grep covers search.

Its size is 1,283 characters against 1,821, which is about 320 tokens against 455 at four characters per token. The saving is smaller than the tool count suggests, because the precise descriptions are longer. The main gain is fewer choices and fewer undocumented parameters, not fewer tokens.

One more rule: tools that the current task cannot need (a web search tool in an offline repository) are left out of the request. Describing them as unavailable would still cost tokens and still invite a call.

Example 8: few-shot examples and their cost

A small model often writes a better call when it has seen one. A few-shot example is a short demonstration placed before the real conversation: a user question, the correct call, its result, and the final answer. This example asks what such a demonstration costs and what risk it adds. The program renders two examples in the Qwen 2.5 chat format, estimates their size, and checks replies for values copied from them.

import java.util.List;
import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class FewShotCost {
    record Shot(String user, String call, String result, String answer) {
        String render() {
            return "<|im_start|>user\n" + user + "<|im_end|>\n<|im_start|>assistant\n<tool_call>\n" + call
                + "\n</tool_call><|im_end|>\n<|im_start|>user\n<tool_response>\n" + result
                + "\n</tool_response><|im_end|>\n<|im_start|>assistant\n" + answer + "<|im_end|>\n";
        }
    }

    static final Pattern PATH = Pattern.compile("\"path\": \"([^\"]+)\"");

    public static void main(String[] args) {
        List<Shot> shots = List.of(
            new Shot("Which Java version does the build use?",
                "{\"name\": \"read_file\", \"arguments\": {\"path\": \"example/build.gradle\"}}",
                "1| plugins { id 'java' }\n2| java { toolchain { languageVersion = JavaLanguageVersion.of(21) } }",
                "The build uses Java 21 (line 2 of example/build.gradle)."),
            new Shot("Do the tests pass?",
                "{\"name\": \"shell\", \"arguments\": {\"command\": \"mvn -q test\"}}",
                "exit code 1\n[ERROR] Tests run: 12, Failures: 1\n[ERROR] OrderTest.total:31 expected:<30> but was:<27>",
                "No. OrderTest.total fails at line 31: expected 30, got 27."));
        StringBuilder block = new StringBuilder();
        for (Shot s : shots) block.append(s.render());
        if (args.length > 0 && args[0].equals("--print")) { System.out.print(block); return; }
        int chars = block.length();
        int tokens = chars / 3;                                    // conservative ratio for code and JSON
        System.out.printf("few-shot block: %d shots, %d characters, about %d tokens%n", shots.size(), chars, tokens);
        int requests = 15;
        System.out.printf("one 15-request session: %d prompt tokens without prefix reuse, %d with it%n", tokens * requests, tokens);
        for (int window : new int[] {8_192, 32_768})
            System.out.printf(Locale.ROOT, "share of a %,d-token window: %.1f%%%n", window, 100.0 * tokens / window);

        // A small model can copy values from an example instead of using the user's request.
        String userAsked = "Which Java version does pom.xml use?";
        String[] replies = {
            "{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}",
            "{\"name\": \"read_file\", \"arguments\": {\"path\": \"example/build.gradle\"}}"};
        for (String reply : replies) {
            Matcher m = PATH.matcher(reply);
            String path = m.find() ? m.group(1) : "";
            boolean copied = shots.stream().anyMatch(s -> s.call().contains("\"" + path + "\""));
            System.out.println(path + (copied ? " -> copied from a few-shot example, not in the request: \"" + userAsked + "\"" : " -> ok"));
        }
    }
}

few-shot block: 2 shots, 851 characters, about 283 tokens
one 15-request session: 4245 prompt tokens without prefix reuse, 283 with it
share of a 8,192-token window: 3.5%
share of a 32,768-token window: 0.9%
pom.xml -> ok
example/build.gradle -> copied from a few-shot example, not in the request: "Which Java version does pom.xml use?"

The first line measures the block: 851 characters, estimated at about 283 tokens with three characters per token. That is the conservative ratio for code and JSON that Context Engineering for Coding Agents uses. The real count with the Qwen2 tokenizer is lower, because each chat marker such as <|im_start|> is one token. The next commands print the block and count it with the real tokenizer:

java FewShotCost.java --print > fewshot.txt
llama-tokenize -m models/ggml-vocab-qwen2.gguf -f fewshot.txt --show-count --log-disable | tail -1

Real output:

Total number of tokens: 230

The real count is 230 tokens, so the estimate of 283 was safe. The second output line shows the cost across a session. The block is part of every request, so a session of 15 model requests sends it 15 times. That is 4,245 estimated prompt tokens if the server processes them every time, and 283 if they sit in a stable prefix that the server's prompt cache reuses. The block therefore belongs at the start of the prompt, after the system prompt and the tools, where it never changes. The next two lines put the size in proportion: 3.5% of an 8K window and 0.9% of a 32K window. The cost is small in a large window and noticeable in a small one.

The last two lines show the risk. The user asked about pom.xml. The first reply reads pom.xml. The second reads example/build.gradle, the path from the first example. A small model can treat the examples as part of the task and copy their values. Two countermeasures follow. First, use example values that do not exist in any real workspace (the example/ prefix here), so that a copied value fails fast with a "file not found" result instead of reading a wrong file. Second, use a copy check like the one in the program to flag such calls in tests.

Few-shot examples matter most with a prompt-based call format, where the model has no other guide to the format (Function Calling from First Principles built one). With native support and a grammar, the format is already enforced, and the examples mainly teach the order of steps.

Example 9: quantization and tool-call accuracy, what to measure and how

Quantization stores a model's weights with fewer bits so that the model needs less memory (Quantization Fundamentals). The price is accuracy: each format changes the model's next-token probabilities by a different amount. This example asks how to tell whether a quantized file is still good enough for tool calls. Two measurements answer the question. One works on text and needs no task. The other works on tool calls and needs a prompt set.

The text measurement is the mean KL divergence (KLD). KL divergence is a number that says how far one probability distribution is from another. Here it compares the quantized model's next-token probabilities with those of the 16-bit original, token by token, and the mean is taken over a text sample. A value of 0 means identical, and larger values mean more damage. llama-perplexity computes it in two runs. Making, Quantizing, and Testing Your Own GGUF runs these commands and explains every line of the report:

llama-perplexity -m model-F16.gguf -f code-sample.txt -c 2048 --kl-divergence-base logits-f16.bin
llama-perplexity -m model-Q4_K_M.gguf --kl-divergence-base logits-f16.bin --kl-divergence

For tool calls, the test text should be code and JSON close to what the agent produces. A quantization error that changes " into a space or closes a string early matters more there than in prose. The high percentiles of KLD in the report (99%, 99.9%, maximum) show the rare large errors that break a call, which the mean can hide.

The tool-call measurement sends the same prompts to each file and counts three outcomes. The call parsed. The call is schema-valid (the tool and arguments match the schema). The call is the right call (the expected tool with the expected arguments). Each count is a sample, so the true rate is uncertain: 198 parsed out of 200 does not mean the real rate is exactly 99%. A Wilson interval is a range that very likely contains the true rate, and it is wider when the sample is smaller. The program turns each count into a rate with a 95% Wilson interval (Evaluating Models and Agents derives it) and compares the files. The counts are an example input that shows the arithmetic, not measurements of real files.

import java.util.List;
import java.util.Locale;

public class QuantToolAccuracy {
    static final double Z = 1.959964;                            // 95% confidence

    static double[] wilson(int passed, int n) {
        double p = (double) passed / n, z2 = Z * Z;
        double center = (p + z2 / (2 * n)) / (1 + z2 / n);
        double half = Z * Math.sqrt(p * (1 - p) / n + z2 / (4.0 * n * n)) / (1 + z2 / n);
        return new double[] {center - half, center + half};
    }

    /** Counts from one run of the same prompts against one GGUF file (made-up example numbers). */
    record Run(String file, double meanKld, int prompts, int parsed, int schemaValid, int rightCall) {}

    static String rate(int k, int n) {
        double[] w = wilson(k, n);
        return String.format(Locale.ROOT, "%5.1f%% [%4.1f, %5.1f]", 100.0 * k / n, 100 * w[0], 100 * w[1]);
    }

    public static void main(String[] args) {
        List<Run> runs = List.of(
            new Run("quant-A (8.5 bpw)", 0.0021, 200, 198, 196, 181),
            new Run("quant-B (4.9 bpw)", 0.0190, 200, 197, 193, 176),
            new Run("quant-C (3.5 bpw)", 0.0870, 200, 189, 178, 152));
        System.out.println("file                mean KLD   parsed                 schema-valid           right call");
        for (Run r : runs)
            System.out.printf(Locale.ROOT, "%-18s  %8.4f  %s  %s  %s%n", r.file(), r.meanKld(),
                rate(r.parsed(), r.prompts()), rate(r.schemaValid(), r.prompts()), rate(r.rightCall(), r.prompts()));

        Run base = runs.get(0);
        for (Run r : runs.subList(1, runs.size())) {
            double[] a = wilson(base.rightCall(), base.prompts()), b = wilson(r.rightCall(), r.prompts());
            boolean overlap = b[1] >= a[0] && a[1] >= b[0];
            System.out.println(r.file() + " vs " + base.file() + ": right-call intervals "
                + (overlap ? "overlap, no difference shown at n=" + r.prompts() : "do not overlap, the drop is larger than sampling noise"));
        }
        // How many prompts separate 90% from 85%?
        for (int n = 100; n <= 2000; n += 100) {
            double[] hi = wilson((int) Math.round(0.90 * n), n), lo = wilson((int) Math.round(0.85 * n), n);
            if (lo[1] < hi[0]) { System.out.println("90% vs 85% separate at n=" + n + " prompts per file"); break; }
        }
    }
}

file                mean KLD   parsed                 schema-valid           right call
quant-A (8.5 bpw)     0.0021   99.0% [96.4,  99.7]   98.0% [95.0,  99.2]   90.5% [85.6,  93.8]
quant-B (4.9 bpw)     0.0190   98.5% [95.7,  99.5]   96.5% [93.0,  98.3]   88.0% [82.8,  91.8]
quant-C (3.5 bpw)     0.0870   94.5% [90.4,  96.9]   89.0% [83.9,  92.6]   76.0% [69.6,  81.4]
quant-B (4.9 bpw) vs quant-A (8.5 bpw): right-call intervals overlap, no difference shown at n=200
quant-C (3.5 bpw) vs quant-A (8.5 bpw): right-call intervals do not overlap, the drop is larger than sampling noise
90% vs 85% separate at n=700 prompts per file

Each row is one file. The columns:

• mean KLD comes from the text measurement. It rises as bits per weight (bpw) fall: 0.0021, 0.0190, 0.0870 in this example input.
• parsed is the share of replies whose call parsed, with its interval in brackets. 99.0% [96.4, 99.7] means 198 of 200 calls parsed, and the true rate for this file and prompt set lies between 96.4% and 99.7% with 95% confidence.
• schema-valid is lower than parsed, because some parsed calls had wrong argument names or types.
• right call is the lowest column and the one that matters for the agent.

The comparison lines apply the intervals. quant-B's right-call rate is 88.0% against quant-A's 90.5%, but the intervals [82.8, 91.8] and [85.6, 93.8] overlap. With 200 prompts, the data does not show a difference. quant-C's interval [69.6, 81.4] lies entirely below quant-A's, so its drop is larger than sampling noise. Non-overlapping intervals are a conservative test. Overlapping ones do not prove that the files are equal, only that this sample cannot separate them.

The last line answers how much data a decision needs: a difference between 90% and 85% separates only at about 700 prompts per file. Small differences need many prompts.

Three rules make the measurement valid:

• Measure parse rates without a grammar. With a grammar, the parse rate is 100% by construction (Example 2), so a weaker file looks as good as a stronger one. With a grammar, measure the right-call rate instead, which the grammar cannot force.
• Keep everything else fixed. Use the same prompts, template, sampling settings, and seed list. Only the file changes.
• Report the measurement, not the format name. The same quantization type affects different models differently, so a result for one model does not transfer to another. The mean KLD and the right-call rate with its interval are the numbers to report.

Example 10: keeping the context short

Every token in the prompt costs time on every request, and an agent prompt grows with each tool result. A test log of 9,000 characters is added once and then sent again with every later request. This example asks how fast the prompt grows in a typical session and how much two simple policies save. How tool-call accuracy changes with prompt length is a property of each model, and the harness of Example 13 can measure it. The program simulates ten tool steps and prints the prompt size before each of the eleven model requests under three policies.

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class ShortContext {
    record Step(String tool, String target, int resultChars) {}
    enum Policy { KEEP_ALL, CAP, CAP_AND_STALE }

    static final int FIXED = 3_000, CALL = 120, CAP = 1_200, STUB = 60;   // characters

    /** Prompt size in tokens (characters / 3) before each model request of the session. */
    static List<Integer> promptTokens(List<Step> steps, Policy policy) {
        List<Integer> sizes = new ArrayList<>();
        List<Step> done = new ArrayList<>();
        for (int i = 0; i <= steps.size(); i++) {
            int chars = FIXED;
            for (int j = 0; j < done.size(); j++) {
                Step s = done.get(j);
                int result = policy == Policy.KEEP_ALL ? s.resultChars() : Math.min(s.resultChars(), CAP);
                if (policy == Policy.CAP_AND_STALE && isStale(done, j)) result = STUB;
                chars += CALL + result;
            }
            sizes.add(chars / 3);
            if (i < steps.size()) done.add(steps.get(i));
        }
        return sizes;
    }

    /** A read is stale when a later step edits or rereads the same file; old test logs are stale after a newer run. */
    static boolean isStale(List<Step> done, int j) {
        Step s = done.get(j);
        for (int k = j + 1; k < done.size(); k++) {
            Step later = done.get(k);
            if (s.tool().equals("read_file") && later.target().equals(s.target())) return true;
            if (s.tool().equals("shell") && later.tool().equals("shell")) return true;
        }
        return false;
    }

    public static void main(String[] args) {
        List<Step> steps = List.of(
            new Step("read_file", "OrderService.java", 6_000), new Step("read_file", "OrderTest.java", 4_500),
            new Step("shell", "mvn test", 9_000), new Step("edit_file", "OrderService.java", 300),
            new Step("read_file", "OrderService.java", 6_100), new Step("shell", "mvn test", 8_000),
            new Step("grep", "total(", 1_500), new Step("edit_file", "OrderService.java", 300),
            new Step("read_file", "OrderService.java", 6_100), new Step("shell", "mvn test", 2_500));
        List<List<Integer>> table = new ArrayList<>();
        for (Policy p : Policy.values()) table.add(promptTokens(steps, p));
        System.out.println("request  last step                     keep-all     cap   cap+stale");
        for (int i = 0; i < table.get(0).size(); i++) {
            String last = i == 0 ? "(start)" : steps.get(i - 1).tool() + " " + steps.get(i - 1).target();
            System.out.printf("%7d  %-28s %9d %7d %11d%n", i + 1, last, table.get(0).get(i), table.get(1).get(i), table.get(2).get(i));
        }
        for (int p = 0; p < table.size(); p++)
            System.out.printf(Locale.ROOT, "%-14s total prompt tokens processed: %,d%n", Policy.values()[p],
                table.get(p).stream().mapToInt(Integer::intValue).sum());
    }
}

request  last step                     keep-all     cap   cap+stale
      1  (start)                           1000    1000        1000
      2  read_file OrderService.java       3040    1440        1440
      3  read_file OrderTest.java          4580    1880        1880
      4  shell mvn test                    7620    2320        2320
      5  edit_file OrderService.java       7760    2460        2080
      6  read_file OrderService.java       9833    2900        2520
      7  shell mvn test                   12540    3340        2580
      8  grep total(                      13080    3780        3020
      9  edit_file OrderService.java      13220    3920        2780
     10  read_file OrderService.java      15293    4360        3220
     11  shell mvn test                   16166    4800        3280
KEEP_ALL       total prompt tokens processed: 104,132
CAP            total prompt tokens processed: 32,200
CAP_AND_STALE  total prompt tokens processed: 26,120

The session reads two files, runs the tests, edits, rereads, runs the tests again, searches, edits, rereads, and runs the tests a third time. The fixed part (system prompt and tools) is 3,000 characters, about 1,000 tokens at three characters per token, so every column starts at 1,000.

• keep-all keeps every tool result in full. The prompt grows to 16,166 tokens before the last request, and the eleven requests process 104,132 prompt tokens in total. The two largest jumps are the test logs (9,000 and 8,000 characters, about 3,000 tokens each).
• cap cuts each tool result to 1,200 characters, as the truncation in Project 21 does. Every step now adds at most 440 tokens (the 120-character call plus the 1,200-character result, divided by 3). The last prompt is 4,800 tokens, and the total is 32,200.
• cap+stale also replaces stale results with a 60-character stub. A read of a file is stale once a later step edits or rereads that file. A test log is stale once a newer test run exists. The first difference appears at request 5, after the first edit: the first read of OrderService.java becomes a stub, which saves 380 tokens (1,140 characters). At request 9 the prompt even shrinks, from 3,020 to 2,780, because the second edit makes the second read stale. The last prompt is 3,280 tokens, and the total is 26,120, about a quarter of keep-all.

A stale result is also wrong information. After an edit, the old content no longer exists, and a model that sees both versions can edit against the old one. Replacing the result with a stub ([earlier read of OrderService.java removed; the file changed]) leaves a record of the step without its outdated content.

The trade-off is the prompt cache. Removing a message from the middle of the history changes the prefix, so the server must recompute the tokens after it. The cap has no such cost, because it applies when a result enters the history. Stale-result removal is therefore applied in batches (for example, together with compaction, the step that summarizes old history), not on every step. Context Engineering for Coding Agents builds the ContextManager that applies these policies with a token budget.

Example 11: loop detection for repeated identical calls

A small model that does not understand a tool result often calls the same tool again with the same arguments, gets the same result, and repeats. Each repeat adds an identical result to the context and uses a model request. This example asks how an agent notices that the model is stuck. The program implements a LoopDetector that compares steps by a signature: the tool name, the arguments with sorted keys and no whitespace, and a hash of the result. Including the result is what separates a loop from progress, as Run 3 shows. Three steps with the same signature, or a two-step cycle seen twice, produce a warning. A second detection stops the run.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import java.util.ArrayList;
import java.util.List;

public class LoopDetection {
    static final ObjectMapper SORTED = new ObjectMapper().configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
    record ToolCall(String id, String name, String arguments) {}
    enum Verdict { OK, WARN, STOP }

    static final class LoopDetector {
        private final List<String> signatures = new ArrayList<>();
        private int warnings;

        /** name + arguments with sorted keys and no whitespace + a hash of the result: equal steps compare equal. */
        static String signature(ToolCall c, String result) {
            String args;
            try { args = SORTED.writeValueAsString(SORTED.readValue(c.arguments(), Object.class)); }
            catch (Exception e) { args = c.arguments(); }
            return c.name() + args + "#" + result.hashCode();
        }

        Verdict check(ToolCall call, String result) {
            signatures.add(signature(call, result));
            int n = signatures.size();
            boolean repeat3 = n >= 3 && signatures.get(n - 1).equals(signatures.get(n - 2)) && signatures.get(n - 2).equals(signatures.get(n - 3));
            boolean cycle2 = n >= 4 && signatures.get(n - 1).equals(signatures.get(n - 3)) && signatures.get(n - 2).equals(signatures.get(n - 4))
                             && !signatures.get(n - 1).equals(signatures.get(n - 2));
            if (!repeat3 && !cycle2) return Verdict.OK;
            return ++warnings > 1 ? Verdict.STOP : Verdict.WARN;
        }
    }

    record Step(ToolCall call, String result) {}

    static void run(String title, List<Step> script) {
        System.out.println(title);
        LoopDetector detector = new LoopDetector();
        for (Step s : script) {
            ToolCall call = s.call();
            Verdict v = detector.check(call, s.result());
            System.out.printf("  %-3s %-9s %-28s %-22s %s%n", call.id(), call.name(), call.arguments(), s.result(), v);
            if (v == Verdict.WARN)
                System.out.println("       -> result gets a note: \"This call repeats earlier calls with the same arguments and results."
                    + " Change the approach or answer.\"");
            if (v == Verdict.STOP) { System.out.println("       -> run stopped: repeated calls after a warning"); return; }
        }
        System.out.println("  -> run finished normally");
    }

    static Step step(String id, String name, String args, String result) { return new Step(new ToolCall(id, name, args), result); }

    public static void main(String[] args) {
        String pom = "{\"path\": \"pom.xml\"}", test = "{\"command\": \"mvn -q test\"}";
        run("1. one call three times, with different whitespace; the model changes course after the note", List.of(
            step("c1", "read_file", pom, "<release>17</release>"),
            step("c2", "read_file", "{ \"path\":\"pom.xml\" }", "<release>17</release>"),
            step("c3", "read_file", pom, "<release>17</release>"),
            step("c4", "edit_file", "{\"path\": \"pom.xml\", ...}", "replaced 1 occurrence"),
            step("c5", "shell", test, "exit 0")));
        run("2. a two-step cycle that continues after the note", List.of(
            step("c1", "shell", test, "exit 1, OrderTest:31"), step("c2", "read_file", pom, "<release>17</release>"),
            step("c3", "shell", test, "exit 1, OrderTest:31"), step("c4", "read_file", pom, "<release>17</release>"),
            step("c5", "shell", test, "exit 1, OrderTest:31"), step("c6", "read_file", pom, "<release>17</release>")));
        run("3. the same command three times with different results is progress", List.of(
            step("c1", "shell", test, "exit 1, 3 failures"), step("c2", "shell", test, "exit 1, 1 failure"),
            step("c3", "shell", test, "exit 0")));
    }
}

1. one call three times, with different whitespace; the model changes course after the note
  c1  read_file {"path": "pom.xml"}          <release>17</release>  OK
  c2  read_file { "path":"pom.xml" }         <release>17</release>  OK
  c3  read_file {"path": "pom.xml"}          <release>17</release>  WARN
       -> result gets a note: "This call repeats earlier calls with the same arguments and results. Change the approach or answer."
  c4  edit_file {"path": "pom.xml", ...}     replaced 1 occurrence  OK
  c5  shell     {"command": "mvn -q test"}   exit 0                 OK
  -> run finished normally
2. a two-step cycle that continues after the note
  c1  shell     {"command": "mvn -q test"}   exit 1, OrderTest:31   OK
  c2  read_file {"path": "pom.xml"}          <release>17</release>  OK
  c3  shell     {"command": "mvn -q test"}   exit 1, OrderTest:31   OK
  c4  read_file {"path": "pom.xml"}          <release>17</release>  WARN
       -> result gets a note: "This call repeats earlier calls with the same arguments and results. Change the approach or answer."
  c5  shell     {"command": "mvn -q test"}   exit 1, OrderTest:31   STOP
       -> run stopped: repeated calls after a warning
3. the same command three times with different results is progress
  c1  shell     {"command": "mvn -q test"}   exit 1, 3 failures     OK
  c2  shell     {"command": "mvn -q test"}   exit 1, 1 failure      OK
  c3  shell     {"command": "mvn -q test"}   exit 0                 OK
  -> run finished normally

Run 1 reads pom.xml three times. The second call is written with different whitespace ({ "path":"pom.xml" }), but the signature normalizes it: Jackson reads the arguments and writes them back with sorted keys and no spaces. The third identical step triggers WARN. The agent adds a note to the tool result that tells the model the call repeats earlier ones with the same results. In this scripted run the model changes course: it edits pom.xml, runs the tests, and the run finishes normally.

Run 2 alternates between running the tests and reading pom.xml, with the same results each time. No single call repeats three times in a row, so the repeat check alone would miss it. The cycle check compares each step with the one two positions back. At c4, steps 4 and 2 match and steps 3 and 1 match, so the detector warns. The model continues the cycle, and at c5 the second detection stops the run. Not stopping would cost every remaining step up to the limit, each one adding an identical result to the context.

Run 3 runs the same test command three times with different results (3 failures, 1 failure, success). The result hash is part of the signature, so the three steps differ, and the detector reports nothing. Without the result in the signature, a normal edit-and-test cycle would look like a loop.

Real harnesses implement the same idea with different thresholds:

• Crush v0.97.1 (internal/agent/loop_detection.go) hashes the tool name, input, and output of each step with SHA-256 and stops when one signature appears more than 5 times in the last 10 steps.
• The OpenHands SDK 1.51.0 has a StuckDetector (openhands/sdk/conversation/stuck_detector.py) with default thresholds of 4 for repeated action and observation pairs, 3 for repeated actions that end in errors, and 6 for alternating patterns (StuckDetectionThresholds in conversation/types.py).

Low thresholds stop loops early and occasionally stop a legitimate retry. High thresholds waste more steps. The harness of Example 13 measures this choice on recorded scenarios.

Example 12: model choice criteria for local coding agents

Many models exist, and a reputation or a leaderboard score does not say whether one works as a local agent on your machine. This example shows how to pick a model by facts. The choice has two stages. Hard constraints remove candidates that cannot work at all, for example a model that does not fit in memory. A measurement then ranks the rest. The first hard constraint can be read from a running server: llama-server reports what the model's chat template supports in /props.

curl -s 127.0.0.1:$PORT/props | jq -c '.chat_template_caps'

Real output (the tiny model with the Qwen 2.5 template):

{"supports_object_arguments":true,"supports_parallel_tool_calls":true,"supports_preserve_reasoning":false,"supports_reasoning_effort":false,"supports_string_content":true,"supports_system_role":true,"supports_tool_calls":true,"supports_tools":true,"supports_typed_content":false}

supports_tools and supports_tool_calls are true because the template renders tool definitions and tool calls. A template that reports false leaves only a prompt-based format, without the server-built grammar.

The program applies the constraints to five candidates and ranks the survivors. The numbers are an example input, not measurements of real models.

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Locale;

public class ModelChoice {
    /** One GGUF file of one model; the numbers are an example, not measurements of real models. */
    record Candidate(String name, double paramsB, int layers, int kvHeads, int headDim, double bpw,
                     int trainedContext, boolean templateHasTools, int tasksPassed, int tasks) {
        double weightsGb() { return paramsB * bpw / 8; }
        double kvGb(int context) { return 2.0 * layers * kvHeads * headDim * context * 2 / 1e9; }   // K and V, F16
    }

    static double wilsonLow(int k, int n) {
        double z = 1.959964, p = (double) k / n, z2 = z * z;
        return ((p + z2 / (2 * n)) - z * Math.sqrt(p * (1 - p) / n + z2 / (4.0 * n * n))) / (1 + z2 / n);
    }

    public static void main(String[] args) {
        double budgetGb = 12.0;
        int context = 32_768;
        List<Candidate> all = List.of(
            new Candidate("cand-1", 7.6, 28, 4, 128, 4.9, 32_768, true, 31, 50),
            new Candidate("cand-2", 14.8, 48, 8, 128, 4.9, 32_768, true, 36, 50),
            new Candidate("cand-3", 3.1, 36, 2, 128, 8.5, 32_768, true, 19, 50),
            new Candidate("cand-4", 8.0, 32, 8, 128, 4.9, 131_072, false, 0, 0),
            new Candidate("cand-5", 6.7, 32, 4, 128, 4.9, 16_384, true, 0, 0));
        List<Candidate> kept = new ArrayList<>();
        for (Candidate c : all) {
            double total = c.weightsGb() + c.kvGb(context);
            String reason = total > budgetGb ? String.format(Locale.ROOT, "needs %.1f GB > %.0f GB", total, budgetGb)
                : c.trainedContext() < context ? "trained context " + c.trainedContext() + " < " + context
                : !c.templateHasTools() ? "chat template has no tool support"
                : null;
            System.out.printf(Locale.ROOT, "%s  weights %.2f GB + KV %.2f GB = %.2f GB  %s%n", c.name(), c.weightsGb(),
                c.kvGb(context), total, reason == null ? "kept" : "rejected: " + reason);
            if (reason == null) kept.add(c);
        }
        kept.sort(Comparator.comparingDouble((Candidate c) -> wilsonLow(c.tasksPassed(), c.tasks())).reversed());
        System.out.println("ranked by the lower 95% bound of the task pass rate on the team's own suite:");
        for (Candidate c : kept)
            System.out.printf(Locale.ROOT, "  %s  %d/%d passed, lower bound %.2f%n", c.name(), c.tasksPassed(), c.tasks(),
                wilsonLow(c.tasksPassed(), c.tasks()));
    }
}

cand-1  weights 4.66 GB + KV 1.88 GB = 6.53 GB  kept
cand-2  weights 9.07 GB + KV 6.44 GB = 15.51 GB  rejected: needs 15.5 GB > 12 GB
cand-3  weights 3.29 GB + KV 1.21 GB = 4.50 GB  kept
cand-4  weights 4.90 GB + KV 4.29 GB = 9.19 GB  rejected: chat template has no tool support
cand-5  weights 4.10 GB + KV 2.15 GB = 6.25 GB  rejected: trained context 16384 < 32768
ranked by the lower 95% bound of the task pass rate on the team's own suite:
  cand-1  31/50 passed, lower bound 0.48
  cand-3  19/50 passed, lower bound 0.26

The machine has 12 GB for the model, and the agent needs a 32,768-token context. Each candidate line shows the memory estimate from Memory Math: Weights, KV Cache, and What Fits. Weights take parameters times bits per weight divided by 8. The KV cache (key-value cache, the memory that holds the model's notes about every token already read) takes 2 (K and V) × layers × KV heads × head dimension × context × 2 bytes (F16).

For cand-1 the weights are 7.6 × 4.9 / 8 = 4.66 GB. The KV cache is 2 × 28 × 4 × 128 × 32,768 × 2 bytes, about 1.88 GB. The sum is 6.53 GB. The program checks the constraints in order and reports the first one that fails.

• cand-1 (7.6B parameters at 4.9 bits per weight) needs 4.66 GB of weights and 1.88 GB of KV cache, 6.53 GB in total. It is kept.
• cand-2 (14.8B) needs 9.07 GB of weights, and its 48 layers with 8 KV heads need 6.44 GB of KV cache at 32K. The total, 15.51 GB, exceeds the budget. A shorter context or a quantized KV cache would change this result.
• cand-3 (3.1B at 8.5 bits) needs 4.50 GB. It is kept.
• cand-4 fits in memory but its chat template has no tool support, which /props shows before any test.
• cand-5 fits, but it was trained for 16,384 tokens. At 32,768 tokens it runs beyond its training length, where quality is not guaranteed.

The two survivors are ranked by the lower bound of the 95% Wilson interval of their pass rate on the team's own task suite, not by the pass rate itself. The lower bound rewards both a high rate and enough tasks to trust it. cand-1 passed 31 of 50 (lower bound 0.48), and cand-3 passed 19 of 50 (lower bound 0.26).

The criteria, as a table:

Public benchmark scores and labels such as "coder" help choose which candidates to measure. The measurements on the agent's own tools and tasks choose which one to use.

Example 13: a reliability harness over scripted scenarios

The earlier examples each fix one failure. The remaining question is which technique helps, and by how much, when failures are mixed. This example builds a harness: a test program that runs the same scenarios against several agent configurations and reports pass or fail for each pair. The harness runs seven scripted scenarios against four configurations. Each scenario is a list of model replies that contains one failure from Example 1. The fake workspace has a bug (total() returns 27 instead of 30) that an edit fixes, and the tests pass only after that edit.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
import java.util.Map;

public class ReliabilityHarness {
    static final ObjectMapper JSON = new ObjectMapper().configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
    static final Map<String, List<String>> TOOLS = Map.of(
        "read_file", List.of("path"), "edit_file", List.of("path", "old_string", "new_string"), "shell", List.of("command"));
    record Config(String name, boolean repair, int retries, boolean loopDetection) {}
    record Scenario(String name, List<String> replies, List<String> afterNote) {}

    static String call(String json) { return "<tool_call>\n" + json + "\n</tool_call>"; }

    /** A reduced form of Example 3: trailing commas and single quotes. Truncated calls are never repaired. */
    static String repair(String s) {
        s = s.replaceAll(",\\s*([}\\]])", "$1");
        return s.contains("\"") ? s : s.replace('\'', '"');
    }

    /** Runs one scenario; returns PASS(steps) or FAIL:reason. */
    static String run(Scenario sc, Config cfg) throws Exception {
        Deque<String> replies = new ArrayDeque<>(sc.replies());
        boolean edited = false, testsPass = false;
        int retriesLeft = cfg.retries(), warnings = 0;
        List<String> signatures = new ArrayList<>();
        for (int step = 1; step <= 8; step++) {
            String reply = replies.poll();
            if (!reply.contains("<tool_call>")) return testsPass ? "PASS(" + step + ")" : "FAIL:unverified";
            String error = null;
            JsonNode c = null;
            if (!reply.contains("</tool_call>")) error = "cut off";
            else {
                String body = reply.substring(reply.indexOf('{'), reply.lastIndexOf('}') + 1);
                try { c = JSON.readTree(body); } catch (Exception e) {
                    try { if (cfg.repair()) c = JSON.readTree(repair(body)); else error = "json"; } catch (Exception e2) { error = "json"; }
                }
            }
            if (c != null) {
                List<String> req = TOOLS.get(c.path("name").asText());
                if (req == null) error = "tool";
                else for (String r : req) if (!c.path("arguments").path(r).isTextual()) error = "args";
            }
            if (error != null) {
                if (retriesLeft-- > 0) continue;                    // the error goes back; the model answers again
                return "FAIL:" + error;
            }
            JsonNode a = c.get("arguments");
            String result = switch (c.get("name").asText()) {
                case "read_file" -> "int total() { return 27; }";
                case "edit_file" -> { edited |= a.get("old_string").asText().equals("27"); yield "edited"; }
                default -> { testsPass = edited; yield edited ? "exit 0" : "exit 1, expected 30 but was 27"; }
            };
            signatures.add(c.get("name").asText() + JSON.writeValueAsString(JSON.treeToValue(a, Object.class)) + "#" + result);
            int n = signatures.size();
            if (cfg.loopDetection() && n >= 3 && signatures.get(n - 1).equals(signatures.get(n - 2))
                    && signatures.get(n - 2).equals(signatures.get(n - 3))) {
                if (++warnings > 1) return "FAIL:loop";
                replies = new ArrayDeque<>(sc.afterNote());          // the model reads the note and changes course
            }
        }
        return "FAIL:steps";
    }

    public static void main(String[] args) throws Exception {
        String read = call("{\"name\": \"read_file\", \"arguments\": {\"path\": \"Order.java\"}}");
        String edit = call("{\"name\": \"edit_file\", \"arguments\": {\"path\": \"Order.java\", \"old_string\": \"27\", \"new_string\": \"30\"}}");
        String test = call("{\"name\": \"shell\", \"arguments\": {\"command\": \"mvn -q test\"}}");
        String done = "Fixed: total() returns 30 and the tests pass.";
        List<Scenario> scenarios = List.of(
            new Scenario("clean run", List.of(read, edit, test, done), List.of()),
            new Scenario("trailing comma", List.of(read, edit.replace("\"30\"}", "\"30\",}"), test, done), List.of()),
            new Scenario("single quotes", List.of(read, edit, test.replace('"', '\''), done), List.of()),
            new Scenario("wrong argument name", List.of(read, edit.replace("\"path\"", "\"file\""), edit, test, done), List.of()),
            new Scenario("cut-off call", List.of(read, edit.substring(0, 60), edit, test, done), List.of()),
            new Scenario("repeats read_file", List.of(read, read, read, read, read, read, read, read), List.of(edit, test, done)),
            new Scenario("answers before testing", List.of(read, edit, done), List.of()));
        List<Config> configs = List.of(new Config("raw", false, 0, false), new Config("+repair", true, 0, false),
            new Config("+retry", true, 2, false), new Config("+loop", true, 2, true));
        StringBuilder out = new StringBuilder(String.format("%-24s", "scenario"));
        configs.forEach(c -> out.append(String.format("%-16s", c.name())));
        int[] passed = new int[configs.size()];
        for (Scenario sc : scenarios) {
            out.append(String.format("%n%-24s", sc.name()));
            for (int i = 0; i < configs.size(); i++) {
                String r = run(sc, configs.get(i));
                if (r.startsWith("PASS")) passed[i]++;
                out.append(String.format("%-16s", r));
            }
        }
        out.append(String.format("%n%-24s", "pass rate"));
        for (int p : passed) out.append(String.format("%-16s", p + "/" + scenarios.size()));
        out.toString().lines().forEach(line -> System.out.println(line.stripTrailing()));
    }
}

scenario                raw             +repair         +retry          +loop
clean run               PASS(4)         PASS(4)         PASS(4)         PASS(4)
trailing comma          FAIL:json       PASS(4)         PASS(4)         PASS(4)
single quotes           FAIL:json       PASS(4)         PASS(4)         PASS(4)
wrong argument name     FAIL:args       FAIL:args       PASS(5)         PASS(5)
cut-off call            FAIL:cut off    FAIL:cut off    PASS(5)         PASS(5)
repeats read_file       FAIL:steps      FAIL:steps      FAIL:steps      PASS(6)
answers before testing  FAIL:unverified FAIL:unverified FAIL:unverified FAIL:unverified
pass rate               1/7             3/7             5/7             6/7

A run passes when the model gives a final answer after a test run that passed. The number in parentheses is the count of model requests. A failed run shows the reason:

• json: the call did not parse.
• args: wrong or missing arguments.
• cut off: no closing tag.
• steps: the limit of 8 requests was reached.
• unverified: an answer before the tests passed.

The four configurations add one technique each. raw runs only correct calls. +repair adds the syntax repair. +retry adds two retries with the error sent back. +loop adds loop detection.

Reading the table by row:

• clean run passes everywhere in 4 requests (read, edit, test, answer). This row checks that no technique changes a correct run.
• trailing comma and single quotes fail in raw with json and pass from +repair on, still in 4 requests. The repair costs no model call.
• wrong argument name fails until +retry, then passes in 5 requests. The failed attempt costs one extra request.
• cut-off call also needs a retry. The repair deliberately does not touch it (Example 4), so +repair still fails, and +retry passes in 5.
• repeats read_file reaches the step limit in every configuration without loop detection. With it, the third identical read triggers the note, the scripted model switches to the edit, and the run passes in 6 requests.
• answers before testing fails everywhere. The model claims success without running the tests. No format technique detects a false claim. Running the tests after the agent finishes does, as Evaluating Models and Agents shows.

The pass rate rises from 1/7 to 3/7, 5/7, and 6/7. These numbers describe the scenarios, not a model. Each scenario was written to contain one failure, so the harness is a regression test. When a change breaks repair or loop detection, a row changes from PASS to FAIL. For a real model, the task-based runner of Evaluating Models and Agents supplies the frequency of each failure.

The techniques also differ in cost. Repair is free. A retry costs a model request. Loop detection costs nothing until it fires. A grammar costs some sampling time per token on the server and removes the first two failure rows before they reach the agent. A later workshop lesson assembles these pieces into loop agent.

Interview points

• Small-model tool-call failures are: wrong format, invalid JSON, unknown tool, wrong or missing arguments, cut-off calls, wrong values, repeated calls, and unverified answers. Each has a different remedy.
• llama-server builds a tool-call grammar from tools. For auto the grammar is lazy and starts when the model writes <tool_call> and a newline. For required it is active from the first token. At commit 0c1e57098 the required grammar allows free text before the first call and allows one or more calls even with parallel_tool_calls: false.
• A grammar guarantees the shape of a call, not its correctness or completeness. A random model produced only schema-valid calls, mostly with the wrong path, and one call cut off by max_tokens. An agent must always check finish_reason.
• JSON repair should fix syntax errors (trailing commas, quotes, control characters, Python literals, surrounding text) and refuse truncation, because a repaired truncation parses but contains invented content.
• Validation errors go back as tool results that name the field, the expected shape, and the next action. Retries need a limit because each one costs a full model request.
• Per-token error rates compound over a call. Low temperature, min_p, and no repeat penalty keep calls on the most likely path. llama.cpp applies temperature last in its default chain.
• Fewer tools with precise descriptions and flat string arguments reduce wrong choices. The token saving is secondary.
• Few-shot examples cost tokens on every request unless they sit in a cached prefix, and small models can copy their values.
• Quantization effects are measured with mean KLD on code-like text and with parse, schema-valid, and right-call rates without a grammar. The rates are compared with Wilson intervals. A difference between 90% and 85% needs about 700 prompts per file to separate.
• Loop detection compares normalized call signatures that include the result. The agent adds a note on the first detection and stops on the second.
• Model choice starts with hard constraints (memory at the target context, template tool support, trained context, license) and then ranks the survivors on the team's own measurements.

Summary

• A catalog of eleven replies maps each small-model failure to grammar, repair, retry, loop detection, or the tool's own error result.
• In the 18 requests without both a prefill and a required grammar, the random model produced no parseable call. With tool_choice: "required", a prefill, and a logit bias, it produced 11 complete schema-valid calls and 1 cut-off call in 6 requests.
• "verbose": true returns the server's grammar. The required grammar has until rules for free text, (tool-call space)+ for the calls, and literal alternatives for enum values.
• The repair scan handles trailing commas, single quotes, raw newlines and tabs, Python literals, and text around the object. Truncation is reported and not executed.
• Retry sends the error as a ToolMessage with isError. Pruning the failed attempt afterwards shortens the history at the cost of prompt-cache reuse.
• In llama.cpp a repeat penalty divides positive logits. At 1.5 it lowered " from 0.821 to 0.629. The server defaults are temperature 0.8, top_k 40, top_p 0.95, min_p 0.05, and repeat penalty 1.0 (off).
• The lint found 18 problems in an 8-tool set and none in a 4-tool set of 1,283 characters.
• Two few-shot examples were 230 real tokens with the Qwen2 tokenizer (283 estimated).
• Capping tool output and removing stale reads and logs cut the prompt tokens of an 11-request session from 104,132 to 26,120.
• Crush v0.97.1 stops when one tool-call signature appears more than 5 times in 10 steps. The OpenHands SDK 1.51.0 StuckDetector uses thresholds 4, 3, and 6.
• /props reports chat_template_caps.supports_tool_calls, a hard constraint that can be read before any test.
• The harness passed 1, 3, 5, and 6 of 7 scenarios as repair, retry, and loop detection were added. An unverified final answer failed in every configuration.