Function Calling from First Principles

A language model can only produce text. It cannot open a file, run a command, or search the web. Yet a coding agent reads files and runs builds. The trick is a division of labor. The model writes a request in a fixed text format, for example "please call read_file with the path pom.xml". A normal program outside the model reads that text, does the work, and sends the result back to the model as more text. This pattern is called tool calling, or function calling. A tool is a function that the program offers to the model.

This lesson follows one tool call from start to end:

1. The program describes the tool (the tool definition).
2. The chat template turns that description into text in the prompt.
3. The model writes a call as text.
4. A parser (code that reads text and extracts its parts) turns the text into a structured call. llama-server can do this, or the client can.
5. The program runs the tool and sends the result back.

After the lesson, you can define tools, parse calls from the common formats, send results back in the right order, and recognize the ways a call goes wrong.

The real outputs come from llama-server (the HTTP server that ships with llama.cpp) at the pinned llama.cpp commit. The server runs the course's tiny model with random weights (llama-server: Endpoints, Slots, and Prompt Cache introduced it). A model with random weights writes meaningless text, so several requests add a grammar (a rule that restricts which text the model may produce) or otherwise constrain the output. The text says where. The structure of every response is real, only the words are noise.

Example 1: a tool call is generated text

The first question is what a tool call looks like from the model's side. The commands start the server with a chat template, render a prompt that contains one tool through the /apply-template endpoint (which shows the prompt without running the model), and then send that prompt to the raw /completion endpoint. /completion returns whatever text the model generates, with no parsing at all. Watch two things: what the template adds to the prompt, and what the model writes back.

llama-server -m tiny-random-qwen2vocab-f32.gguf --port 8080 -c 2048 \
  --jinja --chat-template-file models/templates/Qwen-Qwen2.5-7B-Instruct.jinja

TOOL='{"type": "function", "function": {"name": "read_file", "description": "Read a text file from the workspace",
  "parameters": {"type": "object", "properties": {"path": {"type": "string", "description": "Path relative to the workspace root"}},
                 "required": ["path"]}}}'
curl -s http://127.0.0.1:8080/apply-template \
  -d "{\"messages\": [{\"role\": \"user\", \"content\": \"What is in pom.xml?\"}], \"tools\": [$TOOL]}" > prompt.json
jq -r .prompt prompt.json

The command prints the prompt that the template built:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "read_file", "description": "Read a text file from the workspace", "parameters": {"type": "object", "properties": {"path": {"type": "string", "description": "Path relative to the workspace root"}}, "required": ["path"]}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
What is in pom.xml?<|im_end|>
<|im_start|>assistant

/apply-template applies the chat template and returns the prompt as one string, without running the model. Look at the system turn. The Qwen 2.5 template added a # Tools section with three parts:

• the tool definition as one line of JSON between <tools> tags,
• a sentence saying that the model may call one or more functions,
• the reply format: a JSON object {"name": ..., "arguments": ...} between <tool_call> tags.

The request had no system message, so the template used its default identity sentence ("You are Qwen, created by Alibaba Cloud"). The prompt ends with <|im_start|>assistant and a newline. That is the point where the model starts writing. Chat Templates, Roles, and Special Tokens explained these markers: each template wraps every message in special marker tokens that show who is speaking.

The model sees all of this as tokens. A token is a small piece of text, often a word or part of a word, and it is the unit that the model reads and writes. There is no separate channel for tools. The tool definition, the instructions, and the user's question are one sequence of tokens.

The server's /tokenize endpoint counts them with the Qwen2 tokenizer (the rule set that splits text into tokens). The prompt above is 190 tokens. The same request without tools renders to 35 tokens. The difference is 190 - 35 = 155 tokens. So the tool section with one small tool costs 155 tokens on every request. A model can only read a limited number of tokens at once. That limit is the context window, and this server started with -c 2048, which sets it to 2,048 tokens. For local models with small context windows, the number and size of tool definitions matter for this reason.

The next commands send the prompt to /completion twice: once as it is, and once with a grammar.

cat > toolcall.gbnf <<'EOF'
root ::= "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"" [a-zA-Z./]{1,20} "\"}}\n</tool_call>"
EOF
jq -n --slurpfile p prompt.json '{prompt: $p[0].prompt, n_predict: 16, temperature: 0}' \
  | curl -s http://127.0.0.1:8080/completion -d @- | jq '{content, stop_type, tokens_predicted}'
jq -n --slurpfile p prompt.json --rawfile g toolcall.gbnf '{prompt: $p[0].prompt, n_predict: 60, temperature: 0, grammar: $g}' \
  | curl -s http://127.0.0.1:8080/completion -d @- | jq '{content, stop_type, tokens_predicted, tokens_evaluated}'

The server printed one JSON result for each request:

{
  "content": " المياهshelf(Product;j_generation wikipedia纪委(step(step(step(step(step(step(step(step(step",
  "stop_type": "limit",
  "tokens_predicted": 16
}
{
  "content": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"profileshelf.modelsJ\"}}\n</tool_call>",
  "stop_type": "eos",
  "tokens_predicted": 49,
  "tokens_evaluated": 190
}

Read the first result. Without a grammar, the random model wrote 16 tokens of noise and stopped because it reached the n_predict limit (the maximum number of tokens to generate). The field stop_type: "limit" reports that reason. A trained model at this point would write <tool_call> followed by a JSON object, because it was trained on prompts of this shape.

Now read the second result. A grammar is a set of rules that says which text is allowed. The file toolcall.gbnf uses the GBNF format of llama.cpp, and its single rule allows exactly one string: the tool-call text with a path of 1 to 20 characters from [a-zA-Z./]. During generation, the server only lets the model choose tokens that keep the text inside the grammar. The model therefore produced <tool_call>, a newline, {"name": "read_file", "arguments": {"path": "profileshelf.modelsJ"}}, a newline, and </tool_call>. The path is nonsense because the weights are random, but the text has the shape of a call.

stop_type is eos this time. EOS means end of sequence, the special token that tells the model to stop. Once the grammar was complete, the only token it allowed was that end-of-generation token. tokens_evaluated is 190, the size of the prompt that was read. tokens_predicted is 49, the number of tokens the model wrote.

This output shows the whole mechanism. A tool call is ordinary generated text that follows a convention. The model does not run anything. Something outside the model must recognize the convention, extract the name and arguments, run the tool, and send the result back as more text. That "something" is either llama-server's chat endpoint or the agent itself. The rest of the lesson covers each of those steps.

Example 2: a tool definition as JSON Schema, built with Jackson

The next question is how to describe a tool so that every OpenAI-compatible server accepts it. A tool definition is a JSON object with three parts: type: "function", and a function object that holds a name, a description, and parameters. The parameters field is a JSON Schema. JSON Schema is a standard vocabulary for describing JSON values: their types, which properties are required, and which values are allowed. For read_file, the schema says "an object with one required string property called path".

The program builds the definition with Jackson's tree model, in which ObjectNode and ArrayNode are JSON objects and arrays that you assemble in code. It then places the definition in a request. Look at the pretty-printed definition first, then at the full request body and its size.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class ToolDefinition {
    static final ObjectMapper JSON = new ObjectMapper();

    public static void main(String[] args) throws Exception {
        ObjectNode parameters = JSON.createObjectNode().put("type", "object");
        parameters.putObject("properties").putObject("path")
            .put("type", "string").put("description", "Path relative to the workspace root");
        parameters.putArray("required").add("path");
        parameters.put("additionalProperties", false);

        ObjectNode tool = JSON.createObjectNode().put("type", "function");
        tool.putObject("function")
            .put("name", "read_file")
            .put("description", "Read a text file from the workspace")
            .set("parameters", parameters);
        System.out.println(JSON.writerWithDefaultPrettyPrinter().writeValueAsString(tool));

        ObjectNode request = JSON.createObjectNode().put("model", "local");
        request.putArray("messages").addObject().put("role", "user").put("content", "What is in pom.xml?");
        request.putArray("tools").add(tool);
        request.put("tool_choice", "auto");
        String body = JSON.writeValueAsString(request);
        System.out.println(body);
        System.out.println("request body: " + body.length() + " characters, of which the tool is "
            + JSON.writeValueAsString(tool).length());
    }
}

{
  "type" : "function",
  "function" : {
    "name" : "read_file",
    "description" : "Read a text file from the workspace",
    "parameters" : {
      "type" : "object",
      "properties" : {
        "path" : {
          "type" : "string",
          "description" : "Path relative to the workspace root"
        }
      },
      "required" : [ "path" ],
      "additionalProperties" : false
    }
  }
}
{"model":"local","messages":[{"role":"user","content":"What is in pom.xml?"}],"tools":[{"type":"function","function":{"name":"read_file","description":"Read a text file from the workspace","parameters":{"type":"object","properties":{"path":{"type":"string","description":"Path relative to the workspace root"}},"required":["path"],"additionalProperties":false}}}],"tool_choice":"auto"}
request body: 385 characters, of which the tool is 275

The pretty-printed definition has two levels. The outer object says that the tool is a function. The inner function object has the three fields that a template reads. parameters is a schema for one JSON object. It has a path property of type string with its own description, required lists path, and additionalProperties: false says that no other keys are allowed.

Every field is text that the model reads. The tool description tells the model when to use the tool, and the property description tells it what to put in the argument. A model chooses tools and fills in arguments from these strings and nothing else. So "Path relative to the workspace root" prevents absolute paths more reliably than code that rejects them afterwards.

The second printed line is a complete request body. tools is an array, so a request can offer many tools. tool_choice: "auto" lets the model decide between calling a tool and answering in text. Example 9 covers the other values.

The last line shows that the one tool is 275 of the request's 385 characters, about 71 percent. Tool definitions are usually the largest fixed part of an agent's prompt, and they are sent with every request. The Agent Loop and a Tool Framework generates these schemas from Java records, so that the definitions stay in step with the code.

Example 3: how templates render tools into the prompt

Do all models see tools in the same way? They do not. Each model family was trained with its own prompt layout, so each chat template renders the same definition differently. The command renders the request of Example 1 with three more templates from llama.cpp's models/templates directory. Each render starts with <|endoftext|>, which the templates print as the beginning-of-sequence token of the tiny model's Qwen2 vocabulary. That token is removed from the listings. Compare where the tools appear and which reply format each template asks for.

for t in meta-llama-Llama-3.1-8B-Instruct mistralai-Mistral-Nemo-Instruct-2407 NousResearch-Hermes-3-Llama-3.1-8B-tool_use; do
  # restart llama-server with --chat-template-file models/templates/$t.jinja, then:
  curl -s http://127.0.0.1:8080/apply-template \
    -d "{\"messages\": [{\"role\": \"user\", \"content\": \"What is in pom.xml?\"}], \"tools\": [$TOOL]}" | jq -r .prompt
done

First, the render for Llama 3.1 (the date line is the day of the run):

<|start_header_id|>system<|end_header_id|>

Environment: ipython
Cutting Knowledge Date: December 2023
Today Date: 03 Oct 2026

<|eot_id|><|start_header_id|>user<|end_header_id|>

Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.Do not use variables.

{
    "type": "function",
    "function": {
        "name": "read_file",
        "description": "Read a text file from the workspace",
        "parameters": {
            "type": "object",
            "properties": {
                "path": {
                    "type": "string",
                    "description": "Path relative to the workspace root"
                }
            },
            "required": [
                "path"
            ]
        }
    }
}

What is in pom.xml?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Next, the render for Mistral Nemo:

[AVAILABLE_TOOLS][{"type": "function", "function": {"name": "read_file", "description": "Read a text file from the workspace", "parameters": {"type": "object", "properties": {"path": {"type": "string", "description": "Path relative to the workspace root"}}, "required": ["path"]}}}][/AVAILABLE_TOOLS][INST]What is in pom.xml?[/INST]

Last, the render for Hermes 3:

<|im_start|>system
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> {"type": "function", "function": {"name": "read_file", "description": "read_file(path: str) - Read a text file from the workspace

    Args:
        path(str): Path relative to the workspace root", "parameters": {"type": "object", "properties": {"path": {"type": "string", "description": "Path relative to the workspace root"}}, "required": ["path"]}} </tools>Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}}
For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
<tool_call>
{"name": <function-name>, "arguments": <args-dict>}
</tool_call><|im_end|>
<|im_start|>user
What is in pom.xml?<|im_end|>
<|im_start|>assistant

The three renders differ in where the tools go, how they are written, and what reply format the template asks for. Together with Example 1, the four templates compare like this:

The Llama 3.1 template sets Environment: ipython in the system turn and puts the tools and the instructions into the user's message. It asks for parameters, not arguments, as the key of the argument object. The date lines come from the template's date handling, so they change with the day of the run.

The Mistral Nemo template writes the tool list in a block of special markers and gives no instructions at all. The reply format, [TOOL_CALLS] followed by a JSON array, is something the model learned in training. The template only renders that format when the history contains earlier calls (Example 7). So a client that wants to parse Mistral output must know the format from the template's history rendering or from the model's documentation.

The Hermes 3 template rewrote the tool description into a Python-style docstring: read_file(path: str) - Read a text file from the workspace, followed by an Args: section. It also added a pydantic JSON schema (a schema written for the Python library pydantic) for the call object itself. The reply format is the same <tool_call> tag form that Qwen 2.5 uses, so one parser serves both families.

Two consequences follow:

• The same tool costs a different number of tokens per model. Verbose renders, such as the indented JSON of Llama 3.1, cost more.
• A parser must match the format of the model it serves.

llama-server handles both sides when it runs with --jinja. This flag makes the server render prompts with the model's own Jinja template (Jinja is the template language that chat templates are written in). Then, at this commit, the server analyses the template to find the format of its tool calls and builds a parser for it. The analysis is called a differential autoparser. It lives in common/chat-diff-analyzer.cpp and common/chat-auto-parser-generator.cpp, and it renders variants of a conversation and compares the outputs to find the markers. For formats that the analysis cannot handle, such as Harmony, the server uses hand-written parsers, for example common/parsers/gpt-oss.cpp.

Example 4: four model-specific formats and a parser for each

Sometimes a client receives raw text and has to read the call itself. That happens with the raw /completion endpoint, or with an engine that does not parse calls. The program defines one parser interface and four implementations, one per format family, and runs each on a sample written in that family's form. The Hermes, Llama 3, and Mistral samples follow the renders of Examples 3 and 7. The Harmony sample follows the forms that common/parsers/gpt-oss.cpp accepts (Harmony is the message format of the gpt-oss models). Each parser returns the text outside the call (content) and a list of calls.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class FormatParsers {
    static final ObjectMapper JSON = new ObjectMapper();
    record ParsedCall(String id, String name, String arguments) {}      // arguments as compact JSON text
    record Parsed(String content, List<ParsedCall> calls) {}
    interface ToolCallParser { Parsed parse(String text) throws Exception; }

    // Hermes 2/3 and Qwen 2.5: <tool_call>{"name": ..., "arguments": {...}}</tool_call>, repeated for parallel calls
    static final Pattern HERMES_CALL = Pattern.compile("(?s)<tool_call>\\s*(\\{.*?\\})\\s*</tool_call>");
    static final ToolCallParser HERMES = text -> {
        List<ParsedCall> calls = new ArrayList<>();
        Matcher m = HERMES_CALL.matcher(text);
        while (m.find()) {
            JsonNode o = JSON.readTree(m.group(1));
            calls.add(new ParsedCall(null, o.get("name").asText(), o.get("arguments").toString()));
        }
        int first = text.indexOf("<tool_call>");
        return new Parsed((first < 0 ? text : text.substring(0, first)).strip(), calls);
    };

    // Llama 3.1 custom tools: the whole reply is {"name": ..., "parameters": {...}}
    static final ToolCallParser LLAMA3 = text -> {
        String t = text.strip();
        if (!t.startsWith("{")) return new Parsed(text, List.of());
        JsonNode o = JSON.readTree(t);
        if (!o.has("name") || !o.has("parameters")) return new Parsed(text, List.of());
        return new Parsed("", List.of(new ParsedCall(null, o.get("name").asText(), o.get("parameters").toString())));
    };

    // Mistral Nemo: [TOOL_CALLS] followed by a JSON array of calls with 9-character ids
    static final ToolCallParser MISTRAL = text -> {
        int i = text.indexOf("[TOOL_CALLS]");
        if (i < 0) return new Parsed(text, List.of());
        List<ParsedCall> calls = new ArrayList<>();
        for (JsonNode o : JSON.readTree(text.substring(i + "[TOOL_CALLS]".length())))
            calls.add(new ParsedCall(o.path("id").asText(null), o.get("name").asText(), o.get("arguments").toString()));
        return new Parsed(text.substring(0, i).strip(), calls);
    };

    // Harmony (gpt-oss): a commentary message addressed to functions.NAME, ended by <|call|>
    static final Pattern HARMONY_CALL = Pattern.compile("(?s)(?:<\\|start\\|>assistant)?(?: to=functions\\.(\\w+))?<\\|channel\\|>(?:commentary|analysis)"
        + "(?: to=functions\\.(\\w+))?(?: ?(?:<\\|constrain\\|>)?\\w+)?<\\|message\\|>([^<]*)<\\|call\\|>");
    static final Pattern HARMONY_FINAL = Pattern.compile("(?s)<\\|channel\\|>final<\\|message\\|>(.*?)(?:<\\|return\\|>|<\\|end\\|>|$)");
    static final ToolCallParser HARMONY = text -> {
        List<ParsedCall> calls = new ArrayList<>();
        Matcher m = HARMONY_CALL.matcher(text);
        while (m.find()) {
            String name = m.group(1) != null ? m.group(1) : m.group(2);
            calls.add(new ParsedCall(null, name, JSON.readTree(m.group(3)).toString()));
        }
        Matcher f = HARMONY_FINAL.matcher(text);
        return new Parsed(f.find() ? f.group(1) : "", calls);
    };

    public static void main(String[] args) throws Exception {
        Map<String, ToolCallParser> parsers = new LinkedHashMap<>();
        Map<String, String> samples = new LinkedHashMap<>();
        parsers.put("hermes", HERMES);
        samples.put("hermes", "I will read the build file.\n<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>");
        parsers.put("llama3", LLAMA3);
        samples.put("llama3", "{\"name\": \"read_file\", \"parameters\": {\"path\": \"pom.xml\"}}");
        parsers.put("mistral", MISTRAL);
        samples.put("mistral", "[TOOL_CALLS][{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}, \"id\": \"a1B2c3D4e\"}]");
        parsers.put("harmony", HARMONY);
        samples.put("harmony", "<|channel|>analysis<|message|>The user wants pom.xml.<|end|>"
            + "<|start|>assistant<|channel|>commentary to=functions.read_file <|constrain|>json<|message|>{\"path\": \"pom.xml\"}<|call|>");

        for (String format : parsers.keySet()) {
            Parsed p = parsers.get(format).parse(samples.get(format));
            for (ParsedCall c : p.calls())
                System.out.printf("%-8s name=%s arguments=%s id=%s content=\"%s\"%n", format, c.name(), c.arguments(), c.id(), p.content());
        }
        Parsed answer = HERMES.parse("pom.xml declares one module.");
        System.out.println("plain text: calls=" + answer.calls().size() + " content=\"" + answer.content() + "\"");
        Parsed llamaText = LLAMA3.parse("{\"answer\": 42}");
        System.out.println("llama3 JSON without name: calls=" + llamaText.calls().size());
    }
}

hermes   name=read_file arguments={"path":"pom.xml"} id=null content="I will read the build file."
llama3   name=read_file arguments={"path":"pom.xml"} id=null content=""
mistral  name=read_file arguments={"path":"pom.xml"} id=a1B2c3D4e content=""
harmony  name=read_file arguments={"path":"pom.xml"} id=null content=""
plain text: calls=0 content="pom.xml declares one module."
llama3 JSON without name: calls=0

All four lines show the same call, read_file with {"path":"pom.xml"}, extracted from four different texts. Here is how each parser works.

The Hermes parser finds every <tool_call>...</tool_call> block with a non-greedy regular expression (one that matches as little text as possible), parses the JSON inside, and takes name and arguments. Text before the first tag is content, here I will read the build file., which a client shows to the user. The non-greedy \{.*?\} followed by \s*</tool_call> still handles nested braces, because the regex engine keeps extending the match until the closing tag follows. Several blocks in one reply are parallel calls.

The Llama 3 parser treats the whole reply as one JSON object with name and parameters. If the reply does not start with {, or the object lacks those keys, the parser treats it as a text answer. The last output line confirms that {"answer": 42} is not a call. This format is ambiguous by design: a model that answers with a JSON document whose keys happen to be name and parameters looks like a call. Llama 3.1 also has built-in tools, which the template renders as <|python_tag|> followed by name.call(...). The parser here handles only the JSON form used for custom tools.

The Mistral parser finds [TOOL_CALLS] and parses the JSON array after it. It is the only format of the four in which the model writes the call id itself (a1B2c3D4e, 9 characters, as the template requires). The id is a label that later links a result to its call. In the other formats, id is null, and the client or server must generate ids.

The Harmony parser looks for a message on the commentary or analysis channel (Harmony splits the model's output into named channels) that is addressed to functions.NAME. The address can be in the role header (<|start|>assistant to=functions.read_file<|channel|>...) or in the channel header (<|channel|>commentary to=functions.read_file). An optional <|constrain|>json marker may follow. The parser takes the text up to <|call|> as the arguments. The analysis message before it is the model's reasoning and is not part of the content. content is empty here because no final message exists yet.

The plain text line shows that text without markers parses to zero calls and keeps the text as content. Every parser must handle this case, since most turns of a conversation are not calls.

Example 5: parsing tool calls from text, including malformed cases

Real models make mistakes. They produce cut-off output, wrong quotes, double-encoded arguments, and calls in the wrong wrapper. The question is what a parser should do when the text is almost, but not quite, a call. The program extends the Hermes parser so that it reports each problem in an errors list instead of throwing an exception. It runs on nine texts. Compare each numbered output line with the case in the list that follows the output.

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;

public class MalformedCalls {
    static final ObjectMapper JSON = new ObjectMapper();
    record ParsedCall(String name, String arguments) {}
    record Result(String content, List<ParsedCall> calls, List<String> errors) {}

    static Result parse(String text) {
        StringBuilder content = new StringBuilder();
        List<ParsedCall> calls = new ArrayList<>();
        List<String> errors = new ArrayList<>();
        int pos = 0;
        while (true) {
            int open = text.indexOf("<tool_call>", pos);
            if (open < 0) { content.append(text.substring(pos)); break; }
            content.append(text, pos, open);
            int close = text.indexOf("</tool_call>", open);
            if (close < 0) { errors.add("unterminated <tool_call> (the output was cut off)"); break; }
            String body = text.substring(open + "<tool_call>".length(), close).strip();
            pos = close + "</tool_call>".length();
            JsonNode o;
            try {
                o = JSON.readTree(body);
            } catch (JsonProcessingException e) {
                errors.add("not valid JSON: " + e.getOriginalMessage().lines().findFirst().orElse(""));
                continue;
            }
            if (!o.path("name").isTextual()) { errors.add("call without a \"name\" string"); continue; }
            JsonNode a = o.has("arguments") ? o.get("arguments") : o.path("parameters");
            if (a.isTextual()) {                                  // arguments encoded twice, as a JSON string
                try { a = JSON.readTree(a.asText()); } catch (JsonProcessingException e) { errors.add("arguments string is not JSON"); continue; }
            }
            if (a.isMissingNode()) a = JSON.createObjectNode();
            if (!a.isObject()) { errors.add("arguments of " + o.get("name").asText() + " are not an object"); continue; }
            calls.add(new ParsedCall(o.get("name").asText(), a.toString()));
        }
        return new Result(content.toString().strip(), calls, errors);
    }

    public static void main(String[] args) {
        String call = "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>";
        String[] cases = {
            "Let me check.\n" + call,
            call + "\n" + call.replace("pom.xml", "README.md"),
            "<tool_call>\n{\"name\": \"read_file\", \"argum",
            "<tool_call>\n{'name': 'read_file', 'arguments': {'path': 'pom.xml'}}\n</tool_call>",
            "<tool_call>\n{\"name\": \"read_file\", \"arguments\": \"{\\\"path\\\": \\\"pom.xml\\\"}\"}\n</tool_call>",
            "<tool_call>\n{\"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>",
            "<tool_call>\n{\"name\": \"read_file\", \"arguments\": [\"pom.xml\"]}\n</tool_call>",
            "```json\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n```",
            call + "\nI hope this helps."};
        for (int i = 0; i < cases.length; i++) {
            Result r = parse(cases[i]);
            System.out.println((i + 1) + ". calls=" + r.calls() + " errors=" + r.errors()
                + " content=\"" + r.content().replace("\n", "\\n") + "\"");
        }
    }
}

1. calls=[ParsedCall[name=read_file, arguments={"path":"pom.xml"}]] errors=[] content="Let me check."
2. calls=[ParsedCall[name=read_file, arguments={"path":"pom.xml"}], ParsedCall[name=read_file, arguments={"path":"README.md"}]] errors=[] content=""
3. calls=[] errors=[unterminated <tool_call> (the output was cut off)] content=""
4. calls=[] errors=[not valid JSON: Unexpected character (''' (code 39)): was expecting double-quote to start field name] content=""
5. calls=[ParsedCall[name=read_file, arguments={"path":"pom.xml"}]] errors=[] content=""
6. calls=[] errors=[call without a "name" string] content=""
7. calls=[] errors=[arguments of read_file are not an object] content=""
8. calls=[] errors=[] content="```json\n{"name": "read_file", "arguments": {"path": "pom.xml"}}\n```"
9. calls=[ParsedCall[name=read_file, arguments={"path":"pom.xml"}]] errors=[] content="I hope this helps."

Each numbered line is one input.

1. Text followed by a call: one call, and Let me check. as content.
2. Two blocks: two calls in order, which is how parallel calls look in this format.
3. The output stops inside the call, as it does when max_tokens is reached. The parser finds <tool_call> without </tool_call> and reports unterminated. Running a half-written call is never correct, because the missing part could change its meaning.
4. Single quotes. JSON requires double quotes, so Jackson rejects the text at the first '. Python-trained models write Python dictionaries this way. A repair step that converts quotes can fix some cases. A later lesson on making small local models reliable covers JSON repair.
5. arguments is a string that contains JSON ("{\"path\": \"pom.xml\"}"). This is the encoding of the API's arguments field leaking into the model's text. The parser detects a string, parses it a second time, and accepts the call. Accepting this form costs nothing and avoids a retry.
6. A call without name cannot be dispatched, so it is reported.
7. arguments is an array. Tools take an object of named arguments, so this is rejected.
8. JSON in a Markdown code fence without tags. The format requires tags, so the parser treats the text as plain content, and calls is empty. A model that writes this has not followed the template's instructions. An agent can detect the pattern and reply with a short reminder of the format.
9. A call followed by text. The call is parsed, and I hope this helps. becomes content, because the parser appends the text after each closing tag.

The errors list matters as much as the calls. An agent sends errors back to the model as a message, in the same way it sends tool errors (the agent loop lesson does this), so the model can try again with a corrected call. If the agent discards a malformed call without a word, the model keeps waiting for a result that never comes.

Example 6: server-side parsing into tool_calls

Now the server does the parsing. With --jinja (the default at this commit) and tools in the request, the chat endpoint parses the model's output with the template's format and returns structured calls. The question is what that response looks like.

Because the model has random weights, the request uses several steps from llama-server: Endpoints, Slots, and Prompt Cache to make it produce a call anyway:

• tool_choice: "required" makes the server add a grammar that forces a call.
• The final assistant message prefills <tool_call>\n. The server continues a final assistant message by default, so the model starts inside the call.
• An enum limits path to two values.
• A logit bias of +100 on the </ token (id 522) and on <|im_end|> (id 151645) makes the model close the call as soon as the grammar allows. A logit bias raises or lowers the chance of a specific token.

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{"role": "user", "content": "What is in pom.xml?"},
               {"role": "assistant", "content": "<tool_call>\n"}],
  "tools": [{"type": "function", "function": {"name": "read_file", "description": "Read a text file from the workspace",
             "parameters": {"type": "object", "properties": {"path": {"type": "string", "enum": ["pom.xml", "src/Main.java"]}},
                            "required": ["path"]}}}],
  "tool_choice": "required", "parallel_tool_calls": false,
  "logit_bias": {"151645": 100, "522": 100}, "temperature": 0, "max_tokens": 200}' | jq '{choices, usage}'

The real response, with prompt_tokens_details removed:

{
  "choices": [
    {
      "finish_reason": "tool_calls",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "type": "function",
            "function": {
              "name": "read_file",
              "arguments": "{\n  \t\t\t  \t\t\t  \t\t\t  \t\t\t\"path\":\"src/Main.java\" }"
            },
            "id": "4tchrKKLmaYPKFydOBferm8vtWKKy0lH"
          }
        ]
      }
    }
  ],
  "usage": {
    "completion_tokens": 61,
    "prompt_tokens": 196,
    "total_tokens": 257
  }
}

The message has content: "" and a tool_calls array with one call, and finish_reason is tool_calls. That value tells the client that the model stopped to call a tool, not to give a final answer. The call has four fields:

• id: a 32-character string that the server generated (gen_tool_call_id in tools/server/server-common.cpp returns a random string),
• type: "function",
• the function name,
• arguments.

The random model picked src/Main.java, the other value that the enum allowed. The grammar guarantees a call that matches the schema. It does not guarantee the right call.

arguments is a string. Its content is JSON with odd whitespace (a newline, spaces, and tabs), which the grammar permits between tokens and the random model chose. A trained model writes compact JSON. The program below parses the response the way a client does.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;

public class ServerToolCalls {
    static final ObjectMapper JSON = new ObjectMapper();
    record ToolCall(String id, String name, String arguments) {}

    // the response above, as the server sent it
    static final String RESPONSE = """
        {"choices": [{"finish_reason": "tool_calls", "index": 0, "message": {"role": "assistant", "content": "", "tool_calls": [{"type": "function", "function": {"name": "read_file", "arguments": "{\\n  \\t\\t\\t  \\t\\t\\t  \\t\\t\\t  \\t\\t\\t\\"path\\":\\"src/Main.java\\" }"}, "id": "4tchrKKLmaYPKFydOBferm8vtWKKy0lH"}]}}], "usage": {"completion_tokens": 61, "prompt_tokens": 196, "total_tokens": 257}}""";

    public static void main(String[] args) throws Exception {
        JsonNode choice = JSON.readTree(RESPONSE).at("/choices/0");
        System.out.println("finish_reason: " + choice.get("finish_reason").asText());
        List<ToolCall> calls = new ArrayList<>();
        for (JsonNode c : choice.at("/message/tool_calls"))
            calls.add(new ToolCall(c.get("id").asText(), c.at("/function/name").asText(), c.at("/function/arguments").asText()));
        for (ToolCall call : calls) {
            System.out.println("id: " + call.id() + " (" + call.id().length() + " characters)");
            System.out.println("name: " + call.name());
            System.out.println("arguments is a string of " + call.arguments().length() + " characters");
            JsonNode parsed = JSON.readTree(call.arguments());           // the second parse: string -> object
            System.out.println("parsed arguments: " + parsed + ", path = " + parsed.get("path").asText());
        }
    }
}

finish_reason: tool_calls
id: 4tchrKKLmaYPKFydOBferm8vtWKKy0lH (32 characters)
name: read_file
arguments is a string of 46 characters
parsed arguments: {"path":"src/Main.java"}, path = src/Main.java

The client parses twice. The first parse reads the whole response body, and arguments comes out as a string of 46 characters, including the whitespace. The second parse turns that string into an object, {"path":"src/Main.java"}. The OpenAI API defines arguments as a string so that a server can return exactly what the model wrote, even when it is not valid JSON. A client must therefore handle a parse failure at this second step (Example 11).

The 32-character id identifies the call for the rest of the conversation. The client does not interpret it. It copies the id into the result message (Example 7). Most of the 61 completion tokens in the usage are whitespace tokens that a trained model does not write.

Example 7: tool results and tool_call_id, a full round trip

How does a tool result go back to the model? The client appends the assistant message with its tool_calls unchanged. It runs each call. Then it appends one message per call with role: "tool", the call's id in a field named tool_call_id, and the result in content. The program builds the second request's messages from the first response. Look at the last printed line, which checks that the ids match.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.util.Map;

public class RoundTrip {
    static final ObjectMapper JSON = new ObjectMapper();

    public static void main(String[] args) throws Exception {
        Map<String, String> files = Map.of("pom.xml", "<project>demo</project>");
        ArrayNode messages = JSON.createArrayNode();
        messages.addObject().put("role", "user").put("content", "What is in pom.xml?");

        // request 1 went out; the assistant message of response 1 came back with one call
        ObjectNode assistant = (ObjectNode) JSON.readTree("""
            {"role": "assistant", "content": "", "tool_calls": [{"id": "call0abc1", "type": "function",
             "function": {"name": "read_file", "arguments": "{\\"path\\": \\"pom.xml\\"}"}}]}""");
        messages.add(assistant);                                    // 1. append the assistant message unchanged

        for (var call : assistant.get("tool_calls")) {               // 2. run each call, 3. append one tool message per call
            String path = JSON.readTree(call.at("/function/arguments").asText()).get("path").asText();
            messages.addObject().put("role", "tool")
                .put("tool_call_id", call.get("id").asText())
                .put("content", files.getOrDefault(path, "Error: no such file " + path));
        }
        for (var m : messages) System.out.println(m);

        // the pairing a server and a template rely on
        String callId = messages.get(1).at("/tool_calls/0/id").asText();
        String resultId = messages.get(2).get("tool_call_id").asText();
        System.out.println("result answers call: " + callId.equals(resultId));
    }
}

{"role":"user","content":"What is in pom.xml?"}
{"role":"assistant","content":"","tool_calls":[{"id":"call0abc1","type":"function","function":{"name":"read_file","arguments":"{\"path\": \"pom.xml\"}"}}]}
{"role":"tool","tool_call_id":"call0abc1","content":"<project>demo</project>"}
result answers call: true

The three printed messages are the history for the second request: the user's question, the assistant's call, and the tool's result. The assistant message is copied unchanged, including arguments as a string. The tool message has tool_call_id: "call0abc1", the same value as the call's id, and the last line confirms the pairing. The id here has 9 alphanumeric characters because the Mistral template below checks that length.

Before the next model call, the server renders this history with the model's chat template. The command sends the history to /apply-template with each of four templates, so that you can see how each template writes the call and the result.

curl -s http://127.0.0.1:8080/apply-template -d '{"messages": [
  {"role": "user", "content": "What is in pom.xml?"},
  {"role": "assistant", "content": "", "tool_calls": [{"id": "call0abc1", "type": "function",
    "function": {"name": "read_file", "arguments": "{\"path\": \"pom.xml\"}"}}]},
  {"role": "tool", "tool_call_id": "call0abc1", "content": "<project>demo</project>"}],
  "tools": ['"$TOOL"']}' | jq -r .prompt

The part of each prompt after the user message, for four templates:

Qwen 2.5:
<|im_start|>assistant
<tool_call>
{"name": "read_file", "arguments": {"path": "pom.xml"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
<project>demo</project>
</tool_response><|im_end|>
<|im_start|>assistant

Hermes 3:
<|im_start|>assistant
<tool_call>
{"name": "read_file", "arguments": {"path": "pom.xml"}}
</tool_call><|im_end|>
<|im_start|>tool
<tool_response>
<project>demo</project>
</tool_response><|im_end|><|im_start|>assistant

Llama 3.1:
<|start_header_id|>assistant<|end_header_id|>

{"name": "read_file", "parameters": {"path": "pom.xml"}}<|eot_id|><|start_header_id|>ipython<|end_header_id|>

"<project>demo</project>"<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Mistral Nemo:
[TOOL_CALLS][{"name": "read_file", "arguments": {"path": "pom.xml"}, "id": "call0abc1"}]<|endoftext|>[TOOL_RESULTS]{"content": <project>demo</project>, "call_id": "call0abc1"}[/TOOL_RESULTS]

The four renders show four ways to write the same result:

The template rewrites the API's arguments string as a JSON object inside the call text. llama-server prepares this before rendering, for templates that expect objects (the func_args_not_string workaround in common/chat.cpp).

Only the Mistral template writes the id into the text. For the other three, the model can match a result to its call only by position, which is why the order of results matters (Example 8). The Mistral render also writes the content without quotes ({"content": <project>demo</project>, ...}), because the template inserts the string as it is. A result that is itself JSON stays valid, but plain text does not. Between the call and the results, the Mistral render prints the end-of-sequence token. That token is <|endoftext|> in the tiny model's vocabulary and </s> in a real Mistral GGUF file.

The Mistral template also enforces its id format. The same history with the 32-character id from Example 6 fails with a real error from llama-server (trimmed): {"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: ... Error: Jinja Exception: Tool call IDs should be alphanumeric strings with length 9!"}}. A client that uses one id scheme for all models, or that switches models in the middle of a session, has to map ids to the format that the template accepts.

Example 8: parallel tool calls, ids, and result order

A model can ask for several tools in one turn, for example "read these two files". How are such calls identified and answered? With parallel_tool_calls: true, the server's grammar allows more than one call per reply. If the request does not set the field, the server uses a default that comes from the template's capabilities: /props reports "supports_parallel_tool_calls": true for the Qwen 2.5 template. The request below asks the random model to compare two files. Count the calls in the output and look at finish_reason.

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{"role": "user", "content": "Compare pom.xml and README.md."},
               {"role": "assistant", "content": "<tool_call>\n"}],
  "tools": [{"type": "function", "function": {"name": "read_file", "description": "Read a text file from the workspace",
             "parameters": {"type": "object", "properties": {"path": {"type": "string", "enum": ["pom.xml", "README.md"]}},
                            "required": ["path"]}}}],
  "tool_choice": "required", "parallel_tool_calls": true,
  "logit_bias": {"522": 100}, "temperature": 0, "max_tokens": 300}' \
  | jq -c '.choices[0].finish_reason, (.choices[0].message.tool_calls[] | {id, arguments: .function.arguments})'

The calls, with only the id and the arguments shown:

"length"
{"id":"3br1y9dSC9xmC4ugcI3IAWiolYBI3BKF","arguments":"{\n \t\t\t  \t\t\t  \t\t\t  \t\t\t \"path\":\"pom.xml\" }"}
{"id":"bF3ojBoHJeCpmVMzIVLtoujI1HH7NqPk","arguments":"{\n  \t\t\t  \t\t\t  \t\t\t  \t\t\t\"path\":\"pom.xml\" }"}
{"id":"a6jQ6Lrq9FWUOJGleeFI6HZhZGfEVgs0","arguments":"{\n     \t\t\t\t\t     \t\t\t\t\t\"path\":\"README.md\"}"}
{"id":"o0LtWOxRhvLNGIquLUx0Zntai3lj8Uj0","arguments":"{\n     \t\t\t\t\t     \t\t\t\t\t\"path\":\"README.md\"\n\n     \t\t\t\t\t     \t\t\t\t\t}"}
{"id":"WZsjWUKX9jPKhkQtMfsXVpTfu3bwCeIp","arguments":"{\"path\":\"README.md\"\n\n     \t\t\t\t\t     \t\t\t\t\t}"}

The reply has five calls, each with its own 32-character id. The random model never chose the end-of-turn token, so it kept writing calls until max_tokens (300) stopped it. That is why finish_reason is length, not tool_calls. A trained model writes two calls and stops. The five calls are still useful as data. Two ask for pom.xml and three for README.md, in that order, and the order of the array is the order in which the model wrote them.

An agent answers parallel calls with one tool message per call, each carrying its call's id. The calls can run at the same time (The Agent Loop and a Tool Framework runs read-only calls on virtual threads), and they may finish in any order. This creates a risk. The program below shows what happens when results are sent in completion order to a template that drops ids, such as Qwen 2.5.

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;

public class ResultOrder {
    record ToolCall(String id, String path) {}
    record ToolResult(String callId, String content) {}

    /** What a template without ids (Qwen 2.5) shows the model: calls and results paired by position. */
    static void showPairing(String label, List<ToolCall> calls, List<ToolResult> results) {
        System.out.println(label);
        for (int i = 0; i < calls.size(); i++)
            System.out.println("  call " + (i + 1) + " read_file(" + calls.get(i).path() + ") <- " + results.get(i).content());
    }

    public static void main(String[] args) {
        Map<String, String> files = Map.of("pom.xml", "<project>demo</project>", "README.md", "# Demo");
        List<ToolCall> calls = List.of(new ToolCall("3br1y9dS", "pom.xml"), new ToolCall("a6jQ6Lrq", "README.md"));

        List<ToolResult> completed = new ArrayList<>();               // README.md happened to finish first
        for (ToolCall c : calls.reversed()) completed.add(new ToolResult(c.id(), files.get(c.path())));
        showPairing("results in completion order:", calls, completed);

        List<String> ids = calls.stream().map(ToolCall::id).toList();
        List<ToolResult> ordered = new ArrayList<>(completed);
        ordered.sort(Comparator.comparingInt(r -> ids.indexOf(r.callId())));
        showPairing("results sorted into call order:", calls, ordered);
        System.out.println("ids match positions: " + ordered.stream().map(ToolResult::callId).toList().equals(ids));
    }
}

results in completion order:
  call 1 read_file(pom.xml) <- # Demo
  call 2 read_file(README.md) <- <project>demo</project>
results sorted into call order:
  call 1 read_file(pom.xml) <- <project>demo</project>
  call 2 read_file(README.md) <- # Demo
ids match positions: true

The first block simulates README.md finishing first. The results were appended in completion order, so a template that pairs by position shows the model read_file(pom.xml) followed by # Demo, and read_file(README.md) followed by the content of pom.xml. Nothing fails. The ids in the JSON are still correct, the server accepts the request, and the model receives swapped data. The resulting answer would be wrong in a way that is hard to trace.

The second block sorts the results by the position of their call id in the call list. The pairing is now correct, and the last line confirms that the order of result ids equals the order of call ids. The rule for agents: send results in the order of the calls, whatever order the tools finished in.

Example 9: tool_choice: auto, none, required, and a named function

How does a client control whether the model calls a tool at all? The OpenAI API defines a request field tool_choice for this. It has three string values and one object form:

• auto: the model decides whether to call a tool.
• none: the model must not call a tool.
• required: the model must call at least one tool.
• an object that names one function: the model must call that function.

The commands test each value against llama-server at the pinned commit. In the output, check which requests produced a call and which produced an error.

# none: tools stay in the prompt, the server does not parse calls; a grammar forces call-shaped text
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "What is in pom.xml?"}],
  "tools": ['"$TOOL"'], "tool_choice": "none", "temperature": 0, "max_tokens": 200,
  "grammar": "root ::= \"<tool_call>\\n{\\\"name\\\": \\\"read_file\\\", \\\"arguments\\\": {\\\"path\\\": \\\"pom.xml\\\"}}\\n</tool_call>\""}' \
  | jq -c '.choices[0] | {finish_reason, message}'
# auto and a named function, without a grammar (12 tokens)
for choice in '"auto"' '{"type": "function", "function": {"name": "read_file"}}'; do
  curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "What is in pom.xml?"}],
    "tools": ['"$TOOL"'], "tool_choice": '"$choice"', "temperature": 0, "max_tokens": 12}' | jq -c '.choices[0] | {finish_reason, message}'
done
# an unknown value
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "Hi"}],
  "tools": ['"$TOOL"'], "tool_choice": "bogus"}'

The four responses:

{"finish_reason":"stop","message":{"role":"assistant","content":"<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>"}}
{"finish_reason":"length","message":{"role":"assistant","content":"المياهshelf(Product;j_generation wikipedia纪委(step(step(step(step(step"}}
{"finish_reason":"length","message":{"role":"assistant","content":"المياهshelf(Product;j_generation wikipedia纪委(step(step(step(step(step"}}
{"error":{"code":400,"message":"Invalid tool_choice: bogus","type":"invalid_request_error"}}

The server log for the named-function request (time prefix removed):

W Wrong type supplied for parameter 'tool_choice', using default value: [json.exception.type_error.302] type must be string, but is object

The first line is none with a grammar that forces call-shaped text. The server accepted the grammar, because it rejects custom grammars together with tools only when tool_choice is not none (Cannot use custom grammar constraints with tools. in tools/server/server-common.cpp). With none, the server does not parse tool calls, so the call text arrived as plain content and finish_reason is stop. The tools were still rendered into the prompt: /apply-template returns the same 804-character prompt with none as with required. So none stops parsing and grammar, but it does not make the model forget the tools. A client that wants the model to forget the tools must remove them from the request.

The second line is auto. The server builds a lazy grammar. Sampling is unconstrained until the model writes the trigger word, here <tool_call>, and from then on the grammar applies (grammar_lazy and grammar_triggers in common/chat-auto-parser-generator.cpp). The random model never wrote the trigger, so it produced 12 tokens of noise with no call. This is the normal mode for agents: the model decides, and a trained model writes the trigger when it wants a tool.

The third line is the object form naming read_file. The output is identical to auto, and the server log explains why. The server reads tool_choice as a string, the object has the wrong type, and the server uses the default value auto (the log line above). So at this commit, llama-server does not support forcing a named function through tool_choice. Hosted APIs and vLLM document the object form. The fourth line shows that an unknown string value is an error: HTTP 400 with Invalid tool_choice: bogus.

The program below shows the request variants, including a way to force one named function that works with llama-server. The idea is to offer only that one tool and require a call.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class ToolChoice {
    static final ObjectMapper JSON = new ObjectMapper();

    static ObjectNode tool(String name) {
        ObjectNode t = JSON.createObjectNode().put("type", "function");
        t.putObject("function").put("name", name).putObject("parameters").put("type", "object");
        return t;
    }

    /** Forces one named tool on servers that accept only string values of tool_choice. */
    static ObjectNode forceTool(ObjectNode request, String name) {
        ObjectNode copy = request.deepCopy();
        ArrayNode kept = JSON.createArrayNode();
        for (JsonNode t : request.get("tools")) if (t.at("/function/name").asText().equals(name)) kept.add(t);
        if (kept.isEmpty()) throw new IllegalArgumentException("no tool named " + name);
        copy.set("tools", kept);
        copy.put("tool_choice", "required");
        copy.put("parallel_tool_calls", false);
        return copy;
    }

    static String summary(ObjectNode r) {
        StringBuilder names = new StringBuilder();
        for (JsonNode t : r.get("tools")) names.append(names.isEmpty() ? "" : ",").append(t.at("/function/name").asText());
        String parallel = r.has("parallel_tool_calls") ? r.get("parallel_tool_calls").toString() : "(not sent)";
        return "tools=[" + names + "] tool_choice=" + r.get("tool_choice") + " parallel_tool_calls=" + parallel;
    }

    public static void main(String[] args) {
        ObjectNode base = JSON.createObjectNode();
        base.putArray("messages").addObject().put("role", "user").put("content", "Show me pom.xml");
        base.putArray("tools").add(tool("read_file")).add(tool("list_files")).add(tool("run_command"));
        for (String choice : new String[] {"auto", "none", "required"}) {
            ObjectNode r = base.deepCopy();
            r.put("tool_choice", choice);
            System.out.println(String.format("%-9s", choice) + summary(r));
        }
        ObjectNode named = base.deepCopy();
        named.putObject("tool_choice").put("type", "function").putObject("function").put("name", "read_file");
        System.out.println("named    " + summary(named) + "  (object form, OpenAI style)");
        System.out.println("forced   " + summary(forceTool(base, "read_file")) + "  (works with llama-server)");
    }
}

auto     tools=[read_file,list_files,run_command] tool_choice="auto" parallel_tool_calls=(not sent)
none     tools=[read_file,list_files,run_command] tool_choice="none" parallel_tool_calls=(not sent)
required tools=[read_file,list_files,run_command] tool_choice="required" parallel_tool_calls=(not sent)
named    tools=[read_file,list_files,run_command] tool_choice={"type":"function","function":{"name":"read_file"}} parallel_tool_calls=(not sent)  (object form, OpenAI style)
forced   tools=[read_file] tool_choice="required" parallel_tool_calls=false  (works with llama-server)

The first three lines differ only in tool_choice. The fourth line is the OpenAI object form, which this server ignores. The last line is the replacement: the request keeps only the read_file definition, sets tool_choice: "required", and turns off parallel calls. With one tool offered and a call required, the grammar can produce only a call to read_file, which is the effect of the named form.

The behavior of each value at this commit, in one table:

required has a cost: the model cannot finish a run with a text answer, so an agent that sets it on every step never stops by itself. Agents use it for a single step, then return to auto.

Example 10: streaming tool calls

When a response is streamed, the model's text arrives in many small chunks while it is being generated. A tool call arrives in pieces too. The first piece carries the id and the name, and later pieces carry fragments of the arguments string. The Chat Completions Protocol and Streaming in Java built the accumulator, which is the code that collects pieces into a whole. This program runs it on a real stream from llama-server. The stream comes from the request of Example 6, with stream: true and path limited to pom.xml. Each data: line is one chunk, and a delta holds the new piece.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Map;
import java.util.TreeMap;

public class StreamedCalls {
    static final ObjectMapper JSON = new ObjectMapper();

    // the streamed response from llama-server, data lines only, each chunk reduced to its "choices"
    static final String STREAM = """
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"id":"6aKdJM1U5AYakbQUh2ae9giQx978MzKC","type":"function","function":{"name":"read_file","arguments":"{"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\\n \\t\\t\\t  \\t\\t\\t  \\t\\t\\t  \\t\\t\\t \\""}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"path"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\\":\\""}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"p"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"o"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"m"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":".x"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ml"}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\\""}}]}}]}
        data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":" }"}}]}}]}
        data: {"choices":[{"finish_reason":"tool_calls","index":0,"delta":{}}]}
        data: [DONE]""";

    static final class CallBuilder {
        String id, name;
        final StringBuilder arguments = new StringBuilder();
        int pieces;
    }

    public static void main(String[] args) throws Exception {
        Map<Integer, CallBuilder> calls = new TreeMap<>();
        String finish = null;
        int chunks = 0;
        for (String line : STREAM.lines().toList()) {
            String data = line.substring("data: ".length());
            if (data.equals("[DONE]")) break;
            chunks++;
            JsonNode choice = JSON.readTree(data).at("/choices/0");
            if (!choice.get("finish_reason").isNull()) finish = choice.get("finish_reason").asText();
            for (JsonNode d : choice.path("delta").path("tool_calls")) {
                CallBuilder b = calls.computeIfAbsent(d.get("index").asInt(), i -> new CallBuilder());
                if (d.has("id")) b.id = d.get("id").asText();
                if (d.at("/function/name").isTextual()) b.name = d.at("/function/name").asText();
                if (d.at("/function/arguments").isTextual()) { b.arguments.append(d.at("/function/arguments").asText()); b.pieces++; }
            }
        }
        System.out.println("chunks before [DONE]: " + chunks + ", finish_reason: " + finish);
        for (var e : calls.entrySet()) {
            CallBuilder b = e.getValue();
            System.out.println("call " + e.getKey() + ": id=" + b.id + " name=" + b.name + " argument pieces=" + b.pieces);
            System.out.println("  joined arguments parse to " + JSON.readTree(b.arguments.toString()));
        }
    }
}

chunks before [DONE]: 13, finish_reason: tool_calls
call 0: id=6aKdJM1U5AYakbQUh2ae9giQx978MzKC name=read_file argument pieces=11
  joined arguments parse to {"path":"pom.xml"}

The stream had 13 data chunks before [DONE]:

• The first chunk only set the role.
• The second chunk carried index: 0, the id, type, the name, and the first argument piece {.
• The next ten chunks carried only index and argument fragments: whitespace, path, ":", then p, o, m, .x, ml one or two characters at a time, the closing quote, and }.
• The last chunk had an empty delta and finish_reason: "tool_calls".

The program grouped the pieces by index, so 11 argument pieces joined into one string, which parsed to {"path":"pom.xml"}. Two rules come from this output:

• The arguments string is valid JSON only after the last piece, so a client parses it after the stream ends, never per chunk.
• index, not id, links pieces together, because the id appears only in the first piece of each call. When a reply has parallel calls, their pieces arrive with indexes 0, 1, 2 and so on.

A user interface can show the tool name as soon as the first piece arrives. That tells the user what the agent is about to do before the arguments are complete.

Example 11: failure modes and a validator

Which failures should a client check for before it runs a call? The program validates a response against the declared tools and reports every problem it finds. The first input is a real response from llama-server for the parallel request of Example 8 with max_tokens set to 110. The second input is made up, to show the other failure kinds. Read the code as a checklist: each problems.add line is one failure mode.

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

public class CallValidator {
    static final ObjectMapper JSON = new ObjectMapper();

    /** Checks one choice of a chat completions response against the declared tools. */
    static List<String> validate(JsonNode choice, Map<String, JsonNode> schemas) {
        List<String> problems = new ArrayList<>();
        JsonNode calls = choice.path("tool_calls");
        if ("length".equals(choice.path("finish_reason").asText()))
            problems.add("finish_reason is length: the output hit max_tokens and the last call may be incomplete");
        Set<String> ids = new HashSet<>();
        for (int i = 0; i < calls.size(); i++) {
            JsonNode c = calls.get(i);
            String where = "call " + i + ": ";
            String id = c.path("id").asText("");
            if (id.isEmpty()) problems.add(where + "empty id");
            else if (!ids.add(id)) problems.add(where + "duplicate id " + id);
            String name = c.at("/function/name").asText("");
            JsonNode schema = schemas.get(name);
            if (schema == null) { problems.add(where + "unknown tool '" + name + "'"); continue; }
            JsonNode args;
            try {
                args = JSON.readTree(c.at("/function/arguments").asText(""));
            } catch (JsonProcessingException e) {
                String reason = e.getOriginalMessage().replaceAll("\\s*\\(start marker.*", "");
                problems.add(where + "arguments are not valid JSON (" + reason + ")");
                continue;
            }
            if (args == null || !args.isObject()) { problems.add(where + "arguments are not a JSON object"); continue; }
            for (JsonNode r : schema.path("required"))
                if (!args.has(r.asText())) problems.add(where + "missing required argument '" + r.asText() + "'");
            for (var e : args.properties()) {
                JsonNode p = schema.path("properties").get(e.getKey());
                if (p == null) problems.add(where + "unknown argument '" + e.getKey() + "'");
                else if (p.path("type").asText().equals("string") && !e.getValue().isTextual())
                    problems.add(where + "argument '" + e.getKey() + "' must be a string");
            }
        }
        return problems;
    }

    public static void main(String[] args) throws Exception {
        Map<String, JsonNode> schemas = Map.of("read_file", JSON.readTree(
            "{\"type\":\"object\",\"properties\":{\"path\":{\"type\":\"string\"}},\"required\":[\"path\"]}"));
        String[] responses = {
            // real: llama-server with max_tokens 110; the second call was cut off after "{"
            """
            {"finish_reason": "length", "tool_calls": [{"type": "function", "function": {"name": "read_file", "arguments": "{\\n \\t\\t\\t  \\t\\t\\t  \\t\\t\\t  \\t\\t\\t \\"path\\":\\"pom.xml\\" }"}, "id": "HUlRwgH2mmTv97GCwkugZHjQytYo6GED"}, {"type": "function", "function": {"name": "read_file", "arguments": "{"}, "id": "xxfbWHsmdATilmDyEDVl6PjLkPTkUCbl"}]}""",
            """
            {"finish_reason": "tool_calls", "tool_calls": [
              {"id": "x1", "function": {"name": "read_file", "arguments": "{\\"path\\": \\"pom.xml\\"}"}},
              {"id": "x1", "function": {"name": "read_files", "arguments": "{}"}},
              {"id": "", "function": {"name": "read_file", "arguments": "{\\"file\\": 7}"}},
              {"id": "x3", "function": {"name": "read_file", "arguments": "[\\"pom.xml\\"]"}}]}"""};
        for (String r : responses) {
            List<String> problems = validate(JSON.readTree(r), schemas);
            System.out.println(problems.isEmpty() ? "valid" : String.join("\n", problems));
            System.out.println();
        }
    }
}

finish_reason is length: the output hit max_tokens and the last call may be incomplete
call 1: arguments are not valid JSON (Unexpected end-of-input: expected close marker for Object)

call 1: duplicate id x1
call 1: unknown tool 'read_files'
call 2: empty id
call 2: missing required argument 'path'
call 2: unknown argument 'file'
call 3: arguments are not a JSON object

The real response has two problems. First, finish_reason is length: the output reached max_tokens, so the last call may be incomplete. Second, call 1 shows that it is: its arguments string is only {, and Jackson reports Unexpected end-of-input: expected close marker for Object. Call 0 is complete and valid. An agent can run call 0 and report call 1 as cut off, or it can discard the whole reply and retry with a higher limit. Running call 1 is impossible.

The made-up response shows the remaining kinds:

• Call 1 repeats the id x1, which makes pairing by id ambiguous, and names read_files, a tool that does not exist.
• Call 2 has an empty id, misses the required path, and has an unknown argument file.
• Call 3 has arguments that are valid JSON but an array, not an object.
• Call 0 passes all checks, so it does not appear in the output.

The failure modes in one table:

Grammar-constrained decoding prevents most of these at the source for servers that support it, as the random model's valid calls show. Truncation is the exception: a grammar cannot make a call fit in a token limit.

Example 12: prompt-based tool calling for models without native support

Some models have a chat template with no tool section, and some engines do not parse calls. Can such a model still use tools? Yes. The agent does the work that the template would do. It describes the tools in its own system prompt, defines a reply format, and parses the reply itself. The program uses a format of one JSON object per reply: {"tool": ..., "arguments": ...} for a call and {"answer": ...} for a final answer. The parser returns one of three outcomes (a call, an answer, or "could not parse"), which are the three records of a sealed Step type.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;

public class PromptBasedTools {
    static final ObjectMapper JSON = new ObjectMapper();
    record ToolSpec(String name, String description, String argumentsExample) {}
    sealed interface Step permits Call, Answer, Unparsed {}
    record Call(String tool, JsonNode arguments) implements Step {}
    record Answer(String text) implements Step {}
    record Unparsed(String reason) implements Step {}

    static String systemPrompt(List<ToolSpec> tools) {
        StringBuilder sb = new StringBuilder("You can use these tools:\n");
        for (ToolSpec t : tools) sb.append("- ").append(t.name()).append(": ").append(t.description())
            .append(" Arguments: ").append(t.argumentsExample()).append('\n');
        return sb.append("""
            Reply with exactly one JSON object and nothing else.
            To call a tool: {"tool": "<name>", "arguments": {...}}
            To answer the user: {"answer": "<text>"}""").toString();
    }

    static Step parse(String reply) {
        int start = reply.indexOf('{'), end = reply.lastIndexOf('}');    // tolerate code fences and extra prose
        if (start < 0 || end < start) return new Unparsed("no JSON object in the reply");
        try {
            JsonNode o = JSON.readTree(reply.substring(start, end + 1));
            if (o.path("tool").isTextual()) return new Call(o.get("tool").asText(), o.path("arguments"));
            if (o.path("answer").isTextual()) return new Answer(o.get("answer").asText());
            return new Unparsed("object has neither \"tool\" nor \"answer\"");
        } catch (Exception e) {
            return new Unparsed("invalid JSON");
        }
    }

    public static void main(String[] args) {
        List<ToolSpec> tools = List.of(
            new ToolSpec("read_file", "Read a text file.", "{\"path\": \"src/Main.java\"}"),
            new ToolSpec("list_files", "List files in a directory.", "{\"dir\": \"src\"}"));
        System.out.println(systemPrompt(tools));
        System.out.println("---");
        String[] replies = {
            "{\"tool\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}",
            "Sure, here is the call:\n```json\n{\"tool\": \"list_files\", \"arguments\": {\"dir\": \"src\"}}\n```",
            "{\"answer\": \"pom.xml declares the project demo.\"}",
            "I would read pom.xml first.",
            "{\"tool\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}"};
        for (String r : replies) System.out.println(parse(r));
    }
}

You can use these tools:
- read_file: Read a text file. Arguments: {"path": "src/Main.java"}
- list_files: List files in a directory. Arguments: {"dir": "src"}
Reply with exactly one JSON object and nothing else.
To call a tool: {"tool": "<name>", "arguments": {...}}
To answer the user: {"answer": "<text>"}
---
Call[tool=read_file, arguments={"path":"pom.xml"}]
Call[tool=list_files, arguments={"dir":"src"}]
Answer[text=pom.xml declares the project demo.]
Unparsed[reason=no JSON object in the reply]
Unparsed[reason=invalid JSON]

The first block is the system prompt. Each tool gets one line with its name, its description, and an example of its arguments. An example is shorter than a schema and shows the model the exact shape. The format section asks for exactly one JSON object and gives both forms.

The parser takes the text from the first { to the last }. That tolerates the code fences and the sentence that the second reply wraps around its JSON. The results:

• The first reply is a clean call.
• The second reply wraps a call in prose and a fence, and the parser still finds it.
• The third reply is a final answer.
• The fourth reply is prose without JSON and is reported as no JSON object in the reply.
• The fifth reply is missing its last }. The last } in the text closes the inner object, so the substring is not valid JSON, and the parser reports it.

The sealed Step type makes the three outcomes explicit for the agent: run a tool, finish, or tell the model what was wrong.

A grammar can enforce the format at the source, even without template support. llama-server converts a JSON Schema given in response_format into a grammar. A oneOf (a schema that accepts any one of several alternatives) with one branch per tool and one branch for the answer makes every reply either a valid call or an answer.

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{"role": "system", "content": "Reply with one JSON object: either {\"tool\": ..., \"arguments\": ...} to call a tool, or {\"answer\": ...} to answer."},
               {"role": "user", "content": "What is in pom.xml?"}],
  "response_format": {"type": "json_schema", "json_schema": {"name": "step", "schema": {"oneOf": [
    {"type": "object", "properties": {"tool": {"const": "read_file"},
      "arguments": {"type": "object", "properties": {"path": {"type": "string", "enum": ["pom.xml", "README.md"]}},
                    "required": ["path"], "additionalProperties": false}},
     "required": ["tool", "arguments"], "additionalProperties": false},
    {"type": "object", "properties": {"answer": {"type": "string", "maxLength": 40}},
     "required": ["answer"], "additionalProperties": false}]}}},
  "temperature": 0, "max_tokens": 300}' | jq -c '.choices[0] | {finish_reason, content: .message.content}'

The real response:

{"finish_reason":"stop","content":"{ \"tool\":\"read_file\",\n\n\t     \t\t     \t\t     \"arguments\":{\n \t\t     \t\t     \t \t \t\"path\":\"README.md\"}}"}

The random model's reply is one JSON object that matches the first branch: "tool":"read_file" and arguments with a path from the enum. The whitespace is odd for the same reason as in Example 6. finish_reason is stop because the grammar ended. The reply arrived as content, since no tools were declared, so the agent parses it with parse from the program above.

Prompt-based calling has costs compared with native support. The model was not trained on the format, so a model without native support follows it less reliably. The tool descriptions are the agent's own text, not the format the model saw in training. When a model has native tool support, its template and its training agree on one format, which is the reason to prefer it. A text format has the opposite property: it works with every model and every engine.

mini-swe-agent 2.4.6 ships both approaches. config/mini.yaml gives the model a native bash tool. config/mini_textbased.yaml asks for exactly one fenced code block tagged mswea_bash_command per reply, which the agent extracts with a regular expression (models/utils/actions_text.py).

Interview points

• A tool call is text that the model generates in a format its template taught it. Parsing that text into tool_calls is done by the server (llama-server with --jinja) or by the client.
• A tool definition is {"type": "function", "function": {name, description, parameters}}, with parameters as a JSON Schema. The descriptions are the model's only documentation of the tool.
• Templates render tools differently (system turn, user turn, special markers) and ask for different reply formats: Hermes uses <tool_call> tags, Llama 3 uses {"name", "parameters"}, Mistral uses [TOOL_CALLS] with 9-character ids, and Harmony uses messages to functions.NAME.
• arguments is a JSON string in the API, not an object. Clients parse it a second time and must handle invalid JSON at that step.
• Results go back as role: "tool" messages with tool_call_id. Most templates drop the id, so results must follow the order of the calls.
• tool_choice has these behaviors on llama-server. auto uses a lazy grammar triggered by the call marker, required constrains the output from the first token, and none disables parsing but leaves the tools in the prompt. At the pinned commit, the named-function object form is treated as auto.
• Streamed calls are joined by index. The id and name arrive in the first piece, and the arguments are parsed after the last piece.
• finish_reason: "length" means that the last call may be cut off. A client never runs a partial call.
• Grammars guarantee the shape of a call, not its correctness. A random model produces valid calls with wrong arguments.
• Models without native tool support can use a format that the prompt defines. The format can be enforced with response_format and a oneOf schema.

Summary

• The model sees tools as text inside the rendered prompt. One small tool added 155 tokens to a 35-token prompt with the Qwen 2.5 template.
• A grammar-constrained raw completion showed that a call is ordinary generated text, ending with the end-of-generation token.
• Tool definitions use JSON Schema for parameters. required and additionalProperties constrain the argument object.
• Qwen 2.5 and Hermes use <tool_call> tags. Llama 3.1 uses a bare JSON object with parameters. Mistral Nemo uses [TOOL_CALLS] with ids. Harmony uses messages addressed to functions.NAME that end in <|call|>.
• llama-server at commit 0c1e57098 derives call parsers from templates with a differential autoparser, plus specialized parsers such as gpt-oss.cpp.
• A parser should report unterminated calls, invalid JSON, missing names, and non-object arguments instead of throwing, and it should accept double-encoded arguments.
• llama-server returns tool_calls with a 32-character random id, type: "function", name, and arguments as a string, together with finish_reason: "tool_calls".
• Templates render tool results as user, tool, ipython, or [TOOL_RESULTS] turns. Only Mistral writes the id, and it rejects ids that are not 9 alphanumeric characters.
• Parallel calls are answered with one tool message per call, in call order.
• tool_choice: "none" leaves tools in the prompt, and required forces a call. The named object form falls back to auto on llama-server at this commit, and offering one tool with required has the same effect as naming it.
• Streamed call pieces share an index, and the joined arguments string is parsed once at the end.
• A validator checks finish_reason, ids, tool names, JSON validity, object type, required and unknown arguments, and argument types.
• Prompt-based tool calling defines its own JSON reply format. response_format with a oneOf schema enforces it with a grammar.