Chat Templates, Roles, and Special Tokens

When you use a chat API, you send a list of messages. Each message has a role (system, user, or assistant) and some text. The model itself does not receive a list. It receives one long sequence of token ids (numbers that stand for pieces of text). Somewhere between your list and the model, the messages must be flattened into that single sequence, and the roles must be written into it in some way.

The function that does this flattening is the chat template. It matters because the model was trained on one exact format. If the text you give it uses a different format, the model still answers, but it answers worse, and sometimes it never stops. Many "the model is dumb" problems in local setups are template problems.

In this lesson you render chats by hand in Java for the main formats. You check each renderer against the output of real templates, produced by llama-server. You also see how templates handle system prompts, conversation history, reasoning, and tools. After it, you can diagnose a wrong template and write a renderer for a new one. The lesson builds on Tokens, Vocabularies, and Special Tokens (special tokens are single vocabulary entries that act as markers) and Generation, Prefill, Decode, and the KV Cache (stop conditions, and reuse of the cache for an unchanged prompt start).

The table shows where each idea appears.

Idea	Where
Messages become one string, then tokens	Example 1
ChatML; the generation prompt and the stop token	Examples 2 and 3
Llama 3 and Mistral formats	Examples 4 and 5
System prompts and defaults	Example 6
The Jinja template inside a model file	Example 7
Multi-turn history and prefix reuse	Example 8
Reasoning blocks	Example 9
Tool schemas, tool calls, and tool results	Examples 10 and 11
The Harmony format	Example 12
A wrong template	Example 13
Templates in GGUF files and server flags	Example 14
A `ChatTemplate` class	Example 15 (Project 3)

A note on how the checks were made. The real outputs come from llama-server in the pinned build, with the tiny random-weight model as the host. The weights do not matter here, because rendering a template never runs the model. The server endpoint /apply-template takes a request body, renders the prompt, and returns it without generating anything. The template files come from models/templates/ in the llama.cpp source.

The host model supplies the special tokens that a template names through bos_token (beginning of sequence) and eos_token (end of sequence). So some runs show <|endoftext|>, the end-of-text token of the tiny model's Qwen2 vocabulary, in places where the real model would have its own token.

Example 1: a chat becomes one string

The first program does the most naive flattening possible. It writes each message as role: content on its own line and adds assistant: at the end. Look at the output as the model would see it: plain text with nothing in it that marks a message boundary except the words and the newlines.

Java

import java.util.List;

public class FlatChat {
    record Msg(String role, String content) {}

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string."));
        StringBuilder flat = new StringBuilder();
        for (Msg m : chat) flat.append(m.role()).append(": ").append(m.content()).append('\n');
        flat.append("assistant:");
        System.out.println(flat);
        System.out.println("characters: " + flat.length() + ", UTF-8 bytes: " + flat.toString().getBytes(java.nio.charset.StandardCharsets.UTF_8).length);
        System.out.println("what the model receives: one sequence of token ids, with no roles or messages in it");
    }
}

Output

system: You are a careful Java assistant.
user: Reverse a string.
assistant:
characters: 76, UTF-8 bytes: 76
what the model receives: one sequence of token ids, with no roles or messages in it

The result is 76 characters, and in UTF-8 it is also 76 bytes, because every character here is plain ASCII. The last line is the key point. The model has no message objects and no roles. It reads token ids. Any role boundary must be written into the text itself, in a form the model has seen during training.

This naive format works as text, but a model trained on another convention would not recognize it as a conversation. The conventions differ for every model family. Most of them use special tokens as markers, because a marker that is a single vocabulary entry cannot be confused with ordinary text. A user cannot produce it by typing similar characters, since the tokenizer treats the whole marker as one unit (see lesson 6).

Example 2: ChatML by hand

The program renders a four-message chat in ChatML and compares the text with the real output of the Qwen2.5 template for the same chat. The EXPECTED string at the top of the program is that real output. At the end, the program opens the assistant's turn with <|im_start|>assistant\n. Example 3 explains why.

Java

import java.util.List;

public class ChatMl {
    record Msg(String role, String content) {}

    // output of llama-server /apply-template with the Qwen2.5-7B-Instruct template (see the shell example below)
    static final String EXPECTED = """
            <|im_start|>system
            You are a careful Java assistant.<|im_end|>
            <|im_start|>user
            Reverse a string.<|im_end|>
            <|im_start|>assistant
            Use a StringBuilder.<|im_end|>
            <|im_start|>user
            Show code.<|im_end|>
            <|im_start|>assistant
            """;

    static String render(List<Msg> chat) {
        StringBuilder sb = new StringBuilder();
        for (Msg m : chat) sb.append("<|im_start|>").append(m.role()).append('\n').append(m.content()).append("<|im_end|>\n");
        return sb.append("<|im_start|>assistant\n").toString();       // open the assistant's turn (Example 3)
    }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string."),
            new Msg("assistant", "Use a StringBuilder."), new Msg("user", "Show code."));
        String prompt = render(chat);
        System.out.print(prompt);
        System.out.println("matches the real Qwen2.5 template output: " + prompt.equals(EXPECTED));
        int markers = prompt.split("<\\|im_start\\|>", -1).length - 1;
        System.out.println("messages: " + chat.size() + ", <|im_start|> markers: " + markers);
    }
}

Output

<|im_start|>system
You are a careful Java assistant.<|im_end|>
<|im_start|>user
Reverse a string.<|im_end|>
<|im_start|>assistant
Use a StringBuilder.<|im_end|>
<|im_start|>user
Show code.<|im_end|>
<|im_start|>assistant
matches the real Qwen2.5 template output: true
messages: 4, <|im_start|> markers: 5

The comparison prints true. The hand-written renderer produces the same text as the real Jinja template for this chat.

Look at the structure. Each message becomes three parts: the start marker with the role name and a newline, then the content, then the end marker with a newline. The role names system, user, and assistant are ordinary text after the start marker. The model learned in training that the word after <|im_start|> names the speaker.

The text has five start markers for four messages, because the final line opens the assistant's turn.

Notice also that the earlier assistant answer (Use a StringBuilder.) is part of the input, written like every other message. The model has no memory between requests. The client sends the whole history again each time, and that is how the model "remembers" the conversation.

The real output comes from the request below. The server was started with llama-server -m model.gguf --jinja --chat-template-file Qwen-Qwen2.5-7B-Instruct.jinja. The field prompt of the JSON reply holds the text printed above.

Terminal

curl -s http://127.0.0.1:8080/apply-template -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "system", "content": "You are a careful Java assistant."}, {"role": "user", "content": "Reverse a string."}]}'

Example 3: the generation prompt and the end-of-turn token

There are two small details that make chat work. The first is how the model knows it is its turn to talk. The second is how the engine knows the model has finished.

For the first, a template can add a generation prompt after the last message. This is the opening of the assistant's turn, here <|im_start|>assistant\n. It tells the model: "your turn begins now". The model's next token is then the first token of its answer.

For the second, the model ends its answer by writing the end-of-turn token, here <|im_end|>. The engine watches for that token and uses it as the stop condition (when it appears, the engine stops generating).

The program does two things. First it renders the same chat with and without the generation prompt and prints the last 30 characters of each. Then it simulates the engine: it takes a stream of generated pieces, appends them to the answer, and stops when it sees <|im_end|>.

Java

import java.util.List;

public class GenerationPrompt {
    record Msg(String role, String content) {}

    static String render(List<Msg> chat, boolean generationPrompt) {
        StringBuilder sb = new StringBuilder();
        for (Msg m : chat) sb.append("<|im_start|>").append(m.role()).append('\n').append(m.content()).append("<|im_end|>\n");
        if (generationPrompt) sb.append("<|im_start|>assistant\n");
        return sb.toString();
    }

    static String tail(String s) { return s.substring(s.length() - 30).replace("\n", "\\n"); }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("user", "Reverse a string."), new Msg("assistant", "Use a StringBuilder."), new Msg("user", "Show code."));
        System.out.println("with the generation prompt, ends with:    " + tail(render(chat, true)));
        System.out.println("without the generation prompt, ends with: " + tail(render(chat, false)));

        // the engine feeds the prompt, then samples pieces until the end-of-turn token appears
        String[] pieces = {"Here", " is", " the", " code", ":", "<|im_end|>", "\n", "<|im_start|>", "user"};
        StringBuilder answer = new StringBuilder();
        int generated = 0;
        for (String piece : pieces) {
            generated++;
            if (piece.equals("<|im_end|>")) { System.out.println("stop at " + piece + " after " + generated + " pieces"); break; }
            answer.append(piece);
        }
        System.out.println("answer: " + answer);
    }
}

Output

with the generation prompt, ends with:    m_end|>\n<|im_start|>assistant\n
without the generation prompt, ends with: rt|>user\nShow code.<|im_end|>\n
stop at <|im_end|> after 6 pieces
answer: Here is the code:

With the generation prompt, the text ends in the open assistant line, so the model's next token starts its answer. Without the generation prompt, the text ends with <|im_end|>\n, which is a finished turn. That form is correct for rendering history and training data, where every turn is complete. If an inference request leaves out the generation prompt, the model has to write <|im_start|>assistant\n itself first, or it may continue as another role.

The simulated stream stops at <|im_end|>, which is the sixth piece. That token is the end-of-turn marker, and the engine treats it as the stop condition (lesson 11, Example 10). The marker is not part of the answer, so the answer printed is Here is the code:.

The pieces after the marker (\n<|im_start|>user) show what would happen if nothing stopped the model. It would start a new user turn and invent the user's next message. Example 13 shows this failure in full.

Example 4: the Llama 3 format

The Llama 3.1 template has one more habit. It always writes a system header first, with a knowledge cutoff line and a date line, and it puts the user's own system message after them. The date is the day of the run, so it changes from day to day.

The program implements this format. It takes the BOS token and the date as parameters so that it can be compared with the real run. It trims each message's content, as the template does with its trim filter (which removes spaces and newlines at the start and end). The output has two parts: the text a Llama 3 model would see, and two checks.

Java

import java.util.List;

public class Llama3Format {
    record Msg(String role, String content) {}

    // output of llama-server /apply-template with the Llama 3.1 template; the host model there had another BOS token
    static final String EXPECTED = """
            <|endoftext|><|start_header_id|>system<|end_header_id|>

            Cutting Knowledge Date: December 2023
            Today Date: 03 Oct 2026

            You are a careful Java assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

            Reverse a string.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

            Use a StringBuilder.<|eot_id|><|start_header_id|>user<|end_header_id|>

            Show code.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

            """;

    static String render(List<Msg> chat, String bos, String date) {
        StringBuilder sb = new StringBuilder(bos);
        String system = chat.get(0).role().equals("system") ? chat.get(0).content().trim() : "";
        sb.append("<|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: ")
          .append(date).append("\n\n").append(system).append("<|eot_id|>");
        for (Msg m : chat) {
            if (m.role().equals("system")) continue;
            sb.append("<|start_header_id|>").append(m.role()).append("<|end_header_id|>\n\n").append(m.content().trim()).append("<|eot_id|>");
        }
        return sb.append("<|start_header_id|>assistant<|end_header_id|>\n\n").toString();
    }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string."),
            new Msg("assistant", "Use a StringBuilder."), new Msg("user", "Show code."));
        System.out.print(render(chat, "<|begin_of_text|>", "26 Jul 2024"));
        System.out.println("matches the real template output (same BOS and date as that run): " + render(chat, "<|endoftext|>", "03 Oct 2026").equals(EXPECTED));
        System.out.println("without a system message the header is still written: " + render(List.of(new Msg("user", "Hi")), "", "26 Jul 2024").startsWith("<|start_header_id|>system"));
    }
}

Output

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

You are a careful Java assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Reverse a string.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Use a StringBuilder.<|eot_id|><|start_header_id|>user<|end_header_id|>

Show code.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

matches the real template output (same BOS and date as that run): true
without a system message the header is still written: true

The first part of the output is what a Llama 3 model would see, with <|begin_of_text|> and a date of 26 Jul 2024. The system header comes first, then the system text, then each turn, and the prompt ends with the open assistant header.

The first check compares with the real template run. It uses the BOS token and the date of that run, since the host model was not a Llama model and the server fills the date line with the day of the run. It prints true.

The second check shows that the system header is written even when the chat has no system message. The real render without a system message also starts with the header, the cutoff line, and the date, followed by an empty system text.

Example 5: the Mistral format

Mistral templates are the most compact of the three. They put [INST] and [/INST] around user messages, and they write an end-of-sequence token after each assistant message. The Mistral-Nemo template has no role names for assistant messages and no generation prompt. The text ends with [/INST], and the model continues from there.

Mistral-Nemo also has an unusual rule for the system message. It has no block of its own. The template writes the system text, followed by a blank line, into the last user message.

The program implements the template for plain chats and compares it with the real output.

Java

import java.util.List;

public class MistralFormat {
    record Msg(String role, String content) {}

    // output of llama-server /apply-template with the Mistral-Nemo-Instruct-2407 template; the host model had other BOS and EOS tokens
    static final String EXPECTED = """
            <|endoftext|>[INST]Reverse a string.[/INST]Use a StringBuilder.<|endoftext|>[INST]You are a careful Java assistant.

            Show code.[/INST]""";

    static String render(List<Msg> chat, String bos, String eos) {
        String system = chat.get(0).role().equals("system") ? chat.get(0).content() : null;
        List<Msg> turns = system == null ? chat : chat.subList(1, chat.size());
        StringBuilder sb = new StringBuilder(bos);
        for (int i = 0; i < turns.size(); i++) {
            Msg m = turns.get(i);
            if (m.role().equals("user")) {
                boolean last = i == turns.size() - 1;
                sb.append("[INST]").append(last && system != null ? system + "\n\n" : "").append(m.content()).append("[/INST]");
            } else {
                sb.append(m.content()).append(eos);
            }
        }
        return sb.toString();                                      // no generation prompt: [/INST] already ends the text
    }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string."),
            new Msg("assistant", "Use a StringBuilder."), new Msg("user", "Show code."));
        System.out.println(render(chat, "<s>", "</s>"));
        System.out.println("matches the real template output (same BOS and EOS as that run): " + render(chat, "<|endoftext|>", "<|endoftext|>").equals(EXPECTED));
    }
}

Output

<s>[INST]Reverse a string.[/INST]Use a StringBuilder.</s>[INST]You are a careful Java assistant.

Show code.[/INST]
matches the real template output (same BOS and EOS as that run): true

The check prints true when the program uses the host's BOS and EOS tokens, the same as the real run. Reading the output, you can see the system text sits in front of Show code. in the final [INST] block, and the text ends with [/INST].

This design has three consequences.

The system text is in the final user turn, so it moves later in the text as the chat grows. This hurts cache reuse (Example 8).
The template raises an error when roles do not alternate between user and assistant, as its source shows with raise_exception.
There are no spaces: the real template writes [INST] directly followed by the content.

Other Mistral releases use other templates, some with a [SYSTEM_PROMPT] block (the Mistral-Small-3.2 template in the llama.cpp source has one). So the template file of the exact model is the reference.

Example 6: system prompts and default system prompts

The system message is meant to set the behavior of the assistant. Templates treat it in different ways, and some add text of their own when you send none. The table shows what the real templates render with and without a system message.

Template	Without a system message	With a system message
Qwen2.5 (ChatML)	Adds `You are Qwen, created by Alibaba Cloud. You are a helpful assistant.` as the system turn	Uses the given text
Tiny model's Qwen2-style template	Adds `You are a helpful assistant`	Uses the given text
Llama 3.1	Writes the system header with the cutoff and date and an empty text	Cutoff and date lines, then the given text
Mistral-Nemo	Writes nothing	Prepends it to the last user message
Qwen3	Writes nothing	Uses the given text
gpt-oss (Harmony)	Writes a system message (identity, knowledge cutoff, date, reasoning level, channel list)	Puts the given text in a separate `developer` message after the system message

Two lessons follow from the table.

First, a model that is given no system prompt can still see one. An agent that wants control over the behavior should set its own system prompt, so it does not depend on a default that it did not choose.

Second, the system prompt can sit in different places: at the start of the text (ChatML, Llama), in the last user message (Mistral-Nemo), or in a different role (Harmony). This changes how the model treats the text. It also decides whether the beginning of the prompt stays stable between requests, which matters for cache reuse (Example 8).

Example 7: the Jinja template from the model file, read line by line

So far the renderers were written in Java. Real engines do not have one Java class per model. Instead, each model ships its own template as data, written in Jinja, a template language. In Jinja, statements sit between {% %}, and expressions that produce output sit between {{ }}.

The template is stored in the model's files: in tokenizer_config.json for Hugging Face repositories, and in the key tokenizer.chat_template of a GGUF file. The tiny model's GGUF holds it as one line, with real line breaks inside the quoted strings. The block below splits it over lines for reading, and writes the breaks inside the strings as \n.

Text

{% for message in messages %}
  {% if loop.first and messages[0]['role'] != 'system' %}
    {{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}
  {% endif %}
  {{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{% endfor %}
{% if add_generation_prompt %}
  {{ '<|im_start|>assistant\n' }}
{% endif %}

Here is how to read it, line by line.

{% for message in messages %} loops over messages, a variable the engine fills with the chat.
Inside the loop, loop.first is true for the first message. If that message is not a system message, the template writes a default system turn.
The next expression writes the message in ChatML form, joining strings with +.
After the loop, add_generation_prompt (also set by the engine) decides whether to open the assistant turn.

Other variables that engines supply are bos_token, eos_token, tools, and extra settings passed as template keyword arguments.

The program below is the same logic in Java, with the Jinja line in a comment next to each Java line. It is compared with the real engine output, with and without a system message.

Java

import java.util.List;

public class TinyTemplate {
    record Msg(String role, String content) {}

    static final String WITH_SYSTEM = """
            <|im_start|>system
            You are a careful Java assistant.<|im_end|>
            <|im_start|>user
            Reverse a string.<|im_end|>
            <|im_start|>assistant
            Use a StringBuilder.<|im_end|>
            <|im_start|>user
            Show code.<|im_end|>
            <|im_start|>assistant
            """;
    static final String WITHOUT_SYSTEM = """
            <|im_start|>system
            You are a helpful assistant<|im_end|>
            <|im_start|>user
            Reverse a string.<|im_end|>
            <|im_start|>assistant
            """;

    // the same logic as the Jinja template stored in the tiny GGUF file
    static String render(List<Msg> messages, boolean addGenerationPrompt) {
        StringBuilder out = new StringBuilder();
        for (int i = 0; i < messages.size(); i++) {                                        // {% for message in messages %}
            Msg m = messages.get(i);
            if (i == 0 && !messages.get(0).role().equals("system"))                       // {% if loop.first and messages[0]['role'] != 'system' %}
                out.append("<|im_start|>system\nYou are a helpful assistant<|im_end|>\n");
            out.append("<|im_start|>").append(m.role()).append('\n').append(m.content()).append("<|im_end|>").append('\n');
        }
        if (addGenerationPrompt) out.append("<|im_start|>assistant\n");                    // {% if add_generation_prompt %}
        return out.toString();
    }

    public static void main(String[] args) {
        List<Msg> withSystem = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string."),
            new Msg("assistant", "Use a StringBuilder."), new Msg("user", "Show code."));
        List<Msg> withoutSystem = List.of(new Msg("user", "Reverse a string."));
        System.out.println("with a system message, same as the engine: " + render(withSystem, true).equals(WITH_SYSTEM));
        System.out.println("without one, same as the engine: " + render(withoutSystem, true).equals(WITHOUT_SYSTEM));
        System.out.print(render(withoutSystem, true));
    }
}

Output

with a system message, same as the engine: true
without one, same as the engine: true
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Reverse a string.<|im_end|>
<|im_start|>assistant

Both comparisons print true. The last lines of the output show the case with no system message: the template added You are a helpful assistant as the system turn, then the user message, then the open assistant turn.

Writing the logic by hand like this is how a template is understood. Applications do not do this for every model. They run the Jinja text with a template engine, since each model family writes its own template. llama.cpp includes a Jinja engine in C++.

Example 8: multi-turn history and the unchanged prefix

Every request re-sends the whole conversation, so every request renders the whole conversation again. This costs time, because the engine must process all the prompt tokens (the prefill step from lesson 11). The server can avoid most of that work. If the new prompt starts with exactly the same tokens as the previous one, the server can reuse the KV cache of the earlier request and process only the new tokens (lesson 11, Example 13 showed cache_n).

Whether this works depends on the template. If the template leaves earlier text unchanged when new messages are added, the old prompt is a prefix of the new prompt, and the cache is reusable. If the template rewrites earlier text, the shared prefix is short, and the cache is useless.

The program renders the first request and the second request of one chat, with ChatML and with a Mistral-Nemo style template. Then it counts how many leading characters the two prompts share.

Java

import java.util.ArrayList;
import java.util.List;

public class PrefixStability {
    record Msg(String role, String content) {}

    static String chatMl(List<Msg> chat) {
        StringBuilder sb = new StringBuilder();
        for (Msg m : chat) sb.append("<|im_start|>").append(m.role()).append('\n').append(m.content()).append("<|im_end|>\n");
        return sb.append("<|im_start|>assistant\n").toString();
    }

    // Mistral-Nemo style: the system text is written into the LAST user message
    static String mistral(List<Msg> chat) {
        String system = chat.get(0).role().equals("system") ? chat.get(0).content() : null;
        List<Msg> turns = system == null ? chat : chat.subList(1, chat.size());
        StringBuilder sb = new StringBuilder("<s>");
        for (int i = 0; i < turns.size(); i++) {
            Msg m = turns.get(i);
            if (m.role().equals("user")) sb.append("[INST]").append(i == turns.size() - 1 && system != null ? system + "\n\n" : "").append(m.content()).append("[/INST]");
            else sb.append(m.content()).append("</s>");
        }
        return sb.toString();
    }

    static int commonPrefix(String a, String b) {
        int n = 0;
        while (n < Math.min(a.length(), b.length()) && a.charAt(n) == b.charAt(n)) n++;
        return n;
    }

    public static void main(String[] args) {
        List<Msg> turn1 = new ArrayList<>(List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string.")));
        List<Msg> turn2 = new ArrayList<>(turn1);
        turn2.add(new Msg("assistant", "Use a StringBuilder."));
        turn2.add(new Msg("user", "Show code."));
        String[][] renders = {{"ChatML", chatMl(turn1), chatMl(turn2)}, {"Mistral-Nemo style", mistral(turn1), mistral(turn2)}};
        for (String[] r : renders) {
            int common = commonPrefix(r[1], r[2]);
            System.out.printf("%-19s first prompt %3d chars, shared with the second prompt: %3d chars (%.0f%%)%n", r[0], r[1].length(), common, 100.0 * common / r[1].length());
        }
    }
}

Output

ChatML              first prompt 130 chars, shared with the second prompt: 130 chars (100%)
Mistral-Nemo style  first prompt  68 chars, shared with the second prompt:   9 chars (13%)

With ChatML, the first prompt is a prefix of the second. All 130 characters are shared (100%), so the cache covers the whole first prompt.

With the Mistral-Nemo style, the system text moves from the first user message into the new last one. The two prompts share only the 9 characters of <s>[INST], which is 13% of the first prompt, and then the texts differ. The cache for everything after those 9 characters cannot be reused.

What this means for an agent that talks to an engine for many turns:

Prefer a template whose earlier text never changes.
Keep the messages you control (system prompt, tool definitions) at the start and unchanged.
Put anything that varies per request, such as a clock, at the end.

The Llama 3.1 template shows the same risk. It writes a date line near the start, and in the run above the date was the current day, so a prompt rendered on another day starts differently.

Example 9: reasoning blocks and history

Reasoning models write a chain of thought before the answer, often inside <think> tags (see lesson 14, Example 6). When the conversation continues, should the old thinking be sent again? Most templates say no. Thinking is long and costs tokens, and the model was trained on histories without it. So the template removes the thinking from earlier turns when it renders the history.

The program applies the Qwen3 rule: reasoning in assistant turns before the last user message is dropped. It uses a regular expression to remove everything between <think> and </think>, then compares the result with the real Qwen3 template.

Java

import java.util.List;
import java.util.regex.Pattern;

public class ReasoningHistory {
    record Msg(String role, String content) {}

    static final String EXPECTED = """
            <|im_start|>user
            2+2?<|im_end|>
            <|im_start|>assistant
            The answer is 4.<|im_end|>
            <|im_start|>user
            And times 3?<|im_end|>
            <|im_start|>assistant
            """;
    static final Pattern THINK = Pattern.compile("(?s)<think>.*?</think>\\s*");

    // like the Qwen3 template: reasoning of turns before the last user message is not rendered again
    static String render(List<Msg> chat) {
        int lastUser = 0;
        for (int i = 0; i < chat.size(); i++) if (chat.get(i).role().equals("user")) lastUser = i;
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < chat.size(); i++) {
            Msg m = chat.get(i);
            String content = m.role().equals("assistant") && i < lastUser ? THINK.matcher(m.content()).replaceAll("") : m.content();
            sb.append("<|im_start|>").append(m.role()).append('\n').append(content).append("<|im_end|>\n");
        }
        return sb.append("<|im_start|>assistant\n").toString();
    }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("user", "2+2?"),
            new Msg("assistant", "<think>\nAdd the numbers.\n</think>\n\nThe answer is 4."), new Msg("user", "And times 3?"));
        String history = render(chat);
        System.out.print(history);
        System.out.println("matches the real Qwen3 template output: " + history.equals(EXPECTED));
        String before = chat.get(1).content(), after = THINK.matcher(before).replaceAll("");
        System.out.println("the earlier answer shrinks from " + before.length() + " to " + after.length() + " characters in the prompt");
    }
}

Output

<|im_start|>user
2+2?<|im_end|>
<|im_start|>assistant
The answer is 4.<|im_end|>
<|im_start|>user
And times 3?<|im_end|>
<|im_start|>assistant
matches the real Qwen3 template output: true
the earlier answer shrinks from 51 to 16 characters in the prompt

The earlier answer was <think>\nAdd the numbers.\n</think>\n\nThe answer is 4., which is 51 characters. In the prompt it is only The answer is 4., which is 16 characters. The output also equals the real Qwen3 template.

Templates differ in the details. In the Qwen3 template, reasoning stays when the assistant turn comes after the last user message. This matters in an agent loop, where the assistant calls a tool and then continues the same turn. The model needs its own earlier thinking to continue the task correctly. This real output (pinned build, Qwen3 template, an assistant turn with reasoning and a tool call after the user message) shows the kept reasoning at the end of a prompt:

Output

<|im_start|>assistant
<think>
I need the file contents.
</think>

<tool_call>
{"name": "read_file", "arguments": {"path": "pom.xml"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
<project/>
</tool_response><|im_end|>
<|im_start|>assistant

The DeepSeek-R1-Distill template does it another way. It strips the reasoning of earlier turns by keeping the text after </think>, and its generation prompt ends with <think>, so the model starts by reasoning. This is the real output for the chat of the program above (the host's BOS first; the bars are the full-width characters that the model's special tokens use):

Output

<|endoftext|><｜User｜>2+2?<｜Assistant｜>

The answer is 4.<｜end▁of▁sentence｜><｜User｜>And times 3?<｜Assistant｜><think>

The Qwen3 template also accepts a keyword argument enable_thinking. With false it writes an empty <think>\n\n</think> block after the generation prompt, which tells the model to skip reasoning. The gpt-oss template states in a comment that reasoning is dropped in all previous turns, with one exception: an assistant tool-call message that has no later final answer keeps its reasoning (Example 12).

Example 10: tools in the template

A coding agent needs the model to use tools, such as reading a file. A model cannot run anything. It can only write text. So a tool call is text that the model writes in a format its template taught it, and a client program reads that text and runs the tool.

Two steps make this work. First, the template renders the tool schemas (JSON descriptions of each tool's name, purpose, and arguments) into the prompt, usually in the system turn. Second, the model answers with a call in the trained form, and the client parses it.

The program builds the Qwen2.5 prompt for one user message and one tool, read_file. Jackson (a Java JSON library) parses the schema. A small toJson method imitates the template's tojson filter, which separates items with , and follows keys with : . The result is compared with the real template output. A second part parses a reply of the form the model is expected to write.

Java

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.StringJoiner;

public class ToolsInTemplate {
    static final ObjectMapper JSON = new ObjectMapper();
    static final String EXPECTED = """
            <|im_start|>system
            You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

            # Tools

            You may call one or more functions to assist with the user query.

            You are provided with function signatures within <tools></tools> XML tags:
            <tools>
            {"type": "function", "function": {"name": "read_file", "description": "Read a file from the workspace", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}}
            </tools>

            For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
            <tool_call>
            {"name": <function-name>, "arguments": <args-json-object>}
            </tool_call><|im_end|>
            <|im_start|>user
            What is in pom.xml?<|im_end|>
            <|im_start|>assistant
            """;

    // like the tojson filter of the template engine: ", " between items and ": " after keys
    static String toJson(JsonNode n) {
        StringJoiner j = new StringJoiner(", ", n.isObject() ? "{" : "[", n.isObject() ? "}" : "]");
        if (n.isObject()) { n.properties().forEach(e -> j.add(JSON.valueToTree(e.getKey()) + ": " + toJson(e.getValue()))); return j.toString(); }
        if (n.isArray()) { n.forEach(x -> j.add(toJson(x))); return j.toString(); }
        return n.toString();
    }

    public static void main(String[] args) throws Exception {
        // the tool schema, as an OpenAI-style "tools" entry in the request
        JsonNode tool = JSON.readTree("""
            {"type": "function", "function": {"name": "read_file", "description": "Read a file from the workspace",
             "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}}""");
        String system = "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\n# Tools\n\n"
            + "You may call one or more functions to assist with the user query.\n\n"
            + "You are provided with function signatures within <tools></tools> XML tags:\n<tools>\n" + toJson(tool)
            + "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n"
            + "<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>";
        String prompt = "<|im_start|>system\n" + system + "<|im_end|>\n"
            + "<|im_start|>user\nWhat is in pom.xml?<|im_end|>\n<|im_start|>assistant\n";
        System.out.println("matches the real Qwen2.5 template output: " + prompt.equals(EXPECTED));

        // what the model is expected to write next, and how a client reads it
        String modelOutput = "I will read the build file.\n<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"pom.xml\"}}\n</tool_call>";
        int start = modelOutput.indexOf("<tool_call>") + "<tool_call>".length(), end = modelOutput.indexOf("</tool_call>");
        JsonNode parsed = JSON.readTree(modelOutput.substring(start, end));
        System.out.println("text before the call: " + modelOutput.substring(0, modelOutput.indexOf("<tool_call>")).trim());
        System.out.println("tool: " + parsed.get("name").asText() + ", argument path: " + parsed.get("arguments").get("path").asText());
    }
}

Output

matches the real Qwen2.5 template output: true
text before the call: I will read the build file.
tool: read_file, argument path: pom.xml

The prompt matches the real output. The request had no system message, so the template used its default identity text and appended the tool section to it. The section lists the tools between <tools> tags, with one JSON object per line. It also describes the expected reply: a JSON object with name and arguments between <tool_call> tags. The model was trained on prompts of exactly this shape, which is why it answers in that form.

The second part shows the client side. The program finds the text between the tags and parses it as JSON. The text before the tag (I will read the build file.) is ordinary content that a client can show to the user. The client then reads the tool name read_file and the argument path and can run the tool.

Other families use other formats.

Llama 3.1 asks for a reply of the form {"name": function name, "parameters": dictionary of argument name and its value}.
Mistral-Nemo wraps calls in [TOOL_CALLS] and schemas in [AVAILABLE_TOOLS].
Harmony uses recipients (Example 12).

The server's chat endpoint hides these differences from clients. It parses the model output into the OpenAI tool_calls field. A later lesson on function calling covers the details and the parsing of streamed calls.

Example 11: tool results as `tool` role messages

After the client runs a tool, it sends the result back to the model as a message with the role tool. The next request then contains the whole exchange: the question, the assistant's call, and the result.

The program renders such a history with the Qwen2.5 rules. The assistant made two calls in one turn (parallel calls), so two tool messages follow. The rules were read from the template file:

An assistant message with tool calls writes one <tool_call> block per call.
Consecutive tool messages share one user turn, and each result is wrapped in <tool_response> tags.

The program compares its result with the real output for the same request, and it also checks whether the call ids appear in the prompt.

Java

import java.util.List;

public class ToolResults {
    record Call(String id, String name, String argumentsJson) {}
    record Msg(String role, String content, List<Call> calls) {
        Msg(String role, String content) { this(role, content, List.of()); }
    }

    static final String EXPECTED = """
            <|im_start|>system
            You are a careful Java assistant.<|im_end|>
            <|im_start|>user
            Compare pom.xml and README.md.<|im_end|>
            <|im_start|>assistant
            <tool_call>
            {"name": "read_file", "arguments": {"path": "pom.xml"}}
            </tool_call>
            <tool_call>
            {"name": "read_file", "arguments": {"path": "README.md"}}
            </tool_call><|im_end|>
            <|im_start|>user
            <tool_response>
            <project>...</project>
            </tool_response>
            <tool_response>
            # Demo
            </tool_response><|im_end|>
            <|im_start|>assistant
            """;

    // the Qwen2.5 rules for assistant tool calls and tool results
    static String render(List<Msg> msgs) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < msgs.size(); i++) {
            Msg m = msgs.get(i);
            if (m.role().equals("tool")) {
                boolean first = i == 0 || !msgs.get(i - 1).role().equals("tool");
                boolean last = i == msgs.size() - 1 || !msgs.get(i + 1).role().equals("tool");
                if (first) sb.append("<|im_start|>user");                          // a run of tool messages is one user turn
                sb.append("\n<tool_response>\n").append(m.content()).append("\n</tool_response>");
                if (last) sb.append("<|im_end|>\n");
            } else if (!m.calls().isEmpty()) {
                sb.append("<|im_start|>assistant").append(m.content().isEmpty() ? "" : "\n" + m.content());
                for (Call c : m.calls())
                    sb.append("\n<tool_call>\n{\"name\": \"").append(c.name()).append("\", \"arguments\": ").append(c.argumentsJson()).append("}\n</tool_call>");
                sb.append("<|im_end|>\n");
            } else {
                sb.append("<|im_start|>").append(m.role()).append('\n').append(m.content()).append("<|im_end|>\n");
            }
        }
        return sb.append("<|im_start|>assistant\n").toString();
    }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Compare pom.xml and README.md."),
            new Msg("assistant", "", List.of(new Call("call_1", "read_file", "{\"path\": \"pom.xml\"}"),
                                              new Call("call_2", "read_file", "{\"path\": \"README.md\"}"))),
            new Msg("tool", "<project>...</project>"), new Msg("tool", "# Demo"));
        String prompt = render(chat);
        System.out.println("matches the real Qwen2.5 template output: " + prompt.equals(EXPECTED));
        System.out.println("call ids in the prompt: " + (prompt.contains("call_1") || prompt.contains("call_2")));
        System.out.println("user turns: " + (prompt.split("<\\|im_start\\|>user", -1).length - 1) + " (the question and both results)");
    }
}

Output

matches the real Qwen2.5 template output: true
call ids in the prompt: false
user turns: 2 (the question and both results)

The render matches the real output. The two calls appear in one assistant turn, and the two results share one user turn. The API role tool does not exist in this template's text, so the results are written as user input with <tool_response> tags around each. The model tells results apart from user messages by those tags. That is why the program counts 2 user turns: the question, and the block that holds both results.

The call ids (call_1, call_2) do not appear in the prompt. The OpenAI-style API pairs a result with its call through tool_call_id. This template drops the ids, so the model can only match results to calls by their order. A client that sends results in a different order from the calls gives the model wrong information, and nothing reports an error. Other templates, such as Mistral's, do write ids into the text.

The real output came from this request to the server of Example 2:

Terminal

curl -s http://127.0.0.1:8080/apply-template -H "Content-Type: application/json" -d '{"messages": [
  {"role": "system", "content": "You are a careful Java assistant."},
  {"role": "user", "content": "Compare pom.xml and README.md."},
  {"role": "assistant", "content": "", "tool_calls": [
    {"id": "call_1", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\": \"pom.xml\"}"}},
    {"id": "call_2", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\": \"README.md\"}"}}]},
  {"role": "tool", "tool_call_id": "call_1", "content": "<project>...</project>"},
  {"role": "tool", "tool_call_id": "call_2", "content": "# Demo"}]}'

Example 12: the Harmony format

The gpt-oss models use a format called Harmony. It is more structured than the earlier formats, because it separates the different kinds of text the model writes into channels.

analysis: the reasoning.
commentary: tool calls and notes.
final: the answer for the user.

The block below is the real render for a conversation with the read_file tool of Example 10, the question What is in pom.xml?, one call, and its result. The date line is the day of the run.

Output

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2026-10-03

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.
Calls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>developer<|message|># Tools

<h2 id="functions">functions</h2>

namespace functions {

// Read a file from the workspace
type read_file = (_: {
path: string,
}) => any;

} // namespace functions<|end|><|start|>user<|message|>What is in pom.xml?<|end|><|start|>assistant to=functions.read_file<|channel|>commentary json<|message|>{"path": "pom.xml"}<|call|><|start|>functions.read_file to=assistant<|channel|>commentary<|message|>"<project>...</project>"<|end|><|start|>assistant

Read the render from the top. The system message gives the identity, the knowledge cutoff, the date, a reasoning level, and the valid channels. Tool definitions go in a developer message, written as a TypeScript-like namespace. The history shows the tool call as an assistant message addressed to=functions.read_file on the commentary channel. The result is a message from functions.read_file to the assistant. The prompt ends with <|start|>assistant, and the model picks the channel itself.

Now the client side. The program parses generated text in the forms accepted by the llama.cpp parser for these models (the words inside are made up). For each message it extracts the channel, the recipient, the text, and the end marker.

Java

import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class Harmony {
    record Part(String channel, String recipient, String text, String ended) {}

    static final Pattern PART = Pattern.compile("(?s)(?:<\\|start\\|>assistant)?(?: to=([\\w.]+))?<\\|channel\\|>(\\w+)(?: to=([\\w.]+))?(?: ?(?:<\\|constrain\\|>)?\\w+)?<\\|message\\|>(.*?)(<\\|end\\|>|<\\|call\\|>|<\\|return\\|>|$)");

    static List<Part> parse(String generated) {
        List<Part> parts = new ArrayList<>();
        Matcher m = PART.matcher(generated);
        while (m.find()) {
            String recipient = m.group(1) != null ? m.group(1) : m.group(3) != null ? m.group(3) : "-";
            parts.add(new Part(m.group(2), recipient, m.group(4), m.group(5).isEmpty() ? "(open)" : m.group(5)));
        }
        return parts;
    }

    public static void main(String[] args) {
        // text in the forms that the llama.cpp parser for gpt-oss accepts; the model's own words are made up
        String turn1 = "<|channel|>analysis<|message|>The user wants the file, so I should call the tool.<|end|>"
            + "<|start|>assistant<|channel|>commentary to=functions.read_file <|constrain|>json<|message|>{\"path\": \"pom.xml\"}<|call|>";
        String turn2 = "<|channel|>analysis<|message|>The file lists one dependency.<|end|><|start|>assistant<|channel|>final<|message|>It declares one dependency.<|return|>";
        for (String turn : new String[] {turn1, turn2}) {
            for (Part p : parse(turn)) System.out.printf("channel %-10s recipient %-20s ended by %-10s text: %s%n", p.channel(), p.recipient(), p.ended(), p.text());
            System.out.println();
        }
    }
}

Output

channel analysis   recipient -                    ended by <|end|>    text: The user wants the file, so I should call the tool.
channel commentary recipient functions.read_file  ended by <|call|>   text: {"path": "pom.xml"}

channel analysis   recipient -                    ended by <|end|>    text: The file lists one dependency.
channel final      recipient -                    ended by <|return|> text: It declares one dependency.

A client sees the reasoning, the calls, and the answer as separate parts. An engine can stop on <|call|> (the model wants a tool run) or on <|return|> (the model is done).

Example 13: a wrong template, and a model that never stops

A template error does not throw an exception. The wrong text is still valid text, so the model answers anyway, only worse. The worst case involves the end token.

Suppose a model was trained on ChatML, so it ends its turn with <|im_end|>. Now suppose the engine was configured for Llama 3 and watches for <|eot_id|>. The engine never sees the token it expects, so it never stops the model. The program simulates the token stream of such a model with two engines: one that watches the right token, and one that watches the wrong token and has a limit of 12 tokens.

Java

import java.util.List;

public class WrongTemplate {
    // a model trained with ChatML ends its turn with <|im_end|>; this engine was configured for Llama 3 and watches for <|eot_id|>
    static String run(List<String> modelStream, String stopToken, int maxTokens) {
        StringBuilder out = new StringBuilder();
        int count = 0;
        for (String piece : modelStream) {
            if (piece.equals(stopToken)) return "stopped at " + stopToken + " after " + count + " tokens: " + out.toString().replace("\n", "\\n");
            out.append(piece);
            if (++count >= maxTokens) return "hit the limit of " + maxTokens + " tokens: " + out.toString().replace("\n", "\\n");
        }
        return "stream ended: " + out;
    }

    public static void main(String[] args) {
        List<String> stream = List.of("Use", " a", " StringBuilder", ".", "<|im_end|>", "\n", "<|im_start|>", "user", "\n", "Thanks", "<|im_end|>", "\n",
            "<|im_start|>", "assistant", "\n", "You", "'re", " welcome", ".", "<|im_end|>");
        System.out.println("right stop token:  " + run(stream, "<|im_end|>", 40));
        System.out.println("wrong stop token:  " + run(stream, "<|eot_id|>", 12));
        System.out.println("the model keeps going because the token it was trained to end with is not the one the engine checks");
    }
}

Output

right stop token:  stopped at <|im_end|> after 4 tokens: Use a StringBuilder.
wrong stop token:  hit the limit of 12 tokens: Use a StringBuilder.<|im_end|>\n<|im_start|>user\nThanks<|im_end|>\n
the model keeps going because the token it was trained to end with is not the one the engine checks

The engine that watches <|im_end|> stops after 4 tokens, with the answer Use a StringBuilder..

In practice, the symptoms of this kind of error are:

answers that continue with made-up user turns,
role names appearing in the output,
replies that run to the maximum length.

llama.cpp keeps a set of end-of-generation tokens for each vocabulary (llama_vocab_is_eog), so a correct model file ends at its own tokens. The failure appears with a wrong --chat-template, hand-set stop strings, or a client that builds the prompt itself in another format.

Other causes of poor answers are a missing system prompt and a doubled beginning-of-sequence token. The doubled token happens when the template adds bos_token and the tokenizer adds another one.

Example 14: the template inside a GGUF file, and the server flags

A GGUF file (the model file format used by llama.cpp) carries the template in the key tokenizer.chat_template. llama-server reads it when it loads the model. So in the normal case, you do not have to supply a template yourself.

The first command lists the key for the tiny model, with gguf-dump from the gguf-py package. The output after it is representative, and the tool cuts the long value.

Terminal

gguf-dump tiny-random-qwen2vocab-f32.gguf | grep chat_template

Representative output (gguf-py 0.19.0; the value is cut by the tool):

Output

     24: STRING     |        1 | tokenizer.chat_template = '{% for message in messages %}{% if loop.first and message...'

The second command asks the server what it loaded. The server's /props endpoint reports the template, the BOS and EOS tokens, and a set of capabilities that the engine found by testing the template.

Terminal

curl -s http://127.0.0.1:8080/props

Representative output (fields selected; the template text is cut):

Output

"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}...",
"chat_template_caps": {"supports_system_role": true, "supports_tools": false, "supports_tool_calls": false, "supports_parallel_tool_calls": false, "supports_string_content": true, "supports_typed_content": false, "supports_object_arguments": false, "supports_preserve_reasoning": false, "supports_reasoning_effort": false},
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>"

The capabilities of the tiny model's template show that it cannot render tools: supports_tools is false. A server started with this model cannot offer tool calling through its template. This is why the later lesson on function calling starts with templates that support it.

When a model file has a missing or outdated template, you can override it. The relevant flags are in the pinned --help:

--jinja and --no-jinja: use the Jinja engine for chat (the default is enabled).
--chat-template NAME or --chat-template JINJA_TEMPLATE: replace the template from the file. A built-in name such as chatml, llama3, mistral-v7, or gpt-oss selects one of the templates compiled into llama.cpp. A full template string needs --jinja set first.
--chat-template-file FILE: read the template from a file, as in Example 2.
--chat-template-kwargs '{"enable_thinking": false}': pass extra variables to the template.

The request field chat_template_kwargs does the same per request. The /apply-template endpoint used throughout this lesson is the quickest way to check what a template produces.

Example 15: Project 3, a sealed ChatTemplate

The project collects the three renderers behind one Java type, so that the rest of a program does not care which format is in use.

ChatTemplate is a sealed interface (an interface with a fixed list of allowed implementations) with three records: ChatMl, Llama3, and Mistral. Each has a render method that takes a list of messages and a flag for the generation prompt, and each has a stopToken method. A pattern-matching switch over the sealed type names each variant without a default branch. So if a fourth record is added and the switch does not handle it, the code does not compile.

The program renders one conversation with all three templates. For each it prints the format name, the stop token, the prompt length, and the prompt with \n written out so that the line structure is visible.

Java

import java.util.List;

public class TemplateProject {
    record Msg(String role, String content) {}

    sealed interface ChatTemplate permits ChatMl, Llama3, Mistral {
        String render(List<Msg> chat, boolean generationPrompt);
        String stopToken();
    }

    record ChatMl() implements ChatTemplate {
        public String render(List<Msg> chat, boolean generationPrompt) {
            StringBuilder sb = new StringBuilder();
            for (Msg m : chat) sb.append("<|im_start|>").append(m.role()).append('\n').append(m.content()).append("<|im_end|>\n");
            return generationPrompt ? sb.append("<|im_start|>assistant\n").toString() : sb.toString();
        }
        public String stopToken() { return "<|im_end|>"; }
    }

    record Llama3(String date) implements ChatTemplate {
        public String render(List<Msg> chat, boolean generationPrompt) {
            String system = chat.get(0).role().equals("system") ? chat.get(0).content().trim() : "";
            StringBuilder sb = new StringBuilder("<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n"
                + "Cutting Knowledge Date: December 2023\nToday Date: " + date + "\n\n" + system + "<|eot_id|>");
            for (Msg m : chat)
                if (!m.role().equals("system"))
                    sb.append("<|start_header_id|>").append(m.role()).append("<|end_header_id|>\n\n").append(m.content().trim()).append("<|eot_id|>");
            return generationPrompt ? sb.append("<|start_header_id|>assistant<|end_header_id|>\n\n").toString() : sb.toString();
        }
        public String stopToken() { return "<|eot_id|>"; }
    }

    record Mistral() implements ChatTemplate {
        public String render(List<Msg> chat, boolean generationPrompt) {   // no generation prompt: the text ends with [/INST]
            String system = chat.get(0).role().equals("system") ? chat.get(0).content() : null;
            List<Msg> turns = system == null ? chat : chat.subList(1, chat.size());
            StringBuilder sb = new StringBuilder("<s>");
            for (int i = 0; i < turns.size(); i++) {
                Msg m = turns.get(i);
                if (m.role().equals("user")) sb.append("[INST]").append(i == turns.size() - 1 && system != null ? system + "\n\n" : "").append(m.content()).append("[/INST]");
                else sb.append(m.content()).append("</s>");
            }
            return sb.toString();
        }
        public String stopToken() { return "</s>"; }
    }

    static String describe(ChatTemplate t) {
        return switch (t) {
            case ChatMl c -> "ChatML (roles in <|im_start|> lines)";
            case Llama3 l -> "Llama 3 (roles in header tokens)";
            case Mistral m -> "Mistral ([INST] blocks)";
        };
    }

    public static void main(String[] args) {
        List<Msg> chat = List.of(new Msg("system", "You are a careful Java assistant."), new Msg("user", "Reverse a string."),
            new Msg("assistant", "Use a StringBuilder."), new Msg("user", "Show code."));
        for (ChatTemplate t : List.of(new ChatMl(), new Llama3("26 Jul 2024"), new Mistral())) {
            String prompt = t.render(chat, true);
            System.out.println("== " + describe(t) + ", stop token " + t.stopToken() + ", " + prompt.length() + " characters");
            System.out.println(prompt.replace("\n", "\\n\n"));
        }
    }
}

Output

== ChatML (roles in <|im_start|> lines), stop token <|im_end|>, 221 characters
<|im_start|>system\n
You are a careful Java assistant.<|im_end|>\n
<|im_start|>user\n
Reverse a string.<|im_end|>\n
<|im_start|>assistant\n
Use a StringBuilder.<|im_end|>\n
<|im_start|>user\n
Show code.<|im_end|>\n
<|im_start|>assistant\n

== Llama 3 (roles in header tokens), stop token <|eot_id|>, 422 characters
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n
\n
Cutting Knowledge Date: December 2023\n
Today Date: 26 Jul 2024\n
\n
You are a careful Java assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n
\n
Reverse a string.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n
\n
Use a StringBuilder.<|eot_id|><|start_header_id|>user<|end_header_id|>\n
\n
Show code.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n
\n

== Mistral ([INST] blocks), stop token </s>, 115 characters
<s>[INST]Reverse a string.[/INST]Use a StringBuilder.</s>[INST]You are a careful Java assistant.\n
\n
Show code.[/INST]

The three prompts show the same four messages in three forms. ChatML takes 221 characters. Llama 3 takes 422, because of the header tokens and the date lines. Mistral takes 115, because it has no role names. Each variant also has its own stop token: <|im_end|>, <|eot_id|>, and </s>.

An engine written around this type picks a variant from the model's metadata, uses render to build the prompt, and stops generation at stopToken. The chat engine built in pure Java later in this course uses the same structure.

Interview points

A chat template turns a list of role-tagged messages into the token sequence the model was trained on. The model sees no roles, only text and special tokens.
ChatML wraps messages in <|im_start|> and <|im_end|>. Llama 3 uses header tokens and <|eot_id|>. Mistral uses [INST] blocks and an end-of-sequence token.
The generation prompt opens the assistant turn. It is added for inference and left out when rendering history or training data.
The end-of-turn token is the stop condition. A wrong template or a wrong stop token makes generation run on.
Templates are Jinja programs in tokenizer_config.json or in the GGUF key tokenizer.chat_template. Engines supply messages, add_generation_prompt, bos_token, eos_token, and tools.
Some templates add a default system prompt, some always write a system header (Llama 3.1), and Mistral-Nemo moves the system text into the last user message.
Prefix stability decides KV cache reuse across requests. Put stable text first, and avoid templates or content that change earlier text.
Reasoning of earlier turns is usually dropped from history, except inside the current tool-calling turn in some templates.
Tool schemas go into the system turn or a developer message, and each family has its own call format: <tool_call> JSON, [TOOL_CALLS], or to=functions.name in Harmony.
Harmony has analysis, commentary, and final channels, and ends with <|call|> or <|return|>.
llama-server --jinja --chat-template-file replaces a template, and /apply-template shows the rendered prompt without running the model.
A doubled BOS token, a missing system prompt, or an out-of-date template reduce quality without causing an error.

Summary

The model reads one token sequence. Roles exist only as text and special tokens that the template writes.
ChatML: <|im_start|>role\ncontent<|im_end|>\n, with the generation prompt <|im_start|>assistant\n.
Llama 3.1: BOS, a system header with the cutoff and date, then <|start_header_id|>role<|end_header_id|>\n\ncontent<|eot_id|>.
Mistral-Nemo: [INST]content[/INST] per user message, assistant text followed by the EOS token, and the system text in the last user message.
The renderers match the real templates on the test chats, checked through /apply-template of the pinned llama-server.
Qwen2.5 adds a default system prompt, the tiny model's template adds You are a helpful assistant, Llama 3.1 always writes the system header, and Harmony always writes a system message.
Multi-turn requests re-render all messages. ChatML keeps the earlier prompt as an exact prefix. A template that rewrites earlier text prevents cache reuse.
Qwen3 and DeepSeek-R1-Distill templates drop earlier reasoning from history. Qwen3 keeps it inside the current tool turn and supports enable_thinking.
Qwen templates render tools as JSON in <tools> tags, ask for <tool_call> JSON, and render tool results as a user turn with <tool_response>. Consecutive tool results share one turn. The Qwen2.5 template drops the call ids, so results are matched to calls by order.
Harmony messages use <|start|>, <|channel|>, and <|message|>, and end with <|end|>, <|call|>, or <|return|>.
A wrong end token lets generation run on with invented turns until the token limit.
GGUF stores the template in tokenizer.chat_template. /props shows the loaded template and its capabilities, and --chat-template, --chat-template-file, and --chat-template-kwargs override it.
A sealed ChatTemplate type with one record per format makes the supported formats explicit.

Example 1: a chat becomes one string

Example 2: ChatML by hand

Example 3: the generation prompt and the end-of-turn token

Example 4: the Llama 3 format

Example 5: the Mistral format

Example 6: system prompts and default system prompts

Example 7: the Jinja template from the model file, read line by line

Example 8: multi-turn history and the unchanged prefix

Example 9: reasoning blocks and history

Example 10: tools in the template

Example 11: tool results as tool role messages

Example 12: the Harmony format

Example 13: a wrong template, and a model that never stops

Example 14: the template inside a GGUF file, and the server flags

Example 15: Project 3, a sealed ChatTemplate

Interview points

Summary

Example 11: tool results as `tool` role messages