codeloopback
Guide contents

Lesson 36 of 73

llama-server: Endpoints, Slots, and Prompt Cache

Until now, you ran llama.cpp as a command that reads a prompt, prints an answer, and exits. A coding agent needs something different: a model that stays loaded in memory and answers many requests from other programs. Loading a model takes seconds, so you want to do it once and keep it ready.

llama-server does this. It is a small web server that ships with llama.cpp. It loads one GGUF model, then waits for HTTP requests. A Java program (such as the agent you build later in this course) sends a request with the conversation, and the server sends back the model's answer. The server speaks the same request format as the OpenAI API, so a client written for OpenAI works with it, and it adds its own extra endpoints for things OpenAI does not offer.

This lesson sends real requests to the server and explains every field that comes back. After it, you can start the server with the right flags for a client program, read its responses and logs, and predict when a request reuses earlier work instead of recomputing it.

Endpoint or feature What it is for
GET /health, --api-key Readiness check and authentication
GET /v1/models, GET /props The loaded model and the server's settings
POST /v1/chat/completions Chat requests, with or without streaming
POST /completion Native raw-prompt endpoint with detailed timings
POST /tokenize, /detokenize, /apply-template Token ids and the rendered chat prompt
GET /slots, id_slot, -np Parallel request slots
cache_prompt, --cache-ram, --cache-reuse, slot files Reusing KV cache across requests
grammar, json_schema, response_format Constrained output
tools, tool_choice, reasoning_format Tool calls and reasoning fields
GET /metrics, POST /v1/embeddings Prometheus metrics and embeddings

Every response in this lesson comes from the llama-server binary built from llama.cpp commit 0c1e57098 (version string b1-0c1e570), running the course's tiny random-weight model (Llama architecture, 4 layers, 256 dimensions, Qwen2 tokenizer). Because the weights are random, the generated text is meaningless. The structure of every response, the field names, the token counts, the cache counters, and the log lines are real. Timings and ids differ on every run and every machine. Facts about flags and defaults come from llama-server --help and the server source in tools/server/.

Lesson Running llama.cpp: Core Flags covers the general flags (-c, -np, -t, sampling flags). This lesson uses them and adds the server-only ones.

Some words appear in almost every example:

  • HTTP is the protocol web browsers use. A client sends a request to an address and gets a response. An endpoint is one address on the server, such as /health. The server's address is a host (127.0.0.1 means this same computer) and a port (a number, here 8080).
  • JSON is a text format for structured data, such as {"name": "tiny-coder"}. Requests and responses in this lesson are JSON.
  • curl is a command-line tool that sends HTTP requests. -s hides its progress output, -d gives the request body, and -H adds a header.
  • A status code is a number in each response: 200 means success, 400 means the request was wrong, 401 means not allowed, 404 means not found, 500 means a server fault, 503 means temporarily unavailable.
  • A slot is one place where the server processes a request. A server with two slots can work on two requests at the same time.
  • Prefill is the step where the model reads the prompt. Decode is the step where it writes the answer one token at a time. The KV cache (key-value cache) holds the model's notes about the tokens it has already read, so it does not read them again.

Example 1: starting the server and reading its startup log

Problem: a client program needs to know the server's address, how many requests it can run at once, and when it is ready. Idea: start the server with explicit flags for each of these, then read the startup log, which states the final values the server chose.

The question is which flags a client program needs on the server and what the server prints when it is ready. The command binds to the loopback address, names the model tiny-coder for the API, creates two slots, enables metrics, and allows slot files in ./slots.

Terminal
mkdir -p slots
llama-server -m tiny-random-qwen2vocab-f32.gguf \
  --host 127.0.0.1 --port 8080 \
  -a tiny-coder \
  -np 2 -c 4096 \
  --metrics --slot-save-path ./slots

The log below is long, so the rest of this example picks out the lines that matter. Real output (from the pinned build; the first column is time since start and differs by machine):

Output
0.00.001.364 I srv  llama_server: initializing ...
0.00.003.120 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.003.284 W srv  llama_server: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
0.00.005.133 I srv    load_model: loading model 'tiny-random-qwen2vocab-f32.gguf'
0.00.600.425 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.704.654 I cmn          init: llama threadpool init, n_threads = 4
0.00.715.746 I srv    load_model: initializing, n_slots = 2, n_ctx_slot = 2048, kv_unified = 'false'
0.00.718.889 I srv  llama_server: model loaded
0.00.718.905 I srv  llama_server: listening on http://127.0.0.1:8080
0.00.718.912 W srv  llama_server: notice: server default port will be changed to :9931 in a future release (ref: https://github.com/ggml-org/llama.cpp/pull/26508)

Each line starts with the elapsed time, a level letter (I for information, W for warning), and a component (srv for the server, cmn for the shared common library, load for the model loader). The lines that matter for a client are these:

  • security: no API key is set and CORS allows all origins. Without --api-key, any program that can reach the port can use the server. CORS (cross-origin resource sharing) is the browser rule that decides which web pages may call the server; the default --cors-origins value is *, which means every page.
  • n_slots = 2, n_ctx_slot = 2048, kv_unified = 'false'. A slot is one request being processed at a time. -np 2 asks for two. With an explicit -np, the KV cache is not unified, so the 4,096-token context from -c 4096 is divided into two private 2,048-token halves. When -np is left at its default of -1 (auto), the server source sets 4 slots and a unified KV cache shared by all of them.
  • listening on http://127.0.0.1:8080. The server accepts connections from now on. --host defaults to 127.0.0.1 and --port to 8080, so this command repeats the defaults to make them visible.
  • The last warning appears only when the port is 8080. It states that a future release changes the default port, which is a reason for client programs to pass --port explicitly instead of relying on the default.

The control-looking token warning comes from the vocabulary file the tiny model reuses, not from the server. --slot-save-path must name an existing directory: when ./slots does not exist, the server exits at once with error while handling argument "--slot-save-path": not a directory: ./slots.

Example 2: /health states and API keys

Problem: starting the server takes a moment while the model loads. A client that sends a request too early gets an error. Idea: the server has a /health endpoint that says whether it is ready, and a client polls it (asks repeatedly) until it answers 200. An API key is a secret string that a client must send to prove it is allowed to use the server.

The question is how a client knows that the model is loaded, and what changes when the server requires a key. The first command polls /health in a loop from the moment the process starts; the next commands use a server started with --api-key secret-123.

Terminal
while true; do curl -s -w ' %{http_code}\n' http://127.0.0.1:8080/health; done
curl -s http://127.0.0.1:8080/v1/models
curl -s http://127.0.0.1:8080/v1/models -H 'Authorization: Bearer nope'
curl -s http://127.0.0.1:8080/v1/models -H 'Authorization: Bearer secret-123'
curl -s http://127.0.0.1:8080/health

Real output (the 503 line was printed 69 times before the first 200, then the loop was stopped). Read the lines in order: first the server is loading (503), then it is ready (200). The next two lines are /v1/models without a key and with a wrong key, which are both rejected with 401. The fifth line uses the right key and gets the model list, and the last line shows that /health works without a key:

Output
{"error":{"message":"Loading model","type":"unavailable_error","code":503}} 503
{"status":"ok"} 200
{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}
{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}
{"models":[{"name":"tiny-coder","model":"tiny-coder", ...
{"status":"ok"}

The HTTP listener starts before the model finishes loading. During loading, /health answers 503 Service Unavailable with an error object whose message is Loading model. Once the model is ready, it answers 200 with {"status":"ok"}. A program that starts the server as a child process polls /health until it sees 200; a later lesson on the loop serve workshop builds that loop in Java.

With --api-key, every endpoint except /health and /v1/health returns 401 and the error type authentication_error unless the request carries the key. The key can be sent as Authorization: Bearer <key> or, as the source in tools/server/server-http.cpp shows, as an X-Api-Key header. The source strips the Bearer prefix and compares the rest with each configured key; --api-key accepts a comma-separated list, and --api-key-file reads one key per line. /health stays public so that process supervisors can check readiness without a secret.

Example 3: /v1/models and /props

Problem: a client must put the model's name in every request and must not send a prompt longer than the context allows. Idea: two endpoints answer these questions, one for the model and one for the server's settings.

The question is how a client learns the model name to send, the context size, and the server defaults. GET /v1/models lists the model and GET /props returns the server's properties.

Terminal
curl -s http://127.0.0.1:8080/v1/models

Real output (formatted):

Output
{
    "models": [
        {
            "name": "tiny-coder", "model": "tiny-coder", "modified_at": "", "size": "", "digest": "",
            "type": "model", "description": "", "tags": [""], "capabilities": ["completion"], "parameters": "",
            "details": {"parent_model": "", "format": "gguf", "family": "", "families": [""],
                        "parameter_size": "", "quantization_level": ""}
        }
    ],
    "object": "list",
    "data": [
        {
            "id": "tiny-coder",
            "aliases": ["tiny-coder"],
            "tags": [],
            "object": "model",
            "created": 1791032112,
            "owned_by": "llamacpp",
            "meta": {
                "vocab_type": 2, "n_vocab": 151936, "n_ctx": 2048, "n_ctx_train": 4096,
                "n_embd": 256, "n_params": 80611584, "size": 322446336, "ftype": "all F32"
            }
        }
    ]
}

The response holds two lists. data is the OpenAI shape: object: "list" and one entry per model with id, object: "model", created, and owned_by. Its id is the alias from -a; without -a it is the model file path. The models list uses the field names of Ollama's model list (name, digest, details with format and quantization_level, the same names as ListModelResponse and ModelDetails in Ollama's api/types.go), so tools written for Ollama can read it.

meta is llama.cpp's own addition and has the numbers a client needs:

Field Value Meaning
n_ctx 2048 Context available to one request (one slot's share)
n_ctx_train 4096 Context length stored in the GGUF
n_vocab 151936 Vocabulary size (Qwen2 tokenizer)
n_embd 256 Hidden size
n_params 80611584 Parameter count, mostly the two 151,936 x 256 embedding matrices
size 322446336 Bytes of tensor data

A client that checks a prompt's length before sending it compares against n_ctx, not n_ctx_train, because the slot has only 2,048 tokens.

Terminal
curl -s http://127.0.0.1:8080/props

Real output (formatted and trimmed; the sampler parameters and the full template are left out):

Output
{
 "default_generation_settings": {
  "params": {
   "seed": 4294967295, "temperature": 0.800000011920929, "top_k": 40,
   "top_p": 0.949999988079071, "min_p": 0.05000000074505806,
   "repeat_penalty": 1.0, "n_predict": -1, "stream": false,
   "chat_format": "Content-only", "reasoning_format": "none",
   "samplers": ["penalties", "dry", "top_n_sigma", "top_k", "typ_p", "top_p", "min_p", "xtc", "temperature"]
  },
  "n_ctx": 2048
 },
 "total_slots": 2,
 "model_alias": "tiny-coder",
 "modalities": {"vision": false, "video": false, "audio": false},
 "endpoint_slots": true, "endpoint_props": false, "endpoint_metrics": true,
 "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}...",
 "chat_template_caps": {
  "supports_parallel_tool_calls": false, "supports_system_role": true,
  "supports_tool_calls": false, "supports_tools": false, ...
 },
 "bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>",
 "build_info": "b1-0c1e570",
 "is_sleeping": false
}

default_generation_settings.params lists the sampling defaults that apply when a request omits a field: temperature 0.8, top-k 40, top-p 0.95, min-p 0.05, and the sampler order from Penalties, Logit Bias, and Sampler Chains. The odd decimals such as 0.800000011920929 are the 32-bit floats the server stores, printed as 64-bit doubles. The seed 4294967295 is 0xFFFFFFFF, the value llama.cpp uses for "pick a random seed".

chat_template is the Jinja template from the GGUF metadata, and chat_template_caps is what the server detected about it. This vocabulary file carries a plain ChatML template, so supports_tools is false; Example 13 loads a template that supports tools. endpoint_props: false means POST /props is disabled until the server starts with --props. build_info identifies the exact build, which belongs in every bug report and benchmark result.

Example 4: /v1/chat/completions without streaming

Problem: a chat model works with a conversation, which is a list of messages. Each message has a role: system (standing instructions), user (the person), or assistant (the model). The /v1/chat/completions endpoint takes this list and returns the model's next message. In the request below, max_tokens caps the answer length, temperature 0 means "always pick the most likely token" (so the result is repeatable), and seed fixes the random choices.

The question is what a complete chat response contains. The request has a system message and a user message, caps the answer at 8 tokens, and uses temperature 0 so the server picks the most likely token each time.

Terminal
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "tiny-coder",
  "messages": [
    {"role": "system", "content": "You are a coding assistant."},
    {"role": "user", "content": "Write hello world in Java."}
  ],
  "max_tokens": 8, "temperature": 0, "seed": 42
}'

Real output (formatted):

Output
{
    "choices": [
        {
            "finish_reason": "length",
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "\u05ea\u05d7\u05e8ContentAlignmentContentAlignmentContentAlignmentContentAlignmentContentAlignmentContentAlignmentContentAlignment"
            }
        }
    ],
    "created": 1791032120,
    "model": "tiny-coder",
    "system_fingerprint": "b1-0c1e570",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 8,
        "prompt_tokens": 25,
        "total_tokens": 33,
        "prompt_tokens_details": {"cached_tokens": 0}
    },
    "id": "chatcmpl-v8pXBvqeSgK7ciB8dTvN4RxrS4egYISr",
    "timings": {
        "cache_n": 0, "prompt_n": 25, "prompt_ms": 12.4, "prompt_per_token_ms": 0.496,
        "prompt_per_second": 2016.1290322580644,
        "predicted_n": 8, "predicted_ms": 63.761, "predicted_per_token_ms": 9.108714285714287,
        "predicted_per_second": 109.78497827825785
    }
}

The content is one Hebrew-script token followed by ContentAlignment seven times. With random weights and greedy selection, the model falls into a loop of one token, which is the repetition problem that penalties address. The \u05ea escapes are how the server's JSON library writes non-ASCII characters; a JSON parser turns them back into text.

The fields follow the OpenAI chat completion shape:

  • choices has one entry per generated answer. index is its position. message has the role and the content.
  • finish_reason is length because generation stopped at max_tokens (8). A model that emits its end-of-turn token produces stop, and a tool call produces tool_calls (Example 13).
  • usage counts tokens: 25 prompt tokens, 8 completion tokens, 33 in total. prompt_tokens_details.cached_tokens is the number of prompt tokens served from the KV cache, 0 on this first request.
  • id starts with chatcmpl- followed by 32 random characters; created is a Unix timestamp in seconds. system_fingerprint is the build string.
  • timings is a llama.cpp addition. cache_n (0) plus prompt_n (25) equals prompt_tokens; predicted_n equals completion_tokens. prompt_ms is the time to process the prompt (prefill) and predicted_ms the time to generate (decode). Prefill ran at about 2,016 tokens per second and decode at about 110, the gap that Generation, Prefill, Decode, and the KV Cache explains: prefill handles all 25 tokens in one batch while decode handles one token per step.

The prompt is 25 tokens because the server applies the chat template before tokenizing. Example 7 shows the rendered prompt.

Example 5: streaming chunks

Problem: a long answer can take many seconds. If the client waits for the full answer, the user sees nothing for that time. Idea: with "stream": true, the server sends each token as soon as it is produced, so a chat window can show the text as it grows. The format used is server-sent events (SSE): the response stays open and the server writes one event after another.

The question is what arrives on the wire when stream is true. The request asks for 4 tokens; the second request adds stream_options.include_usage.

Terminal
curl -s -N http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "tiny-coder", "messages": [{"role": "user", "content": "Say hi."}],
  "max_tokens": 4, "temperature": 0, "stream": true
}'

Real output (the -N flag turns off curl's buffering):

Output
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],"created":1791032124,"id":"chatcmpl-EUFvYdRRUXr7E3EXLMXfh9eIZpLzWZAs","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"🇷"}}],"created":1791032124,"id":"chatcmpl-EUFvYdRRUXr7E3EXLMXfh9eIZpLzWZAs","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"🇷"}}],"created":1791032124,"id":"chatcmpl-EUFvYdRRUXr7E3EXLMXfh9eIZpLzWZAs","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"-risk"}}],"created":1791032124,"id":"chatcmpl-EUFvYdRRUXr7E3EXLMXfh9eIZpLzWZAs","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"(\"(%"}}],"created":1791032124,"id":"chatcmpl-EUFvYdRRUXr7E3EXLMXfh9eIZpLzWZAs","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":"length","index":0,"delta":{}}],"created":1791032124,"id":"chatcmpl-EUFvYdRRUXr7E3EXLMXfh9eIZpLzWZAs","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk","timings":{"cache_n":6,"prompt_n":15,"prompt_ms":10.74,"prompt_per_token_ms":0.716,"prompt_per_second":1396.6480446927374,"predicted_n":4,"predicted_ms":26.159,"predicted_per_token_ms":8.719666666666667,"predicted_per_second":114.68328300011468}}

data: [DONE]

The response uses server-sent events (SSE): the body stays open, and each event is a line that starts with data: followed by one JSON object and a blank line. The response headers include Content-Type: text/event-stream. The stream has a fixed shape:

  1. The first chunk carries delta.role: "assistant" and no content. It arrives before the first token, so a client can show that the request was accepted.
  2. Each following chunk carries one token's text in delta.content. Every chunk repeats the same id, created, and model, and has object: "chat.completion.chunk" instead of chat.completion.
  3. The last chunk with choices has an empty delta and the finish_reason (length). llama-server attaches timings to it.
  4. The line data: [DONE] ends the stream. It is not JSON, so a client checks for it before parsing.

Each of the two emoji chunks is one regional-indicator symbol (the letter R from the alphabet that flag emoji are built from), a single Unicode character of 4 UTF-8 bytes that the Qwen2 vocabulary stores as one token. The timings show cache_n: 6 even on this new conversation. The template inserts a default system message (Example 7 shows it), so the rendered prompt starts with <|im_start|>system\nYou are a, and those 6 tokens matched the start of Example 4's prompt, which the slot still held.

With "stream_options": {"include_usage": true}, the stream ends differently (real output, middle chunks left out):

Output
data: {"choices":[{"finish_reason":"length","index":0,"delta":{}}],"created":1791032124,"id":"chatcmpl-GlZFInhbPqXB5KsXxl4VhqI9hItTOVDD","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk"}

data: {"choices":[],"created":1791032124,"id":"chatcmpl-GlZFInhbPqXB5KsXxl4VhqI9hItTOVDD","model":"tiny-coder","system_fingerprint":"b1-0c1e570","object":"chat.completion.chunk","usage":{"completion_tokens":3,"prompt_tokens":21,"total_tokens":24,"prompt_tokens_details":{"cached_tokens":20}},"timings":{"cache_n":20,"prompt_n":1,"prompt_ms":7.805,"prompt_per_token_ms":7.805,"prompt_per_second":128.12299807815504,"predicted_n":3,"predicted_ms":14.001,"predicted_per_token_ms":7.0005,"predicted_per_second":142.8469395043211}}

data: [DONE]

An extra chunk with an empty choices array carries usage (and here the timings). A streaming client that wants token counts must request include_usage and must accept a chunk without choices. This second request repeated the same conversation, so 20 of its 21 prompt tokens came from the cache and only 1 was processed. Example 9 explains why one token is always processed.

When the server has nothing to send for a long time (a long prompt being processed), it writes the SSE comment : followed by a blank line every --sse-ping-interval seconds (default 30). Clients that parse SSE ignore lines that start with :.

Example 6: the native /completion endpoint and its timings

Problem: the chat endpoint wraps your messages in the model's chat template before the model sees them, so you cannot control the exact text. Idea: the native endpoint takes a raw string and gives you the exact control, plus more detail in the response. Use it for code completion or experiments, and use the chat endpoint for conversations.

The question is what the non-OpenAI endpoint adds. POST /completion takes a raw prompt string (no chat template) and llama.cpp's own field names such as n_predict.

Terminal
curl -s http://127.0.0.1:8080/completion -H 'Content-Type: application/json' \
  -d '{"prompt": "public static void main", "n_predict": 4, "temperature": 0}'

Real output (formatted; the 50 fields of generation_settings are trimmed to a few):

Output
{
    "index": 0,
    "content": "\u6642\u9593\u6642\u9593\u6642\u9593\u5e7f\u6cdb",
    "tokens": [],
    "id_slot": 0,
    "stop": true,
    "model": "tiny-coder",
    "tokens_predicted": 4,
    "tokens_evaluated": 4,
    "generation_settings": {
        "temperature": 0.0, "top_k": 40, "max_tokens": 4, "n_predict": 4,
        "grammar": "", "chat_format": "Content-only", ...
    },
    "prompt": "public static void main",
    "has_new_line": false,
    "truncated": false,
    "stop_type": "limit",
    "stopping_word": "",
    "tokens_cached": 7,
    "timings": {
        "cache_n": 0, "prompt_n": 4, "prompt_ms": 10.907, "prompt_per_token_ms": 2.72675,
        "prompt_per_second": 366.7369579169341,
        "predicted_n": 4, "predicted_ms": 21.38, "predicted_per_token_ms": 7.126666666666666,
        "predicted_per_second": 140.31805425631433
    }
}

The native response says more about the request than the OpenAI one:

  • id_slot is the slot that ran the request.
  • tokens_evaluated (4) is the prompt length in tokens. The prompt has no <|endoftext|> in front because the Qwen2 tokenizer metadata sets add_bos_token to false.
  • stop_type is limit (stopped by n_predict). The other values are eos (end-of-generation token), word (a string from the request's stop list, reported in stopping_word), and none (still running, seen only in streaming chunks).
  • truncated is true when prompt plus generated tokens exceeded the slot's context.
  • generation_settings echoes every sampling parameter after the server applied defaults; it is the place to check what a request really used.
  • tokens_cached (7) is the number of tokens now held in the slot's KV cache.

The program below reads these numbers from the response and derives the values a client usually wants: the prompt length, the context in use, and the speeds. Watch the difference between two decode speeds in the output, which the text after the output explains.

Java
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Locale;

public class CompletionTimings {
    static final String RESPONSE = """
        {"id_slot":0,"stop":true,"tokens_predicted":4,"tokens_evaluated":4,"stop_type":"limit",
         "tokens_cached":7,
         "timings":{"cache_n":0,"prompt_n":4,"prompt_ms":10.907,"prompt_per_token_ms":2.72675,
                    "prompt_per_second":366.7369579169341,"predicted_n":4,"predicted_ms":21.38,
                    "predicted_per_token_ms":7.126666666666666,"predicted_per_second":140.31805425631433}}
        """;

    public static void main(String[] args) throws Exception {
        JsonNode r = new ObjectMapper().readTree(RESPONSE);
        JsonNode t = r.get("timings");
        int cacheN = t.get("cache_n").asInt(), promptN = t.get("prompt_n").asInt(), predictedN = t.get("predicted_n").asInt();
        double promptMs = t.get("prompt_ms").asDouble(), predictedMs = t.get("predicted_ms").asDouble();

        System.out.println("prompt tokens      = cache_n + prompt_n = " + cacheN + " + " + promptN + " = " + (cacheN + promptN));
        System.out.println("context in use     = prompt + predicted = " + (cacheN + promptN + predictedN));
        System.out.println("tokens in KV cache = " + r.get("tokens_cached").asInt()
                + " (the last sampled token is never decoded)");
        System.out.println(String.format(Locale.ROOT, "prefill speed      = %d / %.3f ms = %.1f tok/s",
                promptN, promptMs, promptN * 1000.0 / promptMs));
        System.out.println(String.format(Locale.ROOT, "naive decode speed = %d / %.3f ms = %.1f tok/s",
                predictedN, predictedMs, predictedN * 1000.0 / predictedMs));
        int steps = predictedN - 1; // the first token comes from the prefill logits
        System.out.println(String.format(Locale.ROOT, "decode steps speed = %d / %.3f ms = %.1f tok/s",
                steps, predictedMs, steps * 1000.0 / predictedMs));
        System.out.println(String.format(Locale.ROOT, "server time        = %.3f ms", promptMs + predictedMs));
    }
}
Output
prompt tokens      = cache_n + prompt_n = 0 + 4 = 4
context in use     = prompt + predicted = 8
tokens in KV cache = 7 (the last sampled token is never decoded)
prefill speed      = 4 / 10.907 ms = 366.7 tok/s
naive decode speed = 4 / 21.380 ms = 187.1 tok/s
decode steps speed = 3 / 21.380 ms = 140.3 tok/s
server time        = 32.287 ms

The prefill speed matches the server's prompt_per_second (366.7). The naive decode speed does not: 4 tokens divided by 21.38 ms gives 187.1, while the server reports 140.3. The server's predicted_per_token_ms (7.127 ms) is 21.38 divided by 3, not by 4. The source in server-common.h explains why: n_gen_steps() returns the number of generated tokens minus one, with the comment that the first token is free because it comes from the logits of the last prompt batch. Only 3 decode steps ran during predicted_ms, and the program's second rate, 3 steps in 21.38 ms, reproduces the server's 140.3. A client that measures speed from the outside must make the same choice: count decode steps, not tokens, or the first token makes short answers look faster than they are.

The KV cache holds 7 tokens, not 8. Each generated token is sampled from the logits of the previous step; the fourth token was sampled but never fed back through the model, so it has no key and value vectors yet. The README describes the total context in use as prompt_n + cache_n + predicted_n, which counts that last token as well.

Example 7: /tokenize, /detokenize, and /apply-template

Problem: a client cannot count how much of the context a prompt will use unless it knows how the text splits into tokens, and it cannot debug a bad answer unless it sees the exact string the model received. Idea: the server exposes its tokenizer and its chat template as endpoints that do no generation, so they answer instantly. A tokenizer turns text into numbered pieces called tokens (Tokens, Vocabularies, and Special Tokens explains how).

The question is how a client counts tokens and sees the exact prompt the model receives. These three endpoints run no inference.

Terminal
curl -s http://127.0.0.1:8080/tokenize -d '{"content": "public static void main"}'
curl -s http://127.0.0.1:8080/tokenize -d '{"content": "public static void main", "with_pieces": true}'
curl -s http://127.0.0.1:8080/tokenize -d '{"content": "<|im_start|>user", "with_pieces": true}'
curl -s http://127.0.0.1:8080/tokenize -d '{"content": "<|im_start|>user", "with_pieces": true, "parse_special": false}'
curl -s http://127.0.0.1:8080/detokenize -d '{"tokens": [888, 1099, 737, 1887]}'
curl -s http://127.0.0.1:8080/apply-template -d '{"messages": [
  {"role": "system", "content": "You are a coding assistant."},
  {"role": "user", "content": "Write hello world in Java."}]}'
curl -s http://127.0.0.1:8080/apply-template -d '{"messages": [{"role": "user", "content": "Say hi."}]}'

Real output:

Output
{"tokens":[888,1099,737,1887]}
{"tokens":[{"id":888,"piece":"public"},{"id":1099,"piece":" static"},{"id":737,"piece":" void"},{"id":1887,"piece":" main"}]}
{"tokens":[{"id":151644,"piece":"<|im_start|>"},{"id":872,"piece":"user"}]}
{"tokens":[{"id":27,"piece":"<"},{"id":91,"piece":"|"},{"id":318,"piece":"im"},{"id":4906,"piece":"_start"},{"id":91,"piece":"|"},{"id":29,"piece":">"},{"id":872,"piece":"user"}]}
{"content":"public static void main"}
{"prompt":"<|im_start|>system\nYou are a coding assistant.<|im_end|>\n<|im_start|>user\nWrite hello world in Java.<|im_end|>\n<|im_start|>assistant\n"}
{"prompt":"<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nSay hi.<|im_end|>\n<|im_start|>assistant\n"}

/tokenize returns the ids that /completion would evaluate: the four ids match tokens_evaluated: 4 in Example 6. With with_pieces, each id comes with its text; the leading spaces belong to the pieces (" static"), as in Tokens, Vocabularies, and Special Tokens. A piece that is not valid UTF-8 on its own is returned as a list of byte values instead of a string.

The third and fourth lines show parse_special. With the default true, the text <|im_start|> becomes the single special token 151644. With false, the same characters are split into 6 ordinary tokens. A server that takes user text should tokenize it with special-token parsing off; otherwise a user can type a control token into the conversation. The default for add_special is false, so /tokenize does not add a BOS token.

/detokenize is the inverse and returns the text. /apply-template renders the messages with the model's chat template, exactly as /v1/chat/completions does before tokenizing, and ends with the generation prompt <|im_start|>assistant\n. The second render shows a behavior of this template: when the conversation has no system message, the template inserts You are a helpful assistant. That inserted text is the reason a one-line user message became 21 prompt tokens in Example 5.

Example 8: slots, id_slot, and concurrent requests

Problem: if one request takes 20 seconds, a second user should not wait 20 seconds before the server even starts on theirs. Idea: think of a shop with two checkout lanes. Each lane (slot) serves one customer (request) at a time, and a third customer waits in line. The analogy fits the slots and the waiting line. It stops at the speed: both lanes share one model, so each request goes slower when both are busy.

The question is how the server runs two requests at once and how a client sees it. Two long requests start together (1,500 tokens each with ignore_eos, so they cannot stop early); the first asks for slot 0 with id_slot. One second later, /slots shows both slots busy.

Terminal
curl -s http://127.0.0.1:8080/completion -d '{"prompt": "first request", "n_predict": 1500, "ignore_eos": true, "temperature": 0, "id_slot": 0}' &
curl -s http://127.0.0.1:8080/completion -d '{"prompt": "second request here", "n_predict": 1500, "ignore_eos": true, "temperature": 0}' &
sleep 1
curl -s http://127.0.0.1:8080/slots
curl -s -w ' %{http_code}\n' 'http://127.0.0.1:8080/slots?fail_on_no_slot=1'

Real output (the params object of each slot is left out):

Output
[{"id": 0, "n_ctx": 2048, "speculative": false, "is_processing": true, "id_task": 24, "n_prompt_tokens": 153, "n_prompt_tokens_processed": 2, "n_prompt_tokens_cache": 0, "next_token": [{"has_next_token": true, "has_new_line": false, "n_remain": 1349, "n_decoded": 151}]},
 {"id": 1, "n_ctx": 2048, "speculative": false, "is_processing": true, "id_task": 26, "n_prompt_tokens": 153, "n_prompt_tokens_processed": 3, "n_prompt_tokens_cache": 0, "next_token": [{"has_next_token": true, "has_new_line": true, "n_remain": 1350, "n_decoded": 150}]}]
{"error":{"code":503,"message":"no slot available","type":"unavailable_error"}} 503

Both slots have is_processing: true. Slot 0 runs the first request (2 prompt tokens, n_prompt_tokens_processed: 2), and slot 1 runs the second (3 prompt tokens). n_decoded (151 and 150) is the number of tokens generated so far, and n_remain is how many are left of the 1,500. n_prompt_tokens (153) is the number of tokens now in the slot's cache: 2 + 151 for slot 0, 3 + 150 for slot 1. The source fills this field from the slot's cached token list, so it grows during generation.

The two counts advance together because both slots share one llama_decode call per step: llama-server's continuous batching puts the next token of every busy slot into the same batch. When both finished, each response reported tokens_predicted: 1500 and its own id_slot (0 and 1).

?fail_on_no_slot=1 turns the /slots call into a capacity check: it answers 503 with no slot available when every slot is busy. A request sent to /v1/chat/completions while all slots are busy is not rejected; it waits in the server's task queue. In a separate run with two 600-token requests in flight, a third chat request made /metrics (Example 15) report llamacpp:requests_processing 2 and llamacpp:requests_deferred 1, and the third request finished with status 200 once a slot was free.

The server assigns a slot in this order, from get_available_slot in server-context.cpp: the slot named by id_slot if the request has one; otherwise the idle slot whose cached tokens share the longest common prefix with the new prompt, if that prefix is more than 10% of the prompt (--slot-prompt-similarity, default 0.10); otherwise the least recently used idle slot. The middle rule keeps a conversation on the slot that already holds its history.

Example 9: prompt cache within a slot and in RAM

Problem: an agent sends the same long beginning (instructions, tool definitions, project files) with every request and changes only the end. Reading the same 10,000 tokens again each time wastes seconds. Idea: the KV cache already holds the model's notes about those tokens. If the new prompt starts with the same tokens as the cached ones, the server keeps those notes and processes only the new part. The shared beginning is called the common prefix. Think of a bookmark: you resume reading at the bookmark instead of from page one.

The question is when a request reuses KV entries from an earlier one. Five requests go to slot 0 on a fresh server. The first two share a 28-token instruction prefix; the third repeats the second with cache_prompt: false; the fourth is unrelated; the fifth repeats the first.

Terminal
PRE="You are a careful Java reviewer. Read the code, list each bug with its line number, and suggest a fix. Keep answers short."
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: int x = 1/0;\", \"n_predict\": 2, \"id_slot\": 0}"
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: String s = null; s.length();\", \"n_predict\": 2, \"id_slot\": 0}"
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: String s = null; s.length();\", \"n_predict\": 2, \"id_slot\": 0, \"cache_prompt\": false}"
curl -s http://127.0.0.1:8080/completion -d '{"prompt": "Unrelated text about cooking pasta.", "n_predict": 2, "id_slot": 0}'
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: int x = 1/0;\", \"n_predict\": 2, \"id_slot\": 0}"

Real output (fields extracted from each response):

Output
slot 0 tokens_evaluated 38 cache_n 0 prompt_n 38
slot 0 tokens_evaluated 38 cache_n 30 prompt_n 8
slot 0 tokens_evaluated 38 cache_n 0 prompt_n 38
slot 0 tokens_evaluated 7 cache_n 0 prompt_n 7
slot 0 tokens_evaluated 38 cache_n 30 prompt_n 8

Line by line:

  1. The slot is empty, so all 38 prompt tokens are processed.
  2. The slot holds the first request's tokens. The server computes the longest common prefix of the cached tokens and the new prompt: the 28 instruction tokens plus Question and : make 30. Those 30 KV entries stay; the server removes the rest from the cache and processes only the 8 new tokens.
  3. With cache_prompt: false, the server discards the slot's cache and processes all 38 tokens again. The field defaults to true; the README adds that reusing the cache can change results slightly because logits computed in a large batch and in a small one are not guaranteed to be identical.
  4. The unrelated prompt shares nothing with the slot, so all 7 tokens are processed and the slot's old content is replaced.
  5. The slot now holds the pasta prompt, yet 30 tokens are reused. They come from the RAM prompt cache.

The RAM prompt cache is a second level behind the slots. When a request is about to replace most of a slot's content (the source's rule: less than half of the old tokens would be kept), the server copies the slot's KV state into host memory, then searches that store for the saved state with the best match for the new prompt and loads it into the slot. Request 4 saved the state of request 3 (instruction plus the String question), and request 5 loaded it back and matched 30 tokens. At log level 4 (-lv 4) the server prints the setting at startup (real lines from the same build):

Output
0.00.762.591 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.00.762.591 I srv    load_model: use `--cache-ram 0` to disable the prompt cache

--cache-ram N sets the limit in MiB (default 8192, -1 for no limit, 0 to disable). With --cache-ram 0, the same requests 3 to 5 give cache_n 0 for request 5, because the evicted state is gone.

A matched prefix never covers the whole prompt. When a new prompt is identical to the cached tokens, the server keeps all but the last token and processes that one again, because generation needs the logits of the last prompt token and the cache stores keys and values, not logits. The log says so directly: need to evaluate at least 1 token for each active slot (n_past = 40, task.n_tokens() = 40) followed by n_past was set to 39. This is why Example 5's repeated conversation showed cache_n: 20, prompt_n: 1.

For a client the consequence is a layout rule: put the parts of the prompt that never change (system prompt, tool definitions, project files) first and the parts that change last. Any change near the beginning invalidates everything after it.

Example 10: --cache-reuse and KV shifting

Problem: the prefix rule fails when the change is at the start. If a client removes the first line of a file from the prompt, every later token moves one position to the left, and the first token already differs. Idea: the server can still reuse the notes for the later tokens if it adjusts their recorded positions. This is called KV shifting.

The question is what happens when a chunk is removed from the front of a prompt. A plain prefix match fails at the first differing token, but --cache-reuse N can move matching chunks of at least N tokens to their new positions. The test sends a prompt that starts with a file header line, then the same prompt without that line, on two servers started with --cache-ram 0 (so the RAM cache does not help).

Terminal
# server A: llama-server ... -np 1 --cache-ram 0
# server B: llama-server ... -np 1 --cache-ram 0 --cache-reuse 8
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"File: Main.java\n$PRE Question: int x = 1/0;\", \"n_predict\": 2}"
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: int x = 1/0;\", \"n_predict\": 2}"

Real output, server A then server B:

Output
evaluated 43 cache_n 0 prompt_n 43
evaluated 38 cache_n 0 prompt_n 38

evaluated 43 cache_n 0 prompt_n 43
evaluated 38 cache_n 37 prompt_n 1

The header File: Main.java\n is 5 tokens (File, :, Main, .java, \n), so the first prompt has 43 tokens and the second 38. On server A the common prefix is 0 (File against You), and the whole second prompt is processed again.

On server B, the reuse loop in server-context.cpp walks the cached tokens. It skips cached tokens until it finds a run of at least 8 tokens that matches the new prompt at the current position, then removes the skipped tokens from the KV cache and shifts the matching run to its new positions with seq_add. Shifting a cached key means rotating it by the RoPE angle difference, which The Transformer Block and the Full Model introduced; llama.cpp applies that correction inside the cache. Here it found 37 matching tokens (38 minus the final one that is always processed) and reused all of them.

The loop has two limits visible in the source. It advances only through the cached tokens, so it handles text removed from the prompt, not text inserted. And it compares token ids, so the removed chunk must end on a token boundary that leaves the following tokens unchanged; with Note. instead of a full line, the next word tokenizes as You with a leading space instead of You, no run matches, and reuse finds nothing. The feature is aimed at code completion clients that drop the oldest lines of a file from the front of the prompt. The reused entries differ from a fresh computation: the keys and values of the kept tokens were computed while the removed text was still in their context, so the result is an approximation of reprocessing the prompt. The default is 0 (off). A request can set its own threshold with n_cache_reuse.

Example 11: saving and restoring a slot to a file

Problem: the KV cache lives in memory and disappears when the server stops. If your system prompt is large, every restart pays for reading it again. Idea: write the slot's cache to a file and load it later. It works like saving a game.

The question is how to keep a processed prompt across server restarts. With --slot-save-path, POST /slots/{id}?action=save writes a slot's KV cache to a file, action=restore reads it back, and action=erase empties the slot.

Terminal
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE\", \"n_predict\": 0, \"id_slot\": 0}"
curl -s -X POST 'http://127.0.0.1:8080/slots/0?action=save' -d '{"filename": "review-prefix2.bin"}'
curl -s -X POST 'http://127.0.0.1:8080/slots/0?action=erase'
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: int x = 1/0;\", \"n_predict\": 2, \"id_slot\": 0}"
curl -s -X POST 'http://127.0.0.1:8080/slots/0?action=erase'
curl -s -X POST 'http://127.0.0.1:8080/slots/0?action=restore' -d '{"filename": "review-prefix2.bin"}'
curl -s http://127.0.0.1:8080/completion -d "{\"prompt\": \"$PRE Question: int x = 1/0;\", \"n_predict\": 2, \"id_slot\": 0}"
curl -s -X POST 'http://127.0.0.1:8080/slots/0?action=restore' -d '{"filename": "../etc/passwd"}'

Real output (the /completion lines show extracted fields):

Output
{"id_slot":0,"filename":"review-prefix2.bin","n_saved":28,"n_written":29264,"timings":{"save_ms":0.12}}
{"id_slot":0,"n_erased":28}
slot 0 tokens_evaluated 38 cache_n 0 prompt_n 38
{"id_slot":0,"n_erased":39}
{"id_slot":0,"filename":"review-prefix2.bin","n_restored":28,"n_read":29264,"timings":{"restore_ms":0.147}}
slot 0 tokens_evaluated 38 cache_n 28 prompt_n 10
{"error":{"code":400,"message":"Invalid filename","type":"invalid_request_error"}}

The first request uses n_predict: 0 to process the 28-token instruction without generating. Saving writes 28 tokens in 29,264 bytes. After erase, the next request finds nothing to reuse. After restore, the same request reuses 28 tokens and processes only the 10 new ones. The server rejects file names that contain path separators or .., so a client cannot write or read outside the save directory.

The file size follows from the model shape. The program below computes the KV bytes per token of the tiny model and a Llama 3.1 8B shape (from Memory Math), then splits the two observed file sizes (1,184 bytes for a 1-token slot saved earlier in the same session, and 29,264 bytes for 28 tokens) into a fixed part and a per-token part.

Java
import java.util.Locale;

public class SlotFileSize {
    static long kvBytesPerToken(int layers, int kvHeads, int headDim, int bytesPerValue) {
        return 2L * layers * kvHeads * headDim * bytesPerValue; // K and V
    }

    public static void main(String[] args) {
        long tiny = kvBytesPerToken(4, 2, 32, 2);    // f16 cache, the server default
        long llama8b = kvBytesPerToken(32, 8, 128, 2);
        System.out.println("tiny model KV per token  = " + tiny + " bytes");
        System.out.println("8B shape KV per token    = " + llama8b + " bytes");

        long size1 = 1184, size28 = 29264;
        long perToken = (size28 - size1) / (28 - 1);
        long fixed = size1 - perToken;
        System.out.println("observed bytes per token = " + perToken
                + " = " + tiny + " KV + 4 token id + " + (perToken - tiny - 4) + " cell metadata");
        System.out.println("observed fixed bytes     = " + fixed + " (12 header + per-layer metadata)");
        System.out.println("check 28 tokens          = " + (fixed + 28 * perToken));
        System.out.println(String.format(Locale.ROOT, "8B shape, 4096-token prefix file = about %.0f MiB",
                4096.0 * llama8b / (1024 * 1024)));
    }
}
Output
tiny model KV per token  = 1024 bytes
8B shape KV per token    = 131072 bytes
observed bytes per token = 1040 = 1024 KV + 4 token id + 12 cell metadata
observed fixed bytes     = 144 (12 header + per-layer metadata)
check 28 tokens          = 29264
8B shape, 4096-token prefix file = about 512 MiB

The tiny model stores 2 vectors (key and value) of 64 values (2 KV heads times head dimension 32) in each of 4 layers, at 2 bytes per value: 1,024 bytes per token. The two observed files fit exactly 1,040 bytes per token plus 144 fixed bytes. The source of state_seq_save_file in src/llama-context.cpp explains the parts: a 12-byte header (magic number, version, token count), then the token ids at 4 bytes each, then the KV data with per-cell metadata. The same arithmetic for the 8B shape gives 128 KiB per token, so a 4,096-token prefix file is about 512 MiB. Restoring such a file from a fast disk is quicker than recomputing a long prompt on a CPU, which is the use case: a large fixed system prompt or document processed once and loaded at every start.

Example 12: grammars and JSON schemas

Problem: an agent needs the model's answer in a form a program can parse, such as JSON. A model may write extra words or break the syntax. Idea: at each step, the model gives a score to every token in its vocabulary. A grammar lists which tokens are allowed next, and the server removes all others before choosing, so the output must follow the grammar. A GBNF grammar is llama.cpp's notation for such rules, written as name ::= pieces. A JSON Schema is a standard description of a JSON shape (which keys, which types), and the server converts it into a grammar.

The question is how to get valid JSON from a model that writes random tokens. llama-server accepts a GBNF grammar in grammar, a JSON Schema in json_schema (native endpoint), or response_format (OpenAI endpoint). The server converts a schema to a grammar, and the sampler removes every token that the grammar does not allow at the current position.

Terminal
curl -s http://127.0.0.1:8080/completion -d '{"prompt": "Answer:", "n_predict": 20, "temperature": 0,
  "grammar": "root ::= (\"yes\" | \"no\") \"\\n\""}'
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{"role": "user", "content": "Give a file name and line count."}],
  "temperature": 0, "max_tokens": 400,
  "response_format": {"type": "json_schema", "json_schema": {"name": "file_info", "schema": {
    "type": "object",
    "properties": {"file": {"type": "string", "maxLength": 12},
                   "lines": {"type": "integer", "minimum": 0, "maximum": 99999}},
    "required": ["file", "lines"], "additionalProperties": false}}}}'

Real output (the content and finish fields extracted):

Output
'yes\n' stop_type=eos tokens_predicted=4
'{\n\n \t\t \t\t \t\t \t\t \t\t \t\t  "file":\n\n"For(res쎅쎅쎅쎅쎅",\n\n \t\t \t\t\t\t\t    \t \t\t\t  "lines":\n\n \t\t\t \t\t\t \t\t\t \t\t\t \t\t\t33333 }' finish_reason=stop
parsed: {'file': 'For(res쎅쎅쎅쎅쎅', 'lines': 33333}

The yes/no grammar allows only two strings followed by a newline. The random model produced yes\n, and when the grammar was complete, the only token left was the end-of-generation token, so stop_type is eos after 4 tokens even though n_predict was 20.

The schema request produced text that parses as JSON with the two required keys, a file string of at most 12 characters (the 5 Hangul characters count as characters, not bytes), and an integer between 0 and 99,999. The whitespace looks odd because the grammar allows it: the server's grammar for this schema (read from generation_settings.grammar of the same schema sent to /completion) contains these rules, trimmed:

Output
file ::= "\"" char{0,12} "\""
file-kv ::= "\"file\"" space ":" space file
lines-kv ::= "\"lines\"" space ":" space lines
root ::= "{" space file-kv "," space lines-kv space "}"
space ::= | " " | "\n"{1,2} [ \t]{0,20}

space permits up to two newlines and 20 spaces or tabs at every gap, and the random model often chose whitespace tokens. A trained model writes compact JSON because its training data does. A grammar guarantees the syntax and the schema's limits; it does not make the values sensible, as "For(res" shows. {"type": "json_object"} without a schema allows any JSON object. The algorithm behind grammars and its cost are the subject of a later lesson on constrained decoding.

Example 13: tool calls with --jinja

Problem: a coding agent needs the model to ask for actions, such as "read this file". Idea: the client describes the available tools (a name, a description, and the arguments each takes) in the request. The model answers with a tool call, which says "run this tool with these arguments". The client runs it and sends the result back as a new message. The model never runs anything itself.

The question is what a tool call looks like in a response. Tool calls need a chat template that describes tools, so this server runs with --jinja --chat-template-file models/templates/Qwen-Qwen2.5-7B-Instruct.jinja from the llama.cpp repository (--jinja is on by default at this commit). /props now reports supports_tools: true and supports_tool_calls: true.

The server parses the model's output for the template's tool-call markers. For the Qwen 2.5 template, the marker is <tool_call> followed by a JSON object with name and arguments. With tool_choice: "required", the server adds a grammar that forces a tool call, but the grammar still allows free text before the first <tool_call> marker, and a random model never writes that marker on its own. The request therefore prefills the start of the assistant turn with <tool_call>\n (a final assistant message is continued, not answered, by default), restricts path to an enum, turns off parallel calls, and adds a logit bias of +100 to the </ token (id 522) and the <|im_end|> token (id 151645) so that the model closes the call as soon as the grammar allows it. A trained model needs none of these extras.

Terminal
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{
  "model": "tiny-coder",
  "messages": [{"role": "user", "content": "Show me pom.xml"},
               {"role": "assistant", "content": "<tool_call>\n"}],
  "tools": [{"type": "function", "function": {"name": "read_file",
             "description": "Read a file from the workspace",
             "parameters": {"type": "object",
                            "properties": {"path": {"type": "string", "enum": ["pom.xml", "src/Main.java"]}},
                            "required": ["path"]}}}],
  "tool_choice": "required", "parallel_tool_calls": false,
  "logit_bias": {"151645": 100, "522": 100},
  "temperature": 0, "max_tokens": 200
}'

Real output (formatted, timings left out):

Output
{
    "choices": [
        {
            "finish_reason": "tool_calls",
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "type": "function",
                        "function": {
                            "name": "read_file",
                            "arguments": "{\n  \t\t\t  \t\t\t  \t\t\t  \t\t\t\"path\":\"src/Main.java\" }"
                        },
                        "id": "M3mfl7vHzOC3GP1WmM9kVxUEbFGmQv23"
                    }
                ]
            }
        }
    ],
    "usage": {"completion_tokens": 61, "prompt_tokens": 193, "total_tokens": 254, ...}
}

The response has finish_reason: "tool_calls" and a tool_calls array instead of text. Each call has an id (32 random characters here), type: "function", the function name, and arguments as a JSON-encoded string, not an object. The client parses arguments itself, runs the tool, and sends the result back. The random model chose src/Main.java instead of the requested file: the grammar guarantees a call to a declared tool with a valid argument, not the right one.

The streamed version of the same request (15 data: lines in total) sends the call in pieces:

Output
data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"id":"5ztHnIb8odTda96BAIowHvag0ZjNWH7b","type":"function","function":{"name":"read_file","arguments":"{"}}]}}], ...}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"path"}}]}}], ...}

data: {"choices":[{"finish_reason":"tool_calls","index":0,"delta":{}}], ...}

The first tool-call chunk carries index, id, type, name, and the first part of arguments; later chunks carry only index and more arguments text. The client joins the pieces by index, which The Chat Completions Protocol and Streaming in Java implements.

To send the tool result back, the client appends the assistant message with its tool_calls and a message with role: "tool". /apply-template shows how this template renders that turn (real output):

Output
<|im_start|>assistant
<tool_call>
{"name": "read_file", "arguments": {"path": "pom.xml"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
<project>...</project>
</tool_response><|im_end|>
<|im_start|>assistant

The template turns the tool message into a user turn wrapped in <tool_response> tags. Chat Templates, Roles, and Special Tokens showed this format; the server does the rendering and the parsing so that the client only handles JSON.

Example 14: reasoning output and --reasoning-format

Problem: a reasoning model writes its thinking before its answer. If the client receives both mixed together, it cannot show the answer cleanly. Idea: the server can split the text at the </think> marker and return the two parts in two fields.

The question is where a reasoning model's thinking goes. This server uses the DeepSeek R1 distill template from models/templates/, whose generation prompt ends with <think>\n, so everything the model writes until </think> is reasoning.

Terminal
# llama-server ... --jinja --chat-template-file models/templates/deepseek-ai-DeepSeek-R1-Distill-Qwen-32B.jinja
curl -s http://127.0.0.1:8080/apply-template -d '{"messages": [{"role": "user", "content": "Is 7 prime?"}]}'
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "Is 7 prime?"}], "temperature": 0, "max_tokens": 12}'
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"messages": [{"role": "user", "content": "Is 7 prime?"}], "temperature": 0, "max_tokens": 12, "reasoning_format": "none"}'

Real output (the choices[0] object of each chat response):

Output
{"prompt":"<|endoftext|><｜User｜>Is 7 prime?<｜Assistant｜><think>\n"}
{"finish_reason": "length", "index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": "\n creek creek creek creek creek\u2862\u2862\u043b\u0438\u0437\u6a21\u7279\u043b\u0438\u0437\u6a21\u7279\u6a21\u7279"}}
{"finish_reason": "length", "index": 0, "message": {"role": "assistant", "content": "<think>\n creek creek creek creek creek\u2862\u2862\u043b\u0438\u0437\u6a21\u7279\u043b\u0438\u0437\u6a21\u7279\u6a21\u7279"}}

With the default format, the server separates the text into reasoning_content (between <think> and </think>) and content (after </think>). The model never wrote </think> in 12 tokens, so all of its output is reasoning and content is empty. With reasoning_format: "none", the server does no separation and returns the raw text, including the <think> from the generation prompt, in content. The server-wide setting is --reasoning-format (none, deepseek, deepseek-legacy; default auto), and the request field overrides it.

Streaming uses the same field name in the delta (real chunk): "delta":{"reasoning_content":" creek"}. A chat client shows reasoning_content deltas separately from content deltas, and an agent usually does not send old reasoning back in later turns.

The flag --reasoning-budget N limits thinking tokens (-1, the default, means no limit; 0 means end the thinking at once). The server passes the budget to the sampler only when the chat parser reports an end-of-thinking tag for the template (thinking_end_tags in server-common.cpp). A run of this server with --reasoning-budget 0 returned the same reasoning_content as the run above, so with this template and vocabulary file the budget had no visible effect. The request fields reasoning_effort and chat_template_kwargs (for example {"enable_thinking": false}) are passed to templates that read them, and the server flag --reasoning on|off sets enable_thinking for every request. All of these depend on the template and model, and the random tiny model does not show their effect.

Example 15: /metrics in Prometheus text format

Problem: when the server is slow, you need numbers to find out why. Idea: with --metrics, the server publishes counters in the text format of Prometheus, a common monitoring tool that collects such numbers regularly and draws charts. A counter only grows (total tokens served). A gauge goes up and down (requests running now).

The question is what the server reports for monitoring. With --metrics, GET /metrics returns counters and gauges in the Prometheus text format. The values below were read from the first server after it had served most of this lesson's earlier requests, including the two 1,500-token requests of Example 8.

Terminal
curl -s http://127.0.0.1:8080/metrics

Real output (the # HELP lines kept for the first two metrics only):

Output
# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed, excluding cached tokens
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total 207
# HELP llamacpp:prompt_tokens_cached_total Number of prompt tokens reused from the cache
# TYPE llamacpp:prompt_tokens_cached_total counter
llamacpp:prompt_tokens_cached_total 176
llamacpp:prompt_seconds_total 0.436448
llamacpp:tokens_predicted_total 3180
llamacpp:tokens_predicted_seconds_total 33.5476
llamacpp:n_decode_total 1681
llamacpp:n_tokens_max 1502
llamacpp:spec_decode_num_draft_tokens_total 0
llamacpp:spec_decode_num_accepted_tokens_total 0
llamacpp:spec_decode_num_drafts_total 0
llamacpp:prompt_tokens_seconds 474.283
llamacpp:predicted_tokens_seconds 94.284
llamacpp:requests_processing 0
llamacpp:requests_deferred 0
llamacpp:n_busy_slots_per_decode 1.89173

Each metric has a # HELP line with a description, a # TYPE line (counter for totals that only grow, gauge for values that go up and down), and a line with the name and value. The names start with the prefix llamacpp:. The program below parses the format and derives two numbers that the server does not print.

Java
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;

public class MetricsParse {
    static final String TEXT = """
        # HELP llamacpp:prompt_tokens_total Number of prompt tokens processed, excluding cached tokens
        # TYPE llamacpp:prompt_tokens_total counter
        llamacpp:prompt_tokens_total 207
        # TYPE llamacpp:prompt_tokens_cached_total counter
        llamacpp:prompt_tokens_cached_total 176
        # TYPE llamacpp:prompt_seconds_total counter
        llamacpp:prompt_seconds_total 0.436448
        # TYPE llamacpp:prompt_tokens_seconds gauge
        llamacpp:prompt_tokens_seconds 474.283
        # TYPE llamacpp:n_busy_slots_per_decode gauge
        llamacpp:n_busy_slots_per_decode 1.89173
        """;

    public static void main(String[] args) {
        Map<String, Double> values = new TreeMap<>();
        Map<String, String> types = new TreeMap<>();
        for (String line : TEXT.split("\n")) {
            if (line.startsWith("# TYPE ")) {
                String[] p = line.substring(7).split(" ");
                types.put(p[0], p[1]);
            } else if (!line.isBlank() && !line.startsWith("#")) {
                int space = line.lastIndexOf(' ');
                values.put(line.substring(0, space), Double.parseDouble(line.substring(space + 1)));
            }
        }
        values.forEach((k, v) -> System.out.printf(Locale.ROOT, "%-38s %-8s %s%n", k, types.get(k), v));

        double processed = values.get("llamacpp:prompt_tokens_total");
        double cached = values.get("llamacpp:prompt_tokens_cached_total");
        System.out.printf(Locale.ROOT, "share of prompt tokens from cache = %.1f%%%n", 100 * cached / (processed + cached));
        System.out.printf(Locale.ROOT, "prompt tokens / prompt seconds    = %.3f%n",
                processed / values.get("llamacpp:prompt_seconds_total"));
    }
}
Output
llamacpp:n_busy_slots_per_decode       gauge    1.89173
llamacpp:prompt_seconds_total          counter  0.436448
llamacpp:prompt_tokens_cached_total    counter  176.0
llamacpp:prompt_tokens_seconds         gauge    474.283
llamacpp:prompt_tokens_total           counter  207.0
share of prompt tokens from cache = 46.0%
prompt tokens / prompt seconds    = 474.283

prompt_tokens_total counts only processed tokens and prompt_tokens_cached_total the reused ones, so the cache supplied 176 of 383 prompt tokens (46.0%). That share is the first number to check when time to first token is high: a low share means prompts do not share prefixes or slots keep being reassigned.

The gauge prompt_tokens_seconds (474.283) equals the program's quotient of the two counters here, because this was the first scrape. The source comment says the gauges "are averaged over the window between two scrapes": every read of /metrics resets the window, so two monitoring systems scraping the same server see different gauge values. Counters are not reset. n_busy_slots_per_decode (1.89) is the average number of slots in each llama_decode call, close to 2 because the two 1,500-token requests of Example 8 dominate the 1,681 decode calls. n_tokens_max (1502) is the longest sequence seen, the 1,500-token generation plus its prompt.

Example 16: /v1/embeddings with --embeddings

Problem: to search code or documents by meaning, a program needs a way to compare texts. Idea: an embedding model turns a text into a list of numbers (a vector) so that similar texts get similar vectors. To compare two vectors, use the cosine similarity: it is 1 for vectors that point the same way, 0 for unrelated ones, and -1 for opposite ones. For example, the vectors [1, 0] and [0, 1] point at right angles and have a cosine similarity of 0.

The question is how to get a vector for a piece of text. An embedding is a fixed-size vector that represents a whole input; this server runs with --embeddings --pooling mean, which averages the final hidden states of all input tokens.

Terminal
# llama-server -m tiny-random-qwen2vocab-f32.gguf -a tiny-coder --embeddings --pooling mean -np 1
curl -s http://127.0.0.1:8080/v1/embeddings -d '{"input": ["public class Main", "fn main()"], "model": "tiny-coder"}'

Real output (summarized: the first 4 of 256 values, the Euclidean norm, and the cosine similarity computed from the response):

Output
{'model': 'tiny-coder', 'object': 'list', 'usage': {'prompt_tokens': 6, 'total_tokens': 6}}
0 embedding 256 [-0.1285, -0.0264, 0.068, 0.0559] norm=1.0000
1 embedding 256 [-0.0146, -0.0142, 0.0032, 0.0096] norm=1.0000
cos=0.0143

The response has object: "list", one data entry per input with index, object: "embedding", and embedding (256 floats, the model's hidden size), and usage with 6 prompt tokens for both inputs together. Each vector has length 1: the OpenAI-compatible endpoint normalizes with the Euclidean norm (--embd-normalize, default 2). With unit vectors, the dot product is the cosine similarity; 0.0143 means the two vectors are nearly unrelated, as expected from random weights.

Without --embeddings, the endpoint answers 501 with This server does not support embeddings. Start it with --embeddings. The help text describes the flag as restricting the server to embeddings and intended for dedicated embedding models; a generative model such as this one produces vectors, but they are not trained for retrieval. The native POST /embeddings returns a different shape, a JSON array with one embedding per input that is itself an array of vectors (one per token with --pooling none, one pooled vector otherwise).

Example 17: error responses

Problem: a client that treats every failure the same either retries requests that can never work or gives up on ones that would work a moment later. Idea: read the status code and the error type, then choose an action for each kind.

The question is what each failure looks like, so that a client can react correctly. These requests hit the first server (slot context 2,048) with a 2,100-word prompt, broken JSON, a body without messages, a wrong type, a missing route, and two disabled features.

Terminal
curl -s -w ' HTTP %{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -d "{\"messages\":[{\"role\":\"user\",\"content\":\"$BIG\"}],\"max_tokens\":4}"
curl -s -w ' HTTP %{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}'
curl -s -w ' HTTP %{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -d '{"prompt":"hi"}'
curl -s -w ' HTTP %{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}],"temperature":"hot"}'
curl -s -w ' HTTP %{http_code}\n' http://127.0.0.1:8080/v1/nothing
curl -s -w ' HTTP %{http_code}\n' -X POST http://127.0.0.1:8080/props -d '{}'
curl -s -w ' HTTP %{http_code}\n' http://127.0.0.1:8080/v1/embeddings -d '{"input":"hi"}'

Real output:

Output
{"error":{"code":400,"message":"request (2118 tokens) exceeds the available context size (2048 tokens), try increasing it","type":"exceed_context_size_error","n_prompt_tokens":2118,"n_ctx":2048}} HTTP 400
{"error":{"code":500,"message":"[json.exception.parse_error.101] parse error at line 1, column 44: syntax error while parsing array - unexpected end of input; expected ']'","type":"server_error"}} HTTP 500
{"error":{"code":400,"message":"'messages' is required","type":"invalid_request_error"}} HTTP 400
{"error":{"code":400,"message":"Field 'temperature': [json.exception.type_error.302] type must be number, but is string","type":"invalid_request_error"}} HTTP 400
{"error":{"message":"File Not Found","type":"not_found_error","code":404}} HTTP 404
{"error":{"code":501,"message":"This server does not support changing global properties. Start it with `--props`","type":"not_supported_error"}} HTTP 501
{"error":{"code":501,"message":"This server does not support embeddings. Start it with `--embeddings`","type":"not_supported_error"}} HTTP 501

Every error is an object with an error member holding code (the HTTP status repeated), message, and type, the same envelope as OpenAI's API. The context error adds n_prompt_tokens and n_ctx, so a client can compute how much to cut: 2,118 tokens against 2,048 available. The broken JSON is reported as 500 server_error, not 400: the exception wrapper in server.cpp maps std::invalid_argument to 400 and every other exception, including the JSON library's parse error, to 500. A client cannot rely on the status alone to separate its own bugs from server faults. A streaming request that fails before the first token gets an ordinary HTTP error response, not an SSE stream. If an error happens after streaming has started, the source sends one more event, data: {"error": {...}}, and closes the stream without [DONE]. When the client disconnects in the middle of a stream, the server cancels the task; the log of such a run shows stop: cancel task, id_task = 1911 and the slot released with 232 tokens.

The program below turns these responses into the action a client takes.

Java
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;

public class ErrorPolicy {
    record Captured(int status, String body) {}

    static String action(int status, JsonNode error) {
        String type = error.path("type").asText();
        return switch (type) {
            case "unavailable_error" -> "retry after a delay (model loading or no slot)";
            case "exceed_context_size_error" -> "shorten prompt by "
                    + (error.get("n_prompt_tokens").asInt() - error.get("n_ctx").asInt()) + "+ tokens";
            case "authentication_error" -> "fix the API key, do not retry";
            case "not_supported_error" -> "server lacks a flag, do not retry";
            case "invalid_request_error", "not_found_error" -> "client bug, do not retry";
            default -> status >= 500 ? "log and report, retry at most once" : "unknown, do not retry";
        };
    }

    public static void main(String[] args) throws Exception {
        List<Captured> errors = List.of(
            new Captured(503, "{\"error\":{\"message\":\"Loading model\",\"type\":\"unavailable_error\",\"code\":503}}"),
            new Captured(400, "{\"error\":{\"code\":400,\"message\":\"request (2118 tokens) exceeds the available context size (2048 tokens), try increasing it\",\"type\":\"exceed_context_size_error\",\"n_prompt_tokens\":2118,\"n_ctx\":2048}}"),
            new Captured(401, "{\"error\":{\"message\":\"Invalid API Key\",\"type\":\"authentication_error\",\"code\":401}}"),
            new Captured(501, "{\"error\":{\"code\":501,\"message\":\"This server does not support embeddings.\",\"type\":\"not_supported_error\"}}"),
            new Captured(400, "{\"error\":{\"code\":400,\"message\":\"'messages' is required\",\"type\":\"invalid_request_error\"}}"),
            new Captured(500, "{\"error\":{\"code\":500,\"message\":\"[json.exception.parse_error.101] ...\",\"type\":\"server_error\"}}"));
        ObjectMapper json = new ObjectMapper();
        for (Captured c : errors) {
            JsonNode error = json.readTree(c.body()).get("error");
            System.out.printf("%d %-26s -> %s%n", c.status(), error.get("type").asText(), action(c.status(), error));
        }
    }
}
Output
503 unavailable_error          -> retry after a delay (model loading or no slot)
400 exceed_context_size_error  -> shorten prompt by 70+ tokens
401 authentication_error       -> fix the API key, do not retry
501 not_supported_error        -> server lacks a flag, do not retry
400 invalid_request_error      -> client bug, do not retry
500 server_error               -> log and report, retry at most once

The decision uses type, which is more specific than the status: both the context error and the missing messages are 400, but only the first can be fixed by trimming the conversation. The context error needs 70 fewer prompt tokens, plus room for the answer, because a slot must hold prompt and output together. The 503 types are the only ones to retry automatically.

Interview points

  • llama-server answers /health with 503 Loading model until the model is loaded, and with 200 {"status":"ok"} afterwards. /health needs no API key, while every other endpoint does when --api-key is set.
  • -np N sets the number of slots. With an explicit -np, the context from -c is split evenly between the slots. With the default (auto), the server uses 4 slots and a unified KV cache. /v1/models reports the per-slot n_ctx.
  • In timings, cache_n + prompt_n is the prompt length. A prompt that matches the cache completely still processes one token, because the cache stores keys and values, not the logits (the scores for the next token).
  • Prompt reuse has three levels: the longest common prefix inside a slot, the RAM prompt cache (--cache-ram, default 8192 MiB) that saves evicted slot states, and slot files (--slot-save-path).
  • --cache-reuse N moves cached chunks of at least N tokens to new positions when text is removed from the prompt after the common prefix, for example when old lines are dropped from the front. It does not help when text is inserted, it adjusts RoPE positions instead of recomputing, and it is off by default.
  • The server picks a slot in this order: the slot named by id_slot, else the idle slot with the longest common prefix above --slot-prompt-similarity (0.10), else the least recently used idle slot.
  • A stream ends with data: [DONE]. Usage arrives only with stream_options.include_usage, in a final chunk with an empty choices array. Keep-alive pings are comment lines that start with :.
  • Tool calls come back as tool_calls, with arguments as a JSON string and finish_reason: "tool_calls". Streamed calls are joined by index. At this commit, tool_choice: "required" still allows text before the first call.
  • Grammars and JSON schemas guarantee syntax and schema limits, but not sensible values. Schema grammars allow whitespace between tokens.
  • /metrics gauges are averaged over the window between two scrapes, so each scrape resets them. Counters are cumulative.

Summary

  • llama-server listens on 127.0.0.1:8080 by default, logs a security warning when no API key is set, and accepts keys as Authorization: Bearer or X-Api-Key.
  • /v1/models returns an OpenAI-style data list and an Ollama-style models list. The value meta.n_ctx is the context of one slot.
  • /props shows the default sampling parameters, the chat template, its detected capabilities, and build_info.
  • Chat responses carry choices, usage with cached_tokens, and llama.cpp's timings. The finish_reason is stop, length, or tool_calls.
  • Streaming chunks start with a role-only delta, carry one token each, end with a chunk that has a finish_reason, and close with [DONE].
  • /completion reports id_slot, tokens_evaluated, tokens_cached, stop_type (eos, limit, word, or none), and every effective sampling parameter.
  • /tokenize with parse_special: false keeps user text from becoming control tokens. /apply-template shows the exact prompt, including text the template inserts.
  • Busy slots are decoded together in one batch. /slots?fail_on_no_slot=1 returns 503 when all slots are busy, while completion requests wait in a queue.
  • The RAM prompt cache restores evicted prefixes, and --cache-ram 0 disables it.
  • Slot files hold a 12-byte header, the token ids, and the KV data. That is 1,040 bytes per token for the tiny model and about 128 KiB per token for an 8B shape.
  • Tool calling needs a tool-capable template. The server parses its markers into tool_calls, and the client sends results back as role: "tool" messages.
  • With a reasoning template, the server splits <think> text into reasoning_content, and reasoning_format: "none" returns the raw text.
  • Embeddings need --embeddings and a pooling type other than none. /v1/embeddings returns unit-length vectors.
  • Errors share the envelope {"error": {"code", "message", "type"}}. The context error adds n_prompt_tokens and n_ctx, and a client retries only the 503 errors automatically.