Introduction and First Run

The first three lessons explained what a language model is, how people talk to one, and the vocabulary of agents, all on paper. This lesson puts a real model on your machine. A local language model is a program on your own computer that reads text and writes text. Nothing is sent to a company's servers. To use one, you need more than a single download. A file of numbers must be loaded by a program, that program must be reachable by other programs, and something must send it requests. This lesson names each of those pieces and then runs the lowest ones on a real machine.

You will run llama-cli, which gives you a conversation in the terminal, and llama-server, which gives you an HTTP API (a way for other programs to send requests over the network and read answers). You will send a request with curl, read the fields of the response, and read the timing lines the program prints. You will also see which later lesson explains each part, and what the loop command-line tool, which this course builds, does for you.

Where the real captures come from

A real chat model is several gigabytes, and this course shows only outputs that were produced by running the programs. To keep that practical, the captures in this lesson come from a tiny test model. The machine has 4 CPU cores, 15 GiB of RAM, and no GPU (graphics processing unit, the chip that most large models run on). It runs llama.cpp at commit 0c1e57098, the version this course pins. A commit is one exact snapshot of the source code, so everyone sees the same flags and log lines.

The tiny model has the Llama architecture (a common design for language models) with 4 layers and 256 dimensions, and it uses the Qwen2 tokenizer (the part that splits text into pieces, explained in lesson 6). Its weights are random. A model with random weights has learned nothing, so its text is meaningless by design.

This does not make the captures less useful. The log lines, token counts, timings, JSON fields, and server behavior are real, and a trained model produces exactly the same kinds of output. Where this lesson shows meaningful chat text, it is labeled as representative output from a real small instruct model, and the structure around it (log lines, JSON fields) is the same.

Timings in the captures belong to this machine. A different CPU gives different numbers. The shape of the output stays the same.

The stack, from file to agent

A local coding assistant is not one program. It is five layers, and each layer has one job and talks to the next through a defined interface. Knowing the layers helps later, because a problem always sits in one of them, and the fix depends on which one.

Think of a restaurant. The recipe book is the weights file. The kitchen is the engine. The waiter's window is the server. The customer ordering is the client. A personal assistant who orders, tastes, and orders again until a goal is met is the agent. The analogy holds for the order of the layers and for who talks to whom. It stops holding for details: a recipe book does not change while the kitchen cooks, and a weights file does not either, but a kitchen can cook many dishes at once while an engine's capacity depends on memory.

Two terms in the table need a definition. HTTP is the protocol web browsers use to talk to servers: a request goes out, a response comes back. JSON is a plain-text format for structured data, such as {"name": "value"}. "OpenAI-compatible" means the server accepts the same requests as the OpenAI API, so a client written for one works with the other.

The names in the products people mention map onto these layers:

• llama.cpp is an engine written in C and C++ with a server (llama-server) and command-line tools. It reads GGUF files. It runs on CPUs and on most GPU types.
• vLLM is an engine written in Python, C++, and CUDA (NVIDIA's GPU programming platform). It is built for serving many users on GPUs. It reads Hugging Face directories of safetensors files and some GGUF files. It starts an OpenAI-compatible server with vllm serve.
• Ollama is a command-line tool and API server that manages a library of models and loads them in separate runner processes.
• LM Studio is a desktop application that downloads models, runs them with llama.cpp or MLX engines, and offers a chat window and a local server.
• MLX is Apple's array framework for Apple silicon (the M-series chips), and mlx-lm is an engine built on it.

The wrapper-server question

A common design question is whether a project like this needs its own "wrapper server" between the engine and the client. A wrapper server would be an extra HTTP program that sits in the middle and forwards requests.

It does not need one. llama-server and vllm serve are already HTTP servers with an OpenAI-compatible API. The loop tool built in this course starts such an engine as a child process (a program launched and controlled by another program). It waits until the engine's health endpoint answers, and then it talks to the engine over HTTP on 127.0.0.1. Ollama uses the same split: a CLI and API server in front, and the model loaded in a separate runner process.

A lesson on the engine landscape later in this course compares three designs with criteria:

• manage an engine subprocess,
• connect to an engine that already runs elsewhere,
• embed the engine in the JVM (the Java virtual machine).

loop implements the first two. Section 7 builds the third in pure Java.

Example 1: check the machine

Before you run a model, you need to know what your machine can hold and how fast it can read memory. A model has to fit in memory, and speed depends heavily on how quickly the machine can read that memory. This example collects the CPU count, the RAM size, and the GPU, if any.

Three numbers decide most choices. RAM or GPU memory decides which weights fit. Memory bandwidth, the amount of data the machine can read per second, decides the speed. Core count decides how fast the model reads a long prompt. Later lessons explain each.

nproc
free -h
nvidia-smi --query-gpu=name,memory.total --format=csv

nproc prints the number of CPU cores, free -h prints memory in human-readable units, and nvidia-smi asks an NVIDIA GPU for its name and memory. Here is the real output on the reference machine. It has no GPU, so the third command is not found:

4
               total        used        free      shared  buff/cache   available
Mem:            15Gi       610Mi        13Gi        13Mi       1.7Gi        15Gi
Swap:             0B          0B          0B
bash: nvidia-smi: command not found

The first line says 4 cores. The Mem: line says 15 GiB in total, with 13 GiB free, so a model of a few gigabytes fits easily. The last line is an error, which is the expected answer on a machine with no NVIDIA GPU. On a machine with an NVIDIA GPU, the last command prints one line per GPU. Representative output (names and sizes differ by machine):

name, memory.total [MiB]
NVIDIA GeForce RTX 4090, 24564 MiB

On macOS, the same facts come from sysctl -n hw.ncpu hw.memsize (CPU count and memory in bytes). Apple silicon uses one pool of memory for the CPU and the GPU, so hw.memsize is also the GPU budget.

This course sorts machines into four tiers, from simple to large:

"0.5B to 8B" means 0.5 billion to 8 billion parameters, the numbers stored in the model. "4-bit" means each of those numbers is stored in 4 bits, which makes the file small (lessons 18 to 22). VRAM is the memory on a GPU. Metal is Apple's GPU programming interface. Tensor parallelism splits one model across several GPUs.

The reference machine here is tier 1. Every lesson of the course can be followed on tier 1. Lessons that use vLLM give the commands and representative output for readers without a GPU.

Example 2: get llama.cpp and print its version

llama.cpp is distributed as source code and as prebuilt programs. Package managers carry it: the pinned docs/install.md lists brew install llama.cpp for Homebrew on macOS and Linux. The project also publishes prebuilt archives for common platforms. A third route is to build from source with CMake, a build tool that turns source code into programs:

cmake -B build
cmake --build build --config Release -j 4

The first command prepares a build directory. The second compiles the code, and -j 4 uses 4 cores to do it faster. The programs land in build/bin. This course pins one version so that flags and log lines match what you see here. The pinned build reports its version like this:

llama-cli --version

The program prints two lines, the version with a commit hash and the compiler it was built with. This is the real output:

version: 0.5.0-dev (build 1, commit 0c1e570)
built with GNU 13.3.0 for Linux x86_64

The commit hash identifies the exact source. When a flag in this course behaves differently on a newer release, comparing hashes is the first check. The build contains these programs, which later lessons use:

Example 3: download a small instruct model

A base language model only continues text. An instruct model is a language model that was trained further to follow instructions and hold a conversation. That is the kind of model you want for a chat or an agent.

The file format is GGUF, the single-file format llama.cpp reads. One GGUF file holds the weights, the tokenizer, and the settings, so there is only one file to manage (lesson 28 covers it). llama.cpp can download a model from Hugging Face, a website that hosts models, with -hf <user>/<model>[:quant]. The pinned --help lists this flag. The optional :quant suffix picks a quantization, which is a smaller number format for the weights (lessons 18 to 22).

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

The first run downloads the file into a cache directory, and later runs read it from the cache. llama-cli --cache-list prints the cached models. The repository name above is the example used in the pinned docs/models.md. Repositories on the Hub can be renamed or removed, so check that the repository exists before relying on it.

A second route is to fetch a file directly with curl from a resolve URL (lesson 26 explains the URL scheme). This container has no access to the Hub, so the rest of this lesson runs the tiny local model and shows representative output where a trained model is needed.

Example 4: one-shot generation

The simplest thing a language model does is continue a text. A completion is that continuation: the model reads the prompt and produces tokens (small pieces of text) until it reaches a limit.

The command uses these flags:

• -p sets the prompt.
• -n sets the maximum number of new tokens.
• --seed sets the random seed, which makes the random choices repeatable.
• -t sets the number of CPU threads.
• -no-cnv turns off conversation mode, so the prompt is continued as plain text instead of being wrapped as a chat message.

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "The capital of France is" -n 16 --seed 7 -t 2 -no-cnv

The prompt is five tokens, and the model adds 16 more. Here is the real output of the generated line. The model has random weights, so the continuation is meaningless:

The capital of France is Conditions哚_rr냇渤海.data왁iter affection остальн，在 dispers.Measure tax stimulates岽

The first words are the prompt, copied back. Everything after them is the model's guess, and with random weights each guess is an arbitrary token. A trained 1B instruct model gives a sensible continuation for the same command. Representative output:

The capital of France is Paris. It is located on the Seine River in the north of the country

The program also prints a block of log lines before the text. The two lines that matter now:

sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = 16, n_keep = 0

The first line is the sampler chain. After the model computes a score for every possible next token, the chain turns the scores into one chosen token. The steps run left to right, and each one narrows or reshapes the choice. A ? in front of a step means it is optional and skipped unless enabled. By default, top-k, top-p, min-p, and temperature are active, and a final dist step draws the token at random. Lessons 12 and 13 build each step.

The second line gives the context size (n_ctx), which is the number of tokens the model can hold at once, and the maximum number of new tokens (n_predict). Here the model can hold 4096 tokens, and it was asked for 16 new ones.

Example 5: conversation mode

A completion continues text, but a chat has turns: a user speaks, the assistant answers. llama-cli starts a chat when the model file contains a chat template. A chat template is a recipe, stored in the GGUF file, that turns a list of messages (system, user, assistant) into the single text the model was trained on (lesson 15). The model never sees "messages". It sees one long text with special markers that show who is speaking.

The program prints a banner, the available commands, and a > prompt. Below is a real session with the tiny model, started with llama-cli -m tiny-random-qwen2vocab-f32.gguf -t 2 --seed 1 -n 4. The flag -n 4 keeps the reply to 4 tokens. The input was Hi and then end of input. The ASCII-art logo printed above the banner is left out.

build      : b1-0c1e570
model      : tiny-random-qwen2vocab-f32.gguf
ftype      : all F32
modalities : text
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern
> Hi
 willingly potassiumbol trustee
[ Prompt: 1890.9 t/s | Generation: 140.9 t/s ]
> 
Exiting...

Read it from the top. The banner lists the build and the model file. The line ftype : all F32 says every weight is stored as a 32-bit float, which is the largest and most precise format. The commands let you restart a reply, clear the history, or add a file to the conversation. After > Hi, the model printed four meaningless tokens, as expected for random weights. The bracketed line reports speed: Prompt is how many prompt tokens the program processed per second, and Generation is how many new tokens it produced per second. The abbreviation t/s means tokens per second.

A trained model gives a readable answer between those lines:

> Write a Java method that reverses a string.
Here is a method that reverses a string using StringBuilder:

public static String reverse(String s) {
    return new StringBuilder(s).reverse().toString();
}
[ Prompt: 74.2 t/s | Generation: 22.8 t/s ]

This last output is representative: the code and the speeds come from a typical 1B model at 4-bit on a laptop CPU, and a different machine prints different speeds. Notice that generation is slower than prompt processing here too (22.8 against 74.2). The next example explains why.

Example 6: read the timing summary

Speed is the number to watch when running a model locally, so it helps to read the timing report correctly. llama-completion prints a timing summary when it stops. These real lines come from the one-shot run in Example 4:

common_perf_print:    sampling time =       6.46 ms
common_perf_print:    samplers time =       2.49 ms /    21 tokens
common_perf_print:        load time =     217.41 ms
common_perf_print: prompt eval time =       8.16 ms /     5 tokens (    1.63 ms per token,   612.75 tokens per second)
common_perf_print:        eval time =      72.70 ms /    15 runs   (    4.85 ms per token,   206.32 tokens per second)
common_perf_print:       total time =      88.27 ms /    20 tokens

The two lines to learn are prompt eval and eval. They measure two different phases of the same request:

The reason the two phases differ is memory. To process a token, the CPU must read all the model's weights from memory. In prefill, many tokens are processed together, so one read of the weights serves all of them. The CPU does a lot of arithmetic for each byte it reads. In decode, each new token needs the weights again, and only one token uses them. The CPU spends more time waiting for memory than calculating.

On large models and large prompts, prefill reaches many times the tokens per second of decode. The tiny model already shows the gap: 613 tokens per second for the prompt against 206 for generation, about 3 times, with a prompt of only 5 tokens. The representative lines of Example 5 show the same ratio for a trained 1B model (74 against 23 tokens per second). The gap grows with longer prompts, because a longer prompt gives each weight read more tokens to work on. Lesson 11 explains prefill and decode in detail, and lesson 17 explains the memory limit.

The other lines are easier. The load time line shows how long the file took to load, about 0.2 seconds here. The total time line counts 20 evaluated tokens: 5 prompt tokens and 15 decode runs.

The command asked for 16 new tokens (-n 16), so why only 15 decode runs? The first new token is chosen from the scores that prefill produced, so it needs no decode run. The last new token is printed but never fed back into the model. So 16 new tokens need only 15 decode runs.

Example 7: start llama-server

Running llama-completion reloads the model on every command. A server solves this. llama-server loads a model once and keeps it in memory, so every request skips the load time.

The command uses these flags:

• --host and --port choose the address. The default port is 8080, and this lesson uses a different port to avoid collisions.
• -c sets the context size.
• -np sets the number of slots, which are independent conversations the server can hold at once.
• --alias sets the model name the API reports.

llama-server -m tiny-random-qwen2vocab-f32.gguf --alias tiny-random \
  --host 127.0.0.1 --port 18431 -t 2 -c 1024 -np 1

The listening address is 127.0.0.1, the loopback interface, so only programs on the same machine can connect. Binding to 0.0.0.0 would make the server reachable from the network. Here is the real startup log:

srv  llama_server: initializing ...
srv  llama_server: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
srv    load_model: loading model 'tiny-random-qwen2vocab-f32.gguf'
load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
cmn          init: llama threadpool init, n_threads = 2
srv    load_model: initializing, n_slots = 1, n_ctx_slot = 1024, kv_unified = 'false'
srv  llama_server: model loaded
srv  llama_server: listening on http://127.0.0.1:18431

Most lines are routine. Three carry information:

• n_slots = 1, n_ctx_slot = 1024 says the server holds one conversation with room for 1024 tokens.
• The control-looking token line is a warning about the random test model, not about the server.
• The security line states that no API key is required. On a laptop bound to 127.0.0.1 this is acceptable. A server reachable from a network needs --api-key.

The last line, listening on http://127.0.0.1:18431, means the server is ready for requests.

The server can also serve a browser chat page. The pinned --help lists --ui and --no-ui (default: enabled), and the source (tools/server/server-http.cpp) serves the page at / and /index.html. The reference build used here was compiled without the web assets, so GET / returns status 404 with a JSON error. A build that includes the assets shows the chat page when a browser opens the server address.

Example 8: check health and list models

A client needs to know two things before it sends work: is the server ready, and which model does it serve? Two endpoints answer these. An endpoint is one URL path on the server that does one job.

/health returns {"status":"ok"} when the model is loaded. While loading is still in progress, it returns status 503 with the error message Loading model. A client that starts the server itself polls this endpoint, which means it asks again and again until the answer is ok. That is what loop serve will do. /v1/models lists the models in the format of the OpenAI API.

curl -s 127.0.0.1:18431/health
curl -s 127.0.0.1:18431/v1/models

The flag -s makes curl silent, so it prints only the response body. Here is the real output of /health:

{"status":"ok"}

The /v1/models response is long because it carries two lists: an Ollama-style models list and an OpenAI-style data list. This is a real data entry, reformatted:

{
  "id": "tiny-random",
  "aliases": ["tiny-random"],
  "object": "model",
  "owned_by": "llamacpp",
  "meta": {
    "vocab_type": 2,
    "n_vocab": 151936,
    "n_ctx": 1024,
    "n_ctx_train": 4096,
    "n_embd": 256,
    "n_params": 80611584,
    "size": 322446336,
    "ftype": "all F32"
  }
}

The id is the name a client puts in its requests. The meta object describes the loaded model:

Example 14 checks the size number with arithmetic.

Example 9: a non-streaming chat completion

The main endpoint is POST /v1/chat/completions. A POST request sends data to the server in its body. This request holds a list of messages, and each message has a role (system, user, or assistant) and content. The server fills the chat template, generates a reply, and returns it as one JSON object. Two more fields matter here: max_tokens caps the number of generated tokens, and seed fixes the random draw.

curl -s 127.0.0.1:18431/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tiny-random","messages":[{"role":"user","content":"Say hello"}],"max_tokens":8,"seed":1}'

The -H flag adds an HTTP header that tells the server the body is JSON, and -d supplies the body. Here is the real response, reformatted. The per-token fields of timings are left out, and the content is random-weight output:

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " האמריק𝐫bol alleyListeningContentAlignment索取民眾"
      }
    }
  ],
  "created": 1791026763,
  "model": "tiny-random",
  "system_fingerprint": "b1-0c1e570",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 8,
    "prompt_tokens": 20,
    "total_tokens": 28,
    "prompt_tokens_details": { "cached_tokens": 0 }
  },
  "id": "chatcmpl-oDjT54CmrR1CsZjHceDJQ0n6u3AuJhLH",
  "timings": {
    "cache_n": 0,
    "prompt_n": 20,
    "prompt_ms": 12.815,
    "prompt_per_second": 1560.67,
    "predicted_n": 8,
    "predicted_ms": 49.986,
    "predicted_per_second": 140.04
  }
}

The response has many fields, but a client reads only a few of them:

In this response, finish_reason is length, because the request set max_tokens to 8 and the model produced exactly 8 tokens. The timings object repeats what Example 6 showed: 20 prompt tokens processed at about 1561 tokens per second, and 8 tokens generated at about 140 per second.

One number needs an explanation. The request has one 2-word user message, yet prompt_tokens is 20. The chat template added role markers, an end-of-turn marker, and a default system message. Lesson 15 shows the exact string. Clients must count the tokens of the final prompt, not of the user text, when they budget the context.

Example 10: streaming

In Example 9 the client waits until the whole reply is finished. For a long answer, that can take many seconds with nothing on screen. Streaming fixes this: the server sends the reply piece by piece as it is produced.

With "stream": true, the server sends server-sent events. These are lines that start with data: , separated by blank lines, over one open HTTP response. Each event is a JSON chunk with a delta, which holds the new piece of text. The last event is the literal data: [DONE]. The flag -N tells curl not to buffer the output, so lines appear as they arrive.

curl -sN 127.0.0.1:18431/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tiny-random","messages":[{"role":"user","content":"Say hello"}],"max_tokens":4,"seed":1,"stream":true}'

This is the real output. The created and id fields are shortened with ..., and one chunk with random-weight text in Hebrew is removed:

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],...}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"𝐫"}}],...}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"bol"}}],...}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":" alley"}}],...}

data: {"choices":[{"finish_reason":"length","index":0,"delta":{}}],...,"timings":{"cache_n":19,"prompt_n":1,"prompt_ms":6.445,"predicted_n":4,"predicted_ms":17.159,...}}

data: [DONE]

The first chunk announces the role and has no text. Each following chunk carries one token piece in delta.content. The chunk with a finish_reason closes the reply and carries the timings. A client builds the full text by appending every delta.content. Streaming makes the first words visible after one prefill and one decode step instead of after the whole reply. Example 15 parses these lines in Java.

The timings in the last chunk show "cache_n":19,"prompt_n":1. This request used the same prompt as Example 9. The server compared it with the prompt it had processed before and found that the first 19 tokens were identical. It kept their saved state and processed only 1 prompt token. That saved state is the KV cache (key-value cache, lesson 11), and the reuse across requests is the prompt cache (covered in the llama-server lesson later in this course). The first request had cache_n 0 and processed all 20 tokens.

This reuse matters for agents. Each step of an agent sends the whole conversation again, and only the new part needs processing.

Example 11: temperature and seed

Two settings control how a reply varies from one request to the next. Temperature controls how random the choice of the next token is. Low values favor the most likely token, and high values spread the choice over more tokens (lesson 12). The seed fixes the random number generator, so the same request gives the same text.

These real results came from the same prompt with max_tokens 5:

The table shows three behaviors:

• Same seed and same settings give the same text (rows 1 and 2).
• A different seed gives a different text when temperature is above 0 (row 3).
• Temperature 0 ignores the seed, because the highest-scoring token is always chosen and no random draw is needed (rows 4 and 5). The output also repeats a pattern, which is typical of greedy choice, meaning always taking the top token (lesson 5).

Identical output for identical seeds holds on one machine with one configuration. A different number of threads, a different batch size, or a different build can change the order of floating-point additions. Computers round decimal numbers, so adding the same numbers in a different order can give a slightly different sum. That can change which token wins when two scores are very close (lesson 8, Example 15). Reproducibility across machines is not a guarantee.

Example 12: the smallest Java client

So far, curl played the client. A Java program can do the same with one HttpClient call. The program below sends the JSON from Example 9 and prints the response body. It needs a running server, so it is marked java,ignore. The checker for this course compiles and runs only programs that need no server.

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class FirstClient {
    public static void main(String[] args) throws Exception {
        String body = """
            {"model":"tiny-random",
             "messages":[{"role":"user","content":"Say hello"}],
             "max_tokens":8,"seed":1}""";
        HttpRequest request = HttpRequest.newBuilder(URI.create("http://127.0.0.1:18431/v1/chat/completions"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.statusCode());
        System.out.println(response.body());
    }
}

The program builds a request with a URL, a header, and a POST body, sends it, and waits for the response as a string. HttpClient is part of the JDK since Java 11, so this client needs no library. The program prints the status code (200 on success) and the JSON from Example 9. Section 5 builds a full client with streaming, cancellation, and typed records.

Example 13: read the response fields with Jackson

A response body is text. To use the fields, a client must parse the JSON, which means turning the text into objects it can ask questions. This program uses Jackson (com.fasterxml.jackson.databind), the JSON library the course uses, on an inline copy of the real response from Example 9.

The content is the real random-weight text, written as the server sent it. In the Java string, \\u05d4 is the six characters ה in the JSON, which Jackson decodes to the single character ה. The program prints counts instead of the text, so the output does not depend on how your terminal shows Hebrew or Chinese characters.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

public class ParseResponse {
    public static void main(String[] args) throws Exception {
        String json = """
            {"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant",
             "content":" \\u05d4\\u05d0\\u05de\\u05e8\\u05d9\\u05e7\\ud835\\udc2bbol alleyListeningContentAlignment\\u7d22\\u53d6\\u6c11\\u773e"}}],
             "model":"tiny-random","object":"chat.completion",
             "usage":{"completion_tokens":8,"prompt_tokens":20,"total_tokens":28,
                      "prompt_tokens_details":{"cached_tokens":0}},
             "timings":{"cache_n":0,"prompt_n":20,"predicted_n":8}}""";
        JsonNode root = new ObjectMapper().readTree(json);
        JsonNode choice = root.path("choices").path(0);
        String content = choice.path("message").path("content").asText();
        System.out.println("finish_reason = " + choice.path("finish_reason").asText());
        System.out.println("chars = " + content.length());
        System.out.println("code points = " + content.codePointCount(0, content.length()));
        JsonNode usage = root.path("usage");
        System.out.println("prompt = " + usage.path("prompt_tokens").asInt()
            + ", completion = " + usage.path("completion_tokens").asInt()
            + ", total = " + usage.path("total_tokens").asInt());
        System.out.println("cached = " + usage.path("prompt_tokens_details").path("cached_tokens").asInt());
    }
}

finish_reason = length
chars = 47
code points = 46
prompt = 20, completion = 8, total = 28
cached = 0

readTree parses JSON into a tree, and path(...) walks it without throwing an exception when a field is missing. The program asks for choices, then item 0, then message, then content.

The output shows two details. First, the string has 47 char values but 46 code points. A code point is one character as Unicode defines it. Java stores text as 16-bit char values, and one character here (𝐫, a mathematical bold letter) lies outside the Basic Multilingual Plane, the first 65,536 code points. Java stores such a character as two char values, so the char count is one higher. Lesson 6 explains this difference between characters, code points, and bytes.

Second, the usage numbers add up: 20 prompt tokens plus 8 completion tokens equal 28 total tokens.

Example 14: predict the file size from the parameter count

Why is a 7B model several gigabytes? The answer is arithmetic. A weight file is mostly one number per parameter, so the size follows from the parameter count and the bytes per number.

The meta object in Example 8 reports n_params of 80,611,584 and a size of 322,446,336 bytes. The model is stored in F32, 32-bit floats, and 32 bits is 4 bytes. So the expected size is 80,611,584 times 4. This program reads both values from the real /v1/models metadata and checks the relation. It then prints the same estimate for models of common sizes at common storage widths.

The widths are approximate. A 4-bit file also stores scale factors, which are extra numbers that restore the right magnitude, so its real average is slightly above 4 bits per weight (lessons 20 and 21 measure it).

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Locale;

public class FileSize {
    public static void main(String[] args) throws Exception {
        String meta = """
            {"vocab_type":2,"n_vocab":151936,"n_ctx":1024,"n_ctx_train":4096,
             "n_embd":256,"n_params":80611584,"size":322446336,"ftype":"all F32"}""";
        JsonNode m = new ObjectMapper().readTree(meta);
        long params = m.get("n_params").asLong();
        long size = m.get("size").asLong();
        System.out.println("params * 4 bytes = " + params * 4);
        System.out.println("reported size    = " + size);
        System.out.println("bytes per weight = " + String.format(Locale.ROOT, "%.2f", (double) size / params));

        double[] billions = {0.5, 1, 7, 8, 32};
        double[] bits = {32, 16, 8, 4};
        System.out.println();
        System.out.println(String.format(Locale.ROOT, "%-8s%10s%10s%10s%10s", "params", "32-bit", "16-bit", "8-bit", "4-bit"));
        for (double b : billions) {
            StringBuilder row = new StringBuilder(String.format(Locale.ROOT, "%-8s", b + "B"));
            for (double w : bits) {
                double gb = b * 1e9 * w / 8 / 1e9;
                row.append(String.format(Locale.ROOT, "%10.1f", gb));
            }
            System.out.println(row);
        }
    }
}

params * 4 bytes = 322446336
reported size    = 322446336
bytes per weight = 4.00

params      32-bit    16-bit     8-bit     4-bit
0.5B           2.0       1.0       0.5       0.3
1.0B           4.0       2.0       1.0       0.5
7.0B          28.0      14.0       7.0       3.5
8.0B          32.0      16.0       8.0       4.0
32.0B        128.0      64.0      32.0      16.0

The first three output lines confirm the rule: 80,611,584 parameters at 4 bytes equal exactly the reported size, and the size divided by the parameter count is 4.00 bytes per weight. The table applies the same rule to larger models. Each cell is parameters times bits divided by 8, in gigabytes. For example, 7 billion parameters at 16 bits is 7 billion times 2 bytes, which is 14 GB.

A 7B model needs 28 GB in 32-bit floats, 14 GB in 16-bit, and 3.5 GB at 4 bits per weight. A laptop with 16 GB of RAM cannot hold the 32-bit version at all, and the 16-bit version leaves almost no room for anything else. The 4-bit version fits with space to spare. This is why quantization (storing each weight in fewer bits) decides what fits on a 16 GB laptop.

The sizes count weights only. The KV cache and the runtime need more memory (lesson 19 completes the budget).

Example 15: accumulate a streamed reply

Example 10 showed the lines a streaming server sends. A client has to turn those lines back into one piece of text. The steps are:

1. Read the response line by line.
2. Skip blank lines and any line that does not start with data: .
3. Strip the data: prefix.
4. Stop at [DONE].
5. Parse the rest as JSON and append every delta.content to the text.

The program below does this on the real chunks of Example 10. The chunk with Hebrew text is left out, and long fields are removed. The ObjectMapper is the same Jackson class as in Example 13.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;

public class StreamParse {
    public static void main(String[] args) throws Exception {
        List<String> lines = List.of(
            "data: {\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null}}]}",
            "",
            "data: {\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\"bol\"}}]}",
            "",
            "data: {\"choices\":[{\"finish_reason\":null,\"index\":0,\"delta\":{\"content\":\" alley\"}}]}",
            "",
            "data: {\"choices\":[{\"finish_reason\":\"length\",\"index\":0,\"delta\":{}}],\"timings\":{\"cache_n\":19,\"prompt_n\":1,\"predicted_n\":4}}",
            "",
            "data: [DONE]");
        ObjectMapper mapper = new ObjectMapper();
        StringBuilder text = new StringBuilder();
        String finish = null;
        int pieces = 0;
        for (String line : lines) {
            if (!line.startsWith("data: ")) continue;
            String payload = line.substring(6);
            if (payload.equals("[DONE]")) break;
            JsonNode choice = mapper.readTree(payload).path("choices").path(0);
            JsonNode piece = choice.path("delta").path("content");
            if (piece.isTextual()) {
                text.append(piece.asText());
                pieces++;
            }
            if (choice.path("finish_reason").isTextual()) finish = choice.path("finish_reason").asText();
        }
        System.out.println("text = [" + text + "]");
        System.out.println("pieces = " + pieces);
        System.out.println("finish_reason = " + finish);
    }
}

The program prints three lines. The text is the joined pieces, the piece count is how many chunks had text, and the finish reason is the one the last chunk carried.

text = [bol alley]
pieces = 2
finish_reason = length

The first chunk has "content":null, so isTextual() is false and the program skips it. That is why pieces is 2 and not 3. The program joins the two pieces into bol alley, with the leading space of the second piece kept. A token piece often starts with a space, so clients must append pieces without trimming them. If they trimmed, the result would be bolalley.

The final chunk has an empty delta and a finish_reason, which the program records as length. Section 5 adds tool-call deltas and cancellation on top of this loop, in the streaming lesson later in this course.

Example 16: from this lesson to the loop command

Each step in this lesson was done by hand with a program someone else wrote. The course replaces those steps one by one with code the reader writes, and collects them in loop, an Ollama-style command-line tool built with picocli (a Java library for command-line programs). The table maps what happened in this lesson to the part of the course that explains and builds it.

The course has ten sections. They go from how a model works, to the hardware and files around it, to building tools on top:

Sections 1 to 3 are theory with small runnable Java programs. A reader who has never used a model locally needs no machine learning background: lessons 1 to 3 explain the ideas and vocabulary in plain language, and lesson 8 teaches the vectors, dot products, matrix products, and softmax that the later lessons use.

To stop the server from Example 7, press Ctrl+C in its terminal, or send SIGTERM (the standard "please stop" signal) to the process. A server left running keeps the whole model in memory.

Interview points

• A local LLM stack has five layers: the weights file (the learned numbers), the engine (the program that computes the next token), the server (an HTTP front end), the client (code that sends requests), and the agent (a loop that lets the model call tools). One product for each: a GGUF file, llama.cpp, llama-server, curl or a Java HttpClient, and an agent such as Aider.
• llama-server and vllm serve already expose an OpenAI-compatible HTTP API, so a client tool does not need its own wrapper server around them.
• Ollama splits into a CLI and API server and a separate runner process that loads the model.
• finish_reason is length when generation stops at max_tokens, and stop when the model emits its end-of-turn token or a stop string.
• usage.prompt_tokens counts the rendered prompt, which includes role markers and any default system message from the chat template, so it is larger than the token count of the user text.
• Prefill (prompt eval) processes many tokens per step. Decode (eval) produces one token per step and reads all weights each time, so it is limited by memory speed.
• With temperature above 0, the same seed on the same machine and settings gives the same text. At temperature 0, the seed has no effect, because the top token is always chosen.
• Weight file size is about parameters times bytes per weight: 7 billion parameters at 16 bits is 14 GB, and at 4 bits about 3.5 GB plus a small overhead.
• Binding a server to 127.0.0.1 limits it to programs on the same machine. Binding to 0.0.0.0 without an API key exposes it to the network.
• A repeated request with the same prefix reports cache_n greater than 0, which means the server reused saved state for that prefix instead of processing it again.

Summary

• A local coding assistant has five layers: weights file, engine, server, client, agent.
• llama.cpp is an engine with a server and command-line tools. vLLM is a GPU serving engine. Ollama and LM Studio manage models and run engines.
• This course pins llama.cpp commit 0c1e57098. llama-cli --version prints 0.5.0-dev (build 1, commit 0c1e570) for that build.
• llama-completion -no-cnv continues a plain prompt. llama-cli opens a chat when the model has a chat template.
• The default sampler chain is penalties, dry, top-n-sigma, top-k, typical, top-p, min-p, xtc, temperature, then a random draw. Optional steps are skipped unless enabled.
• llama-completion prints a timing summary with separate prompt-eval (prefill) and eval (decode) lines. llama-cli prints prompt and generation speeds after each reply.
• llama-server serves /health, /v1/models, and /v1/chat/completions on the address given by --host and --port. The default port is 8080.
• A chat completion response carries choices, finish_reason, usage, and a llama.cpp timings object.
• With "stream": true, the reply arrives as data: lines, each with a delta, ended by data: [DONE].
• temperature and seed control randomness. Temperature 0 always picks the highest-scoring token.
• A second request with a shared prefix reuses the saved KV state, shown as cache_n in timings.
• The tiny model used for the captures has random weights (80,611,584 parameters, F32, 322,446,336 bytes), so its text is meaningless and its logs are real.
• File size equals parameters times bytes per weight, which is why quantization decides what fits in memory.
• loop automates each manual step: pull, list, show, fit, serve, run, chat, bench, agent, eval.