The Engine Landscape and How Ollama Works

A program that generates text from a model is called an inference engine. The two previous lessons covered llama.cpp and vLLM in depth. Those are not the only choices. A tool that runs models locally will meet several more, and a developer needs a way to tell them apart and pick one.

This lesson does three things. First, it opens up Ollama, the tool that loop (the command-line tool built in this course) resembles most, using the Ollama v0.35.1 source. The point is to see what happens between typing ollama run and receiving the first generated token. Second, it describes the other engines briefly: SGLang, LM Studio, MLX, TensorRT-LLM, TGI, ExLlama, ONNX Runtime GenAI, and the engines written for Java. Third, it shows the three ways a Java program can use an engine: manage it as a subprocess, connect to one that is already running, or run it inside the JVM. It ends with the criteria loop uses to choose.

After this lesson you can explain what Ollama does for each request, pick an engine for a set of requirements, and start an engine from Java safely. The table lists the topics and the example for each.

Facts about Ollama come from the v0.35.1 source and facts about SGLang from the 0.5.21 source, with file paths given. Engines whose source is not in the reference set are described briefly from their own documentation, without flags or version-specific claims.

Log lines labelled as real come from the pinned llama.cpp build (0c1e57098) running the course's tiny model. That model has random weights, so its text is meaningless, but its logs and memory numbers are real.

Example 1: Ollama's architecture and the llama-server command it builds

Ollama is one Go program with two roles. ollama serve runs an HTTP server on 127.0.0.1:11434 (envconfig.Host, which OLLAMA_HOST can change). Every other command (ollama run, ollama pull, ollama ps) is a client that sends HTTP requests to that server.

The server does not compute tokens itself. For a GGUF model it starts the upstream llama.cpp llama-server binary as a child process (NewLlamaServer in llm/server.go: "All GGUF models are served via the upstream llama-server subprocess"). The llama.cpp version is pinned to tag b11232 in the file LLAMA_CPP_VERSION. For a safetensors model it starts ollama runner --model <name> --port <port>, an MLX-based runner (mlxrunner/client.go).

The program below builds the argument list the way startLlamaServer in llm/llama_server.go does. It builds two lists: one for default options on a CPU-only machine, and one with custom options. Compare the two lines to see which setting changes which flag.

import java.util.ArrayList;
import java.util.List;

public class OllamaLaunchArgs {
    record Options(int numCtx, int numBatch, int numGpu, int numThread, String kvCacheType) {}

    static List<String> llamaServerArgs(String modelPath, int port, Options o, int numParallel) {
        List<String> p = new ArrayList<>(List.of("--model", modelPath, "--port", Integer.toString(port),
                "--host", "127.0.0.1", "--no-webui", "--offline",
                "-c", Integer.toString(o.numCtx() * numParallel), "-np", Integer.toString(numParallel)));
        p.addAll(List.of("--log-verbosity", "4", "--no-log-prefix", "--no-log-timestamps"));
        if (!o.kvCacheType().isEmpty()) p.addAll(List.of("--cache-type-k", o.kvCacheType(), "--cache-type-v", o.kvCacheType()));
        p.addAll(List.of("--flash-attn", "auto"));                  // no GPUs: every device supports it
        if (o.numBatch() > 0) p.addAll(List.of("-b", Integer.toString(o.numBatch()), "-ub", Integer.toString(o.numBatch())));
        if (o.numGpu() >= 0) p.addAll(List.of("-ngl", Integer.toString(o.numGpu())));   // -1: let llama-server decide
        if (o.numThread() > 0) p.addAll(List.of("-t", Integer.toString(o.numThread())));
        return p;
    }

    public static void main(String[] args) {
        Options defaults = new Options(4096, 512, -1, 0, "");      // api.DefaultOptions, 4K context tier
        System.out.println(String.join(" ", llamaServerArgs("/models/blobs/sha256-6a07", 41237, defaults, 1)));
        Options custom = new Options(8192, 512, 0, 8, "q8_0");      // num_gpu 0, num_thread 8, OLLAMA_KV_CACHE_TYPE
        System.out.println(String.join(" ", llamaServerArgs("/models/blobs/sha256-6a07", 41237, custom, 2)));
    }
}

--model /models/blobs/sha256-6a07 --port 41237 --host 127.0.0.1 --no-webui --offline -c 4096 -np 1 --log-verbosity 4 --no-log-prefix --no-log-timestamps --flash-attn auto -b 512 -ub 512
--model /models/blobs/sha256-6a07 --port 41237 --host 127.0.0.1 --no-webui --offline -c 16384 -np 2 --log-verbosity 4 --no-log-prefix --no-log-timestamps --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn auto -b 512 -ub 512 -ngl 0 -t 8

The first line is what Ollama starts for a request with default options. Taking the flags in order:

• --model is a blob file in Ollama's store (Example 3).
• --port is a free port that Ollama finds by opening a listener on port 0 and closing it. Port 0 asks the operating system to choose any free port. The same ServerSocket(0) method appears in Workshop: loop serve, run, and chat.
• --host 127.0.0.1 keeps the runner private, so only the Ollama server can talk to it.
• --no-webui and --offline turn off llama-server's browser UI and its Hugging Face downloads.
• -c is the context size (how many tokens the model can hold) per request, times the number of parallel slots.
• --log-verbosity 4 --no-log-prefix --no-log-timestamps make llama-server print its memory lines in a fixed shape, because Ollama parses them (Example 5).
• -ngl (the number of layers to put on the GPU) is absent. num_gpu defaults to -1, which means "let llama-server decide", and llama-server's own --fit logic places the layers (Tuning llama.cpp).

The second line shows the options that change the command. num_ctx 8192 with OLLAMA_NUM_PARALLEL=2 gives -c 16384 -np 2, because each of the 2 slots gets 8,192 tokens. num_gpu 0 becomes -ngl 0 (CPU only), num_thread becomes -t, and OLLAMA_KV_CACHE_TYPE=q8_0 sets both cache types.

The program leaves out flags that startLlamaServer adds only in specific cases:

• --no-jinja --chat-template chatml, when Ollama renders the prompt itself.
• --load-mode none, when the request sets use_mmap to false.
• --load-mode dio, for integrated CUDA or ROCm GPUs on Linux.
• --context-shift, --mmproj (for vision models), and the --spec-type flags (for draft models).

The same flags work with the pinned llama.cpp build. To check, start it directly with the tiny model:

llama-server --model tiny-random-qwen2vocab-f32.gguf --port 18433 --host 127.0.0.1 --no-webui --offline \
  -c 4096 -np 1 --log-verbosity 4 --no-log-prefix --no-log-timestamps --flash-attn auto -b 512 -ub 512

The log shows what llama-server did with those flags. Real output (selected lines; pinned llama.cpp build, tiny random-weight model):

load_tensors:   CPU_Mapped model buffer size =   307.51 MiB
llama_context: n_ctx                 = 4096
llama_context: n_batch               = 512
llama_context: n_ubatch              = 512
llama_context: flash_attn            = auto
llama_kv_cache:        CPU KV buffer size =     4.00 MiB
srv    load_model: initializing, n_slots = 1, n_ctx_slot = 4096, kv_unified = 'false'
srv  llama_server: listening on http://127.0.0.1:18433

The log confirms the flags took effect: the context is 4,096, the batch size is 512, there is one slot, and the server listens on the chosen address. The memory lines (307.51 MiB of weights, 4.00 MiB of KV cache) are the ones Ollama parses in Example 5.

Putting it together, the whole path for ollama run qwen2.5-coder:7b is:

1. The CLI sends POST /api/chat to 127.0.0.1:11434.
2. The server resolves the name to a manifest and a GGUF blob.
3. The scheduler (Example 2) finds or starts a llama-server child for that blob and waits for its /health to return {"status":"ok"}.
4. The server sends the chat to the child. Models with Ollama's own renderers or a Go template get a prompt that Ollama renders itself and sends to llama-server's /completion. Other GGUF chat models get messages sent to /v1/chat/completions, so llama-server applies the chat template. The header comment of llm/llama_server.go describes both paths.
5. The server converts the child's SSE stream (server-sent events) into Ollama's NDJSON stream (one JSON object per line) for the CLI.

The "wrapper server" is Ollama itself, and the engine is an ordinary llama.cpp process.

Example 2: Ollama's scheduler: loading, unloading, keep_alive, concurrency

Loading a model takes seconds and uses a lot of memory, so Ollama cannot load a model for every request or keep every model loaded forever. The scheduler in server/sched.go decides which models stay in memory. Each loaded model is a runner. A runner has a reference count (the number of requests in progress) and a session duration (keep_alive, default 5 minutes from envconfig.KeepAlive).

The rules are:

• When the count drops to zero, a timer starts. When it fires, the runner is unloaded.
• When a new model needs room and the limit is reached, the scheduler unloads one runner. The limit is OLLAMA_MAX_LOADED_MODELS, whose default 0 the scheduler replaces with 3 per GPU (defaultModelsPerGPU).
• The scheduler prefers an idle runner, and it sorts by keep-alive duration (findRunnerToUnload).

The program simulates these rules with a fake clock, so the times in the output are seconds on that clock. Follow each line and ask why the model was loaded or unloaded at that moment.

import java.util.ArrayList;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

public class OllamaScheduler {
    static class Runner {
        final String model; int refCount; long keepAlive; Long expiresAt;
        Runner(String model, long keepAlive) { this.model = model; this.keepAlive = keepAlive; }
    }
    static final Map<String, Runner> loaded = new LinkedHashMap<>();
    static final int MAX_RUNNERS = 3 * Math.max(1, 1);              // defaultModelsPerGPU * GPU count

    static void expire(long now) {
        loaded.values().removeIf(r -> {
            boolean due = r.refCount == 0 && r.expiresAt != null && r.expiresAt <= now;
            if (due) System.out.printf("t=%3ds unload %s (keep_alive elapsed)%n", now, r.model);
            return due;
        });
    }

    static void start(long now, String model, long keepAlive) {
        expire(now);
        Runner r = loaded.get(model);
        if (r == null) {
            if (loaded.size() >= MAX_RUNNERS) {
                List<Runner> list = new ArrayList<>(loaded.values());
                list.sort(Comparator.comparingLong((Runner x) -> x.keepAlive).thenComparing(x -> x.model));
                Runner victim = list.stream().filter(x -> x.refCount == 0).findFirst().orElse(list.get(0));
                loaded.remove(victim.model);
                System.out.printf("t=%3ds unload %s (make room)%n", now, victim.model);
            }
            r = new Runner(model, keepAlive);
            loaded.put(model, r);
            System.out.printf("t=%3ds load   %s%n", now, model);
        }
        r.refCount++;
        r.keepAlive = keepAlive;                                    // the latest request's keep_alive wins
        r.expiresAt = null;
    }

    static void finish(long now, String model) {
        Runner r = loaded.get(model);
        if (--r.refCount == 0) {
            if (r.keepAlive <= 0) { loaded.remove(model); System.out.printf("t=%3ds unload %s (keep_alive 0)%n", now, model); }
            else r.expiresAt = now + r.keepAlive;
        }
    }

    public static void main(String[] args) {
        start(0, "coder:7b", 300);  finish(20, "coder:7b");
        start(30, "embed", 0);      finish(31, "embed");
        start(40, "chat:8b", 600);  finish(50, "chat:8b");
        start(60, "vision", 300);   start(70, "draft:1b", 300);
        finish(90, "vision");       finish(95, "draft:1b");
        start(400, "coder:7b", 300);
        System.out.println("loaded at t=400s: " + loaded.keySet());
    }
}

t=  0s load   coder:7b
t= 30s load   embed
t= 31s unload embed (keep_alive 0)
t= 40s load   chat:8b
t= 60s load   vision
t= 70s unload coder:7b (make room)
t= 70s load   draft:1b
t=400s unload vision (keep_alive elapsed)
t=400s unload draft:1b (keep_alive elapsed)
t=400s load   coder:7b
loaded at t=400s: [chat:8b, coder:7b]

Walking through the output:

• At 0 s the coding model loads. It finishes at 20 s and starts its 300-second timer.
• The embedding request at 30 s sets keep_alive to 0, so its runner unloads as soon as the request finishes. Ollama does the same for "keep_alive": 0.
• At 60 s three models are loaded (coder:7b, chat:8b, vision).
• The request for draft:1b at 70 s would make four, which exceeds the limit of 3, so the scheduler picks a victim. It sorts the runners by keep_alive and then by name: coder:7b (300 s), vision (300 s), chat:8b (600 s). (ByDurationAndName in the real scheduler breaks ties by the model's file path.) It takes the first idle one in that order. coder:7b is idle, since its request ended at 20 s, and it comes first, so it is unloaded. vision would have been skipped anyway, because its request runs until 90 s.
• At 400 s the timers of vision and draft:1b have elapsed (set at 90 s and 95 s, plus 300 s), so they unload before coder:7b loads again. chat:8b stays, because its 600-second timer runs until 650 s.

The real scheduler adds parts that the simulation leaves out. The table lists them:

ollama ps (route GET /api/ps) lists the loaded runners with size, size_vram, context_length, and expires_at, sorted by expiry. It is the quickest way to see what the scheduler currently holds.

Example 3: Ollama's model store: names, manifests, blobs, and Modelfile

Ollama has to store large model files and download them from a registry (an online server that hosts models). It uses a layout borrowed from container images, called OCI-style. A manifest is a small JSON file that lists the layers of a model. Each layer is a blob (a binary file) identified by its SHA-256 digest (a fingerprint of the file's content) and a media type. Blobs live once in blobs/ and are shared between models, so two models that use the same weights store them once.

The program parses model names with the defaults from types/model/name.go, computes the file and registry paths from manifest/paths.go and server/images.go, and reads a manifest. Look at how a short name like llama3.2 expands into a full path.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

public class OllamaStore {
    record Name(String host, String namespace, String model, String tag) {
        static Name parse(String s) {                       // ParseName merged with DefaultName
            String tag = "latest";
            if (s.lastIndexOf(':') > s.lastIndexOf('/')) { tag = s.substring(s.lastIndexOf(':') + 1); s = s.substring(0, s.lastIndexOf(':')); }
            String[] parts = s.split("/");
            String model = parts[parts.length - 1];
            String ns = parts.length >= 2 ? parts[parts.length - 2] : "library";
            String host = parts.length >= 3 ? parts[parts.length - 3] : "registry.ollama.ai";
            return new Name(host, ns, model, tag);
        }
        String manifestPath() { return "~/.ollama/models/manifests/" + host + "/" + namespace + "/" + model + "/" + tag; }
        String manifestUrl() { return "https://" + host + "/v2/" + namespace + "/" + model + "/manifests/" + tag; }
    }

    public static void main(String[] args) throws Exception {
        for (String s : new String[] {"qwen2.5-coder:7b", "llama3.2", "myteam/coder:q4", "hf.co/unsloth/Qwen3-8B-GGUF:Q4_K_M"}) {
            Name n = Name.parse(s);
            System.out.println(s + " -> " + n.manifestPath());
        }
        System.out.println(Name.parse("qwen2.5-coder:7b").manifestUrl());
        JsonNode m = new ObjectMapper().readTree("""
            {"schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
             "config": {"mediaType": "application/vnd.docker.container.image.v1+json", "digest": "sha256:d9bb", "size": 487},
             "layers": [
               {"mediaType": "application/vnd.ollama.image.model", "digest": "sha256:6a07", "size": 4683087332},
               {"mediaType": "application/vnd.ollama.image.system", "digest": "sha256:66b9", "size": 68},
               {"mediaType": "application/vnd.ollama.image.template", "digest": "sha256:1e65", "size": 1626},
               {"mediaType": "application/vnd.ollama.image.license", "digest": "sha256:832d", "size": 11343}]}""");
        long total = m.get("config").get("size").asLong();
        for (JsonNode layer : m.get("layers")) {
            total += layer.get("size").asLong();
            String type = layer.get("mediaType").asText().replace("application/vnd.ollama.image.", "");
            System.out.printf("%-9s blobs/%s%n", type, layer.get("digest").asText().replace(':', '-'));
        }
        System.out.println("manifest size (all layers + config): " + total + " bytes");
    }
}

qwen2.5-coder:7b -> ~/.ollama/models/manifests/registry.ollama.ai/library/qwen2.5-coder/7b
llama3.2 -> ~/.ollama/models/manifests/registry.ollama.ai/library/llama3.2/latest
myteam/coder:q4 -> ~/.ollama/models/manifests/registry.ollama.ai/myteam/coder/q4
hf.co/unsloth/Qwen3-8B-GGUF:Q4_K_M -> ~/.ollama/models/manifests/hf.co/unsloth/Qwen3-8B-GGUF/Q4_K_M
https://registry.ollama.ai/v2/library/qwen2.5-coder/manifests/7b
model     blobs/sha256-6a07
system    blobs/sha256-66b9
template  blobs/sha256-1e65
license   blobs/sha256-832d
manifest size (all layers + config): 4683100856 bytes

A bare name gets the host registry.ollama.ai, the namespace library, and the tag latest, so llama3.2 is registry.ollama.ai/library/llama3.2/latest on disk. A name with one slash sets the namespace. A name with two slashes sets the host, which is how Hugging Face GGUF repositories appear (hf.co/<user>/<repo>:<quant>).

The manifest URL follows the OCI distribution layout: /v2/<namespace>/<model>/manifests/<tag>, fetched with the header Accept: application/vnd.docker.distribution.manifest.v2+json. Blobs come from /v2/<namespace>/<model>/blobs/<digest>.

Each layer's media type says what the blob is. In the example it is the GGUF weights (model), the system prompt (system), the chat template (template, in Go template syntax), or the license. Other models also have params (default options as JSON), projector (a vision projector GGUF), adapter (LoRA), and messages.

On disk the digest's colon becomes a hyphen, so sha256:6a07... is the file blobs/sha256-6a07.... The digests and sizes in the program are shortened and invented, and real digests have 64 hexadecimal digits. The printed size is the sum of all layers plus the config: 4,683,087,332 + 68 + 1,626 + 11,343 + 487 = 4,683,100,856 bytes. This is the number ollama list reports.

A Modelfile is a short recipe that builds a new model from an existing one plus overrides. parser/parser.go accepts the commands FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE, REQUIRES, DRAFT, RENDERER, PARSER, and CAPABILITY. ollama create stores each part as a new layer and reuses the weight blob unchanged, so the new model costs almost no disk space:

cat > Modelfile <<'EOF'
FROM qwen2.5-coder:7b
PARAMETER num_ctx 16384
PARAMETER temperature 0.2
SYSTEM You are a careful Java code reviewer.
EOF
ollama create java-reviewer -f Modelfile

loop's registry from the model-pull workshop uses the same idea: a manifest per model and a shared blob store, with Hugging Face as the only source.

Example 4: Ollama's API: /api/pull, /api/chat, and OpenAI compatibility

Ollama's native API streams NDJSON (newline-delimited JSON): one JSON object per line, with no data: prefix. The content type is application/x-ndjson in server/routes.go. A client reads one line at a time and handles each object as it arrives.

The program parses a /api/pull progress stream and the last object of an /api/chat stream, using the field names of api.ProgressResponse, api.ChatResponse, and api.Metrics. Look at how the progress percentage is computed, and at the two speeds in the last line.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Locale;

public class OllamaNdjson {
    public static void main(String[] args) throws Exception {
        ObjectMapper json = new ObjectMapper();
        String pull = """
            {"status":"pulling manifest"}
            {"status":"pulling 6a0746a1ec1a","digest":"sha256:6a0746a1ec1a","total":4683087332,"completed":1170771833}
            {"status":"pulling 6a0746a1ec1a","digest":"sha256:6a0746a1ec1a","total":4683087332,"completed":4683087332}
            {"status":"verifying sha256 digest"}
            {"status":"writing manifest"}
            {"status":"success"}""";
        for (String line : pull.lines().toList()) {
            JsonNode p = json.readTree(line);
            if (p.has("total")) System.out.printf(Locale.ROOT, "%s %5.1f%%%n", p.get("status").asText(),
                    100.0 * p.get("completed").asLong() / p.get("total").asLong());
            else System.out.println(p.get("status").asText());
        }

        JsonNode done = json.readTree("""
            {"model":"qwen2.5-coder:7b","created_at":"2026-10-03T12:00:00Z","message":{"role":"assistant","content":""},
             "done":true,"done_reason":"stop","total_duration":2350000000,"load_duration":41000000,
             "prompt_eval_count":38,"prompt_eval_duration":120000000,"eval_count":96,"eval_duration":2100000000}""");
        double ns = 1e9;
        System.out.printf(Locale.ROOT, "done_reason=%s prompt %d tokens at %.1f tok/s, output %d tokens at %.1f tok/s%n",
                done.get("done_reason").asText(),
                done.get("prompt_eval_count").asInt(), done.get("prompt_eval_count").asInt() / (done.get("prompt_eval_duration").asLong() / ns),
                done.get("eval_count").asInt(), done.get("eval_count").asInt() / (done.get("eval_duration").asLong() / ns));
    }
}

pulling manifest
pulling 6a0746a1ec1a  25.0%
pulling 6a0746a1ec1a 100.0%
verifying sha256 digest
writing manifest
success
done_reason=stop prompt 38 tokens at 316.7 tok/s, output 96 tokens at 45.7 tok/s

The pull stream follows PullModel in server/images.go. It sends pulling manifest, then one pulling <12 hex digits> line per blob with total and completed byte counts (repeated as the download progresses), then verifying sha256 digest, writing manifest, and success. A client draws a progress bar from completed / total for each digest. In the program, 1,170,771,833 of 4,683,087,332 bytes is 25.0%. The values here are invented.

The chat stream ends with an object where done is true. It carries the metrics, and all durations are in nanoseconds (billionths of a second): total_duration, load_duration (time spent loading the model for this request), prompt_eval_count and prompt_eval_duration (prefill, the work of reading the prompt), and eval_count and eval_duration (generation).

Prompt speed is 38 tokens in 0.12 s, which is 316.7 tokens per second. Generation speed is 96 tokens in 2.1 s, which is 45.7 tokens per second. These are the same two numbers llama.cpp reports as prompt and generation speed, because the work happens in llama-server.

Two request forms control loading without generating any text. A /api/chat request with an empty messages array loads the model and returns "done_reason": "load". The same request with "keep_alive": 0 unloads it and returns "done_reason": "unload" (both branches are in ChatHandler). Options such as num_ctx, temperature, and num_predict go in an options object. Their defaults are in api.DefaultOptions: temperature 0.8, top_k 40, top_p 0.9, num_batch 512, and num_predict -1.

The same server also answers OpenAI-style routes, registered in server/routes.go: POST /v1/chat/completions, POST /v1/completions, POST /v1/embeddings, GET /v1/models, GET /v1/models/:model, POST /v1/responses, and an Anthropic-style POST /v1/messages. Middleware converts each request into the native handler's form and converts the answer back. A client written for llama-server and vLLM therefore works with Ollama at http://127.0.0.1:11434/v1, which is how The Chat Completions Protocol and Streaming in Java treats it.

Example 5: Ollama's GPU memory estimate and layer placement

Ollama needs two memory numbers. The first is a prediction made before starting a runner, used to decide whether another model fits beside the ones already loaded. The second is the real use measured after the runner starts, used by ollama ps and by later decisions.

The prediction is PredictServerVRAM: the GGUF file size plus a KV cache estimate of 2 * layers * kvHeads * headDim * numCtx * 2 bytes (the final 2 is for F16, 2 bytes per value). The real use comes from parsing llama-server's log lines with regular expressions copied from memoryParsingWriter.

The program computes both for the tiny model, using the real log lines from Example 1. It also shows the default context length Ollama picks for different amounts of GPU memory.

import java.util.List;
import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class OllamaMemory {
    static final double MIB = 1 << 20;

    static long predict(long fileBytes, long layers, long kvHeads, long embd, long heads, long numCtx) {
        long headDim = embd / heads;
        return fileBytes + 2 * layers * kvHeads * headDim * numCtx * 2;
    }

    public static void main(String[] args) {
        long tiny = predict(328_377_376L, 4, 2, 256, 8, 4096);
        System.out.printf(Locale.ROOT, "predicted tiny model at num_ctx 4096: %.2f MiB%n", tiny / MIB);
        long llama8b = predict(4_920_000_000L, 32, 8, 4096, 32, 4096);
        System.out.printf(Locale.ROOT, "predicted 8B Q4_K_M at num_ctx 4096: %.2f MiB%n", llama8b / MIB);

        Pattern buffer = Pattern.compile("([A-Za-z_][A-Za-z0-9_]*):\\s+(\\S+)\\s+(model|KV|compute|output|RS)\\s+buffer size\\s*=\\s*([\\d.]+)\\s*MiB");
        List<String> log = List.of(
                "load_tensors:   CPU_Mapped model buffer size =   307.51 MiB",
                "llama_context:        CPU  output buffer size =     0.58 MiB",
                "llama_kv_cache:        CPU KV buffer size =     4.00 MiB",
                "sched_reserve:        CPU compute buffer size =    10.14 MiB");
        double total = 0, gpu = 0;
        for (String line : log) {
            Matcher m = buffer.matcher(line);
            if (!m.find()) continue;
            String backend = m.group(2);
            double mib = Double.parseDouble(m.group(4));
            boolean onGpu = !(backend.equals("CPU") || backend.startsWith("CPU_") || backend.endsWith("_Host"));
            total += mib;
            if (onGpu) gpu += mib;
            System.out.printf(Locale.ROOT, "  %-14s %-10s %-7s %8.2f MiB  gpu=%b%n", m.group(1), backend, m.group(3), mib, onGpu);
        }
        System.out.printf(Locale.ROOT, "parsed total %.2f MiB, VRAM %.2f MiB%n", total, gpu);
        for (double vramGiB : new double[] {0, 12, 24, 48}) {
            int numCtx = vramGiB >= 47 ? 262144 : vramGiB >= 23 ? 32768 : 4096;
            System.out.printf(Locale.ROOT, "total VRAM %4.0f GiB -> default num_ctx %d%n", vramGiB, numCtx);
        }
    }
}

predicted tiny model at num_ctx 4096: 317.17 MiB
predicted 8B Q4_K_M at num_ctx 4096: 5204.08 MiB
  load_tensors   CPU_Mapped model     307.51 MiB  gpu=false
  llama_context  CPU        output      0.58 MiB  gpu=false
  llama_kv_cache CPU        KV          4.00 MiB  gpu=false
  sched_reserve  CPU        compute    10.14 MiB  gpu=false
parsed total 322.23 MiB, VRAM 0.00 MiB
total VRAM    0 GiB -> default num_ctx 4096
total VRAM   12 GiB -> default num_ctx 4096
total VRAM   24 GiB -> default num_ctx 32768
total VRAM   48 GiB -> default num_ctx 262144

The prediction for the tiny model is its file size, 313.17 MiB, plus 4 MiB of KV cache. The 4 MiB comes from 4 layers, 2 KV heads, head size 32 (256 dimensions over 8 heads), 4,096 positions, K and V, and 2 bytes each. The real log shows exactly 4.00 MiB for the KV buffer, so the formula matches llama.cpp for this model.

For an 8B model in Q4_K_M (the 4.92 GB file size is a representative value), the same 4K context adds 512 MiB. The source comment calls the prediction "intentionally conservative": "it overestimates to avoid VRAM contention". For example, the whole file counts as weights, although the token embedding stays in system memory (Example 5 of llama.cpp Architecture).

The parsed lines are the four buffers of the real run. Each line names the component (the function that printed it), the backend buffer type, and the kind of buffer. CPU_Mapped is the memory-mapped GGUF file.

Ollama counts a buffer as system memory when its name is CPU or BLAS, starts with CPU_, or ends with _Host. It counts everything else (CUDA0, ROCm0, MTL0_Mapped) as GPU memory (isGPUBuffer). On this CPU-only run the total is 322.23 MiB and the GPU share is zero. Ollama also parses offloaded N/M layers to GPU and the free memory per device from the same log, which is how it learns where llama-server placed the layers.

The last four lines are the default context length from server/routes.go. When neither the request nor OLLAMA_CONTEXT_LENGTH sets num_ctx, the server chooses it from the total GPU memory at startup: 4,096 tokens below 23 GiB, 32,768 from 23 GiB, and 262,144 from 47 GiB. The thresholds are slightly below 24 and 48 GiB, according to the source comment, to allow for small differences in reported memory. The idea is that a bigger GPU can afford a bigger KV cache.

Placement itself is delegated to llama-server. Ollama picks which GPUs a runner may use (selectLlamaServerPlacement: the best single GPU that fits the prediction, otherwise a group of GPUs). It sets visible-device variables for the child (CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, GGML_VK_VISIBLE_DEVICES in ml/device.go). It leaves the layer count to llama-server's --fit unless the user set num_gpu. If the load fails with an out-of-memory error, the scheduler can retry with a smaller automatic num_ctx (reduceAutoNumCtxForLoadOOM, stepping down to 32,768 and then 4,096).

Example 6: LM Studio and MLX in short

Some tools are applications for people rather than libraries for programmers. LM Studio is a desktop application (not open source) for downloading and running models. It has a chat window and a local server that speaks the OpenAI API. Its documentation describes two engines: llama.cpp for GGUF models and, on Apple Silicon, MLX for MLX-format models. It lists port 1234 as the server's default.

MLX is Apple's array framework for Apple Silicon. It uses the unified memory that the CPU and GPU share, so a model does not have to be copied between them. The mlx-lm Python package runs and serves language models with it. Ollama v0.35.1 uses MLX as well: its scheduler starts ollama runner with the MLX runtime for safetensors models (mlxrunner.NewClient in server/sched.go). Its pull refuses such models when MLX is unavailable (this model requires MLX support, but the MLX runtime is not available).

A client does not need to know which of these servers is running. Asking GET /v1/models identifies an OpenAI-compatible server and its model names. The program probes three fake servers that run inside the program on ports chosen by the operating system. Look at how the probe tells them apart.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;

public class ProbeServers {
    static HttpServer fake(String path, int status, String body) throws Exception {
        HttpServer s = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        s.createContext("/", ex -> {
            byte[] b = body.getBytes(StandardCharsets.UTF_8);
            int code = ex.getRequestURI().getPath().equals(path) ? status : 404;
            ex.sendResponseHeaders(code, code == 404 ? -1 : b.length);
            if (code != 404) ex.getResponseBody().write(b);
            ex.close();
        });
        s.start();
        return s;
    }

    static String probe(HttpClient http, int port) throws Exception {
        HttpResponse<String> r = http.send(HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + port + "/v1/models")).build(),
                HttpResponse.BodyHandlers.ofString());
        if (r.statusCode() != 200) return "not OpenAI-compatible (status " + r.statusCode() + ")";
        StringBuilder ids = new StringBuilder();
        for (JsonNode m : new ObjectMapper().readTree(r.body()).path("data")) ids.append(m.get("id").asText()).append(' ');
        return "OpenAI-compatible, models: " + ids.toString().strip();
    }

    public static void main(String[] args) throws Exception {
        HttpServer a = fake("/v1/models", 200, "{\"object\":\"list\",\"data\":[{\"id\":\"qwen2.5-coder-7b-instruct\",\"object\":\"model\"}]}");
        HttpServer b = fake("/v1/models", 200, "{\"object\":\"list\",\"data\":[{\"id\":\"mlx-community/Qwen3-8B-4bit\",\"object\":\"model\"}]}");
        HttpServer c = fake("/api/tags", 200, "{\"models\":[]}");
        HttpClient http = HttpClient.newHttpClient();
        System.out.println("server A: " + probe(http, a.getAddress().getPort()));
        System.out.println("server B: " + probe(http, b.getAddress().getPort()));
        System.out.println("server C: " + probe(http, c.getAddress().getPort()));
        a.stop(0); b.stop(0); c.stop(0);
    }
}

server A: OpenAI-compatible, models: qwen2.5-coder-7b-instruct
server B: OpenAI-compatible, models: mlx-community/Qwen3-8B-4bit
server C: not OpenAI-compatible (status 404)

Servers A and B answer /v1/models with the OpenAI list shape (object: "list", and data with an id per model), so a client can use them with no other knowledge. The model ids show the naming style each tool uses. Server C answers only Ollama's native /api/tags, so the probe reports it as not OpenAI-compatible. Real Ollama answers both routes.

The choice between these tools is made on the desktop side (which application the user runs), not in the client, as long as the client sticks to the OpenAI API.

Example 7: SGLang: RadixAttention, scheduler, and structured outputs

SGLang is a serving engine in the same class as vLLM: Python, CUDA kernels, continuous batching, a paged KV cache, and an OpenAI-compatible server. In 0.5.21 it starts with sglang serve <model> (or python -m sglang.launch_server --model-path <model>) and listens on 127.0.0.1:30000 by default (host and port in srt/arg_groups/fields/serving.py).

Its best-known part is RadixAttention. Many requests start with the same tokens, such as the same system prompt. If the engine remembers the KV cache of finished requests, a new request can skip recomputing the shared start. SGLang keeps that memory in a radix tree, a tree where each step down the tree is one token (srt/mem_cache/radix_cache.py). Any later request that begins with the same tokens walks down the tree and reuses them.

The program builds a small radix tree of token sequences and reports the longest cached prefix for each new request. The tokens are plain numbers here. Look at how many tokens each request has to compute.

import java.util.HashMap;
import java.util.Map;

public class RadixPrefix {
    static class Node { final Map<Integer, Node> children = new HashMap<>(); }
    static final Node root = new Node();

    static int matchPrefix(int[] tokens) {
        Node n = root;
        int i = 0;
        while (i < tokens.length && n.children.containsKey(tokens[i])) n = n.children.get(tokens[i++]);
        return i;
    }

    static void insert(int[] tokens) {
        Node n = root;
        for (int t : tokens) n = n.children.computeIfAbsent(t, k -> new Node());
    }

    static int[] seq(int... parts) { return parts; }

    public static void main(String[] args) {
        int[][] requests = {
            seq(1, 2, 3, 4, 5, 6, 10, 11),          // system prompt (1..6) + question A
            seq(1, 2, 3, 4, 5, 6, 20, 21, 22),      // same system prompt + question B
            seq(1, 2, 3, 4, 5, 6, 10, 11, 30, 31),  // turn 2 of conversation A
            seq(1, 2, 9, 9)                          // different prompt sharing two tokens
        };
        char name = 'a';
        for (int[] r : requests) {
            int hit = matchPrefix(r);
            System.out.printf("request %c: %2d tokens, cached prefix %d, computed %d%n", name++, r.length, hit, r.length - hit);
            insert(r);
        }
    }
}

request a:  8 tokens, cached prefix 0, computed 8
request b:  9 tokens, cached prefix 6, computed 3
request c: 10 tokens, cached prefix 8, computed 2
request d:  4 tokens, cached prefix 2, computed 2

The first request finds nothing. The second shares the six system-prompt tokens, so only 3 of its 9 tokens are computed. The third is the next turn of the first conversation, so its whole earlier history (8 tokens) is cached, and only the new user message is computed. The fourth shares only two tokens, and the tree matches exactly two.

This shows a difference from vLLM. A radix tree matches at token granularity, while vLLM's hash chain in Serving with vLLM matches whole 16-token blocks. The SGLang docstring for match_prefix states that with --page-size above 1, the key is truncated to a multiple of the page size before matching, so the same granularity trade-off applies there.

The real tree does more than the simple version above:

• It compresses chains of single children into one node that holds a run of tokens.
• It splits nodes when a match ends inside a run.
• It keys entries by an extra key (a LoRA adapter, a salt) so that different keys never share nodes.
• It evicts leaves with a policy chosen by --radix-eviction-policy (default lru; also lfu, slru, priority, tlru).

Other parts of SGLang match topics from earlier lessons. The table lists them:

For a single user on one GPU, SGLang and vLLM offer the same features. The differences show under many concurrent requests that share prefixes, which is where the radix tree and the lpm policy apply.

Example 8: TensorRT-LLM, TGI, ExLlamaV2 and V3 in short

Three more engines appear in deployment guides. Their source is not in this course's reference set, so they are described only by what their projects document.

• TensorRT-LLM is NVIDIA's library that optimizes a model for NVIDIA GPUs with fused kernels (several steps combined into one GPU program) and in-flight batching. It runs only on NVIDIA hardware and documents a trtllm-serve command with an OpenAI-compatible API.
• TGI (Text Generation Inference) is Hugging Face's serving stack: a router written in Rust in front of model servers, loading safetensors models from the Hub.
• ExLlamaV2 and ExLlamaV3 are libraries for running quantized models on consumer NVIDIA GPUs, with their own formats, EXL2 and EXL3 (GPTQ, AWQ, EXL3, NF4, and FP8 Serving). TabbyAPI is an OpenAI-compatible server built on them.

The weight format in a repository is the quickest guide to which engines can load it. The program classifies file listings by the format rules from section 3 and prints the engines that fit. Look at which file name decides each case.

import java.util.List;
import java.util.Map;

public class FormatToEngine {
    static String format(List<String> files) {
        if (files.stream().anyMatch(f -> f.endsWith(".gguf"))) return "GGUF";
        if (files.contains("genai_config.json")) return "ONNX Runtime GenAI";
        if (files.stream().anyMatch(f -> f.endsWith(".safetensors"))) {
            return files.contains("measurement.json") ? "safetensors (EXL2)" : "safetensors (Hugging Face layout)";
        }
        return "unknown";
    }

    static final Map<String, String> ENGINES = Map.of(
            "GGUF", "llama.cpp, Ollama, LM Studio",
            "safetensors (Hugging Face layout)", "vLLM, SGLang, TGI, TensorRT-LLM (after its build step), Transformers",
            "safetensors (EXL2)", "ExLlamaV2 (TabbyAPI)",
            "ONNX Runtime GenAI", "ONNX Runtime GenAI",
            "unknown", "inspect the model card");

    public static void main(String[] args) {
        Map<String, List<String>> repos = new java.util.TreeMap<>(Map.of(
                "a/Qwen2.5-Coder-7B-Instruct-GGUF", List.of("qwen2.5-coder-7b-instruct-q4_k_m.gguf", "README.md"),
                "b/Qwen2.5-Coder-7B-Instruct", List.of("config.json", "model-00001-of-00004.safetensors", "tokenizer.json"),
                "c/Qwen2.5-Coder-7B-exl2", List.of("config.json", "measurement.json", "output.safetensors"),
                "d/Phi-3-mini-onnx", List.of("genai_config.json", "model.onnx", "model.onnx.data", "tokenizer.json")));
        repos.forEach((repo, files) -> {
            String f = format(files);
            System.out.printf("%-34s %-34s %s%n", repo, f, ENGINES.get(f));
        });
    }
}

a/Qwen2.5-Coder-7B-Instruct-GGUF   GGUF                               llama.cpp, Ollama, LM Studio
b/Qwen2.5-Coder-7B-Instruct        safetensors (Hugging Face layout)  vLLM, SGLang, TGI, TensorRT-LLM (after its build step), Transformers
c/Qwen2.5-Coder-7B-exl2            safetensors (EXL2)                 ExLlamaV2 (TabbyAPI)
d/Phi-3-mini-onnx                  ONNX Runtime GenAI                 ONNX Runtime GenAI

Each line pairs a repository listing with a format and the engines that read it. GGUF files go to the llama.cpp family. A Hugging Face layout (config.json plus safetensors shards) goes to the GPU servers, although TensorRT-LLM first converts it into its own engine for a specific GPU.

The program treats a measurement.json file as the marker of an EXL2 repository. This is an assumption of the example, not a documented rule. EXL3 repositories also hold safetensors files, and the reliable source for both is the model card. The ONNX case is Example 10.

The rule for loop follows from the table: loop pull downloads GGUF, so its engine is llama.cpp, and a safetensors model is served by vLLM.

Example 9: Java options: JNI bindings, Jlama, single-file engines, LangChain4j

Several projects run models from Java. Their source is not in the reference set, so this list states only their kind, from their own documentation.

• java-llama.cpp provides JNI bindings to llama.cpp. JNI (Java Native Interface) is the older way for Java to call C code, and the native libraries ship inside the jar.
• Jlama is an inference engine written in Java that uses the Vector API and loads safetensors models.
• llama3.java is a single-file pure-Java engine that runs GGUF models with the Vector API. Section 7 of this course builds an engine of the same kind.
• LangChain4j is an application framework with integrations for OpenAI-compatible servers (with a configurable base URL), Ollama, and Jlama. It calls engines rather than being one.

All in-process options share one property: the weights live outside the Java heap, in memory-mapped files or native allocations, so -Xmx (the heap size limit) does not have to grow with the model. A memory-mapped file is a file that the operating system makes appear as memory, loading pages from disk only when they are touched.

The program maps a 64 MiB file, reads every page, and checks that the heap did not grow by the file's size.

import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;

public class OffHeapWeights {
    public static void main(String[] args) throws Exception {
        Path file = Files.createTempFile("weights", ".bin");
        long size = 64L << 20;
        try (RandomAccessFile raf = new RandomAccessFile(file.toFile(), "rw")) {
            raf.setLength(size);
            raf.seek(size - 1);
            raf.write(7);
        }
        Runtime rt = Runtime.getRuntime();
        System.gc();
        long heapBefore = rt.totalMemory() - rt.freeMemory();
        long sum = 0;
        try (FileChannel ch = FileChannel.open(file)) {
            MappedByteBuffer weights = ch.map(FileChannel.MapMode.READ_ONLY, 0, size);
            for (long i = 0; i < size; i += 4096) sum += weights.get((int) i);   // touch every page
            sum += weights.get((int) (size - 1));
        }
        long heapGrowth = (rt.totalMemory() - rt.freeMemory()) - heapBefore;
        System.out.println("mapped bytes: " + size);
        System.out.println("checksum of touched bytes: " + sum);
        System.out.println("heap grew by less than 8 MiB: " + (heapGrowth < (8L << 20)));
        Files.delete(file);
    }
}

mapped bytes: 67108864
checksum of touched bytes: 7
heap grew by less than 8 MiB: true

The file is 64 MiB of zeros with a 7 in the last byte, so the checksum of the touched bytes is 7. Reading every 4 KiB page through the mapping makes the operating system load the whole file into the page cache, but the bytes never enter the Java heap. The heap grows by much less than the file.

This is how llama.cpp loads GGUF (CPU_Mapped model buffer in Example 5). It is also how an FFM-based Java engine can load it (section 7 uses MemorySegment instead of MappedByteBuffer, which is limited to 2 GiB per mapping). It is why a JVM that runs a 5 GB model in process still needs only a small heap.

The cost of in-process engines lies elsewhere. A native crash or an out-of-memory error in the engine ends the whole JVM, which is the main criterion in Example 11.

Example 10: ONNX Runtime GenAI

ONNX Runtime GenAI is Microsoft's library for running generative models exported to ONNX (an open file format for neural networks). According to its documentation, it adds the generation loop, KV cache management, and sampling on top of ONNX Runtime. It has execution providers (backends) for CPU, CUDA, DirectML, and other hardware, and bindings for several languages.

A model for it is a folder with four parts: the ONNX graph, its external weight file, the tokenizer, and a genai_config.json that describes the model and the search settings. The program checks a folder listing for those parts, as a loop command would before refusing it with a clear message. Compare the two listings.

import java.util.List;

public class OnnxGenAiFolder {
    static String check(List<String> files) {
        boolean config = files.contains("genai_config.json");
        boolean graph = files.stream().anyMatch(f -> f.endsWith(".onnx"));
        boolean externalData = files.stream().anyMatch(f -> f.endsWith(".onnx.data"));
        boolean tokenizer = files.contains("tokenizer.json");
        if (!config) return "not an ONNX Runtime GenAI folder (no genai_config.json)";
        return String.format("ONNX Runtime GenAI model: graph=%b external weights=%b tokenizer=%b -> loop: unsupported format, "
                + "use GGUF with llama.cpp or safetensors with vLLM", graph, externalData, tokenizer);
    }

    public static void main(String[] args) {
        System.out.println(check(List.of("genai_config.json", "model.onnx", "model.onnx.data", "tokenizer.json",
                "tokenizer_config.json")));
        System.out.println(check(List.of("model.onnx", "tokenizer.json")));
    }
}

ONNX Runtime GenAI model: graph=true external weights=true tokenizer=true -> loop: unsupported format, use GGUF with llama.cpp or safetensors with vLLM
not an ONNX Runtime GenAI folder (no genai_config.json)

The first folder has all four parts, so it is recognized, and the message names the formats loop does accept. The second has an ONNX graph but no genai_config.json. It may be a plain ONNX export that ONNX Runtime can run one forward pass at a time, but it lacks the generation settings.

ONNX Runtime GenAI fits applications that already use ONNX Runtime, or Windows machines where DirectML reaches GPUs from several vendors. loop stays with GGUF and safetensors, because the course's model sources (Hugging Face GGUF repositories) and engines use them.

Example 11: a decision table by criteria

No engine is best on every axis, so the choice is a filter over requirements. First list what you need (hardware, file format, number of users, whether native code is acceptable), then keep the engines that meet all of it. The program encodes the facts established in this section for five engines and selects those that meet each of five sets of requirements.

import java.util.List;
import java.util.function.Predicate;

public class EngineChoice {
    record Engine(String name, boolean runsOnCpu, boolean appleGpu, boolean nvidia, boolean gguf, boolean safetensors,
                  boolean openAiApi, boolean manyUsers, boolean javaInProcess) {}

    static final List<Engine> ENGINES = List.of(
            new Engine("llama.cpp (llama-server)", true, true, true, true, false, true, false, false),
            new Engine("Ollama", true, true, true, true, true, true, false, false),
            new Engine("vLLM", true, false, true, false, true, true, true, false),
            new Engine("SGLang", true, true, true, false, true, true, true, false),
            new Engine("Pure-Java engine (section 7)", true, false, false, true, false, false, false, true));

    static void choose(String requirement, Predicate<Engine> p) {
        System.out.printf("%-46s %s%n", requirement, ENGINES.stream().filter(p).map(Engine::name).toList());
    }

    public static void main(String[] args) {
        choose("laptop without GPU, GGUF:", e -> e.runsOnCpu() && e.gguf());
        choose("Mac with Apple GPU:", Engine::appleGpu);
        choose("NVIDIA server, 50 concurrent agents:", e -> e.nvidia() && e.manyUsers());
        choose("OpenAI API and GGUF on NVIDIA:", e -> e.openAiApi() && e.gguf() && e.nvidia());
        choose("no native code, runs inside the JVM:", Engine::javaInProcess);
    }
}

laptop without GPU, GGUF:                      [llama.cpp (llama-server), Ollama, Pure-Java engine (section 7)]
Mac with Apple GPU:                            [llama.cpp (llama-server), Ollama, SGLang]
NVIDIA server, 50 concurrent agents:           [vLLM, SGLang]
OpenAI API and GGUF on NVIDIA:                 [llama.cpp (llama-server), Ollama]
no native code, runs inside the JVM:           [Pure-Java engine (section 7)]

Reading the output requirement by requirement:

• Laptop with no GPU and GGUF files. This leaves the llama.cpp family and a pure-Java engine. vLLM and SGLang also have CPU backends (vllm/platforms/cpu.py, sglang/srt/platforms/cpu.py), but they load safetensors, not GGUF.
• Mac with an Apple GPU. The GPU is reachable through llama.cpp's Metal backend, and therefore through Ollama. It is also reachable through SGLang's MLX backend (srt/hardware_backend/mlx/, described in its source as the "MLX model runner for Apple Silicon"). vLLM 0.30.0's platform list (PlatformEnum: CUDA, ROCm, TPU, XPU, CPU) has no Apple GPU.
• Many concurrent users on NVIDIA. This points to the engines with paged KV caches and large-batch schedulers, vLLM and SGLang.
• GGUF behind the OpenAI API on NVIDIA. This points to llama-server or Ollama.
• No native code. This admits only an engine written in Java.

The criteria behind the columns are listed below, each stated as a consequence:

Example 12: Integration A: a supervised subprocess

Integration A is what Ollama does in Example 1. The tool starts the engine as a child process, finds it a free port, reads its log, waits until the engine reports it is healthy, and stops it when done. Each of these steps needs care, and the table after the output explains the reason for each.

The program does this with a small fake engine, a second class in the same file, which it starts with the same java binary that runs the program, so it runs anywhere. Workshop: loop serve, run, and chat replaces the fake with llama-server. The fake answers /health with 503 (not ready) for its first two checks, as llama-server does while loading, and then with 200 (ready).

import com.sun.net.httpserver.HttpServer;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.ConnectException;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

public class SupervisedEngine {
    public static void main(String[] args) throws Exception {
        int port;
        try (ServerSocket s = new ServerSocket(0)) { port = s.getLocalPort(); }
        String javaBin = ProcessHandle.current().info().command().orElse("java");
        ProcessBuilder pb = new ProcessBuilder(javaBin, "-cp", System.getProperty("java.class.path"),
                "FakeEngine", Integer.toString(port)).redirectErrorStream(true);
        pb.environment().remove("JAVA_TOOL_OPTIONS");    // otherwise the child JVM prints a notice
        Process p = pb.start();
        List<String> logs = Collections.synchronizedList(new ArrayList<>());
        Thread reader = Thread.ofVirtual().start(() -> new BufferedReader(new InputStreamReader(p.getInputStream()))
                .lines().forEach(logs::add));

        HttpClient http = HttpClient.newHttpClient();
        URI health = URI.create("http://127.0.0.1:" + port + "/health");
        List<String> states = new ArrayList<>();
        long sleep = 50;
        while (true) {
            String state;
            try {
                state = Integer.toString(http.send(HttpRequest.newBuilder(health).timeout(Duration.ofSeconds(5))
                        .build(), HttpResponse.BodyHandlers.discarding()).statusCode());
            } catch (ConnectException e) {
                state = "refused";                     // not listening yet
            }
            if (!state.equals("refused") && (states.isEmpty() || !states.get(states.size() - 1).equals(state))) states.add(state);
            if (state.equals("200")) break;
            if (!p.isAlive()) throw new IllegalStateException("engine exited: " + p.exitValue());
            Thread.sleep(sleep);
            sleep = Math.min(sleep * 2, 400);
        }
        System.out.println("health states: " + states);
        p.toHandle().destroy();                        // SIGTERM on Linux and macOS
        reader.join();                                 // read the log until the child closes it
        System.out.println("exit code: " + p.waitFor());
        logs.forEach(l -> System.out.println("engine log | " + l));
    }
}

class FakeEngine {
    public static void main(String[] args) throws Exception {
        AtomicInteger checks = new AtomicInteger();
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", Integer.parseInt(args[0])), 0);
        server.createContext("/health", ex -> {
            int code = checks.incrementAndGet() <= 2 ? 503 : 200;   // "loading" for the first two checks
            ex.sendResponseHeaders(code, -1);
            ex.close();
        });
        Runtime.getRuntime().addShutdownHook(new Thread(() -> System.out.println("received SIGTERM, shutting down")));
        System.out.println("loading model");
        server.start();
        System.out.println("listening");
    }
}

health states: [503, 200]
exit code: 143
engine log | loading model
engine log | listening
engine log | received SIGTERM, shutting down

Each step in the program answers one supervision problem:

The output shows two 503 states and then 200. Connection-refused attempts are not recorded, because how many happen depends on how fast the child JVM starts.

The exit code is 143, which is 128 plus 15, where 15 is the number of the SIGTERM signal. The child ended because of the signal, after running its shutdown hook, whose line is the last one in the log. Reading the log to the end before calling waitFor is what keeps that line.

Example 13: Integration B: connecting to an engine that is already running

Integration B skips process management. The engine runs elsewhere (another machine, a container, Ollama, LM Studio), and the tool needs only a base URL and perhaps a key. The tool still checks the server before the first chat, because a wrong URL or key should fail with a clear message at the start, not in the middle of a conversation.

The program checks a fake server that requires a key. It tries a correct key, a wrong key, and a server that has stopped, and prints a different message for each.

import com.sun.net.httpserver.HttpServer;
import java.net.ConnectException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;

public class RemoteCheck {
    static String check(HttpClient http, String baseUrl, String key) throws Exception {
        HttpRequest.Builder b = HttpRequest.newBuilder(URI.create(baseUrl + "/models"));
        if (key != null) b.header("Authorization", "Bearer " + key);
        try {
            HttpResponse<String> r = http.send(b.build(), HttpResponse.BodyHandlers.ofString());
            return switch (r.statusCode()) {
                case 200 -> "ok: " + r.body();
                case 401 -> "rejected: check the API key";
                default -> "unexpected status " + r.statusCode();
            };
        } catch (ConnectException e) {
            return "no server at the base URL";
        }
    }

    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        server.createContext("/v1/models", ex -> {
            boolean ok = "Bearer sk-local".equals(ex.getRequestHeaders().getFirst("Authorization"));
            byte[] body = (ok ? "{\"object\":\"list\",\"data\":[{\"id\":\"coder\"}]}" : "{\"error\":\"Unauthorized\"}")
                    .getBytes(StandardCharsets.UTF_8);
            ex.sendResponseHeaders(ok ? 200 : 401, body.length);
            ex.getResponseBody().write(body);
            ex.close();
        });
        server.start();
        String base = "http://127.0.0.1:" + server.getAddress().getPort() + "/v1";
        HttpClient http = HttpClient.newHttpClient();
        System.out.println(check(http, base, "sk-local"));
        System.out.println(check(http, base, "wrong"));
        server.stop(0);
        System.out.println(check(http, base, "sk-local"));
    }
}

ok: {"object":"list","data":[{"id":"coder"}]}
rejected: check the API key
no server at the base URL

The first check succeeds and lists the served model name, which the tool then uses in every request. The second gets 401, the status vLLM's middleware returns for a wrong key ({"error": "Unauthorized"} in authenticate.py). llama-server also answers 401 for an invalid key. The third runs after the server has stopped and gets connection refused, which is reported as "no server".

The three messages map to three user actions: none, fix the key, or start or reach the server. Integration B has no supervision work, but the tool also cannot restart the engine, change its flags, or unload it when idle.

Example 14: Integration C: calling libllama in process with FFM

Integration C runs the engine inside the JVM. With native llama.cpp, this means calling the C functions of libllama (llama.cpp's shared library) through the Foreign Function and Memory (FFM) API, final in Java 22. The course teaches it in section 7, and the Java Deep Dive lesson Foreign Function and Memory API covers it in general.

In one sentence: FFM finds a symbol (a function name) in a shared library, describes its C signature with a FunctionDescriptor, and returns a MethodHandle that calls it.

The program loads libllama.so from the pinned build and calls five functions from include/llama.h that take no arguments. It is marked java,ignore because it needs that native library and JDK 22 or later. It was run once with JDK 25.

import java.lang.foreign.Arena;
import java.lang.foreign.FunctionDescriptor;
import java.lang.foreign.Linker;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.SymbolLookup;
import java.lang.foreign.ValueLayout;
import java.lang.invoke.MethodHandle;
import java.nio.file.Path;

public class LlamaSystemInfo {
    public static void main(String[] args) throws Throwable {
        Path lib = Path.of(System.getenv("LLAMA_LIB_DIR"), "libllama.so");
        Linker linker = Linker.nativeLinker();
        try (Arena arena = Arena.ofConfined()) {
            SymbolLookup llama = SymbolLookup.libraryLookup(lib, arena);
            MethodHandle backendInit = linker.downcallHandle(
                    llama.find("llama_backend_init").orElseThrow(), FunctionDescriptor.ofVoid());
            MethodHandle systemInfo = linker.downcallHandle(
                    llama.find("llama_print_system_info").orElseThrow(),
                    FunctionDescriptor.of(ValueLayout.ADDRESS));
            MethodHandle supportsMmap = linker.downcallHandle(
                    llama.find("llama_supports_mmap").orElseThrow(),
                    FunctionDescriptor.of(ValueLayout.JAVA_BOOLEAN));
            MethodHandle supportsGpu = linker.downcallHandle(
                    llama.find("llama_supports_gpu_offload").orElseThrow(),
                    FunctionDescriptor.of(ValueLayout.JAVA_BOOLEAN));
            MethodHandle maxDevices = linker.downcallHandle(
                    llama.find("llama_max_devices").orElseThrow(),
                    FunctionDescriptor.of(ValueLayout.JAVA_LONG));
            MethodHandle backendFree = linker.downcallHandle(
                    llama.find("llama_backend_free").orElseThrow(), FunctionDescriptor.ofVoid());

            backendInit.invokeExact();
            MemorySegment text = (MemorySegment) systemInfo.invokeExact();
            System.out.println("system info: " + text.reinterpret(Long.MAX_VALUE).getString(0));
            System.out.println("supports mmap: " + (boolean) supportsMmap.invokeExact());
            System.out.println("supports GPU offload: " + (boolean) supportsGpu.invokeExact());
            System.out.println("max devices: " + (long) maxDevices.invokeExact());
            backendFree.invokeExact();
        }
    }
}

LLAMA_LIB_DIR=$REF/llama_cpp_python-0.3.36/vendor/llama.cpp/build/bin \
  java --enable-native-access=ALL-UNNAMED LlamaSystemInfo.java

The program is started with the command above. Real output (pinned llama.cpp CPU build, JDK 25; the CPU feature list depends on the machine):

system info: CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
supports mmap: true
supports GPU offload: false
max devices: 16

llama_print_system_info returns a const char *, which FFM sees as an address of unknown size. reinterpret(Long.MAX_VALUE) declares the memory readable, so getString(0) can read up to the terminating zero byte. The string is the same system_info line that llama-server logs at startup: the CPU features this build uses.

llama_supports_gpu_offload is false because the build is CPU-only. llama_max_devices is 16, the size of the per-device arrays in llama.cpp's model parameters. --enable-native-access=ALL-UNNAMED allows code on the class path to call native functions without a warning. libllama.so finds libggml.so and the CPU backend in the same directory through the run path set by the build.

These five calls take no arguments, which makes them the easiest case. Loading a model takes llama_model_load_from_file(path, params), where params is a C struct passed by value. Its Java layout must match the C struct field by field for the exact llama.cpp commit, and the struct changes between releases. The rest of the API (contexts, batches, samplers, the chat template) adds more such structs and pointers whose lifetimes Java must manage.

A JNI binding such as java-llama.cpp hides this behind a C++ layer compiled for each platform. Either way, a segmentation fault in native code ends the JVM, and the tool's chat history and state go with it. This is why Integration A is the default for loop.

Example 15: the criteria for loop's choice

loop implements Integration A for llama-server and vLLM, and Integration B for any OpenAI-compatible server. Section 7 adds a pure-Java engine, which is a form of Integration C without native code.

The program encodes the rule loop applies when it starts a chat, in order:

1. A configured remote URL wins.
2. Otherwise, a local engine binary that matches the model's format is used.
3. Otherwise, the Java engine is used for small GGUF models.

It runs five example setups through that rule.

import java.util.Optional;

public class LoopBackendChoice {
    record Env(Optional<String> remoteUrl, boolean hasLlamaServer, boolean hasVllm, boolean nvidiaGpu,
               String modelFormat, double modelGiB) {}

    static String choose(Env e) {
        if (e.remoteUrl().isPresent()) return "B: remote OpenAI-compatible server at " + e.remoteUrl().get();
        if (e.modelFormat().equals("gguf") && e.hasLlamaServer()) return "A: supervise llama-server";
        if (e.modelFormat().equals("safetensors") && e.hasVllm() && e.nvidiaGpu()) return "A: supervise vllm serve";
        if (e.modelFormat().equals("gguf") && e.modelGiB() <= 2) return "C: pure-Java engine (section 7)";
        return "none: install llama-server or set --url";
    }

    public static void main(String[] args) {
        System.out.println(choose(new Env(Optional.of("http://gpu-box:8000/v1"), true, false, false, "gguf", 4.7)));
        System.out.println(choose(new Env(Optional.empty(), true, false, false, "gguf", 4.7)));
        System.out.println(choose(new Env(Optional.empty(), false, true, true, "safetensors", 15.2)));
        System.out.println(choose(new Env(Optional.empty(), false, false, false, "gguf", 0.6)));
        System.out.println(choose(new Env(Optional.empty(), false, false, false, "gguf", 4.7)));
    }
}

B: remote OpenAI-compatible server at http://gpu-box:8000/v1
A: supervise llama-server
A: supervise vllm serve
C: pure-Java engine (section 7)
none: install llama-server or set --url

The five lines match the five setups in order:

1. The first case has a remote URL, so loop connects and manages nothing.
2. The second has a GGUF model and a llama-server binary, the common laptop case, so loop supervises it.
3. The third has a safetensors model, vLLM, and an NVIDIA GPU, so loop supervises vllm serve.
4. The fourth has no engine binary, but the model (0.6 GiB) is small enough for the pure-Java engine.
5. The last has a 4.7 GiB model and no engine, so loop stops with an instruction instead of starting something that would be too slow.

The criteria behind the order of the rule are:

This is the same split Ollama makes: a long-running server in front, the engine in a child process, and the HTTP API between them. Workshop: loop serve, run, and chat builds loop serve and loop chat on Integration A and B.

Interview points

• Ollama v0.35.1 runs every GGUF model in an upstream llama-server child process, bound to 127.0.0.1 on a free port, and runs safetensors models in an MLX runner process. The Ollama server itself is the API and scheduler in front.
• Ollama's scheduler keeps one runner per model, with a reference count and a keep_alive timer (default 5 minutes). keep_alive: 0 unloads the model after the request, and negative values keep it loaded forever.
• When OLLAMA_MAX_LOADED_MODELS is unset, the limit is 3 models per GPU, and eviction prefers idle runners with the shortest keep_alive.
• Ollama's store is OCI-like. Manifests live under manifests/<host>/<namespace>/<model>/<tag>, layers are content-addressed blobs named blobs/sha256-<hex>, and media types mark model, template, system, params, and license layers.
• The native API streams NDJSON with durations in nanoseconds. Tokens per second is eval_count / eval_duration * 1e9.
• Ollama predicts memory as the file size plus an F16 KV cache estimate, then measures real use by parsing llama-server's buffer-size log lines.
• RadixAttention keeps the KV cache in a radix tree of tokens and matches at token or page granularity. The lpm scheduling policy runs requests with the longest cached prefix first.
• The weight format decides the engine family: GGUF for llama.cpp, Ollama, and LM Studio; safetensors for vLLM, SGLang, TGI, and TensorRT-LLM; EXL2 and EXL3 for ExLlama.
• A supervised subprocess needs a free port, continuous log reading, health polling that treats connection refused and 503 as "not ready", crash detection, and SIGTERM followed by a wait.
• In-process engines (FFM, JNI) remove the process boundary but tie the JVM's life to native code, and APIs that pass structs by value change between llama.cpp releases.

Summary

• ollama serve listens on 127.0.0.1:11434, and the CLI is an HTTP client of it.
• Ollama starts llama-server with --host 127.0.0.1 --no-webui --offline -c <num_ctx * parallel> -np <parallel> plus log flags it parses. -ngl is passed only when num_gpu is set.
• Defaults from envconfig: OLLAMA_KEEP_ALIVE 5m, OLLAMA_NUM_PARALLEL 1, OLLAMA_MAX_QUEUE 512, OLLAMA_LOAD_TIMEOUT 5m, and OLLAMA_MAX_LOADED_MODELS 0 (which means 3 per GPU).
• The default num_ctx is 4,096 below 23 GiB of total VRAM, 32,768 from 23 GiB, and 262,144 from 47 GiB.
• Model names default to host registry.ollama.ai, namespace library, and tag latest.
• Modelfile commands: FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE, REQUIRES, DRAFT, RENDERER, PARSER, CAPABILITY.
• Ollama also serves /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, /v1/responses, and /v1/messages.
• SGLang 0.5.21 listens on 127.0.0.1:30000, schedules fcfs by default, and uses xgrammar for structured outputs unless told otherwise.
• The tiny model's real buffers were 307.51 MiB of mapped weights, 4.00 MiB of KV cache, 10.14 MiB of compute buffer, and 0.58 MiB of output buffer.
• A child process stopped with SIGTERM exits with code 143, which is 128 + 15.
• FFM calls libllama through SymbolLookup.libraryLookup, Linker.downcallHandle, and FunctionDescriptor, and it needs --enable-native-access.