An OpenAI-Compatible API in Front of llama-server

An engine such as llama-server speaks the OpenAI chat protocol on one port, serves one model, and reports its own errors in its own words. That is enough for one person and one model. It is not enough when several models run in several engines and many tools want to use them.

A gateway is a program in front of one or more engines. It gives clients a single address, one model list, one error format, and the same API no matter which engine answers. This lesson builds the first version of that gateway as a Spring Boot application.

A client that works with the OpenAI API, an OpenAI-compatible tool, or loop chat from the serve-and-chat workshop can then point at the gateway. It can use the models that loop pull downloaded without knowing which engine runs them.

The gateway handles non-streaming requests in this lesson. A non-streaming request waits until the engine has produced the whole answer and then returns it in one piece. Streaming goes through the gateway in Streaming Responses Through the Gateway, and starting and stopping engines belongs to Loading, Unloading, and Scheduling Models. Here the engines are processes that already run, and the gateway is told where they are.

The table lists the endpoints and the example that builds each one.

The project is the Maven project of Spring Boot Essentials for an LLM Gateway: Spring Boot 4.1.1 (Spring Framework 7) on JDK 25, with @ConfigurationProperties, RestClient, @RestControllerAdvice, and tests with MockMvc. It uses Spring MVC on virtual threads and no reactive stack. A reader who has not seen those terms should read that lesson first.

Spring code blocks are marked java,ignore because the single-file Java checker cannot run Spring. Each of them compiled and ran in the project, and each output block under them is the real output of a curl call or a Maven run. Every class is in the package dev.loop.gateway, and the blocks leave out the package and import lines to save space.

Two earlier lessons supply input data. The registry comes from the model-pull workshop, and the GGUF key-value format from The GGUF Format.

The engines are fake OpenAI-compatible servers started on 127.0.0.1 (the local machine) with fixed ports, plus one real llama-server with the course's tiny model. The tiny model has random weights, so its answers are meaningless text while the protocol and the token counts are real.

Example 1: a fake OpenAI-compatible engine

Testing a gateway against a real engine makes the tests slow, and their results depend on the model. A fake engine answers instantly and does exactly what the test needs. This program is a complete engine for the purposes of this lesson.

It answers /health and /v1/chat/completions. It echoes the last user message and lists the top-level field names it received, so a test can see what the gateway forwarded. It also misbehaves on request: a last message of status:400 makes it return that status, and sleep:300 makes it wait 300 ms.

Started with a port and a name (java FakeEngine.java 18661 alpha), it serves until killed, and the later examples use it this way. Started without arguments, it runs the self-test below.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;

public class FakeEngine {
    static final ObjectMapper JSON = new ObjectMapper();

    /** Starts a fake OpenAI-compatible engine named alias on 127.0.0.1. Port 0 picks a free port. */
    static HttpServer start(int port, String alias) throws IOException {
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", port), 0);
        server.setExecutor(java.util.concurrent.Executors.newVirtualThreadPerTaskExecutor());
        server.createContext("/health", ex -> reply(ex, 200, "{\"status\":\"ok\"}"));
        server.createContext("/v1/chat/completions", ex -> {
            JsonNode request = JSON.readTree(ex.getRequestBody());
            String last = request.path("messages").path(request.path("messages").size() - 1).path("content").asText("");
            if (last.startsWith("sleep:")) sleep(Long.parseLong(last.substring(6)));
            if (last.startsWith("status:")) {
                int status = Integer.parseInt(last.substring(7));
                reply(ex, status, "{\"error\":{\"code\":" + status + ",\"message\":\"fake engine returned " + status
                        + "\",\"type\":\"server_error\"}}");
                return;
            }
            ArrayNode received = JSON.createArrayNode();
            request.fieldNames().forEachRemaining(received::add);
            ObjectNode answer = JSON.createObjectNode();
            answer.put("id", "chatcmpl-fake").put("object", "chat.completion").put("created", 1700000000).put("model", alias);
            ObjectNode message = JSON.createObjectNode().put("role", "assistant").put("content", alias + " heard: " + last);
            answer.putArray("choices").addObject().put("index", 0).put("finish_reason", "stop").set("message", message);
            answer.putObject("usage").put("prompt_tokens", 7).put("completion_tokens", 5).put("total_tokens", 12);
            answer.set("received_fields", received);
            reply(ex, 200, JSON.writeValueAsString(answer));
        });
        server.start();
        return server;
    }

    static void sleep(long millis) {
        try { Thread.sleep(millis); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    static void reply(com.sun.net.httpserver.HttpExchange ex, int status, String body) throws IOException {
        byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
        ex.getResponseHeaders().add("Content-Type", "application/json; charset=utf-8");
        ex.sendResponseHeaders(status, bytes.length);
        ex.getResponseBody().write(bytes);
        ex.close();
    }

    static String post(int port, String json) throws Exception {
        HttpResponse<String> r = HttpClient.newHttpClient().send(HttpRequest.newBuilder(
                URI.create("http://127.0.0.1:" + port + "/v1/chat/completions"))
                .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString(json)).build(),
                HttpResponse.BodyHandlers.ofString());
        return r.statusCode() + " " + r.body();
    }

    public static void main(String[] args) throws Exception {
        if (args.length == 2) {                          // java FakeEngine.java 18661 alpha: serve until killed
            start(Integer.parseInt(args[0]), args[1]);
            Thread.currentThread().join();
        }
        HttpServer server = start(0, "alpha");
        int port = server.getAddress().getPort();
        System.out.println(post(port, "{\"model\":\"m\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"top_k\":40}"));
        System.out.println(post(port, "{\"model\":\"m\",\"messages\":[{\"role\":\"user\",\"content\":\"status:400\"}]}"));
        long start = System.nanoTime();
        post(port, "{\"model\":\"m\",\"messages\":[{\"role\":\"user\",\"content\":\"sleep:300\"}]}");
        System.out.println("sleep:300 took at least 300 ms: " + ((System.nanoTime() - start) / 1_000_000 >= 300));
        server.stop(0);
    }
}

200 {"id":"chatcmpl-fake","object":"chat.completion","created":1700000000,"model":"alpha","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"alpha heard: hi"}}],"usage":{"prompt_tokens":7,"completion_tokens":5,"total_tokens":12},"received_fields":["model","messages","top_k"]}
400 {"error":{"code":400,"message":"fake engine returned 400","type":"server_error"}}
sleep:300 took at least 300 ms: true

Read the output line by line. The first request carries a field the OpenAI protocol does not define, top_k. The answer's received_fields array shows that the engine saw model, messages, and top_k. The gateway's job in Examples 9 and 10 is to keep that array intact.

The second request shows the error format of the fake, which copies the one llama-server uses: {"error": {"code", "message", "type"}}, as listed in The Chat Completions Protocol and Streaming in Java.

The third line confirms that the sleep is real, which Example 12 needs to trigger a timeout. The program prints a boolean instead of the elapsed time because durations change between runs.

Example 2: the Spring Boot project and its configuration

The gateway needs five starter dependencies. A starter is one dependency that pulls in a group of libraries. They are the web starter (Spring MVC on Tomcat), validation (Jakarta Bean Validation), the RestClient starter, and two test starters. WebFlux, the reactive web stack, is not part of the project.

<dependencies>
  <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency>
  <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-validation</artifactId></dependency>
  <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-restclient</artifactId></dependency>
  <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency>
  <dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-webmvc-test</artifactId><scope>test</scope></dependency>
</dependencies>

Spring Boot 4 splits the test support by technology. @AutoConfigureMockMvc lives in spring-boot-webmvc-test, which the spring-boot-starter-webmvc-test starter brings.

The JSON library is Jackson 3.1.5, whose packages start with tools.jackson. The annotations still use com.fasterxml.jackson.annotation, and Jackson 3 exceptions are unchecked, so the code needs no throws clauses for them.

The application class and the configuration record are small.

@SpringBootApplication
@ConfigurationPropertiesScan
public class GatewayApplication {
    public static void main(String[] args) {
        SpringApplication.run(GatewayApplication.class, args);
    }
}

@ConfigurationProperties(prefix = "loop")
public record GatewayProperties(
        Path home,
        @DefaultValue("2s") Duration connectTimeout,
        @DefaultValue("120s") Duration readTimeout,
        @DefaultValue("5m") Duration keepAlive,
        @DefaultValue List<Engine> engines) {

    /** One engine process: the registry model names it serves and its base URL. */
    public record Engine(String url, List<String> models) {}
}

@ConfigurationPropertiesScan finds the record, and Spring binds the loop.* properties into it. Each component has a purpose.

• home is the ~/.loop directory of loop pull.
• engines lists the engine processes. Each has a base URL and the registry model names it serves.
• The timeouts and the keep-alive period get values from the @DefaultValue annotations when the file does not set them.

The application.yml file turns on virtual threads. A virtual thread is a lightweight Java thread, so a request that waits for an engine does not hold a platform thread (an operating-system thread) while it waits.

spring:
  threads:
    virtual:
      enabled: true
loop:
  home: ${user.home}/.loop
  read-timeout: 120s

A second file, demo.yml, is used by the curl examples. It changes the port, the home directory, the read timeout, and the engines. Its structure is the same as a production file, and only the values differ.

server:
  port: 18660
loop:
  home: /tmp/claude-0/gw66/home
  read-timeout: 1s
  engines:
    - url: http://127.0.0.1:18661
      models:
        - example-org/Tiny-Coder-GGUF:F32
    - url: http://127.0.0.1:18662
      models:
        - example-org/Tiny-Chat-GGUF:F32

The gateway starts with java -jar target/loop-gateway-0.1.0.jar --spring.config.additional-location=file:demo.yml, after two fake engines were started on ports 18661 (alpha) and 18662 (beta). The option --spring.config.additional-location adds demo.yml on top of the normal configuration. The one log line that matters says the server is listening:

Tomcat started on port 18660 (http) with context path '/'

The log line confirms port 18660. The two engines serve two different models, and the read timeout of 1 second is short on purpose so that Example 12 stays quick. A real deployment uses the default of 120 seconds or more, because a large model can take a long time to produce the first byte of an answer to a long prompt.

Example 3: reading GGUF metadata for /api/show

/api/show and /api/tags report facts about a model: its architecture, its context length (how many tokens it can read at once), its parameter count. Those facts are not in the manifest. They are stored inside the GGUF file, in the key-value section at the start (The GGUF Format). The gateway needs a reader for that section.

This program is the reader as plain Java with no framework. It reads the header, loops over the key-value pairs, and skips arrays. Arrays are skipped because the tokenizer's vocabulary is an array with tens of thousands of strings that no client of /api/show needs. A writer in the same program builds a small file with five keys in memory, one of which is an array, so that the run is checkable.

import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.Map;

public class GgufDemo {
    // ---- the reader the gateway uses (arrays are skipped) ----
    static Map<String, Object> read(DataInputStream in) throws IOException {
        if (!new String(in.readNBytes(4), StandardCharsets.US_ASCII).equals("GGUF")) throw new IOException("not a GGUF file");
        in.readNBytes(4);                         // version
        in.readNBytes(8);                         // tensor count
        long kvCount = le64(in);
        Map<String, Object> kv = new LinkedHashMap<>();
        for (long i = 0; i < kvCount; i++) {
            String key = string(in);
            Object value = value(in, le32(in));
            if (value != null) kv.put(key, value);
        }
        return kv;
    }

    static Object value(DataInputStream in, int type) throws IOException {
        return switch (type) {
            case 0 -> in.readUnsignedByte();
            case 1 -> (int) in.readByte();
            case 2 -> le16(in);
            case 3 -> (int) (short) le16(in);
            case 4 -> le32(in) & 0xFFFFFFFFL;
            case 5 -> le32(in);
            case 6 -> Float.intBitsToFloat(le32(in));
            case 7 -> in.readUnsignedByte() != 0;
            case 8 -> string(in);
            case 9 -> {                                   // array: element type, count, elements
                int elementType = le32(in);
                long count = le64(in);
                for (long i = 0; i < count; i++) value(in, elementType);
                yield null;
            }
            case 10, 11 -> le64(in);
            case 12 -> Double.longBitsToDouble(le64(in));
            default -> throw new IOException("unknown value type " + type);
        };
    }

    static int le16(DataInputStream in) throws IOException { return ByteBuffer.wrap(in.readNBytes(2)).order(ByteOrder.LITTLE_ENDIAN).getShort() & 0xFFFF; }
    static int le32(DataInputStream in) throws IOException { return ByteBuffer.wrap(in.readNBytes(4)).order(ByteOrder.LITTLE_ENDIAN).getInt(); }
    static long le64(DataInputStream in) throws IOException { return ByteBuffer.wrap(in.readNBytes(8)).order(ByteOrder.LITTLE_ENDIAN).getLong(); }
    static String string(DataInputStream in) throws IOException { return new String(in.readNBytes((int) le64(in)), StandardCharsets.UTF_8); }

    // ---- a writer for the demo file ----
    static ByteBuffer buf(int n) { return ByteBuffer.allocate(n).order(ByteOrder.LITTLE_ENDIAN); }
    static void str(ByteArrayOutputStream o, String s) { byte[] b = s.getBytes(StandardCharsets.UTF_8); o.writeBytes(buf(8).putLong(b.length).array()); o.writeBytes(b); }
    static void key(ByteArrayOutputStream o, String k, int type) { str(o, k); o.writeBytes(buf(4).putInt(type).array()); }

    public static void main(String[] args) throws IOException {
        ByteArrayOutputStream o = new ByteArrayOutputStream();
        o.writeBytes("GGUF".getBytes(StandardCharsets.US_ASCII));
        o.writeBytes(buf(4).putInt(3).array());           // version
        o.writeBytes(buf(8).putLong(0).array());          // tensor count
        o.writeBytes(buf(8).putLong(5).array());          // key-value count
        key(o, "general.architecture", 8); str(o, "llama");
        key(o, "general.size_label", 8); str(o, "7B");
        key(o, "llama.context_length", 4); o.writeBytes(buf(4).putInt(32768).array());
        key(o, "tokenizer.ggml.tokens", 9);               // an array of strings: skipped
        o.writeBytes(buf(4).putInt(8).array()); o.writeBytes(buf(8).putLong(2).array()); str(o, "a"); str(o, "bc");
        key(o, "llama.rope.freq_base", 6); o.writeBytes(buf(4).putFloat(10000f).array());

        Map<String, Object> kv = read(new DataInputStream(new java.io.ByteArrayInputStream(o.toByteArray())));
        kv.forEach((k, v) -> System.out.println(k + " = " + v + " (" + v.getClass().getSimpleName() + ")"));
        try { read(new DataInputStream(new java.io.ByteArrayInputStream("GGML".getBytes()))); }
        catch (IOException e) { System.out.println("GGML file: " + e.getMessage()); }
    }
}

general.architecture = llama (String)
general.size_label = 7B (String)
llama.context_length = 32768 (Long)
llama.rope.freq_base = 10000.0 (Float)
GGML file: not a GGUF file

The reader returns the four scalar keys in file order and no tokenizer.ggml.tokens, which proves the array was read and dropped.

GGUF stores numbers little-endian, which means the least significant byte comes first. This is why le32 and le64 read four and eight bytes and set the byte order before converting them to numbers.

The value types decide the Java type. Type 4 (uint32, an unsigned 32-bit integer) becomes a Long, so that values above 2,147,483,647 stay positive. This is why llama.context_length prints as Long. Type 6 (float32) becomes a Float.

A file that does not start with GGUF is rejected before any key is read. A mistake such as pointing the registry at a wrong file then turns into one clear message. The class in the gateway, GgufMetadata, is this reader with a Path entry point.

Example 4: the registry as a Spring bean

The registry that loop pull writes is a directory: models/<name>/manifest.json plus one symbolic link per file. ModelRegistry is a bean (an object that Spring creates and manages) that reads it.

It scans the directory on each call. A model pulled while the gateway runs therefore shows up without a restart, and a model removed with loop rm disappears. A directory without a manifest.json is an unfinished pull and is ignored, as in the model-pull workshop.

/** The manifest.json written by loop pull (lesson 29). */
public record Manifest(String name, String repo, String revision, String commit, String quant,
                       List<FileEntry> files, String pulledAt) {
    public record FileEntry(String path, long size, String sha256) {}

    public long totalBytes() {
        return files.stream().mapToLong(FileEntry::size).sum();
    }
}

/** Reads the registry directory of loop pull: home/models/<dir>/manifest.json. */
@Component
public class ModelRegistry {
    /** A manifest and the directory that holds its file links. */
    public record Entry(Manifest manifest, Path dir) {
        /** llama.cpp finds the other parts of a split model next to the first one. */
        public Path firstFile() {
            return dir.resolve(manifest.files().get(0).path());
        }
    }

    private final Path models;
    private final JsonMapper json;

    public ModelRegistry(GatewayProperties props, JsonMapper json) {
        this.models = props.home().resolve("models");
        this.json = json;
    }

    /** Every model with a manifest, sorted by name. A directory without a manifest is an unfinished pull. */
    public List<Entry> list() {
        if (!Files.isDirectory(models)) return List.of();
        try (Stream<Path> dirs = Files.list(models)) {
            return dirs.filter(d -> Files.isRegularFile(d.resolve("manifest.json")))
                    .map(d -> new Entry(json.readValue(d.resolve("manifest.json").toFile(), Manifest.class), d))
                    .sorted(Comparator.comparing(e -> e.manifest().name()))
                    .toList();
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }

    public Optional<Entry> find(String name) {
        return list().stream().filter(e -> e.manifest().name().equals(name)).findFirst();
    }
}

Manifest has the fields of the manifest.json from the model-pull workshop. Jackson 3 reads JSON into a record through its canonical constructor, so no annotations are needed. totalBytes is the sum of the file sizes, which is the size that /api/tags reports.

Entry pairs a manifest with its directory. The first GGUF file of a model is a link inside that directory, and both the engine and the metadata reader open that file. The JsonMapper is the one Spring Boot configures, injected like any bean.

The demo home directory has two models and one unfinished pull. Both models are links to the course's tiny model, so the 328 MB file exists once in blobs/. The listing below shows the directory, with the 64-digit hash shortened:

home/blobs/sha256-<sha>
home/models/example-org--Half-Pulled--Q4_0
home/models/example-org--Tiny-Chat-GGUF--F32/Tiny-Chat-F32.gguf
home/models/example-org--Tiny-Chat-GGUF--F32/manifest.json
home/models/example-org--Tiny-Coder-GGUF--F32/Tiny-Coder-F32.gguf
home/models/example-org--Tiny-Coder-GGUF--F32/manifest.json

example-org--Half-Pulled--Q4_0 is an empty directory: the pull started and never wrote its manifest. The next example shows that the gateway does not list it.

Example 5: GET /v1/models from the registry

OpenAI clients call GET /v1/models to fill a model picker or to check that a name exists. The response is an object with object: "list" and a data array. Every item has id, object: "model", created (Unix seconds, the number of seconds since 1 January 1970), and owned_by. The gateway fills the array from the registry.

public record ModelObject(String id, String object, long created, String owned_by) {}
public record ModelList(String object, List<ModelObject> data) {}

@GetMapping("/v1/models")
ModelList models() {
    List<ModelObject> data = registry.list().stream()
            .map(e -> new ModelObject(e.manifest().name(), "model",
                    java.time.Instant.parse(e.manifest().pulledAt()).getEpochSecond(), "loop"))
            .toList();
    return new ModelList("list", data);
}

Spring serializes the returned record with Jackson, so the record component names are the JSON field names. This is why owned_by is written with an underscore in Java. The created value is the pulledAt time of the manifest converted to Unix seconds. The call:

curl -s localhost:18660/v1/models

{"object":"list","data":[{"id":"example-org/Tiny-Chat-GGUF:F32","object":"model","created":1790949600,"owned_by":"loop"},{"id":"example-org/Tiny-Coder-GGUF:F32","object":"model","created":1790847000,"owned_by":"loop"}]}

The list has two items, sorted by name, and no entry for the half-pulled model. The id is the registry name, so a client that sends this string as model in a chat request addresses exactly that model. The names contain / and :, which JSON strings carry without escaping. Examples 8 and 13 use these names as route keys.

Example 6: one error shape for the whole gateway

Engines report errors in different shapes, and the gateway adds errors of its own: an unknown model, an invalid request, an engine that does not answer. A client library for the OpenAI API expects one envelope, {"error": {"message", "type", "param", "code"}}. If the shapes varied, every client would need code for each of them.

The gateway therefore throws one exception type, GatewayException, from anywhere in the request path. One @RestControllerAdvice (a class that handles exceptions for every controller) converts it to the envelope. The envelope is a record, and the fields that do not apply are written as null, as the OpenAI API does.

/** The OpenAI error envelope: {"error": {"message", "type", "param", "code"}}. */
@JsonInclude(JsonInclude.Include.ALWAYS)
public record OpenAiError(Body error) {
    public record Body(String message, String type, String param, String code) {}

    public static OpenAiError of(String message, String type, String param, String code) {
        return new OpenAiError(new Body(message, type, param, code));
    }
}

/** An error the gateway reports itself, as opposed to one an engine reported. */
public class GatewayException extends RuntimeException {
    private final HttpStatus status;
    private final String type;
    private final String param;
    private final String code;

    public GatewayException(HttpStatus status, String type, String code, String param, String message) {
        super(message);
        this.status = status;
        this.type = type;
        this.code = code;
        this.param = param;
    }

    public HttpStatus status() { return status; }
    public String type() { return type; }
    public String param() { return param; }
    public String code() { return code; }
}

/** One place that turns exceptions into the error shape of the API the client called. */
@RestControllerAdvice
public class ApiErrorAdvice {

    @ExceptionHandler(GatewayException.class)
    ResponseEntity<Object> gateway(GatewayException e, HttpServletRequest request) {
        return ResponseEntity.status(e.status()).body(body(request, e));
    }

    @ExceptionHandler(HttpMessageNotReadableException.class)
    ResponseEntity<Object> unreadable(HttpMessageNotReadableException e, HttpServletRequest request) {
        GatewayException ge = new GatewayException(HttpStatus.BAD_REQUEST, "invalid_request_error", null, null,
                "the request body is not valid JSON");
        return ResponseEntity.badRequest().body(body(request, ge));
    }

    /** /v1/* uses the OpenAI envelope; /api/* uses Ollama's {"error": "text"}. */
    private Object body(HttpServletRequest request, GatewayException e) {
        if (request.getRequestURI().startsWith("/api/")) return Map.of("error", e.getMessage());
        return OpenAiError.of(e.getMessage(), e.type(), e.param(), e.code());
    }
}

The advice has two handlers. The first converts a GatewayException. The second converts HttpMessageNotReadableException, the exception Spring throws when the body is not valid JSON, so that even that error uses the envelope.

The body method picks the shape from the path. Ollama's native endpoints under /api/ report {"error": "text"} with a plain string (The Chat Completions Protocol and Streaming in Java), and Examples 14 and 15 use that shape. The /v1/ paths use the OpenAI envelope. A request with a broken body shows the first behavior:

curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{not json'

HTTP/1.1 400 
Content-Type: application/json

{"error":{"message":"the request body is not valid JSON","type":"invalid_request_error","param":null,"code":null}}

The status line, the content type, and the envelope are all the gateway's. type is invalid_request_error, the type that names a mistake in the request itself. Spring Boot Essentials for an LLM Gateway covers ProblemDetail for the gateway's own management endpoints. The /v1 API uses the OpenAI envelope instead, because OpenAI client libraries parse that shape.

Example 7: a non-streaming chat completion through RestClient

The core of the gateway is a method that takes a chat request, sends it to an engine, and returns the engine's answer. EngineClient builds one RestClient per engine base URL and sends the body as JSON. Two decisions shape it.

1. Bytes in, bytes out. The response is read as bytes and returned as bytes, so the gateway does not parse and rewrite the answer. Every field the engine sets, including timings from llama-server, reaches the client.
2. exchange instead of retrieve. retrieve throws an exception for a 4xx or 5xx status before the code can look at the answer. The gateway wants to look at the status and the body first, so it uses exchange. The single-argument exchange also closes the response when the lambda returns.

/** Sends a JSON body to an engine and returns its answer, or throws the GatewayException that describes the failure. */
@Component
public class EngineClient {
    private final Map<String, RestClient> clients = new ConcurrentHashMap<>();
    private final RestClient.Builder builder;
    private final GatewayProperties props;
    private final JsonMapper json;

    public EngineClient(RestClient.Builder builder, GatewayProperties props, JsonMapper json) {
        this.builder = builder;
        this.props = props;
        this.json = json;
    }

    private RestClient client(String baseUrl) {
        return clients.computeIfAbsent(baseUrl, url -> builder.clone()
                .baseUrl(url)
                .requestFactory(ClientHttpRequestFactoryBuilder.jdk().build(
                        HttpClientSettings.defaults().withTimeouts(props.connectTimeout(), props.readTimeout())))
                .build());
    }

    public ResponseEntity<byte[]> post(String baseUrl, String path, JsonNode body) {
        try {
            return client(baseUrl).post().uri(path).contentType(MediaType.APPLICATION_JSON).body(body)
                    .exchange((request, response) -> {
                        byte[] bytes = response.getBody().readAllBytes();
                        HttpStatusCode status = response.getStatusCode();
                        if (status.is5xxServerError() && status.value() != 503) {
                            throw new GatewayException(HttpStatus.BAD_GATEWAY, "server_error", "upstream_error", null,
                                    "the engine answered " + status.value() + ": " + engineMessage(bytes));
                        }
                        return ResponseEntity.status(status).contentType(response.getHeaders().getContentType()).body(bytes);
                    });
        } catch (ResourceAccessException e) {
            Throwable cause = e.getCause();
            if (cause instanceof HttpTimeoutException || cause instanceof SocketTimeoutException) {
                throw new GatewayException(HttpStatus.GATEWAY_TIMEOUT, "server_error", "engine_timeout", null,
                        "the engine did not answer within " + props.readTimeout().toMillis() + " ms");
            }
            if (cause instanceof ConnectException) {
                throw new GatewayException(HttpStatus.BAD_GATEWAY, "server_error", "engine_unavailable", null,
                        "no engine is listening at " + baseUrl);
            }
            throw new GatewayException(HttpStatus.BAD_GATEWAY, "server_error", "upstream_error", null,
                    "the connection to the engine failed: " + e.getMessage());
        }
    }

    /** The message of an llama-server style error envelope, or the first 200 characters of the body. */
    private String engineMessage(byte[] body) {
        try {
            JsonNode message = json.readTree(body).path("error").path("message");
            if (message.isString()) return message.asString();
        } catch (RuntimeException ignored) {
            // not JSON: fall through to the raw text
        }
        String text = new String(body, StandardCharsets.UTF_8);
        return text.length() > 200 ? text.substring(0, 200) : text;
    }
}

Read the class from the top.

• client creates the RestClient for a base URL on first use and keeps it in a map, so each engine has one client for the whole run.
• The request factory comes from ClientHttpRequestFactoryBuilder.jdk(). It builds a factory on the JDK's java.net.http.HttpClient, with the connect and read timeouts from the configuration.
• The lambda reads the whole body and applies the status policy of Example 11. Otherwise it returns a ResponseEntity<byte[]> with the engine's status, the engine's content type, and the engine's bytes.
• The catch block at the bottom is the subject of Example 12.

The controller validates (Example 13), finds the engine (Example 8), and calls the client.

@PostMapping("/v1/chat/completions")
ResponseEntity<byte[]> chat(@RequestBody ObjectNode body) {
    ChatCompletionRequest request = validator.check(body);
    String url = router.urlFor(request.model()).orElseThrow(() -> new GatewayException(HttpStatus.NOT_FOUND,
            "invalid_request_error", "model_not_found", "model", "the model '" + request.model() + "' does not exist"));
    ResponseEntity<byte[]> answer = engines.post(url, "/v1/chat/completions", body);
    if (answer.getStatusCode().is2xxSuccessful()) loaded.touch(request.model());
    return answer;
}

The controller validates the request (Example 13), finds the engine (Example 8), and calls the client. @RequestBody ObjectNode body receives the request as a Jackson tree, not as a Java class. A tree is a generic structure of JSON objects and values, and it is what makes Example 9 work. A call through the gateway to the engine alpha (the model example-org/Tiny-Coder-GGUF:F32):

curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"hello"}]}'

HTTP/1.1 200 
Content-Type: application/json;charset=utf-8
Content-Length: 302

{"id":"chatcmpl-fake","object":"chat.completion","created":1700000000,"model":"alpha","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"alpha heard: hello"}}],"usage":{"prompt_tokens":7,"completion_tokens":5,"total_tokens":12},"received_fields":["model","messages"]}

The body is the fake engine's answer, unchanged. "model":"alpha" is the engine's own name for itself, not the registry name the client sent. Content-Length: 302 is the length of the engine's bytes. The gateway did not rename anything.

A real llama-server reports whatever name it was started with in --alias. This is why Example 18 starts it with the registry name as its alias: the client then sees the name it sent.

Example 8: routing by the model field

A gateway with several engines needs a rule that maps a request to an engine. The rule in this version is the model field. The configuration lists which model names each engine serves, and EngineRouter looks the name up. A name no engine serves is a client error. The gateway reports it in the OpenAI envelope with HTTP 404 (Not Found), param: "model", and a code a client can test.

/** Maps the model field of a request to the base URL of the engine that serves it. */
@Component
public class EngineRouter {
    private final GatewayProperties props;

    public EngineRouter(GatewayProperties props) {
        this.props = props;
    }

    public Optional<String> urlFor(String model) {
        return props.engines().stream()
                .filter(e -> e.models().contains(model))
                .map(GatewayProperties.Engine::url)
                .findFirst();
    }
}

The router returns an Optional, which is either a value or empty. The controller turns an empty one into a GatewayException(404, ..., "model_not_found", "model", ...), which is the orElseThrow in Example 7. Two requests, one per case:

curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Chat-GGUF:F32","messages":[{"role":"user","content":"hello"}]}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"nope","messages":[{"role":"user","content":"x"}]}'

{"id":"chatcmpl-fake","object":"chat.completion","created":1700000000,"model":"beta","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"beta heard: hello"}}],"usage":{"prompt_tokens":7,"completion_tokens":5,"total_tokens":12},"received_fields":["model","messages"]}
{"error":{"message":"the model 'nope' does not exist","type":"invalid_request_error","param":"model","code":"model_not_found"}}

The first request names the second model, and the answer comes from beta on port 18662. Example 7's identical request with the first model came from alpha on port 18661. The second request names no model of any engine and gets the 404 envelope.

The router knows only the configuration. Whether a name is in the registry is a separate question that /v1/models answers. Two mismatches are possible. A name can be in the registry and have no engine, which routes to the same 404. A name can have an engine and be missing from the registry, which still routes. A later lesson makes the registry the single source and starts engines on demand.

Example 9: passing unknown fields through

The OpenAI protocol has many optional fields, and engines add their own: llama-server accepts top_k, min_p, cache_prompt, grammar, and more. Consider a gateway that deserializes the request into a Java class with a fixed set of fields. It silently drops every other field, and a client that sets top_k gets different sampling without any error.

The gateway avoids this by never converting the request into a class for forwarding. It receives an ObjectNode, validates a typed view of it (Example 13), and forwards the original tree. The request below has three fields that no Java class in the gateway mentions:

curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Chat-GGUF:F32","messages":[{"role":"user","content":"hello"}],"top_k":40,"cache_prompt":true,"seed":7}'

{"id":"chatcmpl-fake","object":"chat.completion","created":1700000000,"model":"beta","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"beta heard: hello"}}],"usage":{"prompt_tokens":7,"completion_tokens":5,"total_tokens":12},"received_fields":["model","messages","top_k","cache_prompt","seed"]}

The engine received five top-level fields, in the order the client sent them. seed is in the OpenAI protocol, and top_k and cache_prompt are engine extensions. The same property holds for the response: it is forwarded as bytes, so fields the gateway has never heard of, such as timings, reach the client.

The cost of this design is that the gateway cannot validate or rewrite what it does not model. The typed view in Example 13 therefore lists only the fields whose wrong values the gateway wants to catch before the engine sees them. A unit test (Example 17) pins the pass-through.

Example 10: engine errors that the client caused

An engine rejects some requests, for example a response_format with an unknown type or a content part of an unknown kind. These are the caller's mistakes, and the engine's message names the problem. The gateway returns a 4xx status from an engine unchanged, with the engine's body and content type, so that the client sees the exact text. The fake engine produces a 400 on request:

curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"status:400"}]}'

HTTP/1.1 400 
Content-Type: application/json;charset=utf-8
Content-Length: 81

{"error":{"code":400,"message":"fake engine returned 400","type":"server_error"}}

The status, the length, and the body are the engine's. The envelope has the engine's shape (code is a number, and there is no param), which differs slightly from the gateway's own. This is the trade-off of pass-through: the exact engine text is kept, and the gateway does not translate it. Clients that read error.message work with both shapes. Example 18 repeats the case with the real llama-server.

Example 11: engine errors that the engine caused

A 5xx status from the engine is not the client's mistake, and it is not the gateway's. The gateway reports it as 502 Bad Gateway, the status for "I asked an upstream server and its answer was an error". The response carries a code that says the engine failed and a message with the engine's status and text.

The one exception is 503. An engine answers 503 when it is up but not ready, for example while it loads a model. In HTTP, 503 means "try again later", so the gateway passes it through. The two cases, from the fake engine:

curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"status:500"}]}'
curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"status:503"}]}'

HTTP/1.1 502 
Content-Type: application/json

{"error":{"message":"the engine answered 500: fake engine returned 500","type":"server_error","param":null,"code":"upstream_error"}}
HTTP/1.1 503 
Content-Type: application/json;charset=utf-8
Content-Length: 81

{"error":{"code":503,"message":"fake engine returned 503","type":"server_error"}}

The first answer has the gateway's envelope and status 502. The engineMessage method of EngineClient pulls error.message out of the engine's JSON. It falls back to the first 200 characters of the body, so an engine that answers with an HTML error page still gives a readable message. The second answer is the engine's own, with the engine's content type and length. The rule has the same shape for every status class:

Example 12: an engine that is down or too slow

Two more failures produce no HTTP response at all. The gateway cannot connect, or it connects and the engine does not answer in time. The JDK client reports both as exceptions, and Spring wraps them in ResourceAccessException.

The catch block in EngineClient.post (Example 7) looks at the cause and maps it.

• A ConnectException becomes 502 with the code engine_unavailable.
• A timeout (HttpTimeoutException, or SocketTimeoutException for other request factories) becomes 504 Gateway Timeout with the code engine_timeout. 504 is the status for "the upstream server did not answer in time".

The first call stops the engine beta (kill on its process) and then asks for a model that it serves. The second call asks the engine alpha to sleep for three seconds, which is longer than the one-second read timeout of demo.yml.

curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Chat-GGUF:F32","messages":[{"role":"user","content":"hello"}]}'
curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"sleep:3000"}]}'

HTTP/1.1 502 
Content-Type: application/json

{"error":{"message":"no engine is listening at http://127.0.0.1:18662","type":"server_error","param":null,"code":"engine_unavailable"}}
HTTP/1.1 504 
Content-Type: application/json

{"error":{"message":"the engine did not answer within 1000 ms","type":"server_error","param":null,"code":"engine_timeout"}}

The two codes let a client decide what to do.

• engine_unavailable means the engine is not running. Waiting a moment may help if something restarts it.
• engine_timeout means the engine is running but slow. A retry of the same long request will probably time out again.

The timeout covers the time until the engine's response arrives, which for a non-streaming request is the whole generation. A model that needs two minutes for a long answer needs a read timeout above two minutes. Streaming (the next lesson) removes the problem, because the first byte arrives early. The timeouts are set per client in HttpClientSettings, so a later lesson can give different engines different limits.

Example 13: validating the request

Without validation, a malformed request reaches the engine, which answers with its own wording or, worse, accepts a value it should reject. The gateway validates the fields it can name.

The check uses Bean Validation, the Jakarta standard that Spring Boot configures through the validation starter. Rules are written as annotations on a record, and a Validator checks them. The record is a view of the request. It lists the fields to check, and Jackson ignores every other field when it fills the view.

/**
 * The fields the gateway checks. It is a view: the request that goes to the engine is the original JSON,
 * so every field not listed here still reaches the engine.
 */
public record ChatCompletionRequest(
        @NotBlank String model,
        @NotEmpty List<@Valid Message> messages,
        @DecimalMin("0.0") @DecimalMax("2.0") Double temperature,
        @Positive Integer max_tokens,
        Boolean stream) {

    public record Message(@NotBlank @Pattern(regexp = "system|developer|user|assistant|tool") String role) {}
}

@Component
public class ChatRequestValidator {
    private final JsonMapper json;
    private final Validator validator;

    public ChatRequestValidator(JsonMapper json, Validator validator) {
        this.json = json;
        this.validator = validator;
    }

    /** Throws a 400 for the first problem, in a stable order, and returns the typed view when the request is valid. */
    public ChatCompletionRequest check(JsonNode body) {
        ChatCompletionRequest request;
        try {
            request = json.treeToValue(body, ChatCompletionRequest.class);
        } catch (JacksonException e) {
            String param = e.getPath().stream().map(r -> r.getPropertyName() != null ? r.getPropertyName() : "[" + r.getIndex() + "]")
                    .collect(java.util.stream.Collectors.joining("."));
            throw invalid(param.isEmpty() ? null : param, "'" + param + "' has the wrong JSON type");
        }
        Set<ConstraintViolation<ChatCompletionRequest>> violations = validator.validate(request);
        ConstraintViolation<ChatCompletionRequest> first = violations.stream()
                .min(Comparator.comparing((ConstraintViolation<ChatCompletionRequest> v) -> v.getPropertyPath().toString()))
                .orElse(null);
        if (first != null) {
            String param = first.getPropertyPath().toString();
            throw invalid(param, "'" + param + "' " + first.getMessage());
        }
        if (Boolean.TRUE.equals(request.stream())) {
            throw invalid("stream", "'stream' is not supported by this gateway version");
        }
        return request;
    }

    private static GatewayException invalid(String param, String message) {
        return new GatewayException(HttpStatus.BAD_REQUEST, "invalid_request_error", null, param, message);
    }
}

check works in four steps.

1. It converts the tree into the view with treeToValue. That call throws a JacksonException when a field has the wrong JSON type, for example messages as a string. The exception's path becomes the param.
2. validator.validate returns every violation. The method reports only one: the first by property path, so the same bad request always gives the same message.
3. It rejects stream: true, because streaming arrives in the next lesson. The check is removed there.
4. It returns the view, which the controller uses for the model name.

Each failing request below gets a 400 in the OpenAI envelope with param set:

curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"x"}]}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[]}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"robot","content":"x"}]}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"x"}],"temperature":3}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"x"}],"max_tokens":0}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":"hi"}'
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"x"}],"stream":true}'

{"error":{"message":"'model' must not be blank","type":"invalid_request_error","param":"model","code":null}}
{"error":{"message":"'messages' must not be empty","type":"invalid_request_error","param":"messages","code":null}}
{"error":{"message":"'messages[0].role' must match \"system|developer|user|assistant|tool\"","type":"invalid_request_error","param":"messages[0].role","code":null}}
{"error":{"message":"'temperature' must be less than or equal to 2.0","type":"invalid_request_error","param":"temperature","code":null}}
{"error":{"message":"'max_tokens' must be greater than 0","type":"invalid_request_error","param":"max_tokens","code":null}}
{"error":{"message":"'messages' has the wrong JSON type","type":"invalid_request_error","param":"messages","code":null}}
{"error":{"message":"'stream' is not supported by this gateway version","type":"invalid_request_error","param":"stream","code":null}}

Each line is one failure, and each param is the path of the field. messages[0].role shows that the path includes the list index, so a client can point at the bad element. A missing model is reported as blank, because @NotBlank rejects null.

The Java record component is named max_tokens, with an underscore, so that the property path equals the JSON name without a mapping layer. The messages come from Bean Validation's default text, prefixed with the field name.

What the typed view does not check stays with the engine: the content of the messages, the sampler fields the view omits, and engine-specific fields.

Example 14: Ollama-style GET /api/tags

Ollama is another popular local engine with its own API. Clients built for Ollama list models with GET /api/tags. The response is {"models": [...]}. Each item has name, model, modified_at, size, digest, and details. The details object has parent_model, format, family, families, parameter_size, and quantization_level. These field names come from the ListModelResponse and ModelDetails types in the Ollama source (version 0.35.1) used throughout this course. The gateway fills them from the manifest and the GGUF metadata.

public record Details(String parent_model, String format, String family, List<String> families,
                      String parameter_size, String quantization_level) {}

public record Tag(String name, String model, String modified_at, long size, String digest, Details details) {}
public record Tags(List<Tag> models) {}

public record Show(Details details, Map<String, Object> model_info, String modified_at) {}

public record Running(String name, String model, long size, String digest, Details details,
                      String expires_at, long size_vram, long context_length) {}
public record Ps(List<Running> models) {}

@GetMapping("/api/tags")
Tags tags() {
    return new Tags(registry.list().stream().map(e -> tag(e.manifest(), details(e))).toList());
}

private Details details(ModelRegistry.Entry entry) {
    Meta meta = metadata(entry);
    String size = String.valueOf(meta.info().getOrDefault("general.size_label", ""));
    return new Details("", "gguf", meta.arch(), List.of(meta.arch()), size, entry.manifest().quant());
}

Each field has a source.

• name and model are the registry name.
• modified_at is pulledAt.
• size is the total size of the model's files.
• digest is the SHA-256 of the first file, a stable identity for the content.
• family is general.architecture from the GGUF file.
• quantization_level comes from the manifest, because the quantization is part of the model's registry name.
• parameter_size is the GGUF key general.size_label when the file has it, and an empty string when not.

The call:

curl -s localhost:18660/api/tags

{
    "models": [
        {
            "name": "example-org/Tiny-Chat-GGUF:F32",
            "model": "example-org/Tiny-Chat-GGUF:F32",
            "modified_at": "2026-10-02T14:00:00Z",
            "size": 328377376,
            "digest": "5239405616616846448e0fd1b11e1807bc5693a20f9f04e8fd57853254b1e7b7",
            "details": {
                "parent_model": "",
                "format": "gguf",
                "family": "llama",
                "families": [
                    "llama"
                ],
                "parameter_size": "",
                "quantization_level": "F32"
            }
        },
        {
            "name": "example-org/Tiny-Coder-GGUF:F32",
            "model": "example-org/Tiny-Coder-GGUF:F32",
            "modified_at": "2026-10-01T09:30:00Z",
            "size": 328377376,
            "digest": "5239405616616846448e0fd1b11e1807bc5693a20f9f04e8fd57853254b1e7b7",
            "details": {
                "parent_model": "",
                "format": "gguf",
                "family": "llama",
                "families": [
                    "llama"
                ],
                "parameter_size": "",
                "quantization_level": "F32"
            }
        }
    ]
}

(The response is indented here with python3 -m json.tool. The gateway sends one line.) Both models have the same digest and size because both manifests point at the same tiny file. parameter_size is empty because the tiny model's GGUF has no general.size_label key. A model whose GGUF file carries that key gets the field filled, for example with a size such as 7B. The gateway lists only what the registry knows, so a model that is not pulled is not listed, as with Ollama.

Example 15: Ollama-style POST /api/show

POST /api/show takes {"model": "<name>"} and returns the details of one model. In the Ollama source, the request has the fields model and a deprecated name. A missing model answers 404 with {"error": "model '<name>' not found"}, and an empty name answers 400 with {"error": "model is required"}. The gateway implements these three behaviors. It returns the model's details, modified_at, and model_info, which is the map of GGUF keys.

@PostMapping("/api/show")
Show show(@RequestBody JsonNode body) {
    String name = body.path("model").asString(body.path("name").asString(""));
    if (name.isBlank()) throw new GatewayException(HttpStatus.BAD_REQUEST, null, null, null, "model is required");
    ModelRegistry.Entry entry = registry.find(name).orElseThrow(() ->
            new GatewayException(HttpStatus.NOT_FOUND, null, null, null, "model '" + name + "' not found"));
    return new Show(details(entry), metadata(entry).info(), entry.manifest().pulledAt());
}

private record Meta(Map<String, Object> info, String arch) {}

private Meta metadata(ModelRegistry.Entry entry) {
    try {
        Map<String, Object> kv = GgufMetadata.read(entry.firstFile());
        return new Meta(new LinkedHashMap<>(kv), String.valueOf(kv.getOrDefault("general.architecture", "unknown")));
    } catch (IOException e) {
        throw new GatewayException(HttpStatus.INTERNAL_SERVER_ERROR, null, null, null,
                "cannot read the model file: " + e.getMessage());
    }
}

The method reads model and falls back to name, as Ollama does. A GatewayException with a null type is rendered by the advice (Example 6) in the Ollama shape, because the path starts with /api/. model_info is the map that GgufMetadata reads from the model's first file. The engine is started with the first part of a split model, and the first part is also the file the reader opens. Three calls:

curl -s -X POST localhost:18660/api/show -H 'Content-Type: application/json' -d '{"model":"example-org/Tiny-Coder-GGUF:F32"}'
curl -s -i -X POST localhost:18660/api/show -H 'Content-Type: application/json' -d '{"model":"nope"}'
curl -s -i -X POST localhost:18660/api/show -H 'Content-Type: application/json' -d '{}'

{
  "details": {
    "parent_model": "",
    "format": "gguf",
    "family": "llama",
    "families": [
      "llama"
    ],
    "parameter_size": "",
    "quantization_level": "F32"
  },
  "model_info": {
    "general.architecture": "llama",
    "general.name": "tiny-random-llama",
    "llama.context_length": 4096,
    "llama.embedding_length": 256,
    "llama.block_count": 4,
    "llama.feed_forward_length": 704,
    "llama.attention.head_count": 8,
    "llama.attention.head_count_kv": 2,
    "llama.rope.dimension_count": 32,
    "llama.attention.layer_norm_rms_epsilon": 1e-05,
    "llama.rope.freq_base": 500000.0,
    "general.file_type": 0,
    "tokenizer.ggml.model": "gpt2",
    "tokenizer.ggml.pre": "qwen2"
  },
  "modified_at": "2026-10-01T09:30:00Z"
}
HTTP/1.1 404 
Content-Type: application/json

{"error":"model 'nope' not found"}
HTTP/1.1 400 
Content-Type: application/json

{"error":"model is required"}

(The response is reformatted with python3, and the real model_info has 18 keys, of which the first 14 are shown.) The keys are the model's real GGUF metadata, printed as the reader returns them. They describe the tiny model:

• 4 blocks (llama.block_count, the number of transformer layers),
• 256 embedding dimensions (the length of the vector that represents one token),
• 8 attention heads that share 2 key-value heads,
• a context length of 4,096 tokens,
• general.file_type 0, the code for F32.

The key names come from the file, so llama.context_length has the architecture as its prefix. This is why /api/ps in the next example builds the key from general.architecture. The 404 and the 400 are Ollama-shaped, {"error": "text"}, and not the OpenAI envelope.

Example 16: Ollama-style GET /api/ps

GET /api/ps lists the models that are loaded now. Each item has the fields of /api/tags plus expires_at, size_vram, and context_length, from the ProcessModelResponse type in the Ollama source.

The gateway of this lesson does not start or stop engines, so it has no true answer to "loaded". LoadedModels approximates it. It records the time of the last successful chat request per model, and a model counts as loaded until keepAlive (default five minutes) has passed since that time. The Clock is injected so that tests can fix the time. A later lesson replaces this class with the real lifecycle manager.

/** Remembers when each model last served a request. A later lesson replaces it with real engine lifecycle management. */
@Component
public class LoadedModels {
    private final Map<String, Instant> lastUsed = new ConcurrentHashMap<>();
    private final Clock clock;
    private final GatewayProperties props;

    public LoadedModels(Clock clock, GatewayProperties props) {
        this.clock = clock;
        this.props = props;
    }

    public void touch(String model) {
        lastUsed.put(model, clock.instant());
    }

    /** Model name to expiry time, for models used within the keep-alive period. */
    public Map<String, Instant> active() {
        Instant now = clock.instant();
        Map<String, Instant> result = new java.util.TreeMap<>();
        lastUsed.forEach((model, used) -> {
            Instant expires = used.plus(props.keepAlive());
            if (expires.isAfter(now)) result.put(model, expires);
        });
        return result;
    }
}

@GetMapping("/api/ps")
Ps ps() {
    List<Running> running = loaded.active().entrySet().stream().map(a -> {
        ModelRegistry.Entry e = registry.find(a.getKey()).orElseThrow();
        Manifest m = e.manifest();
        long context = ((Number) metadata(e).info().getOrDefault(metadata(e).arch() + ".context_length", 0)).longValue();
        return new Running(m.name(), m.name(), m.totalBytes(), m.files().get(0).sha256(), details(e),
                a.getValue().toString(), 0, context);
    }).toList();
    return new Ps(running);
}

touch is called by the chat controller after every 2xx answer (Example 7). active returns the models whose expiry lies in the future. context_length is the <architecture>.context_length key of the GGUF file. size_vram is 0 because the gateway does not know where the engine put the weights. The call below runs before any request, then after one chat request to the first model:

curl -s localhost:18660/api/ps
curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"hello"}]}' > /dev/null
curl -s localhost:18660/api/ps

{"models":[]}
{"models":[{"name":"example-org/Tiny-Coder-GGUF:F32","model":"example-org/Tiny-Coder-GGUF:F32","size":328377376,"digest":"5239405616616846448e0fd1b11e1807bc5693a20f9f04e8fd57853254b1e7b7","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama"],"parameter_size":"","quantization_level":"F32"},"expires_at":"2026-10-03T16:11:36.725952371Z","size_vram":0,"context_length":4096}]}

The list is empty before the request and holds the one model after it. expires_at is the time of the request plus five minutes. The value changes with every run, and Example 17 fixes the clock to check it. context_length is 4,096, the same number as in /api/show.

Example 17: testing the gateway with MockMvc

Eight tests cover the behavior of this lesson. The test class works in four steps.

• It starts the whole application with @SpringBootTest.
• It adds @AutoConfigureMockMvc to get a MockMvc that calls the controllers without a network port.
• It starts a small engine inside the test with the JDK's HttpServer.
• It uses @DynamicPropertySource to supply the properties that depend on that engine's port, which the operating system chooses. These are the route to the engine, the 500 ms read timeout, and a second route to 127.0.0.1:1, where nothing listens.

A @TestConfiguration replaces the Clock with a fixed one, marked @Primary. It gets its own bean name, because Spring Boot does not allow a bean definition to override another by default.

@SpringBootTest
@AutoConfigureMockMvc
@Import(GatewayTest.FixedClock.class)
class GatewayTest {
    static final String MODEL = "example-org/Tiny-Coder-GGUF:F32";
    static HttpServer engine;
    static Path home;

    @TestConfiguration
    static class FixedClock {
        @Bean @Primary Clock fixedClock() { return Clock.fixed(Instant.parse("2026-10-03T12:00:00Z"), ZoneOffset.UTC); }
    }

    @BeforeAll
    static void start() throws IOException {
        engine = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
        engine.createContext("/v1/chat/completions", ex -> {
            String body = new String(ex.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
            int status = body.contains("status:500") ? 500 : 200;
            if (body.contains("sleep")) try { Thread.sleep(1500); } catch (InterruptedException ignored) { }
            byte[] out = (status == 200 ? "{\"echo\":" + body + "}" : "{\"error\":{\"message\":\"engine crashed\"}}").getBytes(StandardCharsets.UTF_8);
            ex.getResponseHeaders().add("Content-Type", "application/json");
            ex.sendResponseHeaders(status, out.length);
            ex.getResponseBody().write(out);
            ex.close();
        });
        engine.start();
        home = Files.createTempDirectory("loop-home");
        Path dir = Files.createDirectories(home.resolve("models/example-org--Tiny-Coder-GGUF--F32"));
        Files.write(dir.resolve("Tiny-Coder-F32.gguf"), gguf());
        Files.writeString(dir.resolve("manifest.json"), """
                {"name":"%s","repo":"example-org/Tiny-Coder-GGUF","revision":"main","commit":"4b1c","quant":"F32",
                 "files":[{"path":"Tiny-Coder-F32.gguf","size":4096,"sha256":"abc123"}],"pulledAt":"2026-10-01T09:30:00Z"}
                """.formatted(MODEL));
    }

    @AfterAll
    static void stop() { engine.stop(0); }

    @DynamicPropertySource
    static void properties(DynamicPropertyRegistry registry) throws IOException {
        registry.add("loop.home", () -> home == null ? "" : home.toString());
        registry.add("loop.read-timeout", () -> "500ms");
        registry.add("loop.engines[0].url", () -> "http://127.0.0.1:" + engine.getAddress().getPort());
        registry.add("loop.engines[0].models[0]", () -> MODEL);
        registry.add("loop.engines[1].url", () -> "http://127.0.0.1:1");          // nothing listens on port 1
        registry.add("loop.engines[1].models[0]", () -> "down-model");
    }

    // gguf() and the helpers that write the three keys are omitted

    @Autowired MockMvc mvc;

    String chat(String extra) {
        return "{\"model\":\"" + MODEL + "\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]" + extra + "}";
    }

    @Test
    void unknownFieldsReachTheEngine() throws Exception {
        mvc.perform(post("/v1/chat/completions").contentType(MediaType.APPLICATION_JSON).content(chat(",\"top_k\":40,\"cache_prompt\":true")))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.echo.top_k").value(40))
                .andExpect(jsonPath("$.echo.cache_prompt").value(true));
    }

    @Test
    void slowEngineBecomes504() throws Exception {
        mvc.perform(post("/v1/chat/completions").contentType(MediaType.APPLICATION_JSON)
                        .content("{\"model\":\"" + MODEL + "\",\"messages\":[{\"role\":\"user\",\"content\":\"sleep\"}]}"))
                .andExpect(status().isGatewayTimeout())
                .andExpect(jsonPath("$.error.code").value("engine_timeout"));
    }
}

Each test sends a request and asserts the status and a JSON path in the response. The pass-through test sends top_k and cache_prompt and finds both in the echo of the test engine. The 502 test is for an engine answering 500, and a separate test routes to the dead port. The timeout test makes the test engine sleep 1,500 ms against the 500 ms limit.

ollamaEndpoints fixes the clock at 2026-10-03T12:00:00Z, sends one chat request, and expects expires_at of 2026-10-03T12:05:00Z, the fixed time plus the five-minute keep-alive. The test engine's registry holds one model with a small GGUF file that the test writes itself (three keys), so the test needs no model download.

mvn -B test

[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0

The line is the Maven Surefire summary (Surefire is the Maven plugin that runs tests), and it shows eight tests with no failures. Every behavior of the examples above is covered: the model list, the route, pass-through, the 404, the 400, the 502 from a 500, the 502 from a dead port, the 504, and the three Ollama endpoints.

Example 18: the gateway in front of a real llama-server

The last example replaces the fake engine with a real one. llama-server from the course's pinned build loads the tiny model and listens on port 18670. Its --alias is the registry name, so the engine reports the name the client sends.

llama-server -m home/models/example-org--Tiny-Coder-GGUF--F32/Tiny-Coder-F32.gguf \
  --host 127.0.0.1 --port 18670 --alias example-org/Tiny-Coder-GGUF:F32 --no-webui -c 2048

The gateway runs with demo2.yml. That file routes example-org/Tiny-Coder-GGUF:F32 to http://127.0.0.1:18670 with a read timeout of 60 seconds. The request sets max_tokens, temperature, the llama.cpp field top_k, and cache_prompt. A small Python call prints the parts of the answer that are the same on every run:

curl -s -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","max_tokens":8,"temperature":0,"top_k":1,"cache_prompt":false,"messages":[{"role":"user","content":"hello"}]}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(sorted(d.keys())); print(d['choices'][0]['finish_reason'], d['usage'], d['model'], d['timings']['predicted_n'])"

['choices', 'created', 'id', 'model', 'object', 'system_fingerprint', 'timings', 'usage']
length {'completion_tokens': 8, 'prompt_tokens': 19, 'total_tokens': 27, 'prompt_tokens_details': {'cached_tokens': 0}} example-org/Tiny-Coder-GGUF:F32 8

The answer has the keys llama-server sets, including timings and system_fingerprint. The gateway forwarded them without knowing them. finish_reason is length because the 8-token limit stopped the generation. The usage shows 19 prompt tokens, which include the chat template's tokens around the one word hello, and 8 completion tokens. The text of the answer is not shown because the model has random weights. The model is the alias, which is the registry name.

Two requests that the real engine rejects pass the gateway's validation, because the typed view does not check those fields. The first has a response_format of an unknown type. The second has a content part of an unknown type.

curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":"hello"}],"response_format":{"type":"bogus"}}'
curl -s -i -X POST localhost:18660/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"example-org/Tiny-Coder-GGUF:F32","messages":[{"role":"user","content":[{"type":"bogus"}]}]}'

HTTP/1.1 400 
Content-Type: application/json;charset=utf-8
Content-Length: 145

{"error":{"code":400,"message":"response_format type must be one of \"text\" or \"json_object\", but got: bogus","type":"invalid_request_error"}}
HTTP/1.1 400 
Content-Type: application/json;charset=utf-8
Content-Length: 92

{"error":{"code":400,"message":"unsupported content[].type","type":"invalid_request_error"}}

Both answers are llama-server's own 400 envelopes with its messages. They passed through the gateway with status, length, and body intact (Example 10). The gateway, llama-server, and the fake engines were stopped after the run.

Interview points

• A gateway in front of engines gives clients one address, one model list, one error shape, and one API across engines. It also becomes the place for routing, limits, and security, which later lessons add.
• Forward the request as a JSON tree and validate a typed view of it. A fixed request class drops unknown fields silently, and engine-specific sampling fields such as top_k then stop working without an error.
• Forward the response as bytes. The gateway then keeps fields it has not modeled, such as timings, and does not re-serialize numbers or reorder keys.
• Use RestClient.exchange when the status decides the behavior. retrieve throws for 4xx and 5xx before the code can look at the body.
• A policy for engine statuses: pass 2xx, 4xx, and 503 through, and map other 5xx to 502, so that each class of fault has one meaning for the client.
• 502 means the gateway could not get a valid answer from the engine, including a refused connection, and 504 means the engine did not answer in time. The distinct codes engine_unavailable and engine_timeout tell a client whether a retry can help.
• A non-streaming read timeout covers the whole generation. A long answer from a large model needs a long timeout, and streaming moves the first byte earlier.
• The OpenAI error envelope is {"error": {"message", "type", "param", "code"}} and Ollama's native API uses {"error": "text"}. One @RestControllerAdvice can choose the shape from the request path.
• Bean Validation works on a record view with @NotBlank, @NotEmpty, @DecimalMin, @Positive, and @Valid on list elements. Reporting the first violation by property path makes the message stable.
• /api/tags, /api/show, and /api/ps need metadata the engine does not provide. The gateway reads it from the registry manifest and from the GGUF key-value section, skipping array values.
• Spring Boot 4 uses Jackson 3 (tools.jackson), and its test support is split into modules such as spring-boot-webmvc-test.

Summary

• The gateway exposes GET /v1/models, POST /v1/chat/completions, GET /api/tags, POST /api/show, and GET /api/ps, backed by the loop pull registry and a list of engine URLs.
• Configuration is a @ConfigurationProperties record: the home directory, connect and read timeouts, the keep-alive period, and a list of engines with their model names.
• /v1/models lists registry models with id, object, created, and owned_by. A directory without manifest.json is not listed.
• The model field selects the engine. An unknown name gives 404 with the OpenAI envelope, param: "model", and the code model_not_found.
• The request body is received as an ObjectNode and forwarded unchanged, so unknown fields reach the engine. The response is forwarded as bytes with the engine's status and content type.
• Engine 4xx and 503 answers pass through. Other 5xx answers become 502 with the code upstream_error. A refused connection is 502 with engine_unavailable. A read timeout is 504 with engine_timeout.
• Validation checks model, messages, the message roles, temperature between 0 and 2, and a positive max_tokens, and rejects stream: true until the streaming lesson. Failures are 400 with param set.
• ApiErrorAdvice renders GatewayException in the OpenAI envelope for /v1 paths and as {"error": "text"} for /api paths.
• /api/tags reports name, size, digest, and details from the manifest and the GGUF metadata. /api/show adds the GGUF key-value map and answers 404 and 400 in Ollama's shape.
• /api/ps uses a last-used time and a keep-alive period until a later lesson provides a real engine lifecycle.
• The GGUF reader reads the key-value section little-endian and skips arrays, so it never loads the tokenizer's vocabulary.
• The eight MockMvc tests run against an in-process engine, a fixed Clock, and a GGUF file that the test writes.
• A real llama-server with the tiny model works behind the gateway: its timings field and its own 400 messages reach the client unchanged.