Spring Boot Essentials for an LLM Gateway

A local inference engine such as llama-server answers requests for one model on one port. A real setup has more than that: several engines, several models, many clients, and a need for one error format and one list of models. A gateway is an HTTP server that sits in front of the engines and speaks one API to its clients. A client talks only to the gateway, and the gateway decides which engine receives each request.

This lesson builds the Spring Boot base of that server in small steps. Spring Boot is a Java framework that starts a web server, wires objects together, and reads configuration, so a reader does not write that plumbing by hand. Every later lesson of this section adds to the same project: an OpenAI-compatible API (An OpenAI-Compatible API in Front of llama-server), streaming (Streaming Responses Through the Gateway), model loading (Loading, Unloading, and Scheduling Models), security and metrics (Securing and Observing the Gateway), and the final workshop (Workshop: loop gateway).

After this lesson, a reader can create the project, expose JSON endpoints, bind and validate configuration, call an engine with RestClient, return structured errors, test the endpoints, and run the packaged jar. Each example adds one small piece, and the output under it shows the piece working. The table lists the pieces in order.

The code in this lesson compiled and ran in a Maven project with Spring Boot 4.1.1 (Spring Framework 7.0.9, Tomcat 11.0.24, Jackson 3.1.5) on JDK 25. Tomcat is the web server that Spring Boot embeds, and Jackson is the library that converts between Java objects and JSON.

Spring code blocks use the java,ignore flag because the single-file Java checker cannot run Spring. The flag only tells the course tooling to skip those blocks. The outputs are the real results of mvn runs, tests, and curl calls (curl is a command-line HTTP client).

The engines in this lesson are fake OpenAI-compatible servers on 127.0.0.1, which is the local machine. The fake is a 70-line program, shown in Example 10. A real llama-server joins in the workshop lesson. The gateway listens on port 18065, and the fake engines on 18066 and 18067. The ports are fixed so that the outputs do not depend on a random port.

The gateway reuses two earlier pieces. The first is the Backend interface of the serve-and-chat workshop, which hands out a base URL ending in /v1. The engines list in Example 5 holds the same information (a name and a base URL) as configuration. The second is the set of request and response shapes from The Chat Completions Protocol and Streaming in Java. Example 4 declares them as records again, this time for a server that receives them instead of a client that sends them.

Example 1: a Maven project from spring-boot-starter-parent

Maven is the build tool that downloads libraries and compiles a Java project. Its project file is the POM (Project Object Model), an XML file named pom.xml. A Spring Boot project is an ordinary Maven project with one special line: a parent POM, which is another POM that this one inherits from. This example creates the project file and builds it.

The Spring Boot parent supplies default plugin settings and a dependency-management table. That table is a list of library versions that the Spring Boot team tested together. Because of it, the project declares the libraries it needs without version numbers, and the versions can never clash.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>4.1.1</version>
    <relativePath/>
  </parent>

  <groupId>dev.loop</groupId>
  <artifactId>loop-gateway</artifactId>
  <version>0.1.0</version>

  <properties>
    <java.version>25</java.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-restclient</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-validation</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-webmvc-test</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>
</project>

A starter is one dependency that pulls in a whole group of libraries for a single purpose. Without starters, a web application would need a separate line for Spring MVC, Tomcat, Jackson, and more. The project declares four starters:

The property java.version is read by the parent to set the compiler release, so 25 means the code compiles for JDK 25.

The spring-boot-maven-plugin has no configuration. The parent binds its repackage goal to the package phase, which means that mvn package automatically turns the finished jar into one that runs with java -jar (Example 16).

The gateway uses Spring MVC, the classic thread-per-request web framework, together with virtual threads and RestClient. It has no reactive dependency, so the code reads top to bottom like ordinary Java.

The build command and the libraries that ended up inside the jar:

export JAVA_HOME=/usr/lib/jvm/java-25-openjdk-amd64
mvn -q -B -DskipTests package
unzip -l target/loop-gateway-0.1.0.jar | awk '{print $4}' | grep -E "BOOT-INF/lib/(jackson|tomcat-embed-core|spring-(core|web|webmvc|boot)-[0-9]|hibernate-validator|spring-boot-restclient)" | sed 's|BOOT-INF/lib/||'

jackson-databind-3.1.5.jar
jackson-annotations-2.21.jar
jackson-core-3.1.5.jar
tomcat-embed-core-11.0.24.jar
spring-boot-4.1.1.jar
spring-web-7.0.9.jar
spring-webmvc-7.0.9.jar
spring-boot-restclient-4.1.1.jar
hibernate-validator-9.1.3.Final.jar
spring-core-7.0.9.jar

mvn -q (quiet mode) prints nothing on success, so the only output is the library list from unzip and grep. Read the list as proof of which versions the parent chose. Three facts matter for the rest of the section.

1. Spring Boot 4 uses Jackson 3. The databind and core jars are version 3.1.5 and their Java packages start with tools.jackson. The annotation jar stays at 2.21 and keeps the package com.fasterxml.jackson.annotation. This split is why later code imports from both names.
2. Spring Framework is 7.0.9, which every Spring jar in the list shows.
3. All library versions come from the parent. This is why the POM contains no version numbers except the parent's own.

Example 2: @SpringBootApplication and the first start

The smallest Spring Boot program is one class with a main method. This example adds that class and starts it. Look at the log afterwards: it shows what @SpringBootApplication caused Spring to do at startup.

package loop.gateway;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.ConfigurationPropertiesScan;

@SpringBootApplication
@ConfigurationPropertiesScan
public class GatewayApplication {
    public static void main(String[] args) {
        SpringApplication.run(GatewayApplication.class, args);
    }
}

@SpringBootApplication is a shortcut for three annotations. Each solves one problem.

• @SpringBootConfiguration marks the class as a source of bean definitions. A bean is an object that Spring creates and manages for you, so other classes can ask for it instead of creating it themselves.
• @EnableAutoConfiguration makes Spring Boot create beans based on the classes on the class path. With Spring MVC and Tomcat present, it creates a web server and a DispatcherServlet (the object that receives every request and hands it to the right controller method). With Jackson present, it creates the JSON converters.
• @ComponentScan finds classes annotated with @Component, @RestController, and similar annotations in this package and its sub-packages. Every class in loop.gateway is therefore picked up without being listed anywhere.

@ConfigurationPropertiesScan is explained in Example 5.

The first application.yml is the main configuration file. YAML is a text format where indentation shows nesting. This file fixes the port and turns the startup banner off. The log pattern contains only the level, the abbreviated logger name, and the message, so the output below has no timestamps.

server:
  port: 18065
spring:
  main:
    banner-mode: "off"
logging:
  pattern:
    console: "%-5level %logger{20} : %msg%n"

java -jar target/loop-gateway-0.1.0.jar

INFO  l.g.GatewayApplication : No active profile set, falling back to 1 default profile: "default"
INFO  o.s.b.t.TomcatWebServer : Tomcat initialized with port 18065 (http)
INFO  o.a.c.c.StandardService : Starting service [Tomcat]
INFO  o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/11.0.24]
INFO  o.s.b.t.TomcatWebServer : Tomcat started on port 18065 (http) with context path '/'

The real log has three more lines that carry values that change between runs. They are left out here so the output stays the same on every run:

• the first line (Starting GatewayApplication v0.1.0 using Java 25.0.4.1 with PID ...),
• the line Root WebApplicationContext: initialization completed in ... ms before the last,
• the last line (Started GatewayApplication in ... seconds ...).

The lines that remain describe the startup order, and the diagram shows it as one sequence.

1. Spring reports the active profile        (Example 7)
2. Tomcat is created on port 18065
3. The servlet engine starts
4. The web application context initializes  (the container that holds the beans)
5. Tomcat accepts connections -> "started"

The web application context is the container that holds all beans. Only after Tomcat accepts connections does the application count as started. The program is a plain Java process: SpringApplication.run returns after startup, and the Tomcat threads keep the process alive.

Example 3: a @RestController with path, query, and header values

A controller is a class whose methods answer HTTP requests. Spring looks at the annotations on each method to decide which URL and which HTTP method it handles. This example adds two endpoints. One returns a string. The other reads three kinds of input from the request (a path variable, a query parameter with a default, and an optional header) and returns a JSON object.

package loop.gateway;

import java.util.LinkedHashMap;
import java.util.Map;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
class InfoController {

    @GetMapping("/version")
    String version() {
        return "loop-gateway 0.1.0";
    }

    @GetMapping("/models/{name}/tokens")
    Map<String, Object> estimate(@PathVariable String name,
                                 @RequestParam String text,
                                 @RequestParam(defaultValue = "4") int charsPerToken,
                                 @RequestHeader(value = "X-Request-Id", required = false) String requestId) {
        Map<String, Object> body = new LinkedHashMap<>();
        body.put("model", name);
        body.put("chars", text.length());
        body.put("estimatedTokens", (text.length() + charsPerToken - 1) / charsPerToken);
        body.put("requestId", requestId);
        return body;
    }
}

@RestController is @Controller plus @ResponseBody, which means that the return value of each method becomes the response body. A String is written as text/plain. A Map is converted to JSON by Jackson. @RequestMapping("/api") on the class puts the prefix /api in front of every method path.

The second method takes its inputs from three places in the request:

curl -s localhost:18065/api/version
curl -s -H 'X-Request-Id: r-1' 'localhost:18065/api/models/qwen-coder/tokens?text=hello%20gateway%20world'
curl -s -i 'localhost:18065/api/models/qwen-coder/tokens' | sed -n '1p;$p' | jq -R 'select(startswith("{")) | fromjson | del(.timestamp)'

loop-gateway 0.1.0
{"model":"qwen-coder","chars":19,"estimatedTokens":5,"requestId":"r-1"}
{
  "status": 400,
  "error": "Bad Request",
  "path": "/api/models/qwen-coder/tokens"
}

Read the three calls one at a time. The first call returns the string. In the second call, the path variable is qwen-coder. The text hello gateway world has 19 characters, and the endpoint estimates ceil(19 / 4) = 5 tokens with the default of 4 characters per token. The code computes the ceiling with integer arithmetic: (19 + 4 - 1) / 4 = 22 / 4 = 5. The map keeps its insertion order because the code uses a LinkedHashMap.

The third call omits the required text parameter. Spring answers with status 400 and a small JSON body. The jq filter (jq is a command-line JSON tool) removes the timestamp field, which changes on every call. This default error body is plain and carries little information, and Example 12 replaces it with a structured one.

Example 4: records as JSON, snake_case, and unknown fields

A gateway receives JSON and sends JSON, so it needs Java types that match the JSON. Request and response bodies are Java records. A record is a class with fixed components and no setters, which suits data that is only carried around. Jackson 3 reads and writes records without extra annotations.

This example declares the shape of a chat request and a small response. It then shows three behaviors that a gateway depends on.

package loop.gateway;

import java.util.List;

public final class ChatPreview {
    private ChatPreview() { }

    public record Message(String role, String content) { }

    public record Request(String model, List<Message> messages, Double temperature, Integer maxTokens) { }

    public record Response(String model, int messageCount, int promptChars, Integer maxTokens, String note) { }
}

The controller takes the request as @RequestBody, which tells Spring to parse the JSON body into the record:

package loop.gateway;

import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
class ChatPreviewController {

    @PostMapping("/api/chat/preview")
    ChatPreview.Response preview(@RequestBody ChatPreview.Request request) {
        int chars = request.messages().stream().mapToInt(m -> m.content().length()).sum();
        return new ChatPreview.Response(request.model(), request.messages().size(), chars,
                request.maxTokens(), "temperature=" + request.temperature());
    }
}

OpenAI-style APIs use snake_case names such as max_tokens, with words joined by underscores. Java components use camel case, such as maxTokens. Writing a name mapping on every field would be tedious and error-prone. One setting in application.yml converts between the two styles for every JSON body the application reads or writes:

spring:
  jackson:
    property-naming-strategy: SNAKE_CASE

curl -s -X POST localhost:18065/api/chat/preview -H 'Content-Type: application/json' \
  -d '{"model":"qwen-coder","messages":[{"role":"user","content":"hi there"}],"temperature":0.2,"max_tokens":64,"stream":false,"x_extra":1}'
curl -s -X POST localhost:18065/api/chat/preview -H 'Content-Type: application/json' \
  -d '{"model":"qwen-coder","messages":[{"role":"user","content":"hi"}]}'

{"model":"qwen-coder","message_count":1,"prompt_chars":8,"max_tokens":64,"note":"temperature=0.2"}
{"model":"qwen-coder","message_count":1,"prompt_chars":2,"max_tokens":null,"note":"temperature=null"}

The first response shows three things.

1. Name mapping. The request field max_tokens filled the component maxTokens, and the response component messageCount was written as message_count.
2. Unknown fields are ignored. The request contained two fields that the record does not declare (stream and x_extra), and Spring Boot ignored them. The mapper that Spring Boot configures does not fail on unknown fields. A gateway needs this setting because clients send fields that the gateway does not model, and the OpenAI API grows new fields over time.
3. Missing fields become null. The second request leaves out temperature and max_tokens. Boxed types (Double, Integer) make those components null, and the response writes null for the missing max_tokens.

A primitive int would receive 0 for a missing field, and 0 cannot be told apart from a value the client set. Optional fields therefore use boxed types.

Example 5: @ConfigurationProperties with a record

The gateway needs settings: which engines exist, how many models may be loaded at once, and when an idle model unloads. Reading each value with getProperty("loop.gateway.name") would scatter strings through the code and give no type checking. @ConfigurationProperties binds a group of properties with a common prefix to one typed object. A record works as the target, and its components are filled through the constructor.

package loop.gateway;

import jakarta.validation.Valid;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.NotNull;
import java.net.URI;
import java.time.Duration;
import java.util.List;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.bind.DefaultValue;
import org.springframework.validation.annotation.Validated;

@ConfigurationProperties(prefix = "loop.gateway")
@Validated
public record GatewayProperties(
        @NotBlank String name,
        @Min(1) @Max(8) int maxLoadedModels,
        @NotNull Duration idleTimeout,
        @NotEmpty List<@Valid Engine> engines) {

    public record Engine(
            @NotBlank String name,
            @NotNull URI baseUrl,
            @DefaultValue("1") @Min(1) int parallel) { }
}

The engines list mirrors the Backend of the serve-and-chat workshop: a name and a base URL. Spring converts strings to richer types by itself. Duration and URI are converted from strings such as 5m and http://.... @DefaultValue("1") supplies parallel when the property is absent. The constraint annotations (@NotBlank, @Min, and the others) come in Example 6.

@ConfigurationPropertiesScan on the application class (Example 2) registers the record as a bean, so any class can take it as a constructor argument.

loop:
  gateway:
    name: loop-gateway
    max-loaded-models: 2
    idle-timeout: 5m
    engines:
      - name: qwen-coder
        base-url: http://127.0.0.1:18066/v1
        parallel: 2
      - name: tiny
        base-url: http://127.0.0.1:18067/v1

A controller exposes the bound object, which shows what the binder produced:

package loop.gateway;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
class ConfigController {
    private final GatewayProperties properties;

    ConfigController(GatewayProperties properties) {
        this.properties = properties;
    }

    @GetMapping("/api/config")
    GatewayProperties config() {
        return properties;
    }
}

curl -s localhost:18065/api/config | jq .

{
  "name": "loop-gateway",
  "max_loaded_models": 2,
  "idle_timeout": "PT5M",
  "engines": [
    {
      "name": "qwen-coder",
      "base_url": "http://127.0.0.1:18066/v1",
      "parallel": 2
    },
    {
      "name": "tiny",
      "base_url": "http://127.0.0.1:18067/v1",
      "parallel": 1
    }
  ]
}

Three things happened in the binding.

• The kebab-case keys in the file (max-loaded-models, words joined by dashes) were bound to the camel-case components (maxLoadedModels). This is relaxed binding, and files in this project use kebab case.
• idle-timeout: 5m became a Duration, which Jackson writes in ISO-8601 form as PT5M (period of time, 5 minutes).
• The second engine has no parallel entry, so the default of 1 applies.

The output keys are snake_case because the response goes through the same Jackson setting as Example 4. The file uses kebab case and the JSON uses snake case, but the Java code only ever sees camel case.

Example 6: validating configuration at startup

A wrong setting should stop the program at startup, not surface later as a strange runtime error hours into a run. The @Validated annotation on the properties record (Example 5) makes Spring Boot check the constraints right after binding. Validation here means checking values against rules that are written as annotations. The constraints used there are:

The program starts with two invalid values given on the command line, an empty name and max-loaded-models=0:

java -jar target/loop-gateway-0.1.0.jar --loop.gateway.max-loaded-models=0 --loop.gateway.name=

***************************
APPLICATION FAILED TO START
***************************

Description:

Binding to target loop.gateway.GatewayProperties failed:

    Property: loop.gateway.maxLoadedModels
    Value: "0"
    Origin: "loop.gateway.max-loaded-models" from property source "commandLineArgs"
    Reason: must be greater than or equal to 1

    Property: loop.gateway.name
    Value: ""
    Origin: "loop.gateway.name" from property source "commandLineArgs"
    Reason: must not be blank


Action:

Update your application's configuration

This block is the failure report of Spring Boot, cut to the lines after the log messages. Each violation has four parts:

1. the property,
2. the rejected value,
3. the origin, which says which property source supplied the value (here the command-line arguments),
4. the reason, taken from the constraint.

The report lists both errors in one run, so a user fixes all of them at once. The process exits with a non-zero status and never opens the port. Later lessons rely on this behavior: a gateway whose engine list is empty or whose limits are out of range never starts.

Example 7: profiles for development and production

A gateway runs in different places. A laptop needs small limits, and a server can hold more models. Changing the file by hand before every deployment is error-prone. A profile is a named set of beans and properties that is active only when selected. Spring Boot loads application-<profile>.yml on top of application.yml when the profile is active, so the base file holds the shared values and each profile file holds only the differences.

# application-dev.yml
loop:
  gateway:
    max-loaded-models: 1
    idle-timeout: 30s

# application-prod.yml
loop:
  gateway:
    max-loaded-models: 4
    idle-timeout: 30m

A configuration class can also be limited to a profile with @Profile. This one logs a line only in dev:

package loop.gateway;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.ApplicationRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

@Configuration
@Profile("dev")
class DevHints {
    private static final Logger log = LoggerFactory.getLogger(DevHints.class);

    @Bean
    ApplicationRunner devBanner(GatewayProperties properties) {
        return args -> log.info("dev profile: {} engine(s) configured, idle timeout {}",
                properties.engines().size(), properties.idleTimeout());
    }
}

An endpoint that reports the active profiles:

package loop.gateway;

import java.util.Arrays;
import java.util.Map;
import org.springframework.core.env.Environment;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
class ProfileController {
    private final Environment environment;

    ProfileController(Environment environment) {
        this.environment = environment;
    }

    @GetMapping("/api/profile")
    Map<String, Object> profile() {
        return Map.of("active", Arrays.asList(environment.getActiveProfiles()),
                "defaults", Arrays.asList(environment.getDefaultProfiles()));
    }
}

The program starts three times: without a profile, with dev, and with prod. The profile is selected with --spring.profiles.active.

for p in "" dev prod; do
  echo "== profile '$p'"
  java -jar target/loop-gateway-0.1.0.jar ${p:+--spring.profiles.active=$p} > app.log 2>&1 &
  pid=$!
  sleep 6
  grep -E "profile" app.log
  curl -s localhost:18065/api/profile | jq -cS .
  curl -s localhost:18065/api/config | jq -c '{max_loaded_models,idle_timeout}'
  kill $pid; wait $pid 2>/dev/null
done

== profile ''
INFO  l.g.GatewayApplication : No active profile set, falling back to 1 default profile: "default"
{"active":[],"defaults":["default"]}
{"max_loaded_models":2,"idle_timeout":"PT5M"}
== profile 'dev'
INFO  l.g.GatewayApplication : The following 1 profile is active: "dev"
INFO  l.gateway.DevHints : dev profile: 2 engine(s) configured, idle timeout PT30S
{"active":["dev"],"defaults":["default"]}
{"max_loaded_models":1,"idle_timeout":"PT30S"}
== profile 'prod'
INFO  l.g.GatewayApplication : The following 1 profile is active: "prod"
{"active":["prod"],"defaults":["default"]}
{"max_loaded_models":4,"idle_timeout":"PT30M"}

The script waits six seconds for startup, prints the profile log lines from app.log, and stops the process after each case. Compare the three runs.

• No profile. The list of active profiles is empty and Spring uses the default profile. Only application.yml applies, so the values are 2 and 5 minutes.
• dev. The file application-dev.yml overrides the two properties it names (1 and 30 seconds) and leaves the rest (the name and the engine list) from application.yml. The DevHints bean exists only in this run, so its log line is missing in the other two.
• prod. The values are 4 and 30 minutes.

Profile files override only the keys they contain, so they stay short.

Example 8: environment variables and the order of property sources

A deployed gateway is configured from the outside: environment variables in a service file, arguments in a command line. The same setting can appear in several places, so Spring Boot needs a rule for which one wins. It collects properties from several property sources, and the source that is highest in the order wins. The relevant part of the order, from lowest to highest:

The program runs with the dev profile (value 1), and then with an environment variable, and then with both an environment variable and an argument:

LOOP_GATEWAY_MAX_LOADED_MODELS=5 java -jar target/loop-gateway-0.1.0.jar --spring.profiles.active=dev
LOOP_GATEWAY_MAX_LOADED_MODELS=5 java -jar target/loop-gateway-0.1.0.jar --spring.profiles.active=dev --loop.gateway.max-loaded-models=3
curl -s localhost:18065/api/config | jq -c '{max_loaded_models,idle_timeout}'

{"max_loaded_models":5,"idle_timeout":"PT30S"}
{"max_loaded_models":3,"idle_timeout":"PT30S"}

The two output lines are the results of the first and second run. Walk through them in order.

• First run. The environment variable (5) beat the value from application-dev.yml (1). The idle timeout of 30 seconds still came from the profile file, because nothing overrode it.
• Second run. The command-line argument (3) beat the environment variable (5).

Relaxed binding maps the variable name LOOP_GATEWAY_MAX_LOADED_MODELS to the property loop.gateway.max-loaded-models: the name is upper case, with underscores in place of dots and dashes. This order is the reason a service file can set LOOP_GATEWAY_... variables for one machine without editing the jar.

lowest   application.yml in the jar          max-loaded-models: 2
         application-dev.yml                 max-loaded-models: 1
         environment variable                LOOP_GATEWAY_MAX_LOADED_MODELS=5
highest  command-line argument               --loop.gateway.max-loaded-models=3

Example 9: virtual threads for blocking engine calls

A gateway spends most of its time waiting: for the engine to answer, sometimes for minutes during a long generation. Tomcat handles each request on its own thread and blocks that thread while it waits. A normal Java thread (a platform thread) is backed by one operating-system thread, which is expensive. The default Tomcat pool has 200 threads, so at most 200 requests can wait at once.

A virtual thread is a lightweight Java thread that the JVM schedules on a few carrier threads. When it makes a blocking call, the JVM parks the virtual thread and frees the carrier thread for other work, so waiting costs almost nothing. One property switches Tomcat and the Spring task executors to virtual threads:

spring:
  threads:
    virtual:
      enabled: true

A small endpoint reports which kind of thread runs a handler:

package loop.gateway;

import java.util.LinkedHashMap;
import java.util.Map;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
class ThreadController {

    @GetMapping("/api/thread")
    Map<String, Object> thread() {
        Thread t = Thread.currentThread();
        Map<String, Object> body = new LinkedHashMap<>();
        body.put("virtual", t.isVirtual());
        body.put("daemon", t.isDaemon());
        body.put("namePrefix", t.getName().replaceAll("[0-9]+$", ""));
        return body;
    }
}

java -jar target/loop-gateway-0.1.0.jar
curl -s localhost:18065/api/thread
java -jar target/loop-gateway-0.1.0.jar --spring.threads.virtual.enabled=true
curl -s localhost:18065/api/thread

{"virtual":false,"daemon":true,"namePrefix":"http-nio-18065-exec-"}
{"virtual":true,"daemon":true,"namePrefix":"tomcat-handler-"}

Compare the two lines. Without the property, the handler runs on a pooled Tomcat thread named http-nio-18065-exec-N. With it, each request runs on a new virtual thread named tomcat-handler-N, and isVirtual() is true. The thread number is cut from the output because it changes. Virtual threads are daemon threads in both cases.

The code of the handler did not change. Blocking code such as RestClient (Example 10) can therefore run on virtual threads without a reactive framework. The streaming lesson uses this for long-lived responses.

The property stays off in the project's application.yml in this lesson, so the other examples run on the default pool of 200 threads (server.tomcat.threads.max, which has no effect once virtual threads are on).

Example 10: calling an engine with RestClient

The gateway forwards requests to engines over HTTP. RestClient is Spring's synchronous HTTP client with a fluent API, which means calls are chained: get().uri(...).retrieve().body(...). Synchronous means the calling thread waits for the answer, which is fine on a virtual thread. The RestClient.Builder bean comes from spring-boot-starter-restclient and carries Spring Boot's JSON converters, so the Jackson settings from Example 4 apply to outgoing bodies too.

First, the engine. This single-file program is an OpenAI-compatible server with three behaviors: /v1/models lists one model, /v1/chat/completions answers with a fixed message, and the model names missing and slow trigger an error and a three-second delay for Example 11. It uses only the JDK.

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/** A fake OpenAI-compatible engine: java FakeEngine.java <port> <model-name>. */
public class FakeEngine {
    public static void main(String[] args) throws Exception {
        int port = Integer.parseInt(args[0]);
        String name = args[1];
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", port), 0);
        server.createContext("/v1/models", ex -> send(ex, 200,
                "{\"object\":\"list\",\"data\":[{\"id\":\"" + name + "\",\"object\":\"model\",\"owned_by\":\"fake\"}]}"));
        server.createContext("/v1/chat/completions", ex -> {
            String body = new String(ex.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
            String model = field(body, "model");
            if (model.equals("missing")) {
                send(ex, 404, "{\"error\":{\"message\":\"model 'missing' not found\",\"type\":\"not_found_error\"}}");
                return;
            }
            if (model.equals("slow")) {
                try { Thread.sleep(3000); } catch (InterruptedException e) { return; }
            }
            send(ex, 200, "{\"id\":\"chatcmpl-1\",\"object\":\"chat.completion\",\"model\":\"" + model + "\","
                    + "\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"echo from " + name
                    + "\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":4,\"total_tokens\":7}}");
        });
        server.start();
        System.out.println("fake engine " + name + " listening");
    }

    static String field(String json, String key) {
        Matcher m = Pattern.compile("\"" + key + "\"\\s*:\\s*\"([^\"]*)\"").matcher(json);
        return m.find() ? m.group(1) : "";
    }

    static void send(HttpExchange ex, int status, String json) throws IOException {
        byte[] bytes = json.getBytes(StandardCharsets.UTF_8);
        ex.getResponseHeaders().set("Content-Type", "application/json");
        ex.sendResponseHeaders(status, bytes.length);
        ex.getResponseBody().write(bytes);
        ex.close();
    }
}

Start two instances (the program prints one line per start):

java fake/FakeEngine.java 18066 qwen-coder &
java fake/FakeEngine.java 18067 tiny &

fake engine qwen-coder listening
fake engine tiny listening

The client class builds one RestClient per configured engine, with the base URL from the properties (Example 5). builder.clone() copies the injected builder so each engine gets its own base URL without changing the shared one.

package loop.gateway;

import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import tools.jackson.databind.JsonNode;

@Component
class EngineClient {

    record ModelEntry(String id, String ownedBy) { }

    record ModelList(String object, List<ModelEntry> data) { }

    private final Map<String, RestClient> clients;

    EngineClient(GatewayProperties properties, RestClient.Builder builder) {
        this.clients = properties.engines().stream().collect(Collectors.toMap(
                GatewayProperties.Engine::name,
                engine -> builder.clone().baseUrl(engine.baseUrl().toString()).build()));
    }

    ModelList models(String engine) {
        return client(engine).get().uri("/models").retrieve().body(ModelList.class);
    }

    JsonNode chat(String engine, JsonNode request) {
        return client(engine).post().uri("/chat/completions").body(request).retrieve().body(JsonNode.class);
    }

    private RestClient client(String engine) {
        RestClient client = clients.get(engine);
        if (client == null) {
            throw new UnknownEngineException(engine);
        }
        return client;
    }
}

models declares a response type with two components. The engine also sends object for each model, which the record leaves out, and the converter ignores it (the same behavior as Example 4). chat takes and returns a JsonNode, Jackson's tree type, which passes the body through without a Java class for every field; the next lesson uses this to forward fields the gateway does not know. retrieve().body(...) sends the request, throws an exception for a 4xx or 5xx status, and converts the body otherwise. A thin controller connects the paths to the client:

package loop.gateway;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import tools.jackson.databind.JsonNode;

@RestController
@RequestMapping("/api/engines/{engine}")
class EngineController {
    private final EngineClient engines;

    EngineController(EngineClient engines) {
        this.engines = engines;
    }

    @GetMapping("/models")
    EngineClient.ModelList models(@PathVariable String engine) {
        return engines.models(engine);
    }

    @PostMapping("/chat")
    JsonNode chat(@PathVariable String engine, @RequestBody JsonNode request) {
        return engines.chat(engine, request);
    }
}

curl -s localhost:18065/api/engines/qwen-coder/models
curl -s localhost:18065/api/engines/tiny/models
curl -s -X POST localhost:18065/api/engines/qwen-coder/chat -H 'Content-Type: application/json' \
  -d '{"model":"qwen-coder","messages":[{"role":"user","content":"hi"}]}'

{"object":"list","data":[{"id":"qwen-coder","owned_by":"fake"}]}
{"object":"list","data":[{"id":"tiny","owned_by":"fake"}]}
{"id":"chatcmpl-1","object":"chat.completion","model":"qwen-coder","choices":[{"index":0,"message":{"role":"assistant","content":"echo from qwen-coder"},"finish_reason":"stop"}],"usage":{"prompt_tokens":3,"completion_tokens":4,"total_tokens":7}}

The path segment selects the engine: qwen-coder goes to port 18066 and tiny to 18067. Look at the model lists. The field object that the engine sent for each model is gone, because ModelEntry has two components and the converter drops the rest. The owned_by field arrived as ownedBy and went out as owned_by again.

The chat response is the engine's JSON, unchanged, because the JsonNode passed through. The dropped object field shows the cost of typed records: every field not declared is lost. This is why the chat path uses a tree.

Example 11: status errors and timeouts in RestClient

Engines fail. They return error statuses, they are slow, and they are sometimes not running. A gateway must tell these cases apart, so this example shows what RestClient throws in each case and how the timeouts are set. Two properties set the default timeouts for every RestClient that the builder creates:

spring:
  http:
    clients:
      connect-timeout: 1s
      read-timeout: 2s

The connect timeout limits the time to open a connection; the read timeout limits the time waiting for the response. In Spring Boot 4 the older properties spring.http.client.* are deprecated in favor of spring.http.clients.*.

The program is started with the two fake engines from Example 10, but the engine on 18067 is stopped. Three requests follow: one for a model that the engine does not know, one for the slow model, and one to the stopped engine. No error handling exists yet, so Spring answers each with its default. The status codes, then the exception types from the application log:

for b in '{"model":"missing","messages":[]}' '{"model":"slow","messages":[]}'; do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:18065/api/engines/qwen-coder/chat -H 'Content-Type: application/json' -d "$b"
done
curl -s -o /dev/null -w "%{http_code}\n" localhost:18065/api/engines/tiny/models
grep -o "Request processing failed: [A-Za-z.\$]*" app.log

500
500
500
Request processing failed: org.springframework.web.client.HttpClientErrorException$NotFound
Request processing failed: org.springframework.web.client.ResourceAccessException
Request processing failed: org.springframework.web.client.ResourceAccessException

All three requests produced 500, the status for an unhandled exception, which tells the client nothing about the cause. The log shows two kinds of exception, and the table shows which case produces which.

RestClientResponseException means the engine answered with an error status: the status and the body are available from the exception. ResourceAccessException means no usable answer arrived: the cause tells a timeout from a refused connection. Without the read timeout, the slow request would have waited the full three seconds and succeeded (the first run without timeouts returned 200 after 3.0 seconds). The default RestClient configuration has no read timeout, so a gateway must set one.

Example 12: @RestControllerAdvice and ProblemDetail

Clients need to tell a bad request from an engine failure, and a plain 500 hides the difference. ProblemDetail is Spring's class for the standard error body of RFC 9457 (application/problem+json). It has the fields type, title, status, detail, and instance, and room for extra properties.

A class annotated with @RestControllerAdvice applies @ExceptionHandler methods to every controller, so error handling lives in one place. Extending ResponseEntityExceptionHandler also converts Spring's own exceptions (missing parameters, wrong method, unreadable body) to ProblemDetail.

package loop.gateway;

import java.net.http.HttpTimeoutException;
import java.util.Map;
import java.util.TreeMap;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.context.request.WebRequest;
import org.springframework.http.HttpStatus;
import org.springframework.http.ProblemDetail;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.client.ResourceAccessException;
import org.springframework.web.client.RestClientResponseException;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;

@RestControllerAdvice
class ApiExceptionHandler extends ResponseEntityExceptionHandler {

    @ExceptionHandler(UnknownEngineException.class)
    ProblemDetail unknownEngine(UnknownEngineException e) {
        ProblemDetail problem = ProblemDetail.forStatusAndDetail(HttpStatus.NOT_FOUND, e.getMessage());
        problem.setTitle("Unknown engine");
        problem.setProperty("engine", e.engine());
        return problem;
    }

    @ExceptionHandler(RestClientResponseException.class)
    ProblemDetail upstreamError(RestClientResponseException e) {
        ProblemDetail problem = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_GATEWAY,
                "the engine answered with status " + e.getStatusCode().value());
        problem.setTitle("Engine error");
        problem.setProperty("upstream_status", e.getStatusCode().value());
        return problem;
    }

    @ExceptionHandler(ResourceAccessException.class)
    ProblemDetail upstreamUnreachable(ResourceAccessException e) {
        boolean timeout = hasCause(e, HttpTimeoutException.class);
        ProblemDetail problem = timeout
                ? ProblemDetail.forStatusAndDetail(HttpStatus.GATEWAY_TIMEOUT, "the engine did not answer in time")
                : ProblemDetail.forStatusAndDetail(HttpStatus.BAD_GATEWAY, "the engine is not reachable");
        problem.setTitle(timeout ? "Engine timeout" : "Engine unreachable");
        return problem;
    }

    @Override
    protected ResponseEntity<Object> handleMethodArgumentNotValid(MethodArgumentNotValidException e,
            HttpHeaders headers, HttpStatusCode status, WebRequest request) {
        Map<String, String> errors = new TreeMap<>();
        e.getBindingResult().getFieldErrors().forEach(f -> errors.put(f.getField(), f.getDefaultMessage()));
        ProblemDetail problem = e.getBody();
        problem.setDetail("the request body has " + errors.size() + " invalid field(s)");
        problem.setProperty("errors", errors);
        return handleExceptionInternal(e, problem, headers, status, request);
    }

    private static boolean hasCause(Throwable error, Class<? extends Throwable> type) {
        for (Throwable t = error; t != null; t = t.getCause()) {
            if (type.isInstance(t)) {
                return true;
            }
        }
        return false;
    }
}

The three handlers translate the failures of Example 11 into statuses with a defined meaning:

The helper hasCause walks the cause chain, because the timeout is nested inside the ResourceAccessException. setProperty adds a field next to the standard ones. The handleMethodArgumentNotValid override belongs to Example 13.

The same requests as before, now with the advice in place :

show() { curl -s -i "$@" | sed -n '1p;/^Content-Type/p;$p'; echo; echo; }
show localhost:18065/api/engines/nope/models
show localhost:18065/api/engines/tiny/models
show -X POST localhost:18065/api/engines/qwen-coder/chat -H 'Content-Type: application/json' -d '{"model":"missing","messages":[]}'
show -X POST localhost:18065/api/engines/qwen-coder/chat -H 'Content-Type: application/json' -d '{"model":"slow","messages":[]}'
show -X POST localhost:18065/api/chat/preview -H 'Content-Type: application/json' -d '{"model":'
show localhost:18065/api/models/x/tokens
show -X DELETE localhost:18065/api/version

HTTP/1.1 404 
Content-Type: application/problem+json
{"detail":"no engine named 'nope'","instance":"/api/engines/nope/models","status":404,"title":"Unknown engine","engine":"nope"}

HTTP/1.1 502 
Content-Type: application/problem+json
{"detail":"the engine is not reachable","instance":"/api/engines/tiny/models","status":502,"title":"Engine unreachable"}

HTTP/1.1 502 
Content-Type: application/problem+json
{"detail":"the engine answered with status 404","instance":"/api/engines/qwen-coder/chat","status":502,"title":"Engine error","upstream_status":404}

HTTP/1.1 504 
Content-Type: application/problem+json
{"detail":"the engine did not answer in time","instance":"/api/engines/qwen-coder/chat","status":504,"title":"Engine timeout"}

HTTP/1.1 400 
Content-Type: application/problem+json
{"detail":"Failed to read request","instance":"/api/chat/preview","status":400,"title":"Bad Request"}

HTTP/1.1 400 
Content-Type: application/problem+json
{"detail":"Required parameter 'text' is not present.","instance":"/api/models/x/tokens","status":400,"title":"Bad Request"}

HTTP/1.1 405 
Content-Type: application/problem+json
{"detail":"Method 'DELETE' is not supported.","instance":"/api/version","status":405,"title":"Method Not Allowed"}

The function show prints the status line, the content type, and the body of each call, followed by a blank line. The first four bodies come from the handlers written above, and each carries the content type application/problem+json. The extension properties (engine, upstream_status) appear beside the standard fields. The property upstream_status keeps the name that was passed to setProperty, because extra properties are written as given.

The last three come from ResponseEntityExceptionHandler: a truncated JSON body (400), a missing query parameter (400, the same case as Example 3, now with a readable detail), and a wrong HTTP method (405). The instance field holds the request path. The type field is omitted because it has its default value about:blank.

Example 13: validating request bodies with @Valid

Example 4 accepted any body that parsed. A gateway should reject bad values before they reach an engine. Jakarta Validation annotations on the record components state the rules, and @Valid on the controller parameter makes Spring check them.

package loop.gateway;

import jakarta.validation.Valid;
import jakarta.validation.constraints.DecimalMax;
import jakarta.validation.constraints.DecimalMin;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.Positive;
import java.util.List;

public final class ChatPreview {
    private ChatPreview() { }

    public record Message(@NotBlank String role, @NotBlank String content) { }

    public record Request(
            @NotBlank String model,
            @NotEmpty List<@Valid Message> messages,
            @DecimalMin("0.0") @DecimalMax("2.0") Double temperature,
            @Positive Integer maxTokens) { }

    public record Response(String model, int messageCount, int promptChars, Integer maxTokens, String note) { }
}

package loop.gateway;

import jakarta.validation.Valid;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
class ChatPreviewController {

    @PostMapping("/api/chat/preview")
    ChatPreview.Response preview(@Valid @RequestBody ChatPreview.Request request) {
        int chars = request.messages().stream().mapToInt(m -> m.content().length()).sum();
        return new ChatPreview.Response(request.model(), request.messages().size(), chars,
                request.maxTokens(), "temperature=" + request.temperature());
    }
}

The rules in the record are:

• @DecimalMin and @DecimalMax bound the temperature to 0.0 through 2.0.
• @Positive requires maxTokens above zero.
• @Valid inside List<@Valid Message> validates every message in the list.

null passes @DecimalMin, @DecimalMax, and @Positive, so an absent optional field stays valid. By default, the ProblemDetail for a validation failure says only "Invalid request content." and hides the field errors. The handleMethodArgumentNotValid override in Example 12 collects them into a sorted errors map. The request below has four problems:

curl -s -X POST localhost:18065/api/chat/preview -H 'Content-Type: application/json' \
  -d '{"model":"","messages":[{"role":"user","content":""}],"temperature":3,"max_tokens":0}' | jq .

{
  "detail": "the request body has 4 invalid field(s)",
  "instance": "/api/chat/preview",
  "status": 400,
  "title": "Bad Request",
  "errors": {
    "maxTokens": "must be greater than 0",
    "messages[0].content": "must not be blank",
    "model": "must not be blank",
    "temperature": "must be less than or equal to 2.0"
  }
}

The response lists all four violations in one answer, sorted by field path (a TreeMap), so a client fixes everything in one round trip. The path messages[0].content points into the list. The names are the Java property names (maxTokens), not the JSON names (max_tokens), because the validator reports the path of the record component. A gateway that must report JSON names maps them in the handler.

Example 14: @SpringBootTest and MockMvc

Tests protect the gateway when the code changes. A @SpringBootTest starts the complete application context inside the test, with the same auto-configuration and the same application.yml as the real program. MockMvc sends requests straight to the DispatcherServlet without opening a port, and checks the response with fluent matchers. Tests are therefore fast and need no free port.

In Spring Boot 4, @AutoConfigureMockMvc lives in org.springframework.boot.webmvc.test.autoconfigure and requires the spring-boot-starter-webmvc-test dependency from Example 1.

package loop.gateway;

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.http.MediaType;
import org.springframework.test.web.servlet.MockMvc;

@SpringBootTest
@AutoConfigureMockMvc
class GatewayApiTests {

    @Autowired
    MockMvc mvc;

    @Test
    void versionIsPlainText() throws Exception {
        mvc.perform(get("/api/version"))
                .andExpect(status().isOk())
                .andExpect(content().string("loop-gateway 0.1.0"));
    }

    @Test
    void previewUsesSnakeCase() throws Exception {
        String body = """
                {"model":"qwen-coder","messages":[{"role":"user","content":"hello"}],"max_tokens":32}""";
        mvc.perform(post("/api/chat/preview").contentType(MediaType.APPLICATION_JSON).content(body))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.message_count").value(1))
                .andExpect(jsonPath("$.prompt_chars").value(5))
                .andExpect(jsonPath("$.max_tokens").value(32));
    }

    @Test
    void configIsBoundFromApplicationYml() throws Exception {
        mvc.perform(get("/api/config"))
                .andExpect(jsonPath("$.max_loaded_models").value(2))
                .andExpect(jsonPath("$.engines.length()").value(2))
                .andExpect(jsonPath("$.engines[0].parallel").value(2));
    }
}

Each test performs one request and asserts on the status and the body. jsonPath("$.message_count") reads the snake_case name, so the test verifies the Jackson setting of Example 4 as well. The third test verifies that the binding and the engine list of Example 5 work. The text block (""") keeps the JSON readable. @Autowired injects the MockMvc bean that @AutoConfigureMockMvc registers.

mvn -B test

[INFO] Running loop.gateway.GatewayApiTests
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.763 s -- in loop.gateway.GatewayApiTests

The summary line shows three tests and no failures. The elapsed time varies. It is mostly the startup of the application context, which the three tests share: Spring caches the context between test methods of a class, and between test classes with the same configuration.

Example 15: property overrides and @MockitoBean in tests

Tests need two more tools: a way to change a property for one test class, and a way to replace a bean so that no engine is needed. @SpringBootTest(properties = "...") sets properties with a priority above the files. @MockitoBean replaces a bean in the context with a Mockito mock (a stand-in object), and the test then defines what the mock returns or throws.

package loop.gateway;

import static org.mockito.BDDMockito.given;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

import java.net.http.HttpTimeoutException;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
import org.springframework.http.MediaType;
import org.springframework.test.context.bean.override.mockito.MockitoBean;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.web.client.ResourceAccessException;

@SpringBootTest(properties = "loop.gateway.max-loaded-models=3")
@AutoConfigureMockMvc
class ErrorHandlingTests {

    @Autowired
    MockMvc mvc;

    @MockitoBean
    EngineClient engines;

    @Test
    void propertyOverrideApplies() throws Exception {
        mvc.perform(get("/api/config")).andExpect(jsonPath("$.max_loaded_models").value(3));
    }

    @Test
    void invalidBodyListsEveryField() throws Exception {
        mvc.perform(post("/api/chat/preview").contentType(MediaType.APPLICATION_JSON)
                        .content("{\"model\":\"\",\"messages\":[],\"max_tokens\":-1}"))
                .andExpect(status().isBadRequest())
                .andExpect(content().contentTypeCompatibleWith(MediaType.APPLICATION_PROBLEM_JSON))
                .andExpect(jsonPath("$.errors.model").value("must not be blank"))
                .andExpect(jsonPath("$.errors.messages").value("must not be empty"))
                .andExpect(jsonPath("$.errors.maxTokens").value("must be greater than 0"));
    }

    @Test
    void upstreamTimeoutBecomes504() throws Exception {
        given(engines.models("qwen-coder"))
                .willThrow(new ResourceAccessException("I/O error", new HttpTimeoutException("get")));
        mvc.perform(get("/api/engines/qwen-coder/models"))
                .andExpect(status().isGatewayTimeout())
                .andExpect(jsonPath("$.title").value("Engine timeout"));
    }

    @Test
    void unknownEngineBecomes404() throws Exception {
        given(engines.models("nope")).willThrow(new UnknownEngineException("nope"));
        mvc.perform(get("/api/engines/nope/models"))
                .andExpect(status().isNotFound())
                .andExpect(jsonPath("$.engine").value("nope"));
    }
}

The class replaces EngineClient with a mock, so these tests run without the fake engines. Each test shows one idea.

• propertyOverrideApplies shows that the override (max-loaded-models=3) beats the 2 of application.yml. This is the property source order of Example 8, inside a test.
• invalidBodyListsEveryField checks the content type and three of the field errors from Example 13.
• upstreamTimeoutBecomes504 makes the mock throw the exception that RestClient throws on a read timeout, and verifies the mapping of Example 12 without waiting two seconds.
• unknownEngineBecomes404 does the same for the unknown engine.

A mock replaces the real client, so these tests cover the advice but not the HTTP client. Example 11 covered the client with real servers.

mvn -B test

[INFO] Running loop.gateway.ErrorHandlingTests
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.018 s -- in loop.gateway.ErrorHandlingTests
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Four more tests pass and the total is seven. A class with a different configuration (a changed property, a mock bean) cannot reuse the cached context of the first class, so Spring starts a second context for it. The elapsed times of the two classes are not comparable and vary between runs.

Example 16: running and packaging the jar

mvn package runs the tests and then builds the executable jar. A fat jar (Spring Boot calls it an executable jar) contains the application classes, every dependency, and a small launcher. The target machine needs only a JDK and the one command java -jar.

mvn -B clean package | grep -E "Tests run:|Building jar|repackage|BUILD|Replacing" | cut -c1-160 | sed 's|/tmp/claude-0/gw65/||'

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.100 s -- in loop.gateway.GatewayApiTests
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.551 s -- in loop.gateway.ErrorHandlingTests
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
[INFO] Building jar: target/loop-gateway-0.1.0.jar
[INFO] --- spring-boot:4.1.1:repackage (repackage) @ loop-gateway ---
[INFO] Replacing main artifact target/loop-gateway-0.1.0.jar with repackaged archive, adding nested dependencies in BOOT-INF/.
[INFO] BUILD SUCCESS

The build ran the seven tests, built a plain jar, and then the repackage goal replaced it with the executable one. The jar's manifest file shows how it starts:

unzip -p target/loop-gateway-0.1.0.jar META-INF/MANIFEST.MF | tr -d '\r' | grep -E "Main-Class|Start-Class|Spring-Boot-Version|Spring-Boot-Classes|Spring-Boot-Lib"

Main-Class: org.springframework.boot.loader.launch.JarLauncher
Start-Class: loop.gateway.GatewayApplication
Spring-Boot-Version: 4.1.1
Spring-Boot-Classes: BOOT-INF/classes/
Spring-Boot-Lib: BOOT-INF/lib/

java -jar runs JarLauncher. The launcher builds a class path from the nested jars in BOOT-INF/lib/ and calls the Start-Class, which is the application class of Example 2. The application's own classes and the three YAML files sit in BOOT-INF/classes/, so the profiles of Example 7 travel inside the jar.

The next run uses the prod profile and then stops the program with SIGTERM, the standard "please stop" signal:

java -jar target/loop-gateway-0.1.0.jar --spring.profiles.active=prod &
curl -s localhost:18065/api/config | jq -c '{max_loaded_models,idle_timeout}'
kill %1

INFO  l.g.GatewayApplication : The following 1 profile is active: "prod"
INFO  o.s.b.t.TomcatWebServer : Tomcat started on port 18065 (http) with context path '/'
{"max_loaded_models":4,"idle_timeout":"PT30M"}
INFO  o.s.b.t.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete
INFO  o.s.b.t.GracefulShutdown : Graceful shutdown complete

The prod profile gives 4 models and a 30-minute idle timeout, as in Example 7. On SIGTERM (the signal that kill sends and that service managers send on stop), Spring Boot shuts down gracefully. Tomcat stops accepting new connections, waits for requests in progress to finish, and then the context closes. This behavior is on by default in Spring Boot, and it matters for a gateway with long generations, because a stop does not cut a running answer in half.

The log lines between the first two and the last two (the servlet initialization on the first request) are omitted. The wait has a limit, spring.lifecycle.timeout-per-shutdown-phase, whose default is 30 seconds (server.shutdown defaults to graceful). The lesson on model lifecycle adds the step that stops the engines at the same point.

Interview points

• @SpringBootApplication is @SpringBootConfiguration, @EnableAutoConfiguration, and @ComponentScan. Auto-configuration creates beans from the classes on the class path, and user-defined beans take precedence.
• The starter parent supplies dependency versions, plugin configuration, and the compiler release from java.version. Starters declare no versions.
• Spring Boot 4 uses Jackson 3 (tools.jackson packages) for its JSON; annotations stay in com.fasterxml.jackson.annotation. Spring Boot disables failure on unknown properties.
• Boxed types distinguish an absent JSON field (null) from a zero value. Primitives cannot.
• @ConfigurationProperties records bind through the constructor, support relaxed binding (kebab-case files, upper-case environment variables), and validate with @Validated at startup.
• Property precedence, low to high: files inside the jar, profile files, environment variables, command-line arguments. Profile files override only the keys they contain.
• spring.threads.virtual.enabled=true runs Tomcat request handling on virtual threads. Blocking RestClient calls then do not hold operating-system threads while they wait.
• RestClient throws RestClientResponseException for 4xx and 5xx answers and ResourceAccessException for timeouts and connection failures; a gateway maps them to 502 and 504. Timeouts are set with spring.http.clients.connect-timeout and read-timeout.
• ProblemDetail with @RestControllerAdvice gives application/problem+json errors with extension properties. ResponseEntityExceptionHandler covers Spring's own exceptions.
• @SpringBootTest with MockMvc tests the whole stack without a port. @MockitoBean replaces a bean, and tests with different configurations start separate contexts.
• Graceful shutdown is on by default: the server finishes requests in progress up to a time limit before the context closes.

Summary

• A Spring Boot project is a Maven project with spring-boot-starter-parent 4.1.1 as parent; starters declare dependencies without versions.
• This section uses Spring MVC with RestClient; it has no reactive dependency.
• The build produced an executable jar that contains Jackson 3.1.5, Tomcat 11.0.24, and Spring Framework 7.0.9.
• @SpringBootApplication starts auto-configuration and component scanning; SpringApplication.run starts embedded Tomcat.
• @RestController methods return Java objects that Jackson writes as JSON; String returns are plain text.
• Records work as request and response bodies. spring.jackson.property-naming-strategy: SNAKE_CASE maps maxTokens to max_tokens in both directions.
• Unknown request fields are ignored by default in Spring Boot, and undeclared fields in a typed response record are dropped; a JsonNode passes a body through unchanged.
• @ConfigurationProperties with @Validated rejects bad configuration at startup and reports the property, value, origin, and reason.
• Profiles load application-<profile>.yml over application.yml; @Profile limits a bean to a profile.
• Command-line arguments beat environment variables, which beat profile files, which beat the base file.
• spring.threads.virtual.enabled=true changed the handler thread from http-nio-18065-exec-N to a virtual thread.
• The default RestClient has no read timeout; spring.http.clients.read-timeout sets one, and a timeout surfaces as ResourceAccessException with an HttpTimeoutException cause.
• ProblemDetail responses use the content type application/problem+json; the gateway uses 404 for unknown engines, 502 for engine errors and unreachable engines, and 504 for timeouts.
• @Valid plus Jakarta constraints reject invalid bodies with status 400; the handler in this lesson lists all field errors in an errors property.
• @SpringBootTest with MockMvc passed 7 tests; mvn package runs them before building the jar.
• java -jar starts JarLauncher, which starts the Start-Class; SIGTERM triggers a graceful shutdown.