Securing and Observing the Gateway

The gateway from the earlier lessons (Spring Boot Essentials for an LLM Gateway, An OpenAI-Compatible API in Front of llama-server, Streaming Responses Through the Gateway, Loading, Unloading, and Scheduling Models) forwards chat requests to an engine such as llama-server. It has two gaps. It answers anyone who can reach its network port, and it gives no view of how it is doing: whether the engine behind it is healthy, how long answers take, or how many requests are waiting.

This lesson closes both gaps. The first half is about security. An API key is a long secret string that identifies a caller. The gateway checks it, limits how often each key may call, and decides which browser pages may use the gateway. The second half is about observability, which means being able to see from the outside what a running program is doing. That covers health checks, metrics (numbers measured over time), and log lines. The last two examples package the gateway as one jar file and run it as a service on a Linux server.

After the lesson, a reader can require API keys, limit each key, allow browser clients, report health to an orchestrator (a tool that starts, stops, and routes traffic to programs), read metrics for latency and throughput, write one structured log line per request, and run the packaged jar on a Linux server. The table lists the ideas and the APIs used for each.

The gateway module uses Spring MVC, the classic Spring web framework in which each request is handled by one thread. The threads are virtual threads, which are very cheap threads managed by the JVM, so a long streamed answer does not tie up an expensive operating-system thread. Outgoing calls use RestClient, Spring's blocking HTTP client. The module has no reactive stack, which is the other Spring style that handles many requests with a few threads and callbacks.

Two kinds of code blocks appear in the lesson. Every Spring block carries the fence flag java,ignore, because the course's single-file Java checker cannot run Spring. The Spring code was compiled and run in a Maven project on Spring Boot 4.1.1 and JDK 25, and each output block is real output from that run. Helpers that need no framework are plain java blocks, and the course checker runs them.

The engine in every run is a small fake OpenAI-compatible server on 127.0.0.1:18691 (Example 10 shows its code). A fake engine answers instantly and predictably, so the numbers in the examples come from the gateway and not from a model. The gateway listens on 127.0.0.1:18690.

Example 1: what Spring Security does when it is added to the classpath

Spring Security is the Spring library that guards web applications. Before writing any security code, it helps to see what it does by default. The question is what changes when spring-boot-starter-security joins the project and no configuration exists yet. (A starter is a dependency that pulls in a library together with its Spring Boot setup.)

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-security</artifactId>
</dependency>

With only this dependency, Spring Boot builds a default filter chain that requires authentication for every request. It also creates one in-memory user with a generated password, so a developer can log in during development. The startup log prints the password:

Using generated security password: e457548d-f576-411f-bc0a-e0f27d7f5aaf

The password changes on every start, and the user is meant for development only. A gateway does not serve people who type passwords. It serves programs that send API keys. For that reason this project removes the default user with @SpringBootApplication(exclude = UserDetailsServiceAutoConfiguration.class) (in Spring Boot 4 the class lives in org.springframework.boot.security.autoconfigure). After the exclusion the log line no longer appears.

Two Spring Security terms matter from here on. A filter chain is an ordered list of servlet filters that every request passes through before it reaches a controller. A servlet filter is a piece of code that can inspect a request, change it, answer it, or pass it on. An authority is a string such as ROLE_ADMIN that is attached to an authenticated caller. Rules in the chain compare authorities with the request path, for example "paths under /api/pull need ROLE_ADMIN".

request --> [filter 1] --> [filter 2] --> [filter 3] --> controller
                |              |              |
          may answer     may answer     may answer
          instead        instead        instead

Each filter either passes the request to the next one or answers it itself. A request that a filter answers never reaches the controller.

Example 2: generating a key and storing only its hash

The problem in this example is how to keep keys safe on the server. If the gateway stored the keys themselves, anyone who read the configuration file (a backup, a leaked repository) could use them. The idea is to store a hash of each key instead. A hash function such as SHA-256 turns any text into a fixed-size fingerprint. The same text always gives the same fingerprint, but nobody can compute the original text from the fingerprint. When a client sends a key, the gateway hashes it and compares the result with the stored fingerprints.

An API key is a long random string that the client sends in the Authorization header. The program below generates a key, hashes a demo key, and compares a correct and a wrong key against the stored hash.

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.HexFormat;
import java.util.Random;

public class KeyHashing {
    static String newKey(Random random) {
        byte[] bytes = new byte[24];
        random.nextBytes(bytes);
        return "sk-" + HexFormat.of().formatHex(bytes);
    }

    static byte[] sha256(String text) throws Exception {
        return MessageDigest.getInstance("SHA-256").digest(text.getBytes(StandardCharsets.UTF_8));
    }

    public static void main(String[] args) throws Exception {
        String key = newKey(new Random(7));
        System.out.println("key length: " + key.length());
        System.out.println("key prefix: " + key.substring(0, 6));
        String stored = HexFormat.of().formatHex(sha256("sk-alice-demo"));
        System.out.println("stored hash: " + stored);

        byte[] presentedGood = sha256("sk-alice-demo");
        byte[] presentedBad = sha256("sk-alice-demX");
        byte[] expected = HexFormat.of().parseHex(stored);
        System.out.println("good key matches: " + MessageDigest.isEqual(presentedGood, expected));
        System.out.println("bad key matches: " + MessageDigest.isEqual(presentedBad, expected));
    }
}

key length: 51
key prefix: sk-991
stored hash: 4805d60cb37819015d8fffb916fd911836bc573c4c5963d9f1a24a543bd352d7
good key matches: true
bad key matches: false

Walk through the output line by line. The key is sk- followed by 48 hex characters. Each byte takes two hex characters, so 24 random bytes (192 bits) give 48 characters, and with the 3-character prefix the length is 51. The first six characters, sk-991, depend on the seed. The demo seeds Random so the output repeats on every run. A real key must come from java.security.SecureRandom, which is built to be unpredictable.

The stored hash is the 64-character hex form of the 256-bit SHA-256 digest (256 bits divided by 4 bits per hex character is 64). The good key hashes to the same value as the stored one, so isEqual returns true. The bad key differs by one character (X instead of o), and its hash is completely different, so the result is false. A one-character change in the input changes the whole hash. For that reason the stored hash gives no hint about how close a guess was.

A fast hash such as SHA-256 is suitable here because the key is random and long. Password hashes such as bcrypt are slow on purpose, to make guessing short human-chosen passwords expensive. That protection is not needed for a 192-bit random key, which no attacker can guess.

MessageDigest.isEqual compares two byte arrays in a time that does not depend on where they first differ. An ordinary comparison stops at the first different byte. An attacker who measures response times could then learn how many leading bytes of a guess were right (a timing attack). The constant-time comparison removes that signal.

The sha256sum command produces the same stored value from a shell, which is how an operator prepares an entry for the configuration file: printf 'sk-alice-demo' | sha256sum. The printf form avoids the trailing newline that echo would add and change the hash.

Example 3: keys, limits, and the engine address in @ConfigurationProperties

Keys, roles, limits, and the allowed browser origins belong in configuration, so that each machine can have its own values and the code does not change. The record below holds them. It binds the gateway prefix of application.yml, as taught in Spring Boot Essentials for an LLM Gateway. Binding means that Spring reads the file and fills the record's fields with the values.

package demo;

import java.util.List;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties("gateway")
public record GatewayProperties(
        List<Key> keys,
        Engine engine,
        RateLimit rateLimit,
        List<String> corsOrigins) {

    public record Key(String name, String sha256, List<String> roles, Integer requestsPerMinute) {}

    public record Engine(String baseUrl, int maxConcurrent) {}

    public record RateLimit(int defaultRequestsPerMinute, int burst) {}
}

# src/main/resources/application.yml (security part)
gateway:
  engine:
    base-url: http://127.0.0.1:18691
    max-concurrent: 1
  rate-limit:
    default-requests-per-minute: 60
    burst: 5
  cors-origins:
    - http://localhost:3000
  keys:
    - name: alice
      sha256: "4805d60cb37819015d8fffb916fd911836bc573c4c5963d9f1a24a543bd352d7"
      roles: [USER]
    - name: admin
      sha256: "c58c64d2318a6223f34060bdb7b09b4181de662820371d4bddfdcc6042654265"
      roles: [ADMIN]
      requests-per-minute: 600

The YAML defines two keys. alice is a normal client with the role USER, and admin may manage models with the role ADMIN. The sha256 value of alice is the hash printed in Example 2, so the client sends the secret sk-alice-demo and the file holds only its hash.

requests-per-minute is optional. It is an Integer and not an int, so it is null when absent, and then the gateway falls back to rate-limit.default-requests-per-minute. @ConfigurationPropertiesScan on the application class registers the record as a bean without a separate @Bean method.

The KeyStore below is the lookup. It hashes the presented key and compares the hash with every stored hash. It does not stop at the first match, so the time taken is the same whichever entry matched, and a timing measurement reveals nothing about the key list.

package demo;

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
import java.util.Optional;
import org.springframework.stereotype.Component;

@Component
public class KeyStore {
    private final GatewayProperties props;

    public KeyStore(GatewayProperties props) {
        this.props = props;
    }

    public Optional<GatewayProperties.Key> find(String presented) {
        byte[] hash = sha256(presented);
        GatewayProperties.Key found = null;
        for (GatewayProperties.Key key : props.keys()) {
            byte[] expected = HexFormat.of().parseHex(key.sha256());
            if (MessageDigest.isEqual(hash, expected)) {
                found = key;
            }
        }
        return Optional.ofNullable(found);
    }

    static byte[] sha256(String text) {
        try {
            return MessageDigest.getInstance("SHA-256").digest(text.getBytes(StandardCharsets.UTF_8));
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);
        }
    }
}

Example 4: the API key filter and the security filter chain

The next piece is the filter that checks the key. It reads Authorization: Bearer <key> (the standard way to send a token in an HTTP header), looks the key up, and puts an authenticated caller into the SecurityContext, which is Spring Security's holder for "who is making this request". There are three outcomes:

• No Authorization header: the request passes through unauthenticated. The rules in the chain decide afterwards whether an unauthenticated caller may continue, for example on the open health path.
• A header that is not Bearer, or a key that is unknown: the filter rejects the request at once with a 401 problem document and a WWW-Authenticate header, which tells clients which scheme the server expects.
• A known key: the caller is authenticated with the roles of that key.

package demo;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.List;
import java.util.Optional;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.web.filter.OncePerRequestFilter;

public class ApiKeyFilter extends OncePerRequestFilter {
    private final KeyStore keys;

    public ApiKeyFilter(KeyStore keys) {
        this.keys = keys;
    }

    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        String header = req.getHeader(HttpHeaders.AUTHORIZATION);
        if (header == null) {
            chain.doFilter(req, res);
            return;
        }
        if (!header.startsWith("Bearer ")) {
            reject(res, "Authorization must use the Bearer scheme");
            return;
        }
        Optional<GatewayProperties.Key> key = keys.find(header.substring(7).trim());
        if (key.isEmpty()) {
            reject(res, "Invalid API key");
            return;
        }
        List<SimpleGrantedAuthority> roles = key.get().roles().stream()
                .map(r -> new SimpleGrantedAuthority("ROLE_" + r)).toList();
        req.setAttribute("gateway.key", key.get().name());
        SecurityContextHolder.getContext().setAuthentication(
                UsernamePasswordAuthenticationToken.authenticated(key.get().name(), null, roles));
        chain.doFilter(req, res);
    }

    static void reject(HttpServletResponse res, String detail) throws IOException {
        res.setStatus(401);
        res.setHeader(HttpHeaders.WWW_AUTHENTICATE, "Bearer realm=\"loop-gateway\"");
        res.setContentType(MediaType.APPLICATION_PROBLEM_JSON_VALUE);
        res.setCharacterEncoding("UTF-8");
        res.getWriter().write("{\"type\":\"about:blank\",\"title\":\"Unauthorized\",\"status\":401,\"detail\":\""
                + detail + "\"}");
    }
}

Now the chain itself. It registers the API key filter before BasicAuthenticationFilter, turns off the features of a browser login application (CSRF tokens, sessions, form login, HTTP Basic), and ends with denyAll(), so that any path without a rule is closed. The rate limit filter of Example 7 sits right after the key filter. The order of a request through the gateway is:

request
  |
  v
ApiKeyFilter          reads the key, sets caller and roles (or answers 401)
  |
  v
RateLimitFilter       takes a token from the caller's bucket (or answers 429)
  |
  v
authorization rules   compare the path with the roles (or answer 401 / 403)
  |
  v
controller

The code below defines the chain and the CORS settings that the chain uses.

package demo;

import jakarta.servlet.DispatcherType;
import java.util.List;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpMethod;
import org.springframework.security.config.Customizer;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configurers.AbstractHttpConfigurer;
import org.springframework.security.config.http.SessionCreationPolicy;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.authentication.www.BasicAuthenticationFilter;
import org.springframework.web.cors.CorsConfiguration;
import org.springframework.web.cors.CorsConfigurationSource;
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;

@Configuration
public class SecurityConfig {

    @Bean
    SecurityFilterChain chain(HttpSecurity http, KeyStore keys, GatewayProperties props) throws Exception {
        return http
                .csrf(AbstractHttpConfigurer::disable)
                .sessionManagement(s -> s.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
                .cors(Customizer.withDefaults())
                .httpBasic(AbstractHttpConfigurer::disable)
                .formLogin(AbstractHttpConfigurer::disable)
                .addFilterBefore(new ApiKeyFilter(keys), BasicAuthenticationFilter.class)
                .addFilterAfter(new RateLimitFilter(props), ApiKeyFilter.class)
                .authorizeHttpRequests(a -> a
                        .dispatcherTypeMatchers(DispatcherType.ASYNC, DispatcherType.ERROR).permitAll()
                        .requestMatchers("/actuator/health/**").permitAll()
                        .requestMatchers(HttpMethod.OPTIONS, "/**").permitAll()
                        .requestMatchers("/actuator/**").hasRole("ADMIN")
                        .requestMatchers("/api/pull", "/api/delete").hasRole("ADMIN")
                        .requestMatchers("/v1/**", "/api/**").hasAnyRole("USER", "ADMIN")
                        .anyRequest().denyAll())
                .exceptionHandling(e -> e
                        .authenticationEntryPoint((req, res, ex) -> ApiKeyFilter.reject(res, "API key required"))
                        .accessDeniedHandler((req, res, ex) -> {
                            res.setStatus(403);
                            res.setContentType("application/problem+json");
                            res.setCharacterEncoding("UTF-8");
                            res.getWriter().write("{\"type\":\"about:blank\",\"title\":\"Forbidden\",\"status\":403,"
                                    + "\"detail\":\"The key lacks the required role\"}");
                        }))
                .build();
    }

    @Bean
    CorsConfigurationSource corsConfigurationSource(GatewayProperties props) {
        CorsConfiguration cors = new CorsConfiguration();
        cors.setAllowedOrigins(props.corsOrigins());
        cors.setAllowedMethods(List.of("GET", "POST", "DELETE"));
        cors.setAllowedHeaders(List.of("Authorization", "Content-Type"));
        cors.setMaxAge(600L);
        UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
        source.registerCorsConfiguration("/**", cors);
        return source;
    }
}

Three decisions in this code need a reason.

The filter is created with new and is not a Spring bean. Spring Boot registers every Filter bean as a servlet filter on its own, in addition to its place in the security chain, so the filter would run twice.

CSRF protection (cross-site request forgery protection) defends browser sessions that rely on cookies. The gateway has no cookies, and every request carries its own key, so the protection is off. STATELESS tells Spring Security not to store the SecurityContext in an HTTP session, which means each request authenticates again from its key.

Calls without a key, and with a wrong key, return the following. The -D - option of curl prints the response headers and -o /dev/null drops the body, so the first and third commands show only the status line and the WWW-Authenticate header.

curl -s -D - -o /dev/null localhost:18690/v1/chat/completions -d '{}' | grep -E '^(HTTP|WWW)'
curl -s -w ' [HTTP %{http_code}]\n' localhost:18690/v1/chat/completions -d '{}'
curl -s -D - -o /dev/null -H "Authorization: Bearer nope" localhost:18690/v1/chat/completions -d '{}' | grep -E '^(HTTP|WWW)'
curl -s -w ' [HTTP %{http_code}]\n' -H "Authorization: Bearer nope" localhost:18690/v1/chat/completions -d '{}'

HTTP/1.1 401 
WWW-Authenticate: Bearer realm="loop-gateway"
{"type":"about:blank","title":"Unauthorized","status":401,"detail":"API key required"} [HTTP 401]
HTTP/1.1 401 
WWW-Authenticate: Bearer realm="loop-gateway"
{"type":"about:blank","title":"Unauthorized","status":401,"detail":"Invalid API key"} [HTTP 401]

The two responses differ only in detail: "API key required" when no key was sent, and "Invalid API key" when the key is unknown. Both use the ProblemDetail media type application/problem+json that the error handling in Spring Boot Essentials for an LLM Gateway uses, so clients parse one error format for every failure. The -w option of curl appends the status code after the body. The filter sets the character encoding to UTF-8 explicitly. Without it, the servlet container labeled the body ISO-8859-1 in this run.

Example 5: roles and authorization rules

Authentication answers "who is calling". Authorization answers "what may this caller do". A valid key is not always enough. Model management endpoints such as /api/pull change what the machine holds, so only keys with the ADMIN role may call them. The rules in SecurityConfig are checked from top to bottom, and the first rule that matches wins. This is why the specific rules come before the general ones.

curl -s -w ' [HTTP %{http_code}]\n' -X POST -H "Authorization: Bearer sk-alice-demo" localhost:18690/api/pull
curl -s -w ' [HTTP %{http_code}]\n' -X POST -H "Authorization: Bearer sk-admin-demo" localhost:18690/api/pull
curl -s -w ' [HTTP %{http_code}]\n' -H "Authorization: Bearer sk-alice-demo" localhost:18690/nothing

{"type":"about:blank","title":"Forbidden","status":403,"detail":"The key lacks the required role"} [HTTP 403]
{"status":"pulling"} [HTTP 200]
{"type":"about:blank","title":"Forbidden","status":403,"detail":"The key lacks the required role"} [HTTP 403]

The three calls show the three results. alice authenticates but holds only USER, so /api/pull returns 403 Forbidden. The difference between the two error codes is: a 401 means the caller is unknown, and a 403 means the caller is known and lacks permission. The admin key reaches the handler, which answers 200 with a status body. The path /nothing has no rule, so denyAll() answers 403 even for a known caller.

The .dispatcherTypeMatchers(DispatcherType.ASYNC, DispatcherType.ERROR).permitAll() line at the top of the rules has a reason that Example 13 explains: streamed responses finish on a second dispatch of the request, and the security context of that dispatch is empty.

Example 6: a token bucket

Rate limiting needs a rule for how fast one key may call. Without a limit, one client with a bug (or bad intent) can fill the engine's queue and starve everyone else. A token bucket is a common rule. Picture a bucket that holds at most burst tokens. Tokens drip in at a fixed rate, and each request removes one token. A request that finds no token is refused. The bucket allows a short burst of quick requests and then holds the long-run rate. The mapping to the real code is direct: the bucket is the tokens field, the drip is the refill computed from elapsed time, and the refusal is a return value above zero. The picture stops being accurate in one place: no clock drips tokens. The code adds the tokens that would have arrived since the last call.

capacity (burst)   3 tokens
refill             30 per minute = 0.5 tokens per second
each request       takes 1 token
one token needs    1 / 0.5 = 2 seconds to refill from empty

The class below is the one the gateway uses. The program drives it with a fake clock (times passed in by hand), so the output repeats.

public class BucketDemo {
    static final class TokenBucket {
        private final double ratePerNano;
        private final double capacity;
        private double tokens;
        private long lastNanos;

        TokenBucket(int perMinute, int burst, long nowNanos) {
            this.ratePerNano = perMinute / 60.0 / 1_000_000_000.0;
            this.capacity = burst;
            this.tokens = burst;
            this.lastNanos = nowNanos;
        }

        /** Returns 0 when a token was taken, otherwise the nanoseconds until one is available. */
        synchronized long tryAcquire(long nowNanos) {
            tokens = Math.min(capacity, tokens + (nowNanos - lastNanos) * ratePerNano);
            lastNanos = nowNanos;
            if (tokens >= 1.0) {
                tokens -= 1.0;
                return 0;
            }
            return (long) Math.ceil((1.0 - tokens) / ratePerNano);
        }
    }

    public static void main(String[] args) {
        long second = 1_000_000_000L;
        TokenBucket bucket = new TokenBucket(30, 3, 0);
        long[] times = {0, 0, 0, 0, second, 2 * second, 2 * second, 10 * second, 10 * second, 10 * second, 10 * second};
        for (long t : times) {
            long wait = bucket.tryAcquire(t);
            String result = wait == 0 ? "allowed" : String.format("refused, retry in %.1f s", wait / 1e9);
            System.out.printf("t=%4.1f s  %s%n", t / 1e9, result);
        }
    }
}

t= 0.0 s  allowed
t= 0.0 s  allowed
t= 0.0 s  allowed
t= 0.0 s  refused, retry in 2.0 s
t= 1.0 s  refused, retry in 1.0 s
t= 2.0 s  allowed
t= 2.0 s  refused, retry in 2.0 s
t=10.0 s  allowed
t=10.0 s  allowed
t=10.0 s  allowed
t=10.0 s  refused, retry in 2.0 s

Read the output with the numbers from the diagram above. The bucket is configured for 30 requests per minute, which is 0.5 tokens per second, with a burst of 3.

• The first three requests at t=0 use the full bucket, so they are allowed.
• The fourth finds an empty bucket and needs one whole token. At 0.5 tokens per second that takes 2.0 seconds, so it is refused with a 2.0 s wait.
• At t=1.0 half a token has refilled (1 s times 0.5), so the missing amount is 0.5 token and the wait is 1.0 second.
• At t=2.0 one token has refilled, so one request passes. The next request at the same instant finds the bucket empty again and waits 2.0 seconds.
• After eight idle seconds the bucket would hold 4 tokens (8 s times 0.5), but the capacity caps it at 3. Three requests pass and the fourth is refused.

The cap is what keeps a long idle period from storing an unlimited number of tokens, which would allow a huge burst later. The method returns the wait in nanoseconds, so the filter can turn it into a Retry-After header without a second calculation.

Example 7: the rate limit filter and 429

The filter runs after the API key filter. It finds the caller's bucket by key name and answers 429 Too Many Requests when no token is left. The answer carries a Retry-After header, which tells the client how many whole seconds to wait before it tries again. Unauthenticated requests pass through, because the rules in the chain refuse them next, and a refused caller should not use up a bucket.

package demo;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.springframework.http.MediaType;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.web.filter.OncePerRequestFilter;

public class RateLimitFilter extends OncePerRequestFilter {
    private final GatewayProperties props;
    private final Map<String, TokenBucket> buckets = new ConcurrentHashMap<>();

    public RateLimitFilter(GatewayProperties props) {
        this.props = props;
    }

    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        if (auth == null || !auth.isAuthenticated()) {
            chain.doFilter(req, res);
            return;
        }
        String name = auth.getName();
        int perMinute = props.keys().stream().filter(k -> k.name().equals(name)).findFirst()
                .map(GatewayProperties.Key::requestsPerMinute)
                .orElse(props.rateLimit().defaultRequestsPerMinute());
        long now = System.nanoTime();
        TokenBucket bucket = buckets.computeIfAbsent(name,
                n -> new TokenBucket(perMinute, props.rateLimit().burst(), now));
        long waitNanos = bucket.tryAcquire(now);
        if (waitNanos > 0) {
            long seconds = Math.max(1, (waitNanos + 999_999_999L) / 1_000_000_000L);
            res.setStatus(429);
            res.setHeader("Retry-After", Long.toString(seconds));
            res.setContentType(MediaType.APPLICATION_PROBLEM_JSON_VALUE);
            res.setCharacterEncoding("UTF-8");
            res.getWriter().write("{\"type\":\"about:blank\",\"title\":\"Too Many Requests\",\"status\":429,"
                    + "\"detail\":\"Rate limit exceeded for key " + name + "\"}");
            return;
        }
        chain.doFilter(req, res);
    }
}

The gateway ran with --spring.profiles.active=laptop (Example 16: 30 requests per minute, burst 3). A profile is a named set of settings that Spring Boot loads on top of the defaults. After eight idle seconds, which refill the bucket to its burst of 3, the commands send four requests in a row and then a fifth with its headers. A final call uses the admin key:

for i in 1 2 3 4; do
  curl -s -o /dev/null -w "%{http_code} " -H "Authorization: Bearer sk-alice-demo" \
    -H "Content-Type: application/json" localhost:18690/v1/chat/completions \
    -d '{"model":"tiny","messages":[]}'
done; echo
curl -si -H "Authorization: Bearer sk-alice-demo" -H "Content-Type: application/json" \
  localhost:18690/v1/chat/completions -d '{"model":"tiny","messages":[]}' | grep -E '^(HTTP|Retry-After|\{)'
curl -s -w ' [HTTP %{http_code}]\n' -X POST -H "Authorization: Bearer sk-admin-demo" localhost:18690/api/pull

200 200 200 429 
HTTP/1.1 429 
Retry-After: 2
{"type":"about:blank","title":"Too Many Requests","status":429,"detail":"Rate limit exceeded for key alice"}
{"status":"pulling"} [HTTP 200]

The first line shows 200 200 200 429: three requests pass and the fourth is refused, as in Example 6. The fifth request, sent right afterwards, is also refused, and its headers show Retry-After: 2. The value is the refused request's wait rounded up to a whole second, which is the form HTTP clients and SDKs read.

computeIfAbsent creates each bucket the first time its key appears, so the map holds one small object per key. Each key has its own bucket. The call with the admin key right after the refusals returned 200, because admin had a full bucket of its own.

A limit counted in requests does not distinguish a request that generates five tokens from one that generates five thousand. A limit in tokens needs the engine's usage numbers after the response, and a bucket that can go negative (a long answer is charged after it finishes). The metrics in Example 12 record those numbers.

Example 8: CORS for browser clients

CORS (cross-origin resource sharing) is a browser rule. An origin is the combination of scheme, host, and port, for example http://localhost:3000. A web page from one origin may read answers from a server at another origin only if that server allows it. The rule exists so that a malicious page cannot make your browser read data from a service you are logged in to. Command-line tools do not enforce it.

Before a browser sends a request that carries an Authorization header, it first sends a preflight request. This is an OPTIONS call that asks which method and headers the real request will use. The gateway names the allowed origins, methods, and headers, and cors(Customizer.withDefaults()) in the chain picks up the CorsConfigurationSource bean from Example 4. Preflight requests carry no Authorization header, because the browser sends them before it sends the real request, so the rules permit OPTIONS without a key.

The three commands below act like a browser. The first is a preflight from an allowed origin, the second is a preflight from an unknown origin, and the third is the real request from the allowed origin, with a key.

curl -si -X OPTIONS -H "Origin: http://localhost:3000" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: authorization,content-type" \
  localhost:18690/v1/chat/completions | grep -iE '^(HTTP|Access-Control)'
curl -s -w ' [HTTP %{http_code}]\n' -X OPTIONS -H "Origin: http://evil.example" \
  -H "Access-Control-Request-Method: POST" localhost:18690/v1/chat/completions
curl -si -H "Authorization: Bearer sk-alice-demo" -H "Content-Type: application/json" \
  -H "Origin: http://localhost:3000" localhost:18690/v1/chat/completions \
  -d '{"model":"tiny","messages":[]}' | grep -iE '^(HTTP|Access-Control)'

HTTP/1.1 200 
Access-Control-Allow-Origin: http://localhost:3000
Access-Control-Allow-Methods: GET,POST,DELETE
Access-Control-Allow-Headers: authorization, content-type
Access-Control-Max-Age: 600
Invalid CORS request [HTTP 403]
HTTP/1.1 200 
Access-Control-Allow-Origin: http://localhost:3000

The allowed origin receives the allow headers, and Access-Control-Max-Age: 600 lets the browser cache the answer for 600 seconds, so it does not repeat the preflight for every request. The unknown origin receives 403. The third call is the real request from the allowed origin, with a key. Its response also carries Access-Control-Allow-Origin: http://localhost:3000, which the browser requires before it hands the response to the page.

CORS protects users of browsers. It does not authenticate anything, and curl ignores it. The API key rules still apply to every request. The configuration lists exact origins and does not use a wildcard.

Example 9: Actuator endpoints and who may read them

Actuator is Spring Boot's set of operational endpoints: health, metrics, and others. Orchestrators, load balancers, and monitoring tools call them to learn how the program is doing. Endpoints are enabled by default, but only those named in exposure.include are reachable over HTTP. The configuration below exposes three. It also turns on the health probe groups, which are separate health answers for separate questions (explained after the output).

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,info
  endpoint:
    health:
      probes:
        enabled: true
      show-details: never
      group:
        readiness:
          include: readinessState,engine

The security rules from Example 5 leave /actuator/health/** open and require ADMIN for the rest, because metrics reveal model names, request counts, and timing. The setting show-details: never keeps component details out of the open health response, because those details could help an attacker.

curl -s localhost:18690/actuator/health
curl -s localhost:18690/actuator/health/liveness
curl -s localhost:18690/actuator/health/readiness
curl -s -w ' [HTTP %{http_code}]\n' localhost:18690/actuator/metrics

{"groups":["liveness","readiness"],"status":"UP"}
{"status":"UP"}
{"status":"UP"}
{"type":"about:blank","title":"Unauthorized","status":401,"detail":"API key required"} [HTTP 401]

/actuator/health aggregates every health contributor and lists the two groups. The two group endpoints each answer one question.

• Liveness answers "is this process alive". An orchestrator restarts the process when it fails.
• Readiness answers "may this process receive traffic". A load balancer removes the process from rotation when it fails, without restarting it.

The /actuator/metrics call without a key is refused with 401. Exposing metrics on a separate management port (management.server.port) is a second option that keeps them off the public listener. This lesson uses one port and the role rule.

Example 10: readiness that follows the engine

The gateway cannot serve a request when its engine is down or still loading a model. Readiness should therefore depend on the engine. llama-server has a /health endpoint that answers 503 while the model loads and 200 when ready (see The llama-server HTTP API). A custom HealthIndicator is a class that Actuator asks for one health answer. The class below calls the engine's /health with short timeouts, so a hung engine cannot hang the probe. Any exception turns into DOWN.

package demo;

import org.springframework.boot.health.contributor.Health;
import org.springframework.boot.health.contributor.HealthIndicator;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import java.time.Duration;

@Component("engine")
public class EngineHealthIndicator implements HealthIndicator {
    private final RestClient http;
    private final String baseUrl;

    public EngineHealthIndicator(GatewayProperties props) {
        this.baseUrl = props.engine().baseUrl();
        SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(Duration.ofMillis(500));
        factory.setReadTimeout(Duration.ofMillis(500));
        this.http = RestClient.builder().baseUrl(baseUrl).requestFactory(factory).build();
    }

    @Override
    public Health health() {
        try {
            http.get().uri("/health").retrieve().toBodilessEntity();
            return Health.up().withDetail("url", baseUrl).build();
        } catch (Exception e) {
            return Health.down().withDetail("url", baseUrl).withDetail("error", e.getClass().getSimpleName()).build();
        }
    }
}

The bean name engine is the component name, and Actuator uses it as the contributor id. The readiness group in Example 9 lists it. To test the behavior, the fake engine used in every run has a toggle that switches its /health between 200 and 503:

import com.sun.net.httpserver.HttpServer;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.atomic.AtomicBoolean;

public class FakeEngine {
    static void nap(long ms) {
        try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    public static void main(String[] args) throws Exception {
        int port = Integer.parseInt(args[0]);
        AtomicBoolean healthy = new AtomicBoolean(true);
        HttpServer s = HttpServer.create(new InetSocketAddress("127.0.0.1", port), 0);
        s.createContext("/health", ex -> {
            byte[] b = (healthy.get() ? "{\"status\":\"ok\"}" : "{\"error\":\"loading model\"}").getBytes();
            ex.sendResponseHeaders(healthy.get() ? 200 : 503, b.length);
            ex.getResponseBody().write(b);
            ex.close();
        });
        s.createContext("/_toggle", ex -> {
            healthy.set(!healthy.get());
            ex.sendResponseHeaders(204, -1);
            ex.close();
        });
        s.createContext("/v1/chat/completions", ex -> {
            String body = new String(ex.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
            if (body.contains("\"stream\":true")) {
                ex.getResponseHeaders().add("Content-Type", "text/event-stream");
                ex.sendResponseHeaders(200, 0);
                try (OutputStream os = ex.getResponseBody()) {
                    nap(120);
                    for (String w : new String[] {"Hello", " from", " the", " fake", " engine"}) {
                        os.write(("data: {\"choices\":[{\"delta\":{\"content\":\"" + w + "\"}}]}\n\n").getBytes());
                        os.flush();
                        nap(40);
                    }
                    os.write("data: [DONE]\n\n".getBytes());
                }
            } else {
                byte[] b = ("{\"choices\":[{\"message\":{\"role\":\"assistant\",\"content\":\"Hello from the fake engine\"}}],"
                        + "\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":5}}").getBytes();
                ex.getResponseHeaders().add("Content-Type", "application/json");
                ex.sendResponseHeaders(200, b.length);
                ex.getResponseBody().write(b);
                ex.close();
            }
        });
        s.setExecutor(java.util.concurrent.Executors.newCachedThreadPool());
        s.start();
    }
}

The program starts with java FakeEngine.java 18691. It uses a cached thread pool so concurrent requests overlap. The JDK HttpServer default executor runs handlers one at a time, which serialized two parallel streams until the pool was added. The commands below flip the toggle, ask for readiness and liveness, flip the toggle back, and ask for readiness again:

curl -s -X POST localhost:18691/_toggle
curl -s -w ' [HTTP %{http_code}]\n' localhost:18690/actuator/health/readiness
curl -s -w ' [HTTP %{http_code}]\n' localhost:18690/actuator/health/liveness
curl -s -X POST localhost:18691/_toggle
curl -s -w ' [HTTP %{http_code}]\n' localhost:18690/actuator/health/readiness

{"status":"DOWN"} [HTTP 503]
{"status":"UP"} [HTTP 200]
{"status":"UP"} [HTTP 200]

The first answer is 503 DOWN for readiness while the engine is down, and the second answer shows liveness staying 200 UP. This split is the point of the two groups. Restarting the gateway does not fix an engine failure, so liveness does not include the engine. A load balancer sees the failing readiness and stops sending traffic. The last answer shows readiness back to UP as soon as the engine answers again, with no restart.

Example 11: what time to first token and tokens per second measure

Before code records any numbers, the definitions need to be fixed, because "speed" has several meanings. Time to first token (TTFT) is the time from the start of the request to the first generated chunk. It is the delay a user feels before the answer starts to appear. Tokens per second can be measured in two ways. The end-to-end rate is all tokens divided by the whole request time. The decode rate is the tokens after the first divided by the time after the first chunk. The helper computes both from the chunk times that the fake engine produces: a 120 ms wait, then five chunks 40 ms apart.

time (ms):   0        120     160     200     240     280
                      chunk1  chunk2  chunk3  chunk4  chunk5
             |-- wait --|

import java.util.List;
import java.util.Locale;

public class StreamMath {
    public static void main(String[] args) {
        List<Long> chunkMillis = List.of(120L, 160L, 200L, 240L, 280L);
        long start = 0;
        long end = chunkMillis.get(chunkMillis.size() - 1);
        long ttft = chunkMillis.get(0) - start;
        int tokens = chunkMillis.size();

        double endToEnd = tokens / ((end - start) / 1000.0);
        double decode = (tokens - 1) / ((end - chunkMillis.get(0)) / 1000.0);

        System.out.println("ttft ms: " + ttft);
        System.out.println(String.format(Locale.ROOT, "end-to-end tokens/s: %.2f", endToEnd));
        System.out.println(String.format(Locale.ROOT, "decode tokens/s: %.2f", decode));
    }
}

ttft ms: 120
end-to-end tokens/s: 17.86
decode tokens/s: 25.00

Here are the numbers by hand. TTFT is 120 ms, the time of the first chunk. The end-to-end rate is 5 tokens over 0.28 s, which is 17.86 tokens per second. The decode rate leaves out the wait for the first chunk: 4 tokens over 0.16 s (from 120 ms to 280 ms) is 25.00 tokens per second. The gap between the two rates grows with the TTFT.

The gateway records the end-to-end rate, because it includes queueing and prompt processing, which is what a caller experiences. Engines report the decode rate in their own timings (the llama-server fields in The llama-server HTTP API), so a dashboard that compares the two must say which one each number is. A streamed chunk carries one token in llama-server, so the gateway counts chunks as tokens. An engine that groups several tokens per chunk needs the usage field instead.

Example 12: Micrometer meters for the gateway

Micrometer is the metrics facade that Spring Boot configures. A facade is a common front that hides the monitoring system behind it. Code registers meters (named measurements) on a MeterRegistry, and Actuator and exporters read them. Four meter types cover this gateway:

• A Gauge reads a current value that can go up and down. Here it is the queue length.
• A Timer records durations and keeps the count, the total, and the maximum. Here it is TTFT.
• A DistributionSummary does the same for values that are not durations. Here it is tokens per second.
• A Counter only goes up. Here it is the number of tokens produced.

package demo;

import io.micrometer.core.instrument.DistributionSummary;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import java.time.Duration;
import java.util.concurrent.atomic.AtomicInteger;
import org.springframework.stereotype.Component;

@Component
public class GatewayMetrics {
    private final MeterRegistry registry;
    private final AtomicInteger queued = new AtomicInteger();

    public GatewayMetrics(MeterRegistry registry) {
        this.registry = registry;
        Gauge.builder("gateway.queue.length", queued, AtomicInteger::get)
                .description("Requests waiting for an engine slot").register(registry);
    }

    public void queueEntered() { queued.incrementAndGet(); }

    public void queueLeft() { queued.decrementAndGet(); }

    public void ttft(String model, Duration d) {
        Timer.builder("gateway.ttft").tag("model", model)
                .description("Time from request start to the first streamed chunk")
                .register(registry).record(d);
    }

    public void tokensPerSecond(String model, double value) {
        DistributionSummary.builder("gateway.tokens.per.second").tag("model", model)
                .register(registry).record(value);
    }

    public void tokens(String model, String kind, long count) {
        registry.counter("gateway.tokens", "model", model, "kind", kind).increment(count);
    }
}

Each meter can have a tag, a key and value pair such as model=tiny, which splits one meter into series that queries can filter. Tags have a cost, because every distinct tag value creates a new series that the registry keeps in memory. The model tag is acceptable because a gateway serves a handful of models. A per-request id or a per-key tag would grow without bound.

The gauge reads an AtomicInteger. The request code increments it when a request starts waiting for an engine slot and decrements it when the request gets one. The Timer is built with Timer.builder("gateway.ttft"), and its unit in the Actuator output is seconds. The registry deduplicates meters by name and tags, so calling register on every request returns the existing meter and does not create a new one.

Example 13: a controller that records the numbers

The controller ties the pieces together. It proxies one chat request: it waits for an engine slot, copies the engine's response to the client line by line, and records the meters. The slot is a Semaphore (a counter of permits) sized by gateway.engine.max-concurrent. With a value of 1, only one request talks to the engine at a time, and others wait. Loading, Unloading, and Scheduling Models develops per-model queues. To stay short, the controller takes the model name and the stream flag from the raw body with String.contains. An OpenAI-Compatible API in Front of llama-server parses the body properly.

package demo;

import jakarta.servlet.http.HttpServletResponse;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.concurrent.Semaphore;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.springframework.http.MediaType;
import org.springframework.http.client.JdkClientHttpRequestFactory;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.client.RestClient;
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;

@RestController
public class ChatController {
    private static final Logger log = LoggerFactory.getLogger("gateway.chat");
    private final RestClient engine;
    private final GatewayMetrics metrics;
    private final Semaphore slots;

    public ChatController(GatewayProperties props, GatewayMetrics metrics) {
        this.engine = RestClient.builder().baseUrl(props.engine().baseUrl())
                .requestFactory(new JdkClientHttpRequestFactory()).build();
        this.metrics = metrics;
        this.slots = new Semaphore(props.engine().maxConcurrent());
    }

    @PostMapping(path = "/v1/chat/completions")
    ResponseEntity<StreamingResponseBody> chat(@RequestBody String body,
            HttpServletResponse response) {
        String requestId = response.getHeader("X-Request-Id");
        boolean stream = body.contains("\"stream\":true");
        String model = body.contains("\"model\":\"tiny\"") ? "tiny" : "other";
        StreamingResponseBody out = (OutputStream springStream) -> {
            OutputStream os = response.getOutputStream();
            long start = System.nanoTime();
            metrics.queueEntered();
            try {
                slots.acquire();
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                return;
            } finally {
                metrics.queueLeft();
            }
            try {
                engine.post().uri("/v1/chat/completions").contentType(MediaType.APPLICATION_JSON)
                        .body(body).exchange((req, res) -> {
                    long chunks = 0;
                    boolean first = true;
                    try (BufferedReader r = new BufferedReader(
                            new InputStreamReader(res.getBody(), StandardCharsets.UTF_8))) {
                        String line;
                        while ((line = r.readLine()) != null) {
                            os.write((line + "\n").getBytes(StandardCharsets.UTF_8));
                            os.flush();
                            if (stream && line.startsWith("data: {")) {
                                if (first) {
                                    metrics.ttft(model, Duration.ofNanos(System.nanoTime() - start));
                                    first = false;
                                }
                                chunks++;
                            }
                        }
                    }
                    double seconds = (System.nanoTime() - start) / 1e9;
                    if (stream && chunks > 0) {
                        metrics.tokens(model, "completion", chunks);
                        metrics.tokensPerSecond(model, chunks / seconds);
                        MDC.put("request_id", requestId);
                        MDC.put("model", model);
                        MDC.put("completion_tokens", Long.toString(chunks));
                        MDC.put("duration_ms", Long.toString((long) (seconds * 1000)));
                        log.info("stream completed");
                        MDC.clear();
                    }
                    return null;
                });
            } finally {
                slots.release();
            }
        };
        return ResponseEntity.ok()
                .contentType(stream ? MediaType.TEXT_EVENT_STREAM : MediaType.APPLICATION_JSON).body(out);
    }

    @PostMapping("/api/pull")
    String pull() { return "{\"status\":\"pulling\"}"; }
}

Three details of this class came from failures during development. The table gives each detail and the reason for it.

With the flush fixed, curl sees each chunk when the engine produces it. A loop prints the time at which each line arrived:

163 ms data: {"choices":[{"delta":{"content":"H
199 ms data: {"choices":[{"delta":{"content":" 
239 ms data: {"choices":[{"delta":{"content":" 
281 ms data: {"choices":[{"delta":{"content":" 
325 ms data: {"choices":[{"delta":{"content":" 
365 ms data: [DONE]

The first line arrives at 163 ms. That is the engine's 120 ms wait plus connection setup and warm-up. The chunks follow 36 to 44 ms apart, which matches the engine's 40 ms pause between chunks. This shows that the gateway passes each chunk on without holding it back.

The first request after startup is slower because of class loading. It took more than one second in one run, which is why its numbers are not used for the metrics examples. The streaming proxy lesson (Streaming Responses Through the Gateway) covers cancellation and timeouts, which this controller leaves out.

Example 14: reading the metrics

Meters are read from /actuator/metrics/<name>, with the admin key. The three commands below read the TTFT timer, the token counter, and the tokens-per-second summary:

curl -s -H "Authorization: Bearer sk-admin-demo" localhost:18690/actuator/metrics/gateway.ttft
curl -s -H "Authorization: Bearer sk-admin-demo" localhost:18690/actuator/metrics/gateway.tokens
curl -s -H "Authorization: Bearer sk-admin-demo" localhost:18690/actuator/metrics/gateway.tokens.per.second

{"availableTags":[{"tag":"model","values":["tiny"]}],"baseUnit":"seconds","description":"Time from request start to the first streamed chunk","measurements":[{"statistic":"COUNT","value":2.0},{"statistic":"TOTAL_TIME","value":0.589523276},{"statistic":"MAX","value":0.462377967}],"name":"gateway.ttft"}
{"availableTags":[{"tag":"kind","values":["completion"]},{"tag":"model","values":["tiny"]}],"measurements":[{"statistic":"COUNT","value":10.0}],"name":"gateway.tokens"}
{"availableTags":[{"tag":"model","values":["tiny"]}],"measurements":[{"statistic":"COUNT","value":2.0},{"statistic":"TOTAL","value":25.598146279407587},{"statistic":"MAX","value":12.807146166253284}],"name":"gateway.tokens.per.second"}

These lines come from different runs of the gateway, so their counts do not match each other. Read them one at a time.

• TTFT. Two requests were recorded with a total of 0.5895 s and a maximum of 0.4624 s. The average is the total divided by the count, 0.5895 / 2, which is about 0.29 s. The maximum is high because the second request in that run waited for the single engine slot (Example 15).
• Tokens. The gateway.tokens counter holds 10 completion tokens, which is two streams of five chunks.
• Tokens per second. The summary has a total of 25.60 over two samples, so the average is 12.80, and the maximum is 12.81 tokens per second. This is the end-to-end rate from Example 11, including the wait.

Spring Boot also registers per-request metrics on its own. They carry the status and outcome:

curl -s -H "Authorization: Bearer sk-ops-demo" \
  "localhost:18690/actuator/metrics/http.server.requests?tag=uri:/v1/chat/completions"

{"availableTags":[{"tag":"exception","values":["none"]},{"tag":"method","values":["POST"]},{"tag":"error","values":["none"]},{"tag":"outcome","values":["SUCCESS"]},{"tag":"status","values":["200"]}],"baseUnit":"seconds","measurements":[{"statistic":"COUNT","value":2.0},{"statistic":"TOTAL_TIME","value":0.838840879},{"statistic":"MAX","value":0.423450946}],"name":"http.server.requests"}

http.server.requests counts and times every request by uri, method, status, and outcome, without any code in the gateway. The JVM, CPU, and executor meters arrive the same way (83 names in this build, among them jvm.memory.used and process.cpu.usage).

Actuator's /actuator/metrics serves JSON for people and scripts. Monitoring systems usually scrape a different format. A Prometheus registry (micrometer-registry-prometheus) would add a scrape endpoint for a metrics server. It is a separate dependency that this project did not download, so no example here uses it.

Example 15: queue length while requests wait

The gauge from Example 12 answers a question that latency alone cannot: how many requests are waiting right now. With one engine slot (max-concurrent: 1), two parallel streams make the second one wait. The command starts two streams in the background, reads the gauge 0.25 s later (while the first stream still runs), and then prints each stream's total time.

for r in a b; do
  curl -sN -H "Authorization: Bearer sk-alice-demo" -H "Content-Type: application/json" \
    localhost:18690/v1/chat/completions \
    -d '{"model":"tiny","stream":true,"messages":[]}' -o /dev/null -w "$r total=%{time_total}\n" &
done
sleep 0.25
curl -s -H "Authorization: Bearer sk-admin-demo" localhost:18690/actuator/metrics/gateway.queue.length
wait

{"availableTags":[],"description":"Requests waiting for an engine slot","measurements":[{"statistic":"VALUE","value":1.0}],"name":"gateway.queue.length"}
a total=0.347519
b total=0.673133

The gauge reads 1.0 while stream a holds the slot and stream b waits. Stream a takes 0.35 s, the same as one stream alone. Stream b takes 0.67 s, because it waits for a to finish and then needs its own 0.35 s of work (0.35 + 0.35 is about 0.7).

A queue length that stays above zero means requests arrive faster than the engine finishes them. The choices are a smaller model, a larger slot count, or another machine. The TTFT timer includes the queue wait because it starts before the semaphore. This is why the TTFT maximum in Example 14 was high.

Example 16: profiles and environment variables for several machines

A laptop with one small model and a server with a GPU need different limits, but the program is the same. Spring Boot handles this with profiles. For each active profile it reads application-<profile>.yml on top of application.yml, and environment variables override both. The order from weakest to strongest source is: application.yml, then the profile file, then environment variables. The two profile files in this project:

# application-laptop.yml
gateway:
  engine:
    max-concurrent: 1
  rate-limit:
    default-requests-per-minute: 30
    burst: 3

# application-server.yml
gateway:
  engine:
    max-concurrent: 4
  rate-limit:
    default-requests-per-minute: 600
    burst: 20
  cors-origins:
    - https://chat.example.com
logging:
  structured:
    format:
      console: logstash

The server profile allows four concurrent engine calls, raises the default limit, names the production front end, and switches the console log to JSON (Example 17). Keys can come from the environment instead of a file, so secrets stay out of the jar. Spring Boot's relaxed binding maps the variable GATEWAY_KEYS_0_NAME to the property gateway.keys[0].name. A list given in the environment replaces the whole list from the file. The command below builds the hash of a new ops key, passes it as environment variables, and starts the jar with the server profile:

OPS=$(printf 'sk-ops-demo' | sha256sum | cut -d' ' -f1)
GATEWAY_KEYS_0_NAME=ops GATEWAY_KEYS_0_SHA256=$OPS \
GATEWAY_KEYS_0_ROLES_0=ADMIN GATEWAY_KEYS_0_ROLES_1=USER \
SPRING_PROFILES_ACTIVE=server SERVER_PORT=18690 \
java -jar target/loop-gateway-0.1.0.jar

The same two parallel streams as in Example 15 now run against the server profile, with the fake engine on a cached thread pool:

{"availableTags":[],"description":"Requests waiting for an engine slot","measurements":[{"statistic":"VALUE","value":0.0}],"name":"gateway.queue.length"}
b total=0.421253
a total=0.430768

The queue stays at 0 and both streams finish in about 0.42 s, because four slots exist and neither stream waits. The old alice key no longer works in this run ({"type":"about:blank","title":"Unauthorized","status":401,"detail":"Invalid API key"}), which shows that the environment list replaced the file's list. A request with the ops key returned its first chunk.

The two runs differ only in configuration, and the jar is the same file. Keeping the settings that differ per machine in profile files makes those differences easy to review.

Example 17: structured request logs

A log line that a program can parse is easier to search than one written for people. Spring Boot formats console logs as JSON when logging.structured.format.console names a format. logstash is one of the supported formats, and the server profile sets it.

Fields put into the MDC (mapped diagnostic context, a per-thread map that logging reads) appear as JSON fields. This lets code attach facts such as the request id to every log line written while that request is handled. The filter below generates a request id, or accepts a client's X-Request-Id of at most 64 characters, and returns it in the response. It writes one line per request when the response is complete.

package demo;

import jakarta.servlet.AsyncEvent;
import jakarta.servlet.AsyncListener;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.UUID;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class RequestLogFilter extends OncePerRequestFilter {
    private static final Logger log = LoggerFactory.getLogger("gateway.access");

    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        String id = req.getHeader("X-Request-Id");
        if (id == null || id.isBlank() || id.length() > 64) {
            id = UUID.randomUUID().toString();
        }
        res.setHeader("X-Request-Id", id);
        long start = System.nanoTime();
        MDC.put("request_id", id);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
        if (req.isAsyncStarted()) {
            String requestId = id;
            req.getAsyncContext().addListener(new AsyncListener() {
                public void onComplete(AsyncEvent e) { finish(req, res, requestId, start); }
                public void onTimeout(AsyncEvent e) { }
                public void onError(AsyncEvent e) { }
                public void onStartAsync(AsyncEvent e) { }
            });
        } else {
            finish(req, res, id, start);
        }
    }

    private static void finish(HttpServletRequest req, HttpServletResponse res, String id, long start) {
        Object key = req.getAttribute("gateway.key");
        MDC.put("request_id", id);
        MDC.put("api_key", key == null ? "anonymous" : key.toString());
        MDC.put("method", req.getMethod());
        MDC.put("path", req.getRequestURI());
        MDC.put("status", Integer.toString(res.getStatus()));
        MDC.put("duration_ms", Long.toString((System.nanoTime() - start) / 1_000_000));
        try {
            log.info("request completed");
        } finally {
            MDC.clear();
        }
    }
}

The completed-request line cannot be written in a finally block for a streamed response. The first dispatch returns while the body is still streaming, so the status and duration would be those of the first dispatch and not of the whole request. The filter therefore registers an AsyncListener that logs when the asynchronous request completes.

The key name comes from a request attribute that the API key filter sets (gateway.key), because the security chain clears the SecurityContext before control returns to this outer filter. The log below is from the run with the ops key and the request id req-77. There are two lines, one from the controller and one from the access filter:

{"@timestamp":"2026-10-03T16:04:24.29371499Z","@version":"1","message":"stream completed","logger_name":"gateway.chat","thread_name":"task-1","level":"INFO","level_value":20000,"duration_ms":"482","completion_tokens":"5","model":"tiny","request_id":"req-77"}
{"@timestamp":"2026-10-03T16:04:24.316609953Z","@version":"1","message":"request completed","logger_name":"gateway.access","thread_name":"tomcat-handler-4","level":"INFO","level_value":20000,"duration_ms":"551","path":"/v1/chat/completions","method":"POST","api_key":"ops","request_id":"req-77","status":"200"}

Both lines carry request_id, so a search for req-77 finds the access line and the controller's token count together. The access line reports api_key as the key's name, never the secret. The controller's line shows 5 completion tokens in 482 ms, and the access line covers the whole request in 551 ms. The difference is the work before and after the stream. This is the first stream after the process started, so class loading adds to the time.

Logging the request body is left out on purpose. Prompts contain code and personal data, and a log that holds them needs its own access rules.

Example 18: packaging the jar

So far the gateway ran from Maven. On a server, a single file that contains everything is easier to install. spring-boot-maven-plugin (declared in the pom.xml) turns mvn package into an executable jar with its dependencies inside. The commands build the jar, list it, and start it:

JAVA_HOME=/usr/lib/jvm/java-25-openjdk-amd64 mvn -q -B package -DskipTests
ls -l target/loop-gateway-0.1.0.jar
java -jar target/loop-gateway-0.1.0.jar --server.port=18690

-rw-r--r-- 1 root root 27602025 Oct  3 16:04 target/loop-gateway-0.1.0.jar

The jar is about 27.6 MB and holds the application, Tomcat, Spring, Security, Actuator, and Micrometer. Nothing but a JDK 25 is needed on the target machine, and the process is one java -jar. The --server.port argument and the environment variables from Example 16 override the packaged defaults.

The packaging lesson in Spring Boot Essentials for an LLM Gateway covers the plugin's other goals. A container image is one way to ship the jar. Running it directly under a service manager is the other, shown next.

Example 19: a systemd unit

On Linux, systemd is the program that starts services at boot, restarts them when they exit, and collects their output in the journal (its log store). A unit file describes one service: who runs it, which command starts it, and when to restart it. The file below is the layout used for the jar from Example 18. The container in which this lesson was written has no systemd, so this unit was not started here. Its directives are standard systemd options, and a reader checks it on the target machine with systemd-analyze verify /etc/systemd/system/loop-gateway.service.

# /etc/systemd/system/loop-gateway.service
[Unit]
Description=loop gateway
After=network.target

[Service]
User=loop
WorkingDirectory=/opt/loop-gateway
EnvironmentFile=/etc/loop-gateway/env
ExecStart=/usr/lib/jvm/java-25-openjdk-amd64/bin/java -jar /opt/loop-gateway/loop-gateway-0.1.0.jar
SuccessExitStatus=143
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now loop-gateway
journalctl -u loop-gateway -f

The JSON log lines from Example 17 go to the journal, where journalctl -u loop-gateway -o cat prints them for a log shipper (a program that forwards logs to a search system). The engine runs under its own unit, or the gateway starts it as taught in Loading, Unloading, and Scheduling Models. The readiness endpoint from Example 10 is the check that a front proxy or a monitoring script calls. The final lesson, Workshop: loop gateway, assembles these pieces into the loop gateway command.

Interview points

• A 401 means the caller is not authenticated, and a 403 means the caller is authenticated but not allowed. A 401 response carries a WWW-Authenticate header that names the expected scheme.
• The server stores a hash of each API key and not the key itself, so a leaked configuration file does not reveal working keys. A fast hash is suitable for a long random key, while slow password hashes protect short human-chosen secrets. The comparison uses a constant-time method such as MessageDigest.isEqual, so response timing does not leak how much of a guess was right.
• A stateless API disables CSRF protection and sessions because it has no cookies. Each request carries its own credentials.
• A Filter that is a Spring bean is also added by Spring Boot to the servlet chain, so it would run twice inside a security chain. For that reason the filter in the security chain is created with new.
• A token bucket allows a burst up to its capacity and holds a long-run rate. Idle time cannot raise the stored tokens above the capacity. The wait until the next token is available gives the Retry-After value.
• A rate limit counted in requests does not account for the number of generated tokens. A token-based limit needs the usage numbers after the response.
• CORS is enforced by browsers and does not authenticate anyone. A preflight is an OPTIONS request without credentials, so it must be allowed without a key.
• Liveness decides restarts and readiness decides routing. Engine health belongs in readiness. Putting it in liveness would restart a healthy gateway for an engine fault.
• Time to first token starts before the queue wait and ends at the first chunk. End-to-end tokens per second includes the wait and the prompt processing, and the decode rate does not.
• A metric tag with unbounded values (a request id or an API key secret) creates unbounded series and uses memory without limit. Bounded tags such as the model name are safe.
• A streamed servlet response completes on an async dispatch. A filter that logs in a finally block sees the status of the first dispatch and not of the finished response. Authorization on the async dispatch needs a matching rule or a persisted security context.
• The stream that Spring passes to a StreamingResponseBody ignores flush() in Spring Framework 7.0.9 (it is wrapped with StreamUtils.nonFlushing) unless spring.http.response.flush.enabled=true is set in spring.properties. Without the property, a proxy that must deliver each chunk writes to the servlet output stream directly.
• Environment variables override profile files, and profile files override application.yml. A list from a higher-priority source replaces the whole list and is not merged with the lower one.

Summary

• spring-boot-starter-security creates a default filter chain and a development user with a generated password. Excluding UserDetailsServiceAutoConfiguration removes that user.
• An OncePerRequestFilter placed before BasicAuthenticationFilter authenticates Authorization: Bearer keys and sets the roles as ROLE_* authorities.
• Keys are stored as SHA-256 hex digests in @ConfigurationProperties and compared with MessageDigest.isEqual against every entry.
• Unknown keys return 401 with WWW-Authenticate: Bearer and an application/problem+json body. A missing role returns 403. Paths without a rule are denied.
• Management endpoints and /api/pull need the ADMIN role, and /actuator/health/** is open.
• A token bucket per key gives a burst and a long-run rate. Refusals return 429 with Retry-After in whole seconds.
• CORS allows named origins, methods, and headers. Preflight requests are permitted without a key and cached for Access-Control-Max-Age.
• Actuator exposes health, metrics, and info here. The readiness group includes the custom engine health indicator, and the liveness group does not.
• The gateway meters are gateway.ttft (Timer), gateway.tokens.per.second (DistributionSummary), gateway.tokens (Counter), and gateway.queue.length (Gauge), tagged by model where a tag applies.
• http.server.requests and the JVM meters come from Spring Boot without code, and /actuator/metrics/<name> reads them as JSON.
• The queue gauge showed 1 with one slot and two parallel streams, and 0 with four slots.
• logging.structured.format.console: logstash writes JSON logs, and MDC fields (request_id, api_key, path, status, duration_ms) become JSON fields.
• A streamed response needs an AsyncListener for its access log line and an ASYNC dispatch rule in the chain.
• Profiles (laptop, server) and environment variables configure one jar for several machines.
• mvn package builds a self-contained jar of about 27.6 MB that runs with java -jar. A systemd unit with EnvironmentFile, Restart=on-failure, and SuccessExitStatus=143 runs it as a service.