Streaming Responses Through the Gateway

A language model produces its answer one token at a time. A token is a small piece of text, often a word or part of a word. Generating a long answer can take many seconds. If the server waits for the last token and sends everything at once, the user stares at a blank screen for the whole time. If the server sends each token as soon as it exists, the user sees the answer grow on screen. This second way is called streaming, and it is what "stream": true asks for in the OpenAI-style API.

The engine (the program that runs the model) already streams. The gateway sits between the client and the engine, so the gateway must pass every token along as it arrives. A gateway that collects the whole answer first and then replies throws away the benefit of streaming. This lesson builds the streaming path of the gateway with Spring MVC (Spring's classic web framework, which gives each request a thread). The gateway relays each chunk from the engine to the client as the chunk arrives, stops the engine when the client leaves, tells the client when the engine stalls, and converts the OpenAI-style event stream into the newline-delimited JSON that Ollama's /api/chat uses.

After the lesson, you can write a streaming endpoint in Spring MVC. You can show with timestamps that tokens reach the client while the engine is still generating. You can also test that property without depending on timing, which matters because timing-based tests fail on slow machines.

The table lists the ideas and APIs in the order they appear. Each one is explained when it is first used.

The gateway uses Spring MVC only. The Reactive stack (Spring WebFlux) is an alternative that this course does not use.

Every example runs against a fake engine. This is a small OpenAI-compatible server started on 127.0.0.1:18671, and it replaces a real model so that the timing is predictable. The gateway listens on 127.0.0.1:18670. Both ports are fixed for the examples and can be changed.

The examples print times such as + 316 ms. These are milliseconds since the curl command started. They are real measurements, and they differ by a few tens of milliseconds from run to run. For that reason the text describes the pattern (tokens spaced about 200 ms apart, or everything arriving at once) and not the exact values.

The project uses Spring Boot 4.1.1 on JDK 25, with spring-boot-starter-web, spring-boot-starter-restclient, and spring-boot-starter-test. The Spring blocks are marked java,ignore because the single-file Java checker of this course cannot run Spring. They are excerpts of code that compiled and ran in the project.

Example 1: the SSE format and a slow fake engine

Before building a relay, look at what it has to relay. Server-sent events (SSE) is a simple text format for a server that keeps one HTTP response open and sends many small messages over time. Each message is one or more lines that start with data:, and a blank line ends the message. The OpenAI streaming protocol uses it: every message holds a JSON chunk with a few new characters of the answer, and the last message is the text [DONE].

The fake engine answers POST /v1/chat/completions in that format. It sends a role chunk, then one content chunk per word, with a 200 ms pause before each. After the words it sends a finish chunk and [DONE]. It reads max_tokens from the request body to decide how many words to send. The pause between words stands in for the time a real model needs per token.

public class FakeEngine implements AutoCloseable {
    interface Pacing { void pause(int tokenIndex) throws InterruptedException; }
    static final List<String> WORDS = List.of("Hello", " from", " the", " fake", " engine", ".");

    private void handle(HttpExchange exchange) throws IOException {
        String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
        int count = maxTokens(body);                      // regex over "max_tokens", default 6
        exchange.getResponseHeaders().set("Content-Type", "text/event-stream");
        exchange.sendResponseHeaders(200, 0);             // 0 = chunked, length unknown
        try (OutputStream out = exchange.getResponseBody()) {
            send(out, chunk("{\"role\":\"assistant\",\"content\":null}", "null"));
            for (int i = 0; i < count; i++) {
                pacing.pause(i);                          // 200 ms in the standalone engine
                send(out, chunk("{\"content\":\"" + WORDS.get(i % WORDS.size()) + "\"}", "null"));
                System.out.println("engine: sent token " + sent.incrementAndGet());
            }
            send(out, chunk("{}", "\"stop\""));
            send(out, "[DONE]");
            System.out.println("engine: finished");
        } catch (IOException e) {                         // the write fails when the reader is gone
            System.out.println("engine: client gone after token " + sent.get());
        }
    }

    private static void send(OutputStream out, String data) throws IOException {
        out.write(("data: " + data + "\n\n").getBytes(StandardCharsets.UTF_8));
        out.flush();                                      // each event leaves the engine at once
    }
}

A few lines of this code carry the lesson. sendResponseHeaders(200, 0) uses a length of 0, which means "length unknown, send the body in chunks". The send method calls out.flush() after every event, so each event leaves the engine immediately. The catch (IOException e) block matters later: when the reader of the connection is gone, a write fails, and the engine learns that nobody is listening.

The engine is a com.sun.net.httpserver.HttpServer (the small HTTP server that ships with the JDK) with a virtual-thread executor. It lives in the test sources of the Maven project and has a main method, so it can be started for the curl experiments. Two helper scripts make the output readable. show.sh puts the milliseconds since it started in front of every non-empty line. short.sh reduces a long JSON chunk to the token it carries.

#!/bin/bash
# show.sh: print each non-empty input line with the time since the script started
start=$(date +%s%N)
while IFS= read -r line; do
  [ -z "$line" ] && continue
  now=$(date +%s%N)
  printf '+%4d ms  %s\n' $(( (now - start) / 1000000 )) "$line"
done

#!/bin/bash
# short.sh: reduce an OpenAI stream chunk to the text it carries
sed -u -E 's/.*"content":null.*/role chunk/; s/.*"content":"([^"]*)".*/token [\1]/; s/.*"finish_reason":"stop".*/finish stop/'

The first command sends a streaming request straight to the engine. The flag -N turns off the output buffering of curl, so each line appears when it arrives and not when a buffer fills. Look at the numbers at the start of each line.

curl -sN -H 'Content-Type: application/json' \
  -d '{"model":"fake","stream":true,"max_tokens":4}' \
  http://127.0.0.1:18671/v1/chat/completions | ./show.sh

+ 132 ms  data: {"id":"chatcmpl-fake","object":"chat.completion.chunk","created":1791032728,"model":"fake","choices":[{"index":0,"delta":{"role":"assistant","content":null},"finish_reason":null}]}
+ 332 ms  data: {"id":"chatcmpl-fake","object":"chat.completion.chunk","created":1791032728,"model":"fake","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}
+ 533 ms  data: {"id":"chatcmpl-fake","object":"chat.completion.chunk","created":1791032728,"model":"fake","choices":[{"index":0,"delta":{"content":" from"},"finish_reason":null}]}
+ 736 ms  data: {"id":"chatcmpl-fake","object":"chat.completion.chunk","created":1791032728,"model":"fake","choices":[{"index":0,"delta":{"content":" the"},"finish_reason":null}]}
+ 937 ms  data: {"id":"chatcmpl-fake","object":"chat.completion.chunk","created":1791032728,"model":"fake","choices":[{"index":0,"delta":{"content":" fake"},"finish_reason":null}]}
+ 940 ms  data: {"id":"chatcmpl-fake","object":"chat.completion.chunk","created":1791032728,"model":"fake","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
+ 942 ms  data: [DONE]

Each event is one data: line followed by a blank line, which show.sh skips. The four content events are spaced by about 200 ms, which is the pause in the engine. The first line includes the start-up time of curl, so it is a little later than the engine's first send. The finish chunk and [DONE] follow the last token within a few milliseconds, because the engine does not pause after the last token.

This is the stream that every later example must preserve: tokens 200 ms apart at the client, not one block at the end.

Example 2: a buffered proxy delays every token

The simplest way to write a proxy is to call the engine, wait for the response, and return it. That is correct for a normal call and wrong for a streaming one. The controller below forwards the body with RestClient (Spring's HTTP client) and returns the result as a String. The RestClient bean points at the engine's base URL. Compare the timing of its output with Example 1.

@RestController
class BufferedController {
    private final RestClient engine;

    BufferedController(RestClient engine) { this.engine = engine; }

    @PostMapping(path = "/buffered/v1/chat/completions", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    String chat(@RequestBody String body) {
        return engine.post().uri("/v1/chat/completions")
                .contentType(MediaType.APPLICATION_JSON)
                .body(body)
                .retrieve()
                .body(String.class);          // waits for the whole stream
    }
}

curl -sN -H 'Content-Type: application/json' \
  -d '{"model":"fake","stream":true,"max_tokens":4}' \
  http://127.0.0.1:18670/buffered/v1/chat/completions | ./short.sh | ./show.sh

+ 817 ms  role chunk
+ 819 ms  token [Hello]
+ 821 ms  token [ from]
+ 823 ms  token [ the]
+ 825 ms  token [ fake]
+ 828 ms  finish stop
+ 829 ms  data: [DONE]

Every line arrives after 817 ms. That is the time the engine needs for the whole answer: the role chunk plus four tokens at about 200 ms each. After that, the lines are only 2 ms apart, because the gateway had them all in memory and wrote them together.

The client did receive a valid SSE body with the right content type, so a client that only checks the final text would work. A chat window or a coding agent that prints tokens as they arrive would show nothing for the whole generation and then the full answer. The same happens in a test that only compares the body. This is why Example 11 checks the order of arrival instead.

The call body(String.class) is the cause. It reads until the engine closes the response, and only then does Spring write the string, with a Content-Length header. Example 6 shows that header. The first request in a fresh JVM took 1,224 ms in the same experiment because of class loading, so the measurements in this lesson use a second request.

Example 3: SseEmitter on a virtual thread

To stream, the gateway has to send events while it is still reading from the engine. A normal controller method cannot do that, because the response is sent when the method returns. Spring MVC solves this with a type called SseEmitter. The controller returns the emitter at once. This starts a text/event-stream response. A different thread then reads the engine's stream and calls emitter.send for each event, and every call writes one SSE message to the client.

That second thread spends nearly all of its life waiting for the engine to send the next line. A virtual thread suits this. A virtual thread is a JDK thread that costs little memory and may block freely. It is taught in Spring Boot Essentials for an LLM Gateway. One virtual thread runs per open stream, so a thousand open streams do not need a thousand heavy platform threads.

@Configuration
class Clients {
    @Bean
    RestClient engineRestClient(RestClient.Builder builder, GatewayProperties props) {
        SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(Duration.ofSeconds(2));
        factory.setReadTimeout(props.readTimeout());      // longest silence allowed between two reads (Example 9)
        return builder.baseUrl(props.engineUrl()).requestFactory(factory).build();
    }
}

@RestController
class EmitterController {
    private static final Logger log = LoggerFactory.getLogger(EmitterController.class);
    private final RestClient engine;
    private final AsyncTaskExecutor streamExecutor;

    EmitterController(RestClient engine, @Qualifier("applicationTaskExecutor") AsyncTaskExecutor streamExecutor) {
        this.engine = engine;
        this.streamExecutor = streamExecutor;
    }

    @PostMapping(path = "/emitter/v1/chat/completions", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    SseEmitter chat(@RequestBody String body) {
        SseEmitter emitter = new SseEmitter(0L);              // 0 = no async timeout
        streamExecutor.execute(() -> {
            log.info("emitter: relay thread is virtual = {}", Thread.currentThread().isVirtual());
            try {
                engine.post().uri("/v1/chat/completions")
                        .contentType(MediaType.APPLICATION_JSON)
                        .body(body)
                        .exchange((request, response) -> {
                            try (var reader = new BufferedReader(
                                    new InputStreamReader(response.getBody(), StandardCharsets.UTF_8))) {
                                for (String line; (line = reader.readLine()) != null; ) {
                                    if (line.startsWith("data:")) {
                                        emitter.send(SseEmitter.event().data(line.substring(5).strip()));
                                    }
                                }
                            }                                  // closing the reader closes the engine connection
                            return null;
                        });
                emitter.complete();
            } catch (Exception e) {                           // send() fails once the client is gone
                log.info("emitter: relay stopped, closing the engine connection");
                emitter.completeWithError(e);
            }
        });
        return emitter;
    }
}

The application.properties file holds spring.threads.virtual.enabled=true. With that setting, Boot's applicationTaskExecutor creates a virtual thread per task. The controller injects that executor by name, so the relay runs on a virtual thread. In an earlier version of this project, the code defined its own Executor bean for the relay. The log line relay thread is virtual then printed false for the StreamingResponseBody endpoint of Example 4, whose thread was named MvcAsync. After the extra bean was removed, both endpoints printed true. The log line stays in the example so that the setting and the executor can be checked.

Some details of the controller:

• new SseEmitter(0L) means "no async timeout". Example 9 shows how to set a limit.
• exchange gives the callback the open response, so the body can be read while the engine is still writing. retrieve().body(...) is not used here, because it reads to the end.
• Each data: line is sent as one SSE event. The value [DONE] is sent like any other event, because clients expect it as the last one.
• The try (var reader ...) block closes the engine connection when the relay stops. Example 8 explains why this matters.

Now send the same request through the gateway.

curl -sN -H 'Content-Type: application/json' \
  -d '{"model":"fake","stream":true,"max_tokens":4}' \
  http://127.0.0.1:18670/emitter/v1/chat/completions | ./short.sh | ./show.sh

+  16 ms  role chunk
+ 217 ms  token [Hello]
+ 418 ms  token [ from]
+ 618 ms  token [ the]
+ 819 ms  token [ fake]
+ 821 ms  finish stop
+ 823 ms  data:[DONE]

The tokens now reach the client 200 ms apart, the same pattern as Example 1, and the last line is at 823 ms like the buffered version. The two versions finish together, but they differ in when the earlier lines arrive: the first token is about 800 ms earlier than in Example 2. This is the whole point of streaming. The total time is the same, and the time to the first token drops from the full generation time to the time of one token.

The gateway log, started with --logging.pattern.console=%msg%n so that it prints only messages, contains:

emitter: relay thread is virtual = true

SseEmitter writes data: without a space after the colon, so the last line reads data:[DONE] and not data: [DONE]. SSE parsers remove one optional space after the colon, as the parser in lesson 37 does, so both forms mean the same.

Example 4: StreamingResponseBody

Spring MVC has a second type for streaming, StreamingResponseBody. The controller returns a lambda that receives an output stream. Spring runs the lambda on an async thread after the handler method returns, and whatever the lambda writes goes to the client. This type does not know about SSE. The lambda writes plain bytes, and the content type comes from the ResponseEntity that wraps it.

For a relay that does not need to look inside the events, copying bytes is the smallest solution. The engine already produces correct SSE, so the gateway can pass the bytes along unchanged.

@RestController
class BodyController {
    private static final Logger log = LoggerFactory.getLogger(BodyController.class);
    private final RestClient engine;

    BodyController(RestClient engine) { this.engine = engine; }

    @PostMapping(path = "/body/v1/chat/completions", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    ResponseEntity<StreamingResponseBody> chat(@RequestBody String body, HttpServletResponse servletResponse) {
        StreamingResponseBody stream = ignored -> {
            log.info("body: writer thread is virtual = {}", Thread.currentThread().isVirtual());
            OutputStream out = servletResponse.getOutputStream();   // the lambda parameter ignores flush(), see Example 7
            engine.post().uri("/v1/chat/completions")
                    .contentType(MediaType.APPLICATION_JSON)
                    .body(body)
                    .exchange((request, response) -> {
                        try (InputStream in = response.getBody()) {
                            byte[] buffer = new byte[1024];
                            for (int n; (n = in.read(buffer)) >= 0; ) {
                                out.write(buffer, 0, n);
                                out.flush();                   // push every chunk to the client now
                            }
                        }
                        return null;
                    });
        };
        return ResponseEntity.ok().contentType(MediaType.TEXT_EVENT_STREAM).body(stream);
    }
}

The handler takes the HttpServletResponse and writes to its output stream, not to the parameter of the lambda (the parameter is named ignored). Example 7 shows why. The return type must be ResponseEntity<StreamingResponseBody>, because Spring picks the handler for the return value from the declared type.

curl -sN -H 'Content-Type: application/json' \
  -d '{"model":"fake","stream":true,"max_tokens":4}' \
  http://127.0.0.1:18670/body/v1/chat/completions | ./short.sh | ./show.sh

+ 196 ms  role chunk
+ 382 ms  token [Hello]
+ 583 ms  token [ from]
+ 783 ms  token [ the]
+ 985 ms  token [ fake]
+ 987 ms  finish stop
+ 990 ms  data: [DONE]

The pattern matches the emitter: tokens about 200 ms apart. The last line keeps the space of the engine's data: [DONE] because the bytes pass through unchanged. The lambda ran on a virtual thread too (body: writer thread is virtual = true in the gateway log), because the same applicationTaskExecutor runs MVC async work when virtual threads are enabled.

Example 5: heartbeat comments while the engine reads the prompt

Before a model produces its first token, it has to read the whole prompt. For a coding agent the prompt can hold thousands of lines of code, so this step can take several seconds, and during it nothing flows on the connection. A proxy or load balancer between the client and the gateway may decide that a silent connection is dead and close it.

SSE has a tool for this. A line that starts with : is a comment, and SSE parsers skip it (lesson 37, Example 4). The gateway can send a comment at a fixed interval until the first real event arrives. This keeps the connection active and changes nothing for the client. The idea is called a heartbeat, like a pulse that shows the connection is alive.

The fake engine has a model name, slowstart, that waits one second before its first chunk. This stands in for prompt processing.

@PostMapping(path = "/heartbeat/v1/chat/completions", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
SseEmitter chat(@RequestBody String body) {
    SseEmitter emitter = new SseEmitter(0L);
    streamExecutor.execute(() -> {
        long every = props.heartbeatInterval().toMillis();     // gateway.heartbeat-interval, 15s by default
        AtomicBoolean waiting = new AtomicBoolean(true);
        Thread beat = Thread.ofVirtual().start(() -> {
            try {
                while (waiting.get()) {
                    Thread.sleep(every);
                    if (waiting.get()) emitter.send(SseEmitter.event().comment("keep-alive"));
                }
            } catch (IOException | IllegalStateException | InterruptedException e) {
                // the relay loop below notices a closed connection on its next send
            }
        });
        try {
            engine.post().uri("/v1/chat/completions")
                    .contentType(MediaType.APPLICATION_JSON)
                    .body(body)
                    .exchange((request, response) -> {
                        try (var reader = new BufferedReader(
                                new InputStreamReader(response.getBody(), StandardCharsets.UTF_8))) {
                            for (String line; (line = reader.readLine()) != null; ) {
                                if (line.startsWith("data:")) {
                                    waiting.set(false);          // real data flows, stop the pings
                                    emitter.send(SseEmitter.event().data(line.substring(5).strip()));
                                }
                            }
                        }
                        return null;
                    });
            emitter.complete();
        } catch (Exception e) {
            emitter.completeWithError(e);
        } finally {
            waiting.set(false);
            beat.interrupt();
        }
    });
    return emitter;
}

A second virtual thread, named beat, sleeps for the interval and then sends a comment. The flag waiting ends the heartbeat at the first real event. SseEmitter.send is synchronized, so the heartbeat thread and the relay loop do not mix their writes. The demo ran with --gateway.heartbeat-interval=300ms so that the pings are visible within a second.

curl -sN -H 'Content-Type: application/json' \
  -d '{"model":"slowstart","stream":true,"max_tokens":3}' \
  http://127.0.0.1:18670/heartbeat/v1/chat/completions | ./short.sh | ./show.sh

+ 316 ms  :keep-alive
+ 617 ms  :keep-alive
+ 918 ms  :keep-alive
+1191 ms  role chunk
+1391 ms  token [Hello]
+1593 ms  token [ from]
+1794 ms  token [ the]
+1796 ms  finish stop
+1798 ms  data:[DONE]

Three comments arrive during the engine's one-second wait (at about 300, 600, and 900 ms), and the real events follow at the 200 ms rhythm. The short.sh script prints a comment as :keep-alive. For comparison, the same request through /emitter shows nothing until the first chunk:

+1015 ms  role chunk
+1215 ms  token [Hello]
+1416 ms  token [ from]
+1617 ms  token [ the]
+1618 ms  finish stop
+1620 ms  data:[DONE]

On the wire a comment is the line :keep-alive followed by a blank line (od -c on the first bytes showed : k e e p - a l i v e \n \n). A comment also commits the response, which means the client sees the 200 status and the headers at the first ping and not at the first token.

A client library that reads only data: lines ignores the comments. A client that treats every line as JSON would fail on them. This is why the heartbeat belongs behind an option for clients that need it. The default interval of 15 seconds in the properties is a configuration value of the gateway, not a protocol constant.

Example 6: nothing between engine and client may buffer

Streaming works only if every step on the way passes bytes on as they come. Buffering means holding bytes in memory until some amount has collected or the data ends. One buffering step is enough to ruin the stream. The steps are the engine, the gateway's client for the engine, the gateway's response, and every proxy between the gateway and the client.

The response headers show whether a step buffers. A response with a known length has been buffered by definition, because the sender had to see all the bytes to count them. curl -i prints the headers of the same request on three paths: the engine directly, the emitter endpoint, and the buffered endpoint.

B='{"model":"fake","stream":true,"max_tokens":3}'
curl -sN -i -H 'Content-Type: application/json' -d "$B" http://127.0.0.1:18671/v1/chat/completions | head -4
curl -sN -i -H 'Content-Type: application/json' -d "$B" http://127.0.0.1:18670/emitter/v1/chat/completions | head -4
curl -sN -i -H 'Content-Type: application/json' -d "$B" http://127.0.0.1:18670/buffered/v1/chat/completions | head -4

HTTP/1.1 200 OK
Date: Sat, 03 Oct 2026 16:17:02 GMT
Transfer-encoding: chunked
Content-type: text/event-stream
HTTP/1.1 200 
Content-Type: text/event-stream
Transfer-Encoding: chunked
Date: Sat, 03 Oct 2026 16:17:01 GMT
HTTP/1.1 200 
Content-Type: text/event-stream;charset=UTF-8
Content-Length: 874
Date: Sat, 03 Oct 2026 16:17:02 GMT

The engine and the SseEmitter response use Transfer-Encoding: chunked. HTTP/1.1 sends a chunked body as a series of pieces, each with its own length, so the sender does not need to know the total size before it starts. The buffered controller sends Content-Length: 874, because the string was complete before the first byte was written. A missing Content-Length together with chunked is the first thing to check when a stream arrives late. The Date header values vary by run.

Other places where bytes can wait:

The gateway can control the first two rows. It has to test the rest in the real deployment.

Example 7: flushing

When code writes bytes to a response, they do not go to the network at once. The servlet container (here Tomcat, the server inside Spring Boot) first puts them in a memory buffer. It sends the buffer when the buffer is full, when the response ends, or when the code calls flush(). A relay that writes a 170-byte chunk and does not flush leaves the chunk in the buffer. A slow stream of small events can then wait until the stream ends.

SseEmitter flushes after each send, so Example 3 needed no extra work. StreamingResponseBody does not flush for the code. Three controllers show the cases. Each copies the engine's stream to the response in a loop. They differ in the stream they write to and in whether they flush.

// /noflush: copy with transferTo on the lambda's stream, never flush
StreamingResponseBody stream = out -> engine.post().uri("/v1/chat/completions")
        .contentType(MediaType.APPLICATION_JSON)
        .body(body)
        .exchange((request, response) -> {
            response.getBody().transferTo(out);                // writes, but never flushes
            return null;
        });

// /outflush: read loop that calls flush() on the lambda's stream
.exchange((request, response) -> {
    InputStream in = response.getBody();
    byte[] buffer = new byte[1024];
    for (int n; (n = in.read(buffer)) >= 0; ) {
        out.write(buffer, 0, n);
        out.flush();                                           // does nothing for this stream
    }
    return null;
});

// /body: the same loop on servletResponse.getOutputStream(), as in Example 4

The first controller copies with transferTo and never flushes. The second calls flush() after every write, but on the stream that Spring passes to the lambda. The third is /body from Example 4, which writes to servletResponse.getOutputStream(). Run all three.

B='{"model":"fake","stream":true,"max_tokens":4}'
for p in body outflush noflush; do
  echo "== $p"
  curl -sN -H 'Content-Type: application/json' -d "$B" http://127.0.0.1:18670/$p/v1/chat/completions | ./short.sh | ./show.sh
done

== body
+ 196 ms  role chunk
+ 382 ms  token [Hello]
+ 583 ms  token [ from]
+ 783 ms  token [ the]
+ 985 ms  token [ fake]
+ 987 ms  finish stop
+ 990 ms  data: [DONE]
== outflush
+ 818 ms  role chunk
+ 820 ms  token [Hello]
+ 822 ms  token [ from]
+ 824 ms  token [ the]
+ 825 ms  token [ fake]
+ 827 ms  finish stop
+ 829 ms  data: [DONE]
== noflush
+ 818 ms  role chunk
+ 820 ms  token [Hello]
+ 822 ms  token [ from]
+ 824 ms  token [ the]
+ 826 ms  token [ fake]
+ 827 ms  finish stop
+ 829 ms  data: [DONE]

Only /body streams. /outflush calls flush() after every write and still delivers everything at the end, which looks like a contradiction. The reason is in the Spring source. In the spring-web 7.0.9 jar, ServletServerHttpResponse.getBody() returns the servlet output stream wrapped with StreamUtils.nonFlushing, unless the flag spring.http.response.flush.enabled is true. The flag is read with SpringProperties.getFlag, which looks in a spring.properties file on the class path or in a JVM system property.

The stream that StreamingResponseBody passes to the lambda is that wrapped stream, so its flush() does nothing. The servlet response's own output stream, which /body writes to, flushes normally.

There are two fixes. The first is the one in Example 4: write to HttpServletResponse.getOutputStream(). The second sets the flag, either as spring.http.response.flush.enabled=true in src/main/resources/spring.properties or as -Dspring.http.response.flush.enabled=true on the command line. The run below used the command-line form on the unchanged jar:

== outflush
+ 237 ms  role chunk
+ 437 ms  token [Hello]
+ 638 ms  token [ from]
+ 838 ms  token [ the]
+ 1039 ms  token [ fake]
+ 1041 ms  finish stop
+ 1043 ms  data: [DONE]
== noflush
+ 818 ms  role chunk
+ 820 ms  token [Hello]
+ 822 ms  token [ from]
+ 824 ms  token [ the]
+ 826 ms  token [ fake]
+ 827 ms  finish stop
+ 829 ms  data: [DONE]

With the flag, /outflush streams. /noflush still delivers everything at the end, because it never calls flush(): the flag makes flush() work and does not add flushes. Whichever fix is chosen, the loop must call flush() after each write.

Example 8: a client disconnect cancels the upstream request

A user who closes a chat window or presses Ctrl+C expects the model to stop. If the gateway keeps reading, the engine keeps generating for nobody. That occupies one of the engine's slots and a share of the GPU, and other users wait for no reason. So the gateway has to notice that the client is gone and close its own connection to the engine. The engine's next write then fails, and the engine stops, exactly as the catch (IOException e) block of Example 1 shows.

The experiment asks for 40 tokens, which would take 8 seconds at 200 ms each, and curl --max-time 0.7 leaves after 0.7 seconds. Afterwards the engine log is printed.

./engine.sh                       # restarts the fake engine so its log starts empty
curl -sN --max-time 0.7 -H 'Content-Type: application/json' \
  -d '{"model":"fake","stream":true,"max_tokens":40}' \
  http://127.0.0.1:18670/emitter/v1/chat/completions | ./short.sh | ./show.sh
sleep 2
cat engine.log

+ 309 ms  role chunk
+ 491 ms  token [Hello]
+ 692 ms  token [ from]
engine: listening on 127.0.0.1:18671
engine: request received, will generate 40 tokens
engine: sent token 1
engine: sent token 2
engine: sent token 3
engine: sent token 4
engine: sent token 5
engine: client gone after token 5

The client saw three lines before it left. The engine log shows that the engine stopped after 5 of 40 tokens, which is far less than the 40 it was asked for. The same experiment on /body ended with engine: client gone after token 5 as well.

The chain of events has three links, and two of them are explicit in the code:

1. The client closes its TCP connection.
2. The gateway notices on a write. Tomcat reports a closed connection only when the gateway writes to it. curl left at 0.7 seconds. Two tokens later, at about 1.1 seconds, emitter.send threw an exception. In the log the failure is AsyncRequestNotUsableException: ServletResponse failed to flushBuffer: java.io.IOException: Broken pipe. Because of this, detection takes up to one token interval, and the count of tokens the engine sends after the client leaves varies by one or two between runs.
3. The exception leaves the exchange callback, and the try (var reader ...) block has closed the reader. Closing the reader closes the response stream, which closes the connection to the engine. The engine's next write fails and it stops, as the log shows.

The try block is required. Without it, the first version of this controller stopped relaying at the same moment, but the engine ran to the end of its 12-token stream. The reason is a detail of RestClient. RestClient.exchange closes the response when the callback returns. With the JDK-based response class (JdkClientHttpResponse, the class used by the default request factory), close() first reads the remaining body and only then closes the stream. Its bytecode calls StreamUtils.drain and then close. Reading the remaining body waits for the engine to finish, so the connection stays open and the generation continues.

Closing the stream inside the callback makes the later drain fail at once, which ends the connection. The gateway of this lesson uses SimpleClientHttpRequestFactory (Example 3) and keeps the explicit close in every relay, because the behavior has to hold whichever factory the project uses.

Example 9: timeouts

Three different limits protect a streaming endpoint, and they answer different questions. A single "timeout" value cannot do all three jobs.

The read timeout applies to each wait on the socket. It covers both the wait for the first chunk and the pause between two chunks, so a stalled engine is detected. It does not limit how long the whole answer takes, because each arriving chunk starts a new wait. The total limit does. A long answer from a slow model can be legitimate, so the value of the total limit follows from the longest answer the deployment accepts.

The gateway reads the read timeout from its own property gateway.read-timeout. The Boot property spring.http.client.read-timeout did not stop a stalled stream in this project. With the default factory and with spring.http.client.factory=simple, a stalled engine was relayed until the 60-second total limit ended the request. The gateway therefore sets the timeout in code on the factory, as in Example 3.

When the read timeout fires during a stream, the response has usually started. The status line 200 has been sent and cannot change. The error has to travel as an event inside the stream. The guarded controller catches the timeout and sends an OpenAI-style error object followed by [DONE].

@PostMapping(path = "/guarded/v1/chat/completions", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
SseEmitter chat(@RequestBody String body) {
    SseEmitter emitter = new SseEmitter(60_000L);          // total limit for the whole stream
    emitter.onTimeout(() -> log.info("guarded: total time limit reached"));
    streamExecutor.execute(() -> {
        try {
            engine.post().uri("/v1/chat/completions")
                    .contentType(MediaType.APPLICATION_JSON)
                    .body(body)
                    .exchange((request, response) -> {
                        try (var reader = new BufferedReader(
                                new InputStreamReader(response.getBody(), StandardCharsets.UTF_8))) {
                            for (String line; (line = reader.readLine()) != null; ) {
                                if (line.startsWith("data:")) emitter.send(SseEmitter.event().data(line.substring(5).strip()));
                            }
                        }
                        return null;
                    });
            emitter.complete();
        } catch (ResourceAccessException e) {
            if (isTimeout(e)) {
                log.info("guarded: engine read timed out, sending an error event");
                try {
                    emitter.send(SseEmitter.event().data(
                            "{\"error\":{\"type\":\"gateway_timeout\",\"message\":\"the engine stopped sending data\"}}"));
                    emitter.send(SseEmitter.event().data("[DONE]"));
                    emitter.complete();
                } catch (IOException gone) {
                    emitter.completeWithError(gone);
                }
            } else {
                emitter.completeWithError(e);             // for example, the client disconnected
            }
        }
    });
    return emitter;
}

private static boolean isTimeout(Throwable t) {
    for (; t != null; t = t.getCause()) {
        if (t instanceof SocketTimeoutException || t instanceof HttpTimeoutException) return true;
    }
    return false;
}

RestClient wraps every IOException from the callback in a ResourceAccessException. This holds for a write failure toward the client and for a read timeout toward the engine alike. The helper isTimeout walks the cause chain to tell them apart. A timeout gets an error event. Any other failure ends the emitter with an error, because the client is gone and cannot read an event anyway.

The demo started the gateway with --gateway.read-timeout=1s and used the fake engine's stall model, which sends two tokens and then stays silent.

curl -sN -H 'Content-Type: application/json' \
  -d '{"model":"stall","stream":true,"max_tokens":6}' \
  http://127.0.0.1:18670/guarded/v1/chat/completions | ./short.sh | ./show.sh

+ 275 ms  role chunk
+ 470 ms  token [Hello]
+ 671 ms  token [ from]
+1676 ms  data:{"error":{"type":"gateway_timeout","message":"the engine stopped sending data"}}
+1677 ms  data:[DONE]

The error event arrives 1.0 second after the last token, which is the configured silence limit (the token at 671 ms, the error at 1676 ms). The first-chunk case uses the same code path. With --gateway.read-timeout=500ms and the slowstart model (one second before the first chunk), the client received the error event and [DONE] before any token:

curl -sN -i -H 'Content-Type: application/json' \
  -d '{"model":"slowstart","stream":true,"max_tokens":3}' \
  http://127.0.0.1:18670/guarded/v1/chat/completions | cut -c1-120 | ./show.sh

+ 715 ms  HTTP/1.1 200 
+ 717 ms  Content-Type: text/event-stream
+ 719 ms  Transfer-Encoding: chunked
+ 721 ms  Date: Sat, 03 Oct 2026 16:16:50 GMT
+ 723 ms  
+ 725 ms  data:{"error":{"type":"gateway_timeout","message":"the engine stopped sending data"}}
+ 727 ms  data:[DONE]

The status is 200 even though the request failed, because the emitter commits the response when it sends its first event, and the first event here was the error. Clients of the OpenAI protocol handle this case by looking for an error object in a data event.

A timeout that fires before the first event could return a real 504 status instead, if the controller waited for the first line before returning the emitter. That variant needs the first line to be read before the handler returns, which gives up the early commit of the heartbeat in Example 5.

Example 10: NDJSON for an Ollama-style /api/chat

Ollama has its own HTTP API next to the OpenAI-style one. Its /api/chat streams newline-delimited JSON, abbreviated NDJSON. In NDJSON every line is one complete JSON object. There is no data: prefix and no [DONE] marker. A client reads a line, parses it, and shows the text.

The gateway keeps one engine protocol, the OpenAI one, and converts. The request body of /api/chat has model, messages, and options. The controller builds an OpenAI request from it, relays the engine's SSE stream, and writes one NDJSON line per chunk. The field names of the output follow the shape described in lesson 37, Example 13: model, created_at, message, done, and done_reason. The gateway writes only these fields. It does not write the duration and count fields that Ollama adds to the last line.

@RestController
class OllamaChatController {
    record Message(String role, String content) {}

    @JsonInclude(JsonInclude.Include.NON_NULL)
    record Line(String model, @JsonProperty("created_at") String createdAt, Message message,
                boolean done, @JsonProperty("done_reason") String doneReason) {}

    // constructor takes RestClient engine and tools.jackson.databind.json.JsonMapper mapper

    @PostMapping(path = "/api/chat", produces = MediaType.APPLICATION_NDJSON_VALUE)
    ResponseEntity<StreamingResponseBody> chat(@RequestBody JsonNode ollama, HttpServletResponse servletResponse) {
        ObjectNode openai = mapper.createObjectNode();
        openai.set("model", ollama.get("model"));
        openai.set("messages", ollama.get("messages"));
        openai.put("stream", true);
        if (ollama.path("options").has("num_predict")) {
            openai.set("max_tokens", ollama.get("options").get("num_predict"));
        }
        String body = mapper.writeValueAsString(openai);
        StreamingResponseBody stream = ignored -> engine.post().uri("/v1/chat/completions")
                .contentType(MediaType.APPLICATION_JSON)
                .body(body)
                .exchange((request, response) -> {
                    OutputStream out = servletResponse.getOutputStream();   // flush() works on this stream only
                    try (var reader = new BufferedReader(
                            new InputStreamReader(response.getBody(), StandardCharsets.UTF_8))) {
                        for (String line; (line = reader.readLine()) != null; ) {
                            if (!line.startsWith("data:")) continue;
                            String data = line.substring(5).strip();
                            if (data.equals("[DONE]")) break;
                            Line converted = toLine(mapper.readTree(data));
                            if (converted == null) continue;      // the opening role chunk has no text
                            out.write((mapper.writeValueAsString(converted) + "\n").getBytes(StandardCharsets.UTF_8));
                            out.flush();
                        }
                    }
                    return null;
                });
        return ResponseEntity.ok().contentType(MediaType.APPLICATION_NDJSON).body(stream);
    }

    private static Line toLine(JsonNode chunk) {
        JsonNode choice = chunk.path("choices").path(0);
        String model = chunk.path("model").asString();
        String createdAt = Instant.ofEpochSecond(chunk.path("created").asLong()).toString();
        if (choice.path("finish_reason").isString()) {
            return new Line(model, createdAt, new Message("assistant", ""), true, choice.get("finish_reason").asString());
        }
        JsonNode content = choice.path("delta").path("content");
        if (!content.isString()) return null;
        return new Line(model, createdAt, new Message("assistant", content.asString()), false, null);
    }
}

Spring Boot 4 uses Jackson 3 (the JSON library). Its packages start with tools.jackson, as in tools.jackson.databind.JsonNode and JsonMapper, and its methods throw unchecked exceptions. JsonNode.asString() is the Jackson 3 name for text extraction, and isString() replaces isTextual(). The annotations JsonInclude and JsonProperty keep their com.fasterxml.jackson.annotation package. A JsonMapper bean is auto-configured and injected.

The created_at value comes from the engine's created field (Unix seconds), formatted as an ISO-8601 instant. This makes the output the same on every run. A production gateway can use the current time instead.

curl -sN -i -H 'Content-Type: application/json' \
  -d '{"model":"fake","messages":[{"role":"user","content":"hi"}],"options":{"num_predict":3}}' \
  http://127.0.0.1:18670/api/chat | ./show.sh

+ 308 ms  HTTP/1.1 200 
+ 310 ms  Content-Type: application/x-ndjson
+ 312 ms  Transfer-Encoding: chunked
+ 314 ms  Date: Sat, 03 Oct 2026 16:17:02 GMT
+ 316 ms  
+ 317 ms  {"model":"fake","created_at":"2026-10-03T13:05:28Z","message":{"role":"assistant","content":"Hello"},"done":false}
+ 474 ms  {"model":"fake","created_at":"2026-10-03T13:05:28Z","message":{"role":"assistant","content":" from"},"done":false}
+ 675 ms  {"model":"fake","created_at":"2026-10-03T13:05:28Z","message":{"role":"assistant","content":" the"},"done":false}
+ 677 ms  {"model":"fake","created_at":"2026-10-03T13:05:28Z","message":{"role":"assistant","content":""},"done":true,"done_reason":"stop"}

The content type is application/x-ndjson. Each content chunk became one line with done: false and only the new text. The opening role chunk produced no line, because it has no text. The finish chunk became the last line with done: true and done_reason: "stop". The [DONE] marker was dropped, because NDJSON ends when the line with done: true arrives.

The option num_predict from options became max_tokens, which is why three tokens were generated. The controller writes to the servlet output stream for the reason given in Example 7.

Example 11: testing that tokens arrive before the engine finishes

A test that measures time and asserts "the first token came within 300 ms" fails on a loaded machine. A better test proves the property without a clock, by controlling the engine. The test engine sends the first token at once and then waits on a CountDownLatch before it sends the next one. A latch is a gate: threads that wait on it block until another thread opens it with countDown().

The test then does these steps in order. It reads the first token from the gateway. This is possible only if the gateway relays the token while the engine is blocked at the gate. It checks that the engine has not finished. Only then does it open the gate and let the rest through.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class StreamingTest {
    static volatile CountDownLatch release = new CountDownLatch(1);   // the test opens this gate
    static FakeEngine engine;

    @DynamicPropertySource
    static void engineUrl(DynamicPropertyRegistry registry) throws Exception {
        // token 0 is sent at once; every later token waits until the test calls release.countDown()
        engine = new FakeEngine(0, i -> { if (i > 0) { release.await(); Thread.sleep(20); } });
        registry.add("gateway.engine-url", engine::baseUrl);
    }

    @BeforeEach
    void closeTheGate() { release = new CountDownLatch(1); engine.reset(); }

    @AfterAll
    static void stopEngine() { engine.close(); }

    @LocalServerPort int port;
    final HttpClient http = HttpClient.newHttpClient();

    @ParameterizedTest
    @ValueSource(strings = {"/emitter", "/body"})
    void firstTokenArrivesWhileTheEngineIsStillGenerating(String prefix) throws Exception {
        List<String> seen = new ArrayList<>();
        HttpResponse<Stream<String>> response = http.send(request(prefix), HttpResponse.BodyHandlers.ofLines());
        try (Stream<String> lines = response.body()) {
            var it = lines.iterator();
            while (it.hasNext()) {
                String line = it.next();
                if (!line.startsWith("data:")) continue;
                seen.add(line);
                if (line.contains("Hello")) break;                // first content token
            }
            assertEquals(1, engine.finished.getCount(), "the engine must still be generating");
            assertEquals(1, engine.sent.get());
            release.countDown();
            it.forEachRemaining(line -> { if (line.startsWith("data:")) seen.add(line); });
        }
        assertTrue(engine.finished.await(5, TimeUnit.SECONDS));
        assertEquals("data:[DONE]", seen.get(seen.size() - 1).replace(" ", ""));
    }

    @Test
    void aBufferingProxyHoldsEveryTokenUntilTheEngineIsDone() throws Exception {
        CompletableFuture<HttpResponse<String>> pending =
                http.sendAsync(request("/buffered"), HttpResponse.BodyHandlers.ofString());
        assertTrue(engine.firstTokenSent.await(5, TimeUnit.SECONDS));
        assertThrows(TimeoutException.class, () -> pending.get(500, TimeUnit.MILLISECONDS));
        release.countDown();
        String body = pending.get(5, TimeUnit.SECONDS).body();
        assertTrue(body.contains("Hello") && body.contains("[DONE]"));
    }

    @Test
    void ollamaStyleChatStreamsOneJsonObjectPerLine() throws Exception {
        // POST /api/chat, read the first line, check content type, "Hello", done:false,
        // check the engine has not finished, release the gate, check the last line has done:true
    }
}

Some points about this test:

• @SpringBootTest with RANDOM_PORT starts the real server, so the test goes through Tomcat and the relay thread like curl does.
• @DynamicPropertySource starts the test engine first and points gateway.engine-url at its random port.
• BodyHandlers.ofLines() yields lines as they arrive, which lets the test read one line and stop.
• The engine.finished latch is the evidence. It still has a count of 1 while the test holds the first token, so the first token reached the client while the engine was still generating.

The negative test shows the buffering proxy of Example 2 failing the same check. It starts the request, waits until the engine has sent its first token, and then asserts that the response is not complete after 500 ms. The only time-based step is this 500 ms wait. The engine's gate is closed during the wait, so the buffered response cannot complete, and a slow machine cannot change the result.

JAVA_HOME=/usr/lib/jvm/java-25-openjdk-amd64 mvn -B test

[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.028 s -- in demo.StreamingTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

The six test cases are firstTokenArrivesWhileTheEngineIsStillGenerating for /emitter and /body, closingTheClientConnectionStopsTheEngine for the same two (Example 12), ollamaStyleChatStreamsOneJsonObjectPerLine, and aBufferingProxyHoldsEveryTokenUntilTheEngineIsDone. The elapsed time of 4 seconds is the total of the test run and varies by machine.

Example 12: testing that a disconnect stops the engine

Example 8 showed by hand that a disconnect stops the engine. A test can check it automatically. The disconnect test uses the same engine and a second latch, clientGone, which the engine counts down in its IOException handler (the line engine: client gone after token N in Example 8). The test reads one token and closes the response stream, which closes the TCP connection. Then it waits for the engine to report the broken connection.

@ParameterizedTest
@ValueSource(strings = {"/emitter", "/body"})
void closingTheClientConnectionStopsTheEngine(String prefix) throws Exception {
    HttpResponse<Stream<String>> response = http.send(request(prefix, 200), HttpResponse.BodyHandlers.ofLines());
    try (Stream<String> lines = response.body()) {
        lines.filter(line -> line.contains("Hello")).findFirst();   // read one token, then close
    }
    release.countDown();                                  // the engine keeps writing and meets the closed connection
    assertTrue(engine.clientGone.await(5, TimeUnit.SECONDS), "the engine must see the disconnect");
    assertEquals(1, engine.finished.getCount());
}

The request asks for 200 tokens, and the engine pauses 20 ms before each token once the gate is open, so a complete answer would take about 4 seconds. The assertion engine.clientGone.await(5, TimeUnit.SECONDS) waits at most 5 seconds for the disconnect to arrive. It does not assert how soon the disconnect arrives, because the detection time depends on the token interval (Example 8). The second assertion checks that the engine did not reach [DONE].

A relay without the try block of Example 8 leaves the engine running to the end of its stream. The engine would then not report the disconnect, and the first assertion would time out. A test of the timeout path of Example 9 can use the same structure: an engine that goes silent after two tokens, a small gateway.read-timeout set with @DynamicPropertySource, and an assertion on the last two lines.

Example 13: choosing between SseEmitter and StreamingResponseBody

Both types worked in the experiments. The choice depends on how much of the work the code wants Spring to do. SseEmitter formats SSE and flushes for you. StreamingResponseBody hands over raw control, which also means raw responsibility. The table compares them.

The gateway of this course uses SseEmitter for /v1/chat/completions when it needs heartbeats, error events, and its own event sequence. It uses StreamingResponseBody for NDJSON on /api/chat, where it writes its own lines. A byte-for-byte relay of an SSE stream with no heartbeat needs only StreamingResponseBody.

Spring WebFlux with Flux<ServerSentEvent<String>> and WebClient is a third way, and the Spring documentation covers it. It needs a second web stack on the class path, which the gateway avoids. The virtual-thread MVC code above reaches the same behavior with ordinary blocking calls.

The next lesson, Loading, Unloading, and Scheduling Models, adds the part that decides which engine receives the stream. The finished command is assembled in Workshop: loop gateway, and the production concerns of the stream endpoints (API keys, rate limits, metrics) are in Securing and Observing the Gateway. The non-streaming endpoint that these controllers extend is in An OpenAI-Compatible API in Front of llama-server.

Interview points

• A streaming relay forwards each upstream chunk as it arrives. A proxy that reads the whole upstream response and then writes it adds the engine's full generation time to the time to first token.
• Transfer-Encoding: chunked without Content-Length is the header signature of a streamed response. Content-Length on a streamed endpoint means the response was buffered.
• SseEmitter flushes after each send and formats SSE events. StreamingResponseBody writes raw bytes, and Spring wraps the stream it passes to the lambda so that flush() has no effect.
• With StreamingResponseBody, either write to HttpServletResponse.getOutputStream() or set spring.http.response.flush.enabled=true, and call flush() after each write in both cases.
• The servlet container reports a closed client connection only when the application writes. Disconnect detection therefore takes up to one token interval, and a stalled engine does not trigger it.
• Closing the stream to the engine stops the generation, because the engine's next write fails. RestClient drains a response in close() for the JDK-based response class, so the relay has to close the body stream itself before it leaves the callback.
• A read timeout limits each wait on the socket, so it covers the time to first byte and every pause between tokens. A total timeout limits the whole stream. They answer different questions and need different values.
• After the status line is sent, a failure travels as an event (an error object followed by [DONE]), because the status cannot change.
• SSE comment lines (: prefix) keep a connection alive during long prompt processing and are ignored by SSE parsers.
• NDJSON ends with the line that has done: true. It has no data: prefix and no [DONE] marker.
• A streaming test can use a latch-gated engine to prove that the first token reaches the client before the engine finishes, which avoids assertions on elapsed time.
• Virtual threads let one blocking relay per open stream run without a large platform-thread pool. A custom Executor bean in the project changed the thread that ran the relay from virtual to a platform thread, so the executor type is worth logging once.

Summary

• The engine streams OpenAI chunks as SSE events: one data: line per event, a blank line after each, and [DONE] at the end.
• A controller that returns String from RestClient.retrieve().body(...) delivers all tokens after the engine finishes (817 ms for 4 tokens at 200 ms each in the experiment) with a Content-Length header.
• SseEmitter returned from the handler commits a text/event-stream response. A virtual thread reads the engine's lines with RestClient.exchange and calls emitter.send for each data: line.
• With spring.threads.virtual.enabled=true, the applicationTaskExecutor runs relays on virtual threads, for SseEmitter tasks and for StreamingResponseBody.
• StreamingResponseBody needs ResponseEntity<StreamingResponseBody> as its return type for the content type to be set, and its lambda parameter ignores flush() in spring-web 7.0.9.
• Writing to HttpServletResponse.getOutputStream() with flush() after each write, or setting spring.http.response.flush.enabled=true, makes StreamingResponseBody stream. A copy that never calls flush() stays buffered under both fixes.
• A heartbeat comment (SseEmitter.event().comment("keep-alive")) sent from a second virtual thread every interval, until the first data event, keeps the connection active during prompt processing.
• Tomcat detects a closed client only on a write, so a relay notices a disconnect when the next token arrives and send throws.
• RestClient closes the response after the callback and the JDK-based response class drains the rest of the body first. The relay closes the engine stream in a try block to cancel the generation.
• SimpleClientHttpRequestFactory.setReadTimeout limits each wait between reads. spring.http.client.read-timeout did not stop a stalled stream in this project.
• A read timeout during a stream is reported as an error event and [DONE] under status 200. new SseEmitter(60_000L) sets a total time limit.
• /api/chat streams application/x-ndjson. The relay converts each OpenAI chunk to a line with model, created_at, message, done, and done_reason, and drops the role chunk and [DONE].
• Spring Boot 4 uses Jackson 3 (tools.jackson.databind), where asString() and isString() replace asText() and isTextual().
• A test with a latch-gated engine shows that the first token arrives while the engine is blocked, and a second latch shows that closing the client connection makes the engine's write fail.