Client Code: Latency, Async Inference, Streaming

Optimization work usually stays on the server: kernels, batching, GPUs, clusters. A request, though, starts in a client and ends in a client. Time spent in the client code and on the connection counts toward the latency that users feel, and the client side of a call is an area that is often overlooked when optimizing for latency and building for scale.

This lesson covers the client and server roles, the overhead of opening connections, asynchronous inference for jobs that do not need a fast answer, and streaming protocols for text, audio, and video. After it, a reader can estimate how much of a latency budget a connection setup uses, choose between synchronous and asynchronous calls, and choose between HTTP streaming, WebSockets, and gRPC.

1. The two sides of a call

The industry standard for client code is the OpenAI SDK (software development kit, a library for calling an API). It supports a wide range of compatible providers in addition to OpenAI's own models. Popular AI engineering frameworks and libraries such as LangChain, Vercel AI SDK, LiteLLM, and LlamaIndex, and dozens of others, can also act as clients.

Whether a team uses an existing library or its own code, the client can add latency or become a throughput bottleneck. Real-time applications may also need a protocol other than plain HTTP, such as WebSockets, to keep a continuous connection open.

The time of a request splits into parts. End-to-end inference time can be drawn as a row of four blocks, with the model server as the largest block and only a fraction of the whole:

end-to-end time = network (to server) + queue + model server + network (back)

example with invented numbers:  40 ms + 10 ms + 250 ms + 40 ms = 340 ms
model server share = 250 / 340 = about 74 percent

Even in this example, a quarter of the time is outside the model. Tuning the server to be twice as fast removes 125 ms. Cutting the two network legs and the queue time can remove up to 90 ms more.

2. Client latency overhead

Establishing a session between a client and a server takes a few dozen milliseconds, depending on the client's internet connection and the protocol. A TLS handshake (the exchange that sets up an encrypted connection) is a common part of it.

Consider a high-performance system with a P95 end-to-end latency SLA of 300 ms (95 percent of requests must finish within it):

TLS handshake: at least 10 percent of the budget
  300 ms x 0.10 = 30 ms spent before inference starts

The handshake costs at least 30 ms, so one tenth of the whole budget is gone before the model sees the request. The remedy is session re-use: later requests from the same client use the existing session and skip the setup. In the commands below, the first request pays the setup and the second, on the same connection, does not. (curl re-uses a connection only within one command. An application re-uses it by keeping one client object alive.)

# Illustration: two requests in one curl command share one connection.
curl -s https://inference.example.com/health https://inference.example.com/health -o /dev/null -o /dev/null \
  -w "connect: %{time_connect}s  tls: %{time_appconnect}s  total: %{time_total}s\n"

Session re-use is an old idea. Tools such as the OpenAI SDK provide it silently. A team that builds its own client, for example for a non-standard modality, should follow practices like session re-use deliberately.

3. Asynchronous inference

Some systems care about throughput, not latency. Bulk document processing and embedding a whole corpus are not latency sensitive, so asynchronous jobs fit them better.

An asynchronous request is a "fire and forget" way to run inference. An ordinary synchronous request has a timeout, generally a few minutes. After that the request fails. An asynchronous job avoids this: the server acknowledges the request at once, and later delivers the result to a webhook supplied in the original request. A webhook is a URL on the caller's side that the server calls when the work is done.

sequenceDiagram
    participant C as Client
    participant S as Inference service
    C->>S: submit job with webhook URL
    S-->>C: accepted, job id (immediate)
    Note over S: queue and process, minutes to hours
    S->>C: POST result to webhook URL

Asynchronous jobs still have time limits, but these are usually measured in hours, not minutes. Together with strong server-side queuing, asynchronous requests make high-throughput, latency-insensitive systems more reliable and more efficient. The shape of such a call is shown below as an illustration. The field names are generic and differ between providers.

# Illustration only: host, path, and field names are placeholders.
curl -s https://inference.example.com/async/embed \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"input": ["doc one", "doc two"], "webhook_url": "https://my-app.example.com/hooks/embeddings"}'
# immediate reply (illustration):
# {"job_id": "job_123", "status": "queued"}

4. Streaming and protocol support

Streaming makes applications feel instant. A language model writing 200 tokens at 50 tokens per second needs 4 seconds to finish. Without streaming, the user stares at a blank screen for 4 seconds. With streaming, the first words appear after the time to first token (TTFT, see measuring latency and throughput), for example 0.3 seconds, and reading begins while the rest is still being written.

For language models, streaming text over HTTP is enough. The common form is server-sent events (SSE), where one HTTP response stays open and the server writes small data: lines into it. The request and the beginning of the response look like this (an illustration for an OpenAI-compatible server, with placeholder host and model name):

curl -N https://inference.example.com/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "my-model", "stream": true, "messages": [{"role": "user", "content": "Say hello"}]}'

data: {"choices":[{"delta":{"content":"Hel"}}]}

data: {"choices":[{"delta":{"content":"lo"}}]}

data: {"choices":[{"delta":{"content":"!"}}]}

data: [DONE]

Each data: line carries a few new characters. The -N flag tells curl not to buffer the output, so each piece prints as it arrives. Each request has one request and one response. That is a good fit for text chat.

For other modalities, especially live voice and video, both the input and the output streams must carry more data and keep running in both directions at once. Plain one-time HTTP request and response is a poor fit for continuous streaming. The two most common bi-directional streaming protocols are:

flowchart LR
    subgraph HTTP[HTTP request-response]
        A1[Client] -->|one request| B1[Server]
        B1 -->|one response| A1
    end
    subgraph WS[WebSocket or gRPC stream]
        A2[Client] <-->|continuous two-way stream| B2[Server]
    end

WebSockets suit unstructured and real-time data such as audio, where the server that receives the data parses it and processes it downstream. A WebSocket server supports a fixed number of clients, which the developer configures. When that limit is reached, new connections cannot be established. They wait until a slot frees up or another replica scales up. This ties the WebSocket limit to the concurrency target of the autoscaling system.

gRPC also supports bi-directional streaming, but for structured data. A request sent over gRPC must follow a predefined schema, which removes the work of parsing unknown input. That extra validation layer makes gRPC slightly slower than WebSockets.

Choosing between them follows from the data. Audio chunks with no fixed structure and a need for the lowest overhead point to WebSockets. A service-to-service call with a well-defined message format points to gRPC. Voice workloads are covered in speech models.

Interview points

• The client is the browser, agent, or application. The server is the inference service. The OpenAI SDK is the industry standard client, and LangChain, Vercel AI SDK, LiteLLM, and LlamaIndex can serve as clients too.
• On-server inference time is only a fraction of end-to-end time, which also includes network time and queue time.
• A TLS handshake costs at least 10 percent of a 300 ms P95 SLA (30 ms) before inference starts. Re-using sessions avoids it, and custom clients for non-standard modalities must do it deliberately.
• Asynchronous inference is fire and forget. The server acknowledges at once and posts the result to a webhook. It suits bulk document processing and corpus embedding, and its limits are hours instead of the minutes allowed for synchronous requests.
• Streaming text works over HTTP. Live voice and video need bi-directional protocols.
• WebSockets suit unstructured real-time data such as audio and have a developer-configured client limit, after which new connections wait for a free slot or a new replica. gRPC suits structured service-to-service streaming with a predefined schema and is slightly slower because of the validation.

Summary

• Client code and connection setup are part of end-to-end latency.
• Session re-use removes the handshake cost from later requests.
• Asynchronous jobs with webhooks fit throughput-oriented, latency-insensitive work and allow time limits in hours.
• HTTP streaming is enough for text, while voice and video use WebSockets or gRPC.
• WebSockets favor unstructured data and gRPC favors structured data at a small validation cost.