Serving with vLLM

A server such as vLLM must decide how to spend the GPU's memory before it answers anything. The GPU holds three things: the model weights (the numbers the model learned during training), temporary working space for each calculation, and the KV cache (key-value cache). The KV cache stores notes about the tokens already processed, so the model does not have to reread the whole conversation for every new token. Almost every flag in this lesson changes one of these budgets, or changes how the server shapes its answers.

The previous lesson, vLLM Architecture and Installation, explained how vLLM is built. This lesson turns that design into working vllm serve commands. It covers memory, request length, batching, splitting a model over GPUs, quantization, prefix caching, structured outputs, tool calls, reasoning text, metrics, and sampling defaults.

After this lesson you can write a vllm serve command for a given GPU and model, predict the error vLLM prints when the numbers do not fit, and call the server from Java. The table lists each flag or field and the example that explains it.

Hardware tier. vLLM needs Linux with an NVIDIA GPU (tier 3) or better (tier 4). It cannot run in the environment that checks this course. For that reason every vllm serve command in this lesson is a bash block, and the log lines after it are representative: they are built from the format strings in the vLLM 0.30.0 source, and the values differ by machine.

Every flag name, default, parser name, and metric name comes from that source. The files used are vllm/engine/arg_utils.py and vllm/config/ (engine flags), vllm/entrypoints/launchers/cli_args.py (server flags), vllm/tool_parsers/__init__.py and vllm/reasoning/__init__.py (parser names), and vllm/v1/metrics/loggers.py (metrics).

The Java programs do not need a GPU. They compute the same numbers vLLM computes, or they build and parse the same JSON. You can run them anywhere and compare their output with what a real server prints.

Example 1: a basic vllm serve command

Every deployment answers four questions: which model to load, where to listen, who may call it, and what name clients use for the model. These map to the model argument, --host and --port, --api-key, and --served-model-name. A Java launcher passes them to ProcessBuilder as a list of strings, with one string for each flag and one for each value.

The program below builds that list and prints it as one shell line. Look at the argument count, and at what happens to the key in the printed line.

import java.util.ArrayList;
import java.util.List;

public class VllmServeCommand {
    record ServeOptions(String model, String host, int port, String apiKey, List<String> servedNames) {}

    static List<String> command(ServeOptions o) {
        List<String> cmd = new ArrayList<>(List.of("vllm", "serve", o.model()));
        cmd.addAll(List.of("--host", o.host(), "--port", Integer.toString(o.port())));
        if (o.apiKey() != null) cmd.addAll(List.of("--api-key", o.apiKey()));
        if (!o.servedNames().isEmpty()) {
            cmd.add("--served-model-name");
            cmd.addAll(o.servedNames());
        }
        return cmd;
    }

    static String redacted(List<String> cmd) {
        List<String> copy = new ArrayList<>(cmd);
        int i = copy.indexOf("--api-key");
        if (i >= 0) copy.set(i + 1, "***");
        return String.join(" ", copy);
    }

    public static void main(String[] args) {
        ServeOptions o = new ServeOptions("Qwen/Qwen2.5-Coder-7B-Instruct", "127.0.0.1", 8000,
                "local-secret", List.of("coder", "qwen-coder-7b"));
        List<String> cmd = command(o);
        System.out.println("argument count: " + cmd.size());
        System.out.println(redacted(cmd));
    }
}

argument count: 12
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --host 127.0.0.1 --port 8000 --api-key *** --served-model-name coder qwen-coder-7b

The output has 12 arguments because ProcessBuilder needs each flag and each value as its own string. Counting them: vllm, serve, and the model make 3; the host flag and value make 2 more; the port flag and value make 2; the key flag and value make 2; and --served-model-name with two names makes 3.

The printed line replaces the key with ***. vLLM does the same in its own log (redact_sensitive_args in vllm/entrypoints/serve/utils/api_utils.py hides api_key, hf_token, and watermark_config). This matters because log files are often copied into tickets and chat messages, and a key in a log is a leaked key.

Typed in a shell, the same command looks like this:

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --api-key local-secret \
  --served-model-name coder qwen-coder-7b

When the server starts, it prints lines like the following. They are representative: they are built from the vLLM 0.30.0 log format strings, and the values and the order of routes differ by machine and version. A route is a URL path the server answers.

INFO [api_utils.py] non-default args: {'model_tag': 'Qwen/Qwen2.5-Coder-7B-Instruct', 'host': '127.0.0.1', 'api_key': '***', 'served_model_name': ['coder', 'qwen-coder-7b']}
...
INFO [entry.py] Starting vLLM server on http://127.0.0.1:8000
INFO [launcher.py] Available routes are:
INFO [launcher.py] Route: /health, Methods: GET
INFO [launcher.py] Route: /v1/models, Methods: GET
INFO [launcher.py] Route: /v1/chat/completions, Methods: POST
INFO [launcher.py] Route: /v1/completions, Methods: POST
INFO [launcher.py] Route: /metrics, Methods: GET

Each flag has a default and a behavior. The table below comes from FrontendArgs and ModelConfig in the pinned source.

The default host is the main difference from llama-server, which listens on 127.0.0.1 unless told otherwise. The address 127.0.0.1 means "this machine only". With no --host, vLLM listens on all network interfaces (0.0.0.0), so any computer that can reach the machine can send it requests.

The --api-key flag does not close that gap completely. Its help text in the pinned source warns that the key protects only paths under /v1, /v2, and /inference. Other routes on the same server, such as /metrics, stay open. A key is therefore not a replacement for binding to 127.0.0.1 or for putting a proxy in front.

--served-model-name matters for clients. Without it, the model field of every request must be the full repository id, such as Qwen/Qwen2.5-Coder-7B-Instruct. With it, "model": "coder" works, and every response says "model": "coder" because the first name is reported. The Prometheus metrics in Example 13 also carry the first name in their model_name label.

Example 2: --max-model-len and the errors when it does not fit

A request needs room in the KV cache for every token it holds, and that count includes the prompt and the output together. --max-model-len sets the largest number of tokens one request may use. If it is too large, the cache cannot hold even one request of that size, so vLLM refuses to start rather than fail later.

vLLM checks the value twice at startup. The first check compares it with the model's own limit in config.json. The second compares the memory one maximum-length request needs with the KV cache memory left after loading the weights.

The program reproduces both checks for a Llama 3.1 8B shape on a 24 GB GPU and prints the messages in the format of the pinned source. Look at three things in the output: the two error messages, and the number of bytes of cache that one token needs.

import java.util.Locale;

public class MaxModelLenCheck {
    static final long GIB = 1L << 30, MIB = 1L << 20;

    static String gib(long bytes) {                     // Python: str(round(b / GiB, 2))
        return Double.toString(Math.round(bytes * 100.0 / GIB) / 100.0);
    }

    public static void main(String[] args) {
        int maxPositionEmbeddings = 131072;              // from config.json
        long pagePerBlock = 32L * 16 * 2 * 8 * 128 * 2;  // layers * blockSize * (K,V) * kvHeads * headDim * 2 bytes

        int requested = 200_000;
        if (requested > maxPositionEmbeddings) {
            System.out.println("User-specified max_model_len (" + requested + ") is greater than the derived"
                    + " max_model_len (max_position_embeddings=" + maxPositionEmbeddings + " ...)");
        }

        long usable = (long) (24_564 * MIB * 0.92);      // total memory times --gpu-memory-utilization
        long available = usable - 15_316 * MIB - 1_500 * MIB - 400 * MIB; // weights, activations, other
        int maxModelLen = maxPositionEmbeddings;         // default: taken from config.json
        long needed = (maxModelLen + 15) / 16 * pagePerBlock;
        if (needed > available) {
            long estimated = available / pagePerBlock * 16;
            System.out.println("To serve at least one request with the model's max seq len (" + maxModelLen
                    + "), (" + gib(needed) + " GiB KV cache is needed, which is larger than the available KV"
                    + " cache memory (" + gib(available) + " GiB). Based on the available memory, the"
                    + " estimated maximum model length is " + estimated + ".");
        }
        System.out.printf(Locale.ROOT, "KV bytes per 16-token block: %d (%d per token)%n",
                pagePerBlock, pagePerBlock / 16);
    }
}

User-specified max_model_len (200000) is greater than the derived max_model_len (max_position_embeddings=131072 ...)
To serve at least one request with the model's max seq len (131072), (16.0 GiB KV cache is needed, which is larger than the available KV cache memory (5.26 GiB). Based on the available memory, the estimated maximum model length is 43056.
KV bytes per 16-token block: 2097152 (131072 per token)

The first line is the configuration check from _get_and_verify_max_len in vllm/config/model.py. A requested length above max_position_embeddings (the longest sequence the model was trained on) stops startup with a ValueError. The reason is that models using RoPE (rotary position embedding, the method that tells the model where each token sits) give wrong results at positions past the trained range. The full message continues with the variable that overrides the check, VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, and a warning about NaN (not a number) outputs.

The second line is the memory check from _check_enough_kv_cache_memory in vllm/v1/core/kv_cache_utils.py. Without --max-model-len, vLLM takes the model's full 131,072 tokens. The arithmetic behind the message is:

cache one request needs:  131,072 tokens x 128 KiB per token = 16 GiB
usable memory:            24,564 MiB x 0.92                  = 22,599 MiB
minus BF16 weights:       22,599 - 15,316                    =  7,283 MiB
minus activation peak:     7,283 -  1,500                    =  5,783 MiB
minus other buffers:       5,783 -    400                    =  5,383 MiB = 5.26 GiB

The cache needs 16 GiB and only 5.26 GiB is free, so vLLM refuses to start. The 1,500 MiB activation peak and 400 MiB of other buffers are the assumptions used in Example 13 of Memory Math. The reason for refusing is practical: a server that can never finish one maximum-length request would fail later, under load, where the cause is harder to see.

The estimate of 43,056 tokens comes from a binary search in estimate_max_model_len over the lengths that fit. Memory is handed out in whole blocks of 16 tokens, and one block costs 2 MiB across all 32 layers. The estimate is therefore a multiple of 16: 5.26 GiB holds 2,691 blocks, and 2,691 times 16 is 43,056.

The number in the message is also the fix. Start with --max-model-len 43056 or less, or free GPU memory. --max-model-len also accepts 32k (32,000), 32K (32,768), and auto or -1, which picks the largest length that fits, according to the field's documentation in ModelConfig.

The third line shows where 2 MiB per block comes from: 32 layers, times 16 tokens, times 2 (one K and one V), times 8 KV heads, times 128 values per head, times 2 bytes per value. Dividing by 16 tokens gives 131,072 bytes (128 KiB) per token.

The checks above run at startup. A different check runs for every request. A prompt that is too long for the limit gets HTTP 400 with the message This model's maximum context length is ... tokens. However, you requested ... output tokens and your prompt contains ... input tokens, for a total of ... tokens. (from vllm/renderers/params.py).

Example 3: --gpu-memory-utilization and KV cache memory settings

Example 2 showed that the KV cache gets whatever memory is left after the weights and working space. --gpu-memory-utilization is the dial that sets the total. It is the fraction of the GPU's memory that this vLLM instance may use for weights, activations, and KV cache together. Its default in the pinned CacheConfig is 0.92. (Memory Math used 0.9, the value in older vLLM documentation.) The flag --kv-cache-memory-bytes is the alternative: it sets the cache size directly.

The program computes the KV cache size and the maximum concurrency (how many full-length requests fit at once) that vLLM logs at startup, for three settings, with --max-model-len 32768. Compare the first two lines to see what a lower fraction costs, and read the last line to see whether the server would start.

import java.util.Locale;

public class KvCacheBudget {
    static final long MIB = 1L << 20, GIB = 1L << 30;
    static final long PAGE = 32L * 16 * 2 * 8 * 128 * 2;   // bytes per 16-token block, all 32 layers

    static String line(long kvBytes, int maxModelLen) {
        long tokens = kvBytes / PAGE * 16;
        return String.format(Locale.ROOT, "GPU KV cache size: %,d tokens, Maximum concurrency for %,d tokens"
                + " per request: %.2fx", tokens, maxModelLen, (double) tokens / maxModelLen);
    }

    public static void main(String[] args) {
        long total = 24_564 * MIB;
        long nonKv = (15_316 + 1_500 + 400) * MIB;           // weights + activation peak + other buffers
        int len = 32_768;
        long at92 = (long) (total * 0.92) - nonKv;
        long at80 = (long) (total * 0.80) - nonKv;
        System.out.println("util 0.92: " + line(at92, len));
        System.out.println("util 0.80: " + line(at80, len));
        System.out.println("4 GiB set: " + line(4 * GIB, len));
        long oneRequest = (long) len / 16 * PAGE;
        System.out.printf(Locale.ROOT, "one 32K request needs %.2f GiB; 0.80 leaves %.2f GiB -> starts: %b%n",
                (double) oneRequest / GIB, (double) at80 / GIB, at80 >= oneRequest);
    }
}

util 0.92: GPU KV cache size: 43,056 tokens, Maximum concurrency for 32,768 tokens per request: 1.31x
util 0.80: GPU KV cache size: 19,472 tokens, Maximum concurrency for 32,768 tokens per request: 0.59x
4 GiB set: GPU KV cache size: 32,768 tokens, Maximum concurrency for 32,768 tokens per request: 1.00x
one 32K request needs 4.00 GiB; 0.80 leaves 2.38 GiB -> starts: false

The first line is the default. The cache holds 43,056 tokens, the same number as the estimate in Example 2, because the same memory is left over. vLLM logs this line from update_kv_cache_capacity in kv_cache_utils.py. Maximum concurrency is the cache size divided by --max-model-len: how many requests of the full length fit at the same time. Here 43,056 divided by 32,768 is 1.31. Most requests are shorter than the limit and use fewer blocks, so the number of requests that run together in practice is usually higher than this figure.

The second line lowers the fraction to 0.80. The worked numbers are:

0.92: 24,564 x 0.92 = 22,599 MiB;  22,599 - 17,216 = 5,383 MiB of cache
0.80: 24,564 x 0.80 = 19,651 MiB;  19,651 - 17,216 = 2,435 MiB of cache = 2.38 GiB

The budget falls by 12% of 24,564 MiB, about 2,948 MiB, and all of it comes out of the KV cache, because the weights and the activation peak do not shrink. The cache drops to 19,472 tokens, which is below one full 32K request (0.59x).

The last line applies the startup check from Example 2. One 32,768-token request needs 4 GiB, the cache would have only 2.38 GiB, so vLLM refuses to start with this setting and prints the same error message. A lower fraction is meant for sharing one GPU between two instances (for example 0.45 each), and it always costs cache.

The third line uses --kv-cache-memory-bytes 4G. The field's documentation in CacheConfig says that when it is set, --gpu-memory-utilization is ignored for the cache size. A fixed 4 GiB holds 2,048 blocks of 2 MiB, which is 32,768 tokens: exactly one full-length request (1.00x). A fixed size is useful when the memory-profiling step gives different results between runs, or when several processes share a GPU and each must stay within a known amount.

For the default case, the startup log shows lines like these. They come from the format strings in gpu_worker.py and kv_cache_utils.py, and they carry the same numbers as the first output line above:

INFO [gpu_worker.py] Available KV cache memory: 5.26 GiB
INFO [kv_cache_utils.py] GPU KV cache size: 43,056 tokens, Maximum concurrency for 32,768 tokens per request: 1.31x

Example 4: --max-num-seqs and --max-num-batched-tokens

The server does not process requests one at a time. It works in steps. In each scheduler step it picks some requests and runs the model once over all of them together, a single forward pass (one trip of the data through all the model's layers). Two flags limit one step. --max-num-seqs is the largest number of requests in one batch. --max-num-batched-tokens is the largest number of tokens computed in one forward pass. Those tokens are the prompt tokens being prefilled (processed for the first time) plus one new token for each request that is already decoding (generating its answer).

When neither flag is set, get_batch_defaults in arg_utils.py picks values from the GPU's memory and name. The program reproduces that table for the API server and then applies the validation rules from SchedulerConfig. Look at how the defaults change with the GPU, and at which of the last three checks passes.

public class BatchDefaults {
    record Limits(int maxBatchedTokens, int maxSeqs) {}

    static Limits serverDefaults(double gpuGiB, String name, boolean cpu, int worldSize) {
        if (cpu) return new Limits(2048 * worldSize, 128 * worldSize);
        if (gpuGiB >= 160) return new Limits(16384, 1024);
        if (gpuGiB >= 70 && !name.toLowerCase().contains("a100")) return new Limits(8192, 1024);
        return new Limits(2048, 256);
    }

    static String validate(int batched, int seqs, int maxModelLen) {
        if (batched < seqs)
            return "error: max_num_batched_tokens (" + batched + ") must be greater than or equal to max_num_seqs ("
                    + seqs + ").";
        if ((long) batched > (long) seqs * maxModelLen)
            return "warning: max_num_batched_tokens (" + batched + ") exceeds max_num_seqs * max_model_len ("
                    + (long) seqs * maxModelLen + ").";
        return "ok";
    }

    public static void main(String[] args) {
        Object[][] gpus = {{23.6, "NVIDIA GeForce RTX 4090"}, {79.2, "NVIDIA A100-SXM4-80GB"},
                {79.1, "NVIDIA H100 80GB HBM3"}, {178.4, "NVIDIA B200"}};
        for (Object[] g : gpus) {
            Limits l = serverDefaults((double) g[0], (String) g[1], false, 1);
            System.out.printf("%-26s tokens=%-6d seqs=%d%n", g[1], l.maxBatchedTokens(), l.maxSeqs());
        }
        Limits c = serverDefaults(0, "", true, 2);
        System.out.printf("%-26s tokens=%-6d seqs=%d%n", "CPU backend, world size 2", c.maxBatchedTokens(), c.maxSeqs());
        System.out.println(validate(256, 512, 32768));
        System.out.println(validate(8192, 1, 4096));
        System.out.println(validate(2048, 256, 32768));
    }
}

NVIDIA GeForce RTX 4090    tokens=2048   seqs=256
NVIDIA A100-SXM4-80GB      tokens=2048   seqs=256
NVIDIA H100 80GB HBM3      tokens=8192   seqs=1024
NVIDIA B200                tokens=16384  seqs=1024
CPU backend, world size 2  tokens=4096   seqs=256
error: max_num_batched_tokens (256) must be greater than or equal to max_num_seqs (512).
warning: max_num_batched_tokens (8192) exceeds max_num_seqs * max_model_len (4096).
ok

The first four lines are the defaults for vllm serve on four GPUs. A 24 GB card gets 2,048 tokens and 256 sequences per step. An H100 gets 8,192 and 1,024. An A100 with 80 GB gets the small defaults even though it has as much memory as an H100, because the source excludes any device whose name contains a100, with a comment that a larger token budget reduced throughput on that GPU. A B200, at 160 GiB or more, gets 16,384 and 1,024.

The CPU backend scales both numbers by the world size (the number of worker processes), so two workers give 4,096 and 256. The memory numbers in the program are representative values for each card.

The next three lines are the checks. A token budget smaller than the sequence limit is an error. Every decoding request needs at least one token per step, so 512 requests cannot each get a token from a budget of 256. A budget larger than max_num_seqs * max_model_len gets a warning that ends with "This may lead to unexpected behavior". No step can hold more tokens than all sequences at full length, so part of the budget could never be used (here 1 x 4,096 is 4,096, below 8,192). The default pair passes.

The two numbers work together with chunked prefill, which splits a long prompt into pieces so it does not block everyone else. It is enabled by default in the pinned SchedulerConfig (enable_chunked_prefill = True). In each step, decoding requests take one token each, and the rest of the budget goes to prompt chunks. For example, with a budget of 2,048 and 100 requests decoding, 100 tokens go to decoding and 1,948 tokens are left for prompt chunks. The lesson vLLM Architecture and Installation simulates this split.

The trade-off follows from it. A larger token budget prefills long prompts in fewer steps, which lowers time to first token (the wait before the first output token). But each step takes longer, which raises the time between tokens for requests that are already decoding. A larger --max-num-seqs lets more requests decode together, which raises throughput until the KV cache from Example 3 runs out. After that, requests wait or are preempted, which means their blocks are freed and their work is recomputed later.

Example 5: tensor and pipeline parallel sizes

Some models are too big for one GPU. The fix is to split the model over several GPUs, and there are two ways to cut it. An analogy: think of the model as a restaurant kitchen preparing a long recipe with many stages. Tensor parallelism (--tensor-parallel-size, -tp) is like having several cooks work on every stage together, each handling a slice of each dish, and calling out partial results to each other. In the model, every layer's matrices are split across GPUs, and the GPUs exchange partial results inside each layer. Pipeline parallelism (--pipeline-parallel-size, -pp) is like an assembly line: each cook owns a block of consecutive stages and passes the dish on. In the model, each GPU holds a contiguous range of layers, and activations (the intermediate numbers) pass from one GPU to the next. The analogy stops being accurate in one place: real GPUs in tensor parallelism must synchronize at every layer, which is far more communication than cooks calling out.

Both flags default to 1. The program computes, for a 70B shape (80 layers, 64 query heads, 8 KV heads), what each GPU holds under several layouts, and applies the head check from ModelConfig.verify_with_parallel_config. Watch how the weights and the KV cache per GPU change as tp grows.

import java.util.Locale;

public class ParallelLayout {
    static void layout(int tp, int pp) {
        int layers = 80, heads = 64, kvHeads = 8, headDim = 128;
        double weightsGiB = 70.6e9 * 2 / (1L << 30);
        if (heads % tp != 0) {
            System.out.printf("tp=%d pp=%d  Total number of attention heads (%d) must be divisible by tensor"
                    + " parallel size (%d).%n", tp, pp, heads, tp);
            return;
        }
        int kvPerGpu = Math.max(1, kvHeads / tp);          // replicated when tp > kvHeads
        int layersPerGpu = layers / pp;
        long kvBytesPerToken = 2L * layersPerGpu * kvPerGpu * headDim * 2;
        System.out.printf(Locale.ROOT, "tp=%d pp=%d  GPUs=%d  weights/GPU=%.1f GiB  layers/GPU=%d  KV heads/GPU=%d"
                + "  KV/token/GPU=%d KiB%n", tp, pp, tp * pp, weightsGiB / (tp * pp), layersPerGpu, kvPerGpu,
                kvBytesPerToken / 1024);
    }

    public static void main(String[] args) {
        layout(1, 1);
        layout(2, 1);
        layout(4, 1);
        layout(8, 1);
        layout(16, 1);
        layout(4, 2);
        layout(3, 1);
    }
}

tp=1 pp=1  GPUs=1  weights/GPU=131.5 GiB  layers/GPU=80  KV heads/GPU=8  KV/token/GPU=320 KiB
tp=2 pp=1  GPUs=2  weights/GPU=65.8 GiB  layers/GPU=80  KV heads/GPU=4  KV/token/GPU=160 KiB
tp=4 pp=1  GPUs=4  weights/GPU=32.9 GiB  layers/GPU=80  KV heads/GPU=2  KV/token/GPU=80 KiB
tp=8 pp=1  GPUs=8  weights/GPU=16.4 GiB  layers/GPU=80  KV heads/GPU=1  KV/token/GPU=40 KiB
tp=16 pp=1  GPUs=16  weights/GPU=8.2 GiB  layers/GPU=80  KV heads/GPU=1  KV/token/GPU=40 KiB
tp=4 pp=2  GPUs=8  weights/GPU=16.4 GiB  layers/GPU=40  KV heads/GPU=2  KV/token/GPU=40 KiB
tp=3 pp=1  Total number of attention heads (64) must be divisible by tensor parallel size (3).

The first line is the whole model on one GPU. The arithmetic is 70.6 billion weights times 2 bytes (BF16) divided by 2^30, which is 131.5 GiB, more than any single 80 GB card holds. The KV cache per token is 2 (K and V) x 80 layers x 8 KV heads x 128 values x 2 bytes = 327,680 bytes, which is 320 KiB.

Each doubling of tp halves the weights per GPU and halves the KV heads per GPU, so each GPU's cache per token also halves. At tp=8 every GPU holds one KV head. At tp=16 there are only 8 KV heads to share among 16 GPUs, so vLLM replicates them (get_num_kv_heads returns max(1, kvHeads // tp)). Each GPU still stores one head, and the KV cache per GPU stops shrinking.

tp=4 pp=2 uses 8 GPUs like tp=8 and has the same 16.4 GiB of weights per GPU. Each GPU holds 40 layers with two KV heads each, which gives the same 40 KiB of cache per token per GPU. The last line shows the startup error for a tensor-parallel size that does not divide the 64 attention heads, because the heads are dealt out evenly and 64 cannot be split into 3 equal groups.

The choice between the two follows from what travels between GPUs. Tensor parallelism communicates inside every layer, so it needs a fast link such as NVLink (a high-speed connection between GPUs) and is used within one machine. Pipeline parallelism sends activations only at stage boundaries, so it tolerates slower links and is used across machines, but a single request passes through the stages one after another. A later lesson on multi-GPU inference covers the communication costs and data and expert parallelism (--data-parallel-size, --enable-expert-parallel). A command that uses four GPUs looks like this:

vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --max-model-len 32768

Example 6: quantized checkpoints, --quantization, and --dtype

Quantization stores each weight in fewer bits, for example 4 bits instead of 16, so the model takes less memory at a small cost in accuracy. A checkpoint is the set of saved model files. Quantized checkpoints on the Hub (AWQ, GPTQ, FP8, compressed-tensors, and others; see GPTQ, AWQ, EXL3, NF4, and FP8 Serving) describe their format in a quantization_config object inside config.json, the file that lists the model's settings. vLLM reads its quant_method and selects the matching kernels (the GPU programs that do the math) without any flag.

The program parses two config.json fragments with Jackson and applies the rules from ModelConfig._verify_quantization and the dtype documentation. A dtype is the number type used for calculations, such as FP16 or BF16 (two 16-bit formats for decimal numbers). Look at what happens when a flag agrees with the file, and when it does not.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

public class QuantDetection {
    static String resolve(JsonNode config, String quantizationFlag) {
        JsonNode qc = config.path("quantization_config");
        String fromConfig = qc.isMissingNode() ? null : qc.path("quant_method").asText();
        if (fromConfig == null) return quantizationFlag == null ? "none (weights in " + dtype(config) + ")"
                : quantizationFlag;
        if (quantizationFlag != null && !quantizationFlag.equals(fromConfig))
            return "error: Quantization method specified in the model config (" + fromConfig + ") does not match"
                    + " the quantization method specified in the `quantization` argument (" + quantizationFlag + ").";
        return fromConfig + " (from config.json)";
    }

    static String dtype(JsonNode config) {            // --dtype auto: FP16 for FP32 and FP16 models, BF16 for BF16
        String t = config.path("torch_dtype").asText("float32");
        return t.equals("bfloat16") ? "bfloat16" : "float16";
    }

    public static void main(String[] args) throws Exception {
        ObjectMapper json = new ObjectMapper();
        JsonNode awq = json.readTree("""
                {"architectures": ["Qwen2ForCausalLM"], "torch_dtype": "float16",
                 "quantization_config": {"quant_method": "awq", "bits": 4, "group_size": 128,
                                         "zero_point": true, "version": "gemm"}}""");
        JsonNode plain = json.readTree("""
                {"architectures": ["LlamaForCausalLM"], "torch_dtype": "bfloat16"}""");
        System.out.println("AWQ checkpoint, no flag:        " + resolve(awq, null));
        System.out.println("AWQ checkpoint, -q gptq:         " + resolve(awq, "gptq"));
        System.out.println("BF16 checkpoint, no flag:       " + resolve(plain, null));
        System.out.println("BF16 checkpoint, -q fp8:        " + resolve(plain, "fp8"));
        System.out.println("--dtype auto for AWQ (float16): " + dtype(awq));
    }
}

AWQ checkpoint, no flag:        awq (from config.json)
AWQ checkpoint, -q gptq:         error: Quantization method specified in the model config (awq) does not match the quantization method specified in the `quantization` argument (gptq).
BF16 checkpoint, no flag:       none (weights in bfloat16)
BF16 checkpoint, -q fp8:        fp8
--dtype auto for AWQ (float16): float16

The first line is the normal case: the checkpoint says awq, no flag is given, and vLLM uses AWQ. The real code goes one step further and asks each method in a preference list whether it can serve the checkpoint faster. On a GPU with the Marlin kernels, an AWQ checkpoint is served by awq_marlin (one of the names in the overrides list in _verify_quantization).

The second line shows the mismatch error: a flag that contradicts config.json stops startup with that message. The third line is an unquantized BF16 model, so there is no quantization and --dtype auto keeps BF16.

The fourth line is online quantization. The checkpoint is BF16, and --quantization fp8 (short form -q fp8) makes vLLM quantize the weights to FP8 while loading them. The flag accepts the names in QuantizationMethods in vllm/model_executor/layers/quantization/__init__.py. These include awq, gptq, fp8, compressed-tensors, mxfp4, and the online shorthands fp8_per_tensor, fp8_per_block, fp8_per_channel, int8_per_channel_weight_only, nvfp4_per_token, and mxfp8.

The last line shows the --dtype auto rule from ModelConfig: FP16 for FP32 and FP16 checkpoints, and BF16 for BF16 checkpoints. --dtype half (FP16) is the value the documentation suggests for AWQ.

One format is missing from this list. The pinned source has no GGUF loader. LoadConfig.load_format lists auto, pt, safetensors, npcache, dummy, tensorizer, runai_streamer, sharded_state, mistral, and a few others, and no module in vllm/ reads GGUF. Models for vLLM come as safetensors directories. GGUF files are for llama.cpp and Ollama.

Example 7: --kv-cache-dtype

The KV cache stores keys and values as numbers, and each number takes bytes. --kv-cache-dtype sets the type of those numbers. The default auto uses the model's type, which is BF16 (2 bytes) for most current models. FP8 uses 1 byte per value, so it halves the size of every cached token.

The program computes the KV cache from Example 3 with both types. Compare the bytes per token and the number of tokens that fit.

import java.util.Locale;

public class KvCacheDtype {
    public static void main(String[] args) {
        long mib = 1L << 20;
        long kvBudget = (long) (24_564 * mib * 0.92) - (15_316 + 1_500 + 400) * mib;
        String[] names = {"auto (bfloat16)", "fp8 (fp8_e4m3)"};
        int[] bytesPerValue = {2, 1};
        for (int i = 0; i < names.length; i++) {
            long page = 32L * 16 * 2 * 8 * 128 * bytesPerValue[i];
            long tokens = kvBudget / page * 16;
            System.out.printf(Locale.ROOT, "%-16s %4d KiB/token  %,7d tokens  %.2fx at 32,768%n",
                    names[i], page / 16 / 1024, tokens, tokens / 32768.0);
        }
    }
}

auto (bfloat16)   128 KiB/token   43,056 tokens  1.31x at 32,768
fp8 (fp8_e4m3)     64 KiB/token   86,112 tokens  2.63x at 32,768

Halving the value size doubles the tokens per byte, so the same 5.26 GiB holds twice as many tokens (86,112 instead of 43,056) and the maximum concurrency doubles. The per-token size follows the formula from Example 2: 2 x 32 x 8 x 128 x 1 byte = 65,536 bytes, which is 64 KiB.

The cost is precision. When FP8 is chosen, vLLM logs (from the cache_dtype validator in CacheConfig): Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor. The scaling factor is the number each value is divided by before it is rounded to FP8, so that the values fit FP8's small range. Checkpoints that ship calibrated scales lose less accuracy than a cache that uses a default scale of 1.

The pinned CacheDType lists many names: auto, float16, bfloat16, fp8, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_ds_mla, nvfp4_ds_mla, four turboquant_* variants, int4_per_token_head, int8_per_token_head, fp8_per_token_head, nvfp4, and nvfp4_4over6. Its documentation states that fp8 means fp8_e4m3 and that CUDA 11.8 or later supports fp8 and fp8_e5m2. Support for the other types depends on the GPU and the attention backend. In llama.cpp the same trade-off is -ctk q8_0 -ctv q8_0 (Tuning llama.cpp).

Example 8: prefix caching and cached-token counts in usage

Agents send long prompts that start the same way every time: the same system prompt, then the conversation so far. Computing the KV cache for those repeated tokens again on every request wastes GPU time. Prefix caching avoids this. It reuses the KV blocks of a prompt prefix (the beginning of a prompt) that an earlier request already computed.

vLLM enables it by default (enable_prefix_caching = True in CacheConfig). To find matches, it gives each full 16-token block a hash (a short fingerprint of data). The hash covers the block's tokens and the hash of the block before it, so two equal hashes mean the whole prefix up to that block is equal.

The program simulates that hash chain for two requests that share a system prompt, and computes how many tokens of the second request come from the cache. Look at how many tokens match, and why that number is a multiple of 16.

import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.Set;

public class PrefixCacheBlocks {
    static final int BLOCK = 16;

    static List<Integer> blockHashes(int[] tokens) {
        List<Integer> hashes = new ArrayList<>();
        int parent = 0;
        for (int start = 0; start + BLOCK <= tokens.length; start += BLOCK) {   // full blocks only
            int[] block = java.util.Arrays.copyOfRange(tokens, start, start + BLOCK);
            parent = Objects.hash(parent, java.util.Arrays.hashCode(block));
            hashes.add(parent);
        }
        return hashes;
    }

    static int[] prompt(int sharedLen, int userLen, int userSeed) {
        int[] t = new int[sharedLen + userLen];
        for (int i = 0; i < sharedLen; i++) t[i] = 1000 + i;              // same system prompt
        for (int i = 0; i < userLen; i++) t[sharedLen + i] = userSeed + i; // different question
        return t;
    }

    public static void main(String[] args) {
        Set<Integer> cache = new HashSet<>();
        int[] first = prompt(70, 25, 50_000), second = prompt(70, 40, 60_000);
        for (int[] p : new int[][] {first, second}) {
            List<Integer> hashes = blockHashes(p);
            int hits = 0;
            while (hits < hashes.size() && cache.contains(hashes.get(hits))) hits++;
            cache.addAll(hashes);
            System.out.printf("prompt_tokens=%d full_blocks=%d cached_tokens=%d computed=%d%n",
                    p.length, hashes.size(), hits * BLOCK, p.length - hits * BLOCK);
        }
    }
}

prompt_tokens=95 full_blocks=5 cached_tokens=0 computed=95
prompt_tokens=110 full_blocks=6 cached_tokens=64 computed=46

The first request finds an empty cache, so all 95 of its tokens are computed and its full blocks enter the cache. The second request shares 70 tokens with the first. Only whole blocks can match, and 70 tokens fill 4 whole blocks (64 tokens) plus 6 tokens of a fifth. The match therefore covers 4 blocks, 64 tokens. Tokens 65 to 70 sit in a block that also contains the new question, so that block's hash differs. The second request computes only the remaining 46 of its 110 tokens.

The trailing partial block of each prompt (here 15 and 14 tokens) is never hashed, because a block enters the cache only when it is full.

The real implementation (hash_block_tokens and request_block_hasher in vllm/v1/core/kv_cache_utils.py) uses SHA-256 by default (--prefix-caching-hash-algo sha256; xxhash is faster and not cryptographic). The hash also covers extra keys such as LoRA adapters (small add-on weights that change a model's behavior), multimodal inputs, and the request's cache_salt. A cache salt is a per-request string that makes equal prompts hash differently. It keeps one user from detecting another user's cached prompts by noticing faster responses.

The client sees cached-token counts only when the server starts with --enable-prompt-tokens-details (default off, BaseFrontendArgs). With that flag, usage gains a details object. The server is started like this:

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --served-model-name coder --enable-prompt-tokens-details

A response for the second request would then contain usage like this (field names from UsageInfo and PromptTokenUsageInfo). The 64 matches the cached tokens computed above:

"usage": {"prompt_tokens": 110, "total_tokens": 142, "completion_tokens": 32,
          "prompt_tokens_details": {"cached_tokens": 64}}

The server-wide view is in the metrics of Example 13: vllm:prefix_cache_queries and vllm:prefix_cache_hits count tokens, and the periodic log line reports Prefix cache hit rate: %.1f%%. An agent resends a long system prompt and a growing history on every turn, so most of each request is a cache hit. That is why time to first token stays low as the conversation grows. A later lesson on paged KV cache and prefix caching covers eviction and the data structures.

Example 9: structured outputs request fields

A model normally writes free text, but a program often needs a strict shape, such as valid JSON with known fields. Structured outputs guarantee the shape by restricting which tokens the model may pick at each step, so the text must match a JSON schema (a description of allowed fields and types), a regular expression, a list of choices, or a grammar. vLLM accepts the OpenAI field response_format and its own field structured_outputs. The members of structured_outputs come from StructuredOutputsParams in vllm/sampling_params.py: json, regex, choice, grammar, json_object, and structural_tag. A request uses exactly one.

The program builds three request bodies with Jackson, one for a JSON schema, one for a choice, and one for a regular expression. Compare where each constraint sits in the JSON.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class StructuredRequests {
    static final ObjectMapper JSON = new ObjectMapper();

    static ObjectNode base(String question) {
        ObjectNode r = JSON.createObjectNode().put("model", "coder").put("temperature", 0);
        r.putArray("messages").addObject().put("role", "user").put("content", question);
        return r;
    }

    public static void main(String[] args) throws Exception {
        ObjectNode schemaReq = base("Extract the file and line from: NPE at Main.java:42");
        ObjectNode format = schemaReq.putObject("response_format").put("type", "json_schema");
        ObjectNode js = format.putObject("json_schema").put("name", "location").put("strict", true);
        ObjectNode schema = js.putObject("schema").put("type", "object");
        schema.putObject("properties").putObject("file").put("type", "string");
        ((ObjectNode) schema.get("properties")).putObject("line").put("type", "integer");
        schema.putArray("required").add("file").add("line");
        System.out.println(JSON.writeValueAsString(schemaReq));

        ObjectNode choiceReq = base("Is this diff safe to apply? Answer yes or no.");
        choiceReq.putObject("structured_outputs").putArray("choice").add("yes").add("no");
        System.out.println(JSON.writeValueAsString(choiceReq));

        ObjectNode regexReq = base("Give the Java version as major.minor.");
        regexReq.putObject("structured_outputs").put("regex", "[0-9]+\\.[0-9]+");
        System.out.println(JSON.writeValueAsString(regexReq));
    }
}

{"model":"coder","temperature":0,"messages":[{"role":"user","content":"Extract the file and line from: NPE at Main.java:42"}],"response_format":{"type":"json_schema","json_schema":{"name":"location","strict":true,"schema":{"type":"object","properties":{"file":{"type":"string"},"line":{"type":"integer"}},"required":["file","line"]}}}}
{"model":"coder","temperature":0,"messages":[{"role":"user","content":"Is this diff safe to apply? Answer yes or no."}],"structured_outputs":{"choice":["yes","no"]}}
{"model":"coder","temperature":0,"messages":[{"role":"user","content":"Give the Java version as major.minor."}],"structured_outputs":{"regex":"[0-9]+\\.[0-9]+"}}

The first body uses the OpenAI form. response_format.type is json_schema, and json_schema holds a name, the schema, and strict (the fields of JsonSchemaResponseFormat). vLLM converts it into a json constraint. The second body uses structured_outputs.choice, so the model can produce only yes or no. The third uses a regular expression. Its backslash is doubled in the JSON text because JSON requires it: the Java string "[0-9]+\\.[0-9]+" holds one backslash, and the JSON encoder writes it as two.

Mistakes in the constraint fail early. Supplying two kinds in one structured_outputs object fails with You can only use one kind of structured outputs constraint but multiple are specified. An empty object fails with You must use one kind of structured outputs constraint but none are specified. Both come from StructuredOutputsParams.__post_init__.

The server flag --structured-outputs-config selects the backend (the library that enforces the constraint). StructuredOutputsConfig.backend accepts auto, xgrammar, guidance, outlines, and lm-format-enforcer, with default auto. The same idea in llama-server is the json_schema and grammar fields from llama-server: Endpoints, Slots, and Prompt Cache. A later lesson on constrained decoding explains how a grammar masks tokens at each step.

Example 10: tool calling, --enable-auto-tool-choice, and --tool-call-parser

A model cannot run code or read files by itself. When an agent needs that, the model writes a tool call: a request such as "call read_file with this path", and the agent program runs it. The model writes this request as plain text in its own format, and each model family has a different format. A tool call parser finds that text in the output and converts it into the standard OpenAI tool_calls array, which clients know how to read.

vLLM needs two flags for tool_choice: "auto" (the model decides whether to call a tool). --enable-auto-tool-choice turns the feature on, and --tool-call-parser names the parser that matches the model's format. The program maps model families to parser names from the registry in vllm/tool_parsers/__init__.py and builds the flags. It also shows the startup check for a missing parser.

import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

public class ToolParserChoice {
    static final Map<String, String> PARSER_BY_FAMILY = new LinkedHashMap<>();
    static {
        PARSER_BY_FAMILY.put("Qwen2.5 and Hermes-style <tool_call> models", "hermes");
        PARSER_BY_FAMILY.put("Qwen3-Coder", "qwen3_coder");
        PARSER_BY_FAMILY.put("Llama 3.1, 3.2, 3.3 JSON tool calls", "llama3_json");
        PARSER_BY_FAMILY.put("Llama 4 pythonic tool calls", "llama4_pythonic");
        PARSER_BY_FAMILY.put("Mistral", "mistral");
        PARSER_BY_FAMILY.put("DeepSeek V3.1", "deepseek_v31");
        PARSER_BY_FAMILY.put("gpt-oss (harmony format)", "openai");
        PARSER_BY_FAMILY.put("Granite", "granite");
    }

    static List<String> toolFlags(String parser) {
        return List.of("--enable-auto-tool-choice", "--tool-call-parser", parser);
    }

    static String validate(boolean auto, String parser) {   // validate_parsed_serve_args
        if (auto && parser == null) return "TypeError: Error: --enable-auto-tool-choice requires --tool-call-parser";
        return "ok";
    }

    public static void main(String[] args) {
        PARSER_BY_FAMILY.forEach((family, parser) -> System.out.printf("%-44s %s%n", family, parser));
        System.out.println(String.join(" ", toolFlags("hermes")));
        System.out.println(validate(true, null));
        System.out.println(validate(true, "hermes"));
    }
}

Qwen2.5 and Hermes-style <tool_call> models  hermes
Qwen3-Coder                                  qwen3_coder
Llama 3.1, 3.2, 3.3 JSON tool calls          llama3_json
Llama 4 pythonic tool calls                  llama4_pythonic
Mistral                                      mistral
DeepSeek V3.1                                deepseek_v31
gpt-oss (harmony format)                     openai
Granite                                      granite
--enable-auto-tool-choice --tool-call-parser hermes
TypeError: Error: --enable-auto-tool-choice requires --tool-call-parser
ok

The table pairs a model family with a registered parser name. The pinned registry has 51 names, among them hermes, qwen3_coder, qwen3_xml, llama3_json, llama4_json, llama4_pythonic, pythonic, mistral, deepseek_v3, deepseek_v31, deepseek_v32, kimi_k2, glm45, glm47, granite, granite4, openai, phi4_mini_json, minimax_m2, xlam, and gemma4. The parser must match the format in the model's chat template (the text pattern that wraps the conversation for the model). A model's card on the Hub usually names the parser to use. --tool-parser-plugin registers a parser written outside vLLM.

The last two lines are the startup check. --enable-auto-tool-choice without a parser stops vllm serve with that TypeError.

A server started with neither flag still accepts requests with tools and tool_choice: "none". A request with tool_choice: "auto" gets HTTP 400 with "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set. A request with tool_choice: "required" or a named function gets tool_choice=... requires --tool-call-parser to be set (from vllm/renderers/online_renderer.py). For required and named functions, vLLM uses structured outputs (Example 9) to force a valid call. --exclude-tools-when-tool-choice-none removes the tool definitions from the prompt when tool_choice is none. A command with both flags looks like this:

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --served-model-name coder \
  --enable-auto-tool-choice --tool-call-parser hermes

Example 11: what the hermes parser does with model text

Example 10 named the parser. This example shows what it does. The hermes parser (Hermes2ProToolParser) looks for <tool_call> and </tool_call> tags around a JSON object with name and arguments. The program applies the same rule to a fixed model output that contains one sentence and two tool calls, and prints the OpenAI-shaped result. Look at where the sentence goes, and at the type of arguments.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class HermesParse {
    static final Pattern CALL = Pattern.compile("<tool_call>(.*?)</tool_call>", Pattern.DOTALL);

    public static void main(String[] args) throws Exception {
        ObjectMapper json = new ObjectMapper();
        String output = """
                I will read the file first.
                <tool_call>
                {"name": "read_file", "arguments": {"path": "src/Main.java"}}
                </tool_call>
                <tool_call>
                {"name": "list_dir", "arguments": {"path": "src"}}
                </tool_call>""";
        int start = output.indexOf("<tool_call>");
        ObjectNode message = json.createObjectNode().put("role", "assistant");
        message.put("content", start < 0 ? output : output.substring(0, start).strip());
        ArrayNode calls = message.putArray("tool_calls");
        Matcher m = CALL.matcher(output);
        int n = 0;
        while (m.find()) {
            JsonNode call = json.readTree(m.group(1).strip());
            ObjectNode c = calls.addObject().put("id", "call_" + n++).put("type", "function");
            c.putObject("function").put("name", call.get("name").asText())
                    .put("arguments", json.writeValueAsString(call.get("arguments")));
        }
        System.out.println(json.writerWithDefaultPrettyPrinter().writeValueAsString(message));
    }
}

{
  "role" : "assistant",
  "content" : "I will read the file first.",
  "tool_calls" : [ {
    "id" : "call_0",
    "type" : "function",
    "function" : {
      "name" : "read_file",
      "arguments" : "{\"path\":\"src/Main.java\"}"
    }
  }, {
    "id" : "call_1",
    "type" : "function",
    "function" : {
      "name" : "list_dir",
      "arguments" : "{\"path\":\"src\"}"
    }
  } ]
}

The text before the first <tool_call> stays in content (vLLM keeps its trailing newline; the program trims it). Each tagged block becomes one entry in tool_calls. So the model's two tags produce two entries, call_0 and call_1.

arguments is a JSON object inside the model's text but a string in the OpenAI response, so the parser serializes it again. That is why the output shows "{\"path\":\"src/Main.java\"}" with escaped quotes. Clients parse that string back into an object. vLLM generates random ids with a chatcmpl-tool- prefix; the program uses call_0 and call_1 to keep its output fixed.

When the model writes broken JSON inside the tags, the parser logs Error in extracting tool call from response. and returns the whole output as plain content with no tool calls. A client that receives a <tool_call> tag in content is therefore looking at a malformed call, or at a server started without the parser.

Example 12: --reasoning-parser

Reasoning models write their thinking before the answer, between markers such as <think> and </think>. If that thinking arrives mixed into the answer, a client shows the user pages of internal notes. --reasoning-parser tells vLLM to move the thinking out of content into a separate reasoning field.

The pinned registry (vllm/reasoning/__init__.py) has 34 names, among them deepseek_r1, qwen3, glm45, mistral, openai_gptoss, granite, kimi_k2, minimax_m2, olmo3, and seed_oss. The program performs the split that the deepseek_r1 parser performs on a finished output, for three cases: both markers, no markers, and only the end marker.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

public class ReasoningSplit {
    record Split(String reasoning, String content) {}

    static Split split(String text) {
        String start = "<think>", end = "</think>";
        int e = text.indexOf(end);
        if (e < 0) return new Split(null, text);                  // no end marker: no reasoning found
        int s = text.indexOf(start);
        String reasoning = text.substring(s < 0 ? 0 : s + start.length(), e).strip();
        return new Split(reasoning, text.substring(e + end.length()).strip());
    }

    public static void main(String[] args) throws Exception {
        ObjectMapper json = new ObjectMapper();
        String[] outputs = {
            "<think>The loop runs n+1 times because of <=.</think>Change `<=` to `<` on line 7.",
            "The loop condition is fine.",
            "The template already opened the block.</think>Use a StringBuilder."
        };
        for (String out : outputs) {
            Split sp = split(out);
            ObjectNode msg = json.createObjectNode().put("role", "assistant").put("content", sp.content());
            msg.put("reasoning", sp.reasoning());
            System.out.println(json.writeValueAsString(msg));
        }
    }
}

{"role":"assistant","content":"Change `<=` to `<` on line 7.","reasoning":"The loop runs n+1 times because of <=."}
{"role":"assistant","content":"The loop condition is fine.","reasoning":null}
{"role":"assistant","content":"Use a StringBuilder.","reasoning":"The template already opened the block."}

The first output has both markers: the thinking goes to reasoning and the answer stays in content. The second has no markers, so reasoning is null. The third has only an end marker. That case is common, because the chat templates for several reasoning models write <think> into the prompt themselves. The generated text then starts inside the thinking block, and the model emits only </think>. The parser treats everything before the end marker as reasoning.

In the pinned source the response field is reasoning (ChatMessage.reasoning and DeltaMessage.reasoning). A request message that still carries the older name reasoning_content is renamed on input. llama-server returns reasoning_content (llama-server: Endpoints, Slots, and Prompt Cache), so a client for both servers reads both names.

Other details: with a reasoning parser set, the usage object reports completion_tokens_details.reasoning_tokens. The request field include_reasoning (default true) drops the reasoning from the response when it is false. --reasoning-config and the request field thinking_token_budget limit how many tokens the model may spend thinking. A command that sets both parsers looks like this:

vllm serve Qwen/Qwen3-8B --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser hermes

Example 13: Prometheus metrics names

A server in use needs to be watched: how many requests wait, how full the cache is, and how long users wait for the first token. vLLM publishes these numbers at /metrics in the Prometheus text format. Prometheus is a common monitoring system that reads such a page on a schedule and stores the values. Every engine metric has the prefix vllm: and the labels model_name and engine (from PrometheusStatLogger in vllm/v1/metrics/loggers.py).

The program parses a small sample in that format and computes the numbers an operator watches: queue length, KV cache usage, prefix cache hit rate, and mean time to first token (TTFT). Look at how each printed number is derived from the raw lines.

import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;

public class VllmMetrics {
    public static void main(String[] args) {
        String text = """
                # HELP vllm:num_requests_running Number of requests in model execution batches.
                # TYPE vllm:num_requests_running gauge
                vllm:num_requests_running{engine="0",model_name="coder"} 6.0
                vllm:num_requests_waiting{engine="0",model_name="coder"} 2.0
                vllm:kv_cache_usage_perc{engine="0",model_name="coder"} 0.83
                vllm:prefix_cache_queries_total{engine="0",model_name="coder"} 184320.0
                vllm:prefix_cache_hits_total{engine="0",model_name="coder"} 151552.0
                vllm:time_to_first_token_seconds_sum{engine="0",model_name="coder"} 97.5
                vllm:time_to_first_token_seconds_count{engine="0",model_name="coder"} 390.0
                vllm:request_success_total{engine="0",finished_reason="stop",model_name="coder"} 371.0
                vllm:request_success_total{engine="0",finished_reason="length",model_name="coder"} 19.0
                """;
        Map<String, Double> m = new TreeMap<>();
        for (String line : text.lines().toList()) {
            if (line.isBlank() || line.startsWith("#")) continue;
            String series = line.substring(0, line.lastIndexOf(' '));
            String name = series.contains("{") ? series.substring(0, series.indexOf('{')) : series;
            String reason = series.contains("finished_reason=\"") ? "." + series.split("finished_reason=\"")[1]
                    .split("\"")[0] : "";
            m.merge(name + reason, Double.parseDouble(line.substring(line.lastIndexOf(' ') + 1)), Double::sum);
        }
        System.out.printf(Locale.ROOT, "running %.0f, waiting %.0f%n", m.get("vllm:num_requests_running"),
                m.get("vllm:num_requests_waiting"));
        System.out.printf(Locale.ROOT, "KV cache usage %.0f%%%n", 100 * m.get("vllm:kv_cache_usage_perc"));
        System.out.printf(Locale.ROOT, "prefix cache hit rate %.1f%%%n",
                100 * m.get("vllm:prefix_cache_hits_total") / m.get("vllm:prefix_cache_queries_total"));
        System.out.printf(Locale.ROOT, "mean TTFT %.3f s over %.0f requests%n",
                m.get("vllm:time_to_first_token_seconds_sum") / m.get("vllm:time_to_first_token_seconds_count"),
                m.get("vllm:time_to_first_token_seconds_count"));
        System.out.printf(Locale.ROOT, "finished: stop %.0f, length %.0f%n",
                m.get("vllm:request_success_total.stop"), m.get("vllm:request_success_total.length"));
    }
}

running 6, waiting 2
KV cache usage 83%
prefix cache hit rate 82.2%
mean TTFT 0.250 s over 390 requests
finished: stop 371, length 19

Six requests run and two wait, and the cache is 83% full. When waiting stays above zero while kv_cache_usage_perc is near 1, the cache is the limit, and Example 3 and Example 7 are the levers to pull.

The prefix cache served 82.2% of the queried prompt tokens (151,552 hits divided by 184,320 queries). The mean time to first token is the histogram's _sum divided by its _count: 97.5 seconds over 390 requests is 0.250 s. A histogram counts how many values fell into each range (bucket), plus the sum and count of all values. Percentiles come from the _bucket series, which the sample leaves out.

The last line splits finished requests by reason. 19 of 390 stopped at max_tokens or the context limit (length), and the rest ended naturally (stop). FinishReason in vllm/v1/engine/__init__.py also has abort, error, and repetition.

Metric types matter when reading the table. A gauge is a value that goes up and down, such as the queue length. A counter only increases, such as total tokens generated. In the text format, counters gain the suffix _total, which is why the code reads vllm:prefix_cache_hits_total for the counter registered as vllm:prefix_cache_hits. The main names in the pinned source are:

The engine also logs a summary every 10 seconds while it is busy (VLLM_LOG_STATS_INTERVAL, default 10.0). It is built from the format list in LoggingStatLogger.log, and it carries the same numbers in one line:

INFO [loggers.py] Engine 000: Avg prompt throughput: 3120.4 tokens/s, Avg generation throughput: 412.7 tokens/s, Running: 6 reqs, Waiting: 2 reqs, GPU KV cache usage: 83.0%, Prefix cache hit rate: 82.2%

--disable-log-stats turns this line off. llama-server's /metrics (enabled with --metrics) uses the prefix llamacpp: and a smaller set of names.

Example 14: generation config defaults from the model

When a model is created, its authors often find which sampling settings work best, and they save them in a file named generation_config.json. Sampling is the step where the model picks the next token from a list of likely tokens. The common settings are:

• temperature: higher values make the choice more random.
• top_k: keep only the k most likely tokens.
• top_p: keep the smallest group of tokens whose probabilities add up to p.
• min_p: drop tokens whose probability is below this fraction of the best token's.
• repetition_penalty: lower the probability of tokens that were already used.

With the default --generation-config auto, vLLM uses the values in that file as defaults for every request. The program reproduces ModelConfig.get_diff_sampling_param. It keeps only the sampling keys vLLM uses, applies --override-generation-config, renames max_new_tokens, and then fills in a request that sets only temperature. Look at which keys survive and which value wins in the last line.

import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

public class GenerationDefaults {
    static final List<String> KEYS = List.of("repetition_penalty", "temperature", "top_k", "top_p", "min_p",
            "max_new_tokens");

    static Map<String, Object> defaults(String mode, Map<String, Object> fileConfig, Map<String, Object> override) {
        Map<String, Object> config = new LinkedHashMap<>(mode.equals("vllm") ? Map.of() : fileConfig);
        config.putAll(override);
        Map<String, Object> diff = new LinkedHashMap<>();
        for (String k : KEYS) if (config.get(k) != null) diff.put(k.equals("max_new_tokens") ? "max_tokens" : k, config.get(k));
        return diff;
    }

    public static void main(String[] args) {
        Map<String, Object> file = new LinkedHashMap<>();       // generation_config.json of the model
        file.put("bos_token_id", 151643);
        file.put("do_sample", true);
        file.put("repetition_penalty", 1.05);
        file.put("temperature", 0.7);
        file.put("top_k", 20);
        file.put("top_p", 0.8);
        Map<String, Object> auto = defaults("auto", file, Map.of());
        System.out.println("--generation-config auto:  " + auto);
        System.out.println("--generation-config vllm:  " + defaults("vllm", file, Map.of()));
        System.out.println("auto + override top_p=0.95: " + defaults("auto", file, Map.of("top_p", 0.95)));

        Map<String, Object> request = new LinkedHashMap<>(Map.of("temperature", 0.2));
        Map<String, Object> effective = new LinkedHashMap<>(Map.of("temperature", 1.0, "top_p", 1.0, "top_k", 0,
                "min_p", 0.0, "repetition_penalty", 1.0));   // vLLM's neutral values
        effective.putAll(auto);
        effective.putAll(request);
        System.out.println("request {temperature=0.2} runs with: " + new java.util.TreeMap<>(effective));
    }
}

--generation-config auto:  {repetition_penalty=1.05, temperature=0.7, top_k=20, top_p=0.8}
--generation-config vllm:  {}
auto + override top_p=0.95: {repetition_penalty=1.05, temperature=0.7, top_k=20, top_p=0.95}
request {temperature=0.2} runs with: {min_p=0.0, repetition_penalty=1.05, temperature=0.2, top_k=20, top_p=0.8}

With auto, four of the file's keys become defaults. Keys that are not sampling parameters (bos_token_id, do_sample) are dropped. vLLM logs this at startup with the warning Default vLLM sampling parameters have been overridden by the model's generation_config.json: ... If this is not intended, please relaunch vLLM instance with --generation-config vllm.

With vllm, the file is ignored and the defaults map is empty, so requests fall back to vLLM's neutral values: temperature 1.0, top-p 1.0, top-k 0 (meaning off), min-p 0.0, and repetition penalty 1.0 (_DEFAULT_SAMPLING_PARAMS in the chat protocol). --override-generation-config '{"top_p": 0.95}' changes one value and keeps the rest, as the third line shows.

The last line shows the merge for one request. The request's own temperature 0.2 wins, and top_k, top_p, and repetition_penalty come from the model's file. min_p stays at the neutral 0.0 because neither the file nor the request sets it. The order is: neutral values first, then the file, then the request.

Two clients that send the same request to llama-server and vLLM can therefore sample differently. llama-server's defaults come from its own flags (--temp, --top-k, and the rest) and not from generation_config.json. A client that needs the same behavior on both sets every sampling field in the request.

max_new_tokens in the file, when present, becomes a server-wide max_tokens default. Without it, a request with no max_tokens may generate up to max_model_len minus the prompt length (get_max_tokens in api_utils.py). An agent should always set max_tokens, because that bounds the cost of one turn.

Example 15: a vLLM response parsed into records

Before calling a real server, a client must be able to read vLLM's response. A response from vLLM looks like the OpenAI one, with extra fields. The program parses a non-streaming response that contains the vLLM-specific fields from the previous examples (reasoning, stop_reason, prompt_tokens_details, completion_tokens_details) into Java records. The records ignore unknown fields, as in The Chat Completions Protocol and Streaming in Java. Look at which fields the records declare and which fields of the JSON they skip.

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;

public class VllmResponseParse {
    @JsonIgnoreProperties(ignoreUnknown = true)
    record Message(String role, String content, String reasoning,
                   @JsonProperty("reasoning_content") String reasoningContent) {
        String thinking() { return reasoning != null ? reasoning : reasoningContent; }
    }
    @JsonIgnoreProperties(ignoreUnknown = true)
    record Choice(int index, Message message, @JsonProperty("finish_reason") String finishReason) {}
    @JsonIgnoreProperties(ignoreUnknown = true)
    record Details(@JsonProperty("cached_tokens") Integer cachedTokens,
                   @JsonProperty("reasoning_tokens") Integer reasoningTokens) {}
    @JsonIgnoreProperties(ignoreUnknown = true)
    record Usage(@JsonProperty("prompt_tokens") int prompt, @JsonProperty("completion_tokens") int completion,
                 @JsonProperty("prompt_tokens_details") Details promptDetails,
                 @JsonProperty("completion_tokens_details") Details completionDetails) {}
    @JsonIgnoreProperties(ignoreUnknown = true)
    record Response(String id, String model, List<Choice> choices, Usage usage) {}

    public static void main(String[] args) throws Exception {
        String body = """
            {"id":"chatcmpl-0b1c","object":"chat.completion","created":1791000000,"model":"coder",
             "choices":[{"index":0,"message":{"role":"assistant","content":"Use `<` instead of `<=`.",
               "refusal":null,"reasoning":"The loop runs one extra time."},
               "logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],
             "service_tier":null,"system_fingerprint":"vllm-0.30.0-1a2b3c4d","prompt_logprobs":null,
             "usage":{"prompt_tokens":110,"total_tokens":142,"completion_tokens":32,
               "prompt_tokens_details":{"cached_tokens":64},
               "completion_tokens_details":{"reasoning_tokens":12}}}""";
        Response r = new ObjectMapper().readValue(body, Response.class);
        Choice c = r.choices().get(0);
        System.out.println("model=" + r.model() + " finish=" + c.finishReason());
        System.out.println("thinking: " + c.message().thinking());
        System.out.println("answer:   " + c.message().content());
        System.out.println("prompt=" + r.usage().prompt() + " cached=" + r.usage().promptDetails().cachedTokens()
                + " completion=" + r.usage().completion() + " reasoning=" + r.usage().completionDetails().reasoningTokens());
    }
}

model=coder finish=stop
thinking: The loop runs one extra time.
answer:   Use `<` instead of `<=`.
prompt=110 cached=64 completion=32 reasoning=12

The records name only the fields the client uses, and ignoreUnknown skips the rest (refusal, stop_reason, token_ids, service_tier, prompt_logprobs, system_fingerprint). The Message record accepts both reasoning (vLLM) and reasoning_content (llama-server), so one client class reads both servers.

The system_fingerprint follows the pinned --fingerprint-mode full format, vllm-<version>[-<parallelism>]-<hash8>. The usage line shows 64 of 110 prompt tokens served from the prefix cache, which is the result of Example 8, and 12 of 32 completion tokens spent on reasoning, which is the effect of Example 12.

Example 16: calling vLLM from Java and the differences from llama-server

The last step is a real call. The program sends one streaming chat request with the HttpClient and the SSE (server-sent events) parsing taught in The Chat Completions Protocol and Streaming in Java, reduced here to one class. Streaming means the server sends the answer in small pieces as it generates them. The program is marked java,ignore because it needs a running vLLM server on a GPU machine.

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class VllmStreamCall {
    public static void main(String[] args) throws Exception {
        ObjectMapper json = new ObjectMapper();
        String body = """
            {"model": "coder", "stream": true, "stream_options": {"include_usage": true},
             "max_tokens": 64, "temperature": 0.2,
             "messages": [{"role": "user", "content": "Write a Java record for a 2D point."}]}""";
        HttpRequest request = HttpRequest.newBuilder(URI.create("http://127.0.0.1:8000/v1/chat/completions"))
                .header("Content-Type", "application/json")
                .header("Authorization", "Bearer " + System.getenv("VLLM_API_KEY"))
                .timeout(Duration.ofMinutes(5))
                .POST(HttpRequest.BodyPublishers.ofString(body)).build();
        HttpResponse<java.util.stream.Stream<String>> response = HttpClient.newHttpClient()
                .send(request, HttpResponse.BodyHandlers.ofLines());
        System.out.println("status " + response.statusCode());
        response.body().filter(l -> l.startsWith("data: ") && !l.equals("data: [DONE]")).forEach(l -> {
            try {
                JsonNode chunk = json.readTree(l.substring(6));
                for (JsonNode c : chunk.path("choices")) System.out.print(c.path("delta").path("content").asText(""));
                if (chunk.hasNonNull("usage")) System.out.println("\nusage " + chunk.get("usage"));
            } catch (Exception e) { throw new RuntimeException(e); }
        });
    }
}

Run against vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --served-model-name coder --api-key ..., the program prints status 200, the record as it streams, and a final usage line such as usage {"prompt_tokens":36,"total_tokens":58,"completion_tokens":22}. A wrong key gives status 401 before any stream starts.

The same program works against llama-server if you change the port, which is the point of the OpenAI-compatible API. The differences a client meets are listed in the table below. Each one comes from the source of the pinned versions.

The health row matters for a supervisor such as loop serve, a program that starts an engine and waits until it is ready. vLLM binds its socket at startup (setup_server in launcher.py) but starts accepting connections only after the engine is built, the model is loaded, and the KV cache is allocated. A client polling /health therefore sees connection refused for the whole loading time, and then 200. llama-server opens its port first and answers 503 until the model is ready.

A supervisor that treats connection refused and 503 the same way (keep polling until a deadline) works with both. Workshop: loop serve, run, and chat builds it that way.

Interview points

• vllm serve <model> listens on all network interfaces at port 8000 by default. --api-key protects only the /v1, /v2, and /inference paths, so a local deployment should also set --host 127.0.0.1.
• --max-model-len defaults to the model's max_position_embeddings. Startup fails when one full-length request does not fit in the KV cache, and the error message gives the largest length that does fit.
• --gpu-memory-utilization (default 0.92 at 0.30.0) limits weights, activations, and KV cache together, so every byte taken away from it comes out of the KV cache. --kv-cache-memory-bytes sets the cache size directly and makes vLLM ignore the fraction for the cache.
• Maximum concurrency in the startup log is the KV cache size in tokens divided by max_model_len. It is a lower bound on how many requests can run at the same time, because most requests are shorter than the limit.
• --max-num-batched-tokens is the token budget of one scheduler step, and it must be at least --max-num-seqs. The defaults depend on GPU memory, and for the API server on a 24 GB card they are 2,048 tokens and 256 sequences.
• Tensor parallelism splits every layer across GPUs and needs the number of attention heads to be divisible by the tensor-parallel size. When the size is larger than the number of KV heads, the KV heads are copied to several GPUs.
• vLLM detects quantization from quantization_config.quant_method in config.json. --quantization is used for online quantization, or it must agree with the checkpoint. vLLM 0.30.0 has no GGUF loader.
• An FP8 KV cache doubles the number of tokens the cache can hold, at some cost in accuracy.
• Prefix caching is on by default. It matches whole 16-token blocks through a hash chain, and cached-token counts appear in usage only with --enable-prompt-tokens-details.
• Tool calls need --enable-auto-tool-choice plus a --tool-call-parser that matches the model's format. Reasoning text needs --reasoning-parser and arrives in the reasoning field.
• --generation-config auto makes the model's generation_config.json the default set of sampling settings. A client that needs identical behavior across servers sets every sampling field in each request.

Summary

• Flag defaults at vLLM 0.30.0: host None (all interfaces), port 8000, --gpu-memory-utilization 0.92, block size 16, --kv-cache-dtype auto, prefix caching on, chunked prefill on, tensor-parallel and pipeline-parallel size 1, and --generation-config auto.
• --served-model-name accepts several names. Responses and metrics use the first one.
• When the KV cache is too small, the startup error reads To serve at least one request with the model's max seq len (N), (X GiB KV cache is needed, which is larger than the available KV cache memory (Y GiB).
• On a 24,564 MiB GPU with an 8B BF16 model and the assumed overheads, the default budget gives 43,056 tokens of KV cache, which is 1.31 full 32K requests. An FP8 cache doubles that.
• The API server's batch defaults are 2,048 tokens and 256 sequences below 70 GiB or on A100s, 8,192 and 1,024 on H100-class GPUs, and 16,384 and 1,024 at 160 GiB or more.
• --quantization accepts names such as awq, gptq, fp8, compressed-tensors, mxfp4, and online shorthands such as fp8_per_block.
• Structured outputs use response_format or structured_outputs, with exactly one of json, regex, choice, grammar, json_object, or structural_tag.
• The tool parser registry has 51 names (for example hermes, qwen3_coder, llama3_json, mistral, openai). The reasoning parser registry has 34 names (for example deepseek_r1, qwen3, openai_gptoss).
• Metrics use the prefix vllm: and the labels model_name and engine. The key ones are num_requests_running, num_requests_waiting, kv_cache_usage_perc, prefix_cache_hits, and time_to_first_token_seconds.
• A vLLM request without max_tokens may generate up to the remaining context length.
• vLLM accepts connections only after loading finishes. llama-server accepts them first, and its /health returns 503 while the model loads.