vLLM Architecture and Installation

vLLM is an inference server, a program that loads a model and answers requests over HTTP. It is written in Python and C++/CUDA, and it is built for many users at once on GPUs. The earlier lessons used llama.cpp, which is a good fit for one person on one machine. vLLM solves a different problem: when dozens of requests arrive together, how do you keep an expensive GPU busy and not run out of memory?

This lesson reads vLLM's architecture from the source of version 0.30.0 (the vllm package published on PyPI, commit gced6857af according to its _version.py). It then rebuilds the central algorithms as small, deterministic Java programs, so a reader can see each idea run on tiny numbers. Real vLLM needs a GPU, but every Java program here runs on any machine.

After this lesson, a reader can name the processes of a running vLLM server, follow a request through the source files, predict how much KV cache a GPU gets, read the startup log, and decide between vLLM and llama.cpp for a given machine and workload. The KV cache (key-value cache) is the memory where the model keeps the attention data of every token it has already processed, so it does not recompute them. It was introduced in Generation, Prefill, Decode, and the KV Cache, and the memory arithmetic comes from Memory Math. A later lesson on batching and scheduling and a later lesson on the paged KV cache and prefix caching go deeper into the algorithms that this lesson only introduces.

The table lists the main parts of vLLM, what each part does, and where it lives in the source. All paths in this lesson are relative to the installed package.

vLLM needs a supported accelerator (hardware tier 3 or 4 in this course's table). For that reason, the examples that involve a running server show commands and a representative log instead of live output.

Example 1: the process layout

The problem: when you type vllm serve, how many programs actually start, and what does each one do? A process is a separate running program with its own memory. The answer matters when you read logs, monitor the server, or give it several GPUs.

The program applies the naming and executor rules from the source to three configurations. The first uses one GPU. The second uses two GPUs with tensor parallelism (tp), which splits every layer of the model across the GPUs. The third uses four GPUs with two pipeline stages (pp) of two tensor-parallel GPUs each, where pipeline stages hold different layers of the model. Look at the number of processes and how the worker names reflect the layout.

import java.util.ArrayList;
import java.util.List;

public class ProcessLayout {
    /** Processes of one `vllm serve` on one node without data parallelism, named as vLLM 0.30.0 names them. */
    static List<String> processes(int pp, int tp) {
        List<String> out = new ArrayList<>();
        out.add("APIServer   the `vllm serve` process: HTTP, chat template, tokenizer, detokenizer");
        boolean uni = pp * tp == 1;                                  // world size 1 -> "uni" executor backend
        out.add("EngineCore  scheduler and KV cache manager"
                + (uni ? ", model runner on GPU 0 (UniProcExecutor)" : " (MultiprocExecutor)"));
        if (!uni) {
            for (int p = 0; p < pp; p++) {
                for (int t = 0; t < tp; t++) {
                    String name = "Worker" + (pp > 1 ? "_PP" + p : "") + (tp > 1 ? "_TP" + t : "");
                    out.add("  " + name + "  model runner on GPU " + (p * tp + t));
                }
            }
        }
        return out;
    }

    public static void main(String[] args) {
        int[][] configs = {{1, 1}, {1, 2}, {2, 2}};
        for (int[] c : configs) {
            List<String> p = processes(c[0], c[1]);
            System.out.printf("pp=%d tp=%d: %d processes, GPUs used: %d%n", c[0], c[1], p.size(), c[0] * c[1]);
            p.forEach(s -> System.out.println("  " + s));
        }
        System.out.println("log prefix example: (EngineCore pid=4242) INFO 10-03 12:00:00 [core.py:123] ...");
    }
}

pp=1 tp=1: 2 processes, GPUs used: 1
  APIServer   the `vllm serve` process: HTTP, chat template, tokenizer, detokenizer
  EngineCore  scheduler and KV cache manager, model runner on GPU 0 (UniProcExecutor)
pp=1 tp=2: 4 processes, GPUs used: 2
  APIServer   the `vllm serve` process: HTTP, chat template, tokenizer, detokenizer
  EngineCore  scheduler and KV cache manager (MultiprocExecutor)
    Worker_TP0  model runner on GPU 0
    Worker_TP1  model runner on GPU 1
pp=2 tp=2: 6 processes, GPUs used: 4
  APIServer   the `vllm serve` process: HTTP, chat template, tokenizer, detokenizer
  EngineCore  scheduler and KV cache manager (MultiprocExecutor)
    Worker_PP0_TP0  model runner on GPU 0
    Worker_PP0_TP1  model runner on GPU 1
    Worker_PP1_TP0  model runner on GPU 2
    Worker_PP1_TP1  model runner on GPU 3
log prefix example: (EngineCore pid=4242) INFO 10-03 12:00:00 [core.py:123] ...

On one GPU there are two processes. The vllm serve process itself is the API server: vllm/entrypoints/cli/serve.py runs run_server in the same process when one API server is configured (the comment there says "Single API server (this process)"). It hosts the FastAPI application (a Python web framework), applies the chat template, tokenizes prompts, and detokenizes outputs, through the AsyncLLM class in vllm/v1/engine/async_llm.py. Tokenizing turns text into token numbers, and detokenizing turns token numbers back into text. The second process is the engine core. EngineCoreProc.run_engine_core in vllm/v1/engine/core.py names it EngineCore (or EngineCore_DP<n> with data parallelism), and it holds the scheduler and the KV cache manager.

Where the model runs depends on the executor, the component that starts and talks to the workers. ParallelConfig in vllm/config/parallel.py sets the executor backend to "uni" when the world size is 1. The world size is the number of GPUs used by the model. UniProcExecutor runs the worker, and with it the model, inside the engine-core process. For larger world sizes on one node, the default backend is "mp" (multiprocess). MultiprocExecutor in vllm/v1/executor/multiproc_executor.py starts one worker process per GPU. It names them Worker_TP0, Worker_PP1_TP0, and so on, through setup_proc_title_and_log_prefix. Tensor parallelism and pipeline stages are topics of a later lesson on multi-GPU inference.

This split has a purpose. The Python work of the API server (HTTP parsing, templates, detokenization) runs in parallel with the engine core's busy loop, which schedules and launches the GPU steps. Neither waits for the other, except through the sockets between them. The process title gets the prefix VLLM:: (from VLLM_PROCESS_NAME_PREFIX, default VLLM), so ps shows VLLM::EngineCore. Every log line of a process starts with its name and process id, as the last output line shows.

Data parallelism means running several full copies of the model. With data parallelism, each copy gets its own EngineCore_DP<n>, the default number of API server processes equals the data-parallel size, and a DPCoordinator process collects load statistics.

Example 2: the life of a request

The problem: when a chat request arrives, which code handles it, and in what order? Knowing the path makes the source readable and tells you where to look when something is slow or wrong. The program prints the 13 hops of one request through the source files. It then prints the states the request object goes through when the scheduler preempts it once. Preempting a request means pausing it and taking its memory away so other requests can use it.

import java.util.List;
import java.util.Map;

public class RequestPath {
    record Hop(String process, String file, String what) {}

    enum Status { WAITING, RUNNING, PREEMPTED, FINISHED_STOPPED, FINISHED_LENGTH_CAPPED, FINISHED_ABORTED }

    // Finished statuses map to OpenAI finish reasons (vllm/v1/request.py, _FINISHED_REASON_MAP)
    static final Map<Status, String> FINISH_REASON = Map.of(
            Status.FINISHED_STOPPED, "stop", Status.FINISHED_LENGTH_CAPPED, "length", Status.FINISHED_ABORTED, "abort");

    public static void main(String[] args) {
        List<Hop> hops = List.of(
            new Hop("APIServer", "entrypoints/openai/chat_completion/api_router.py", "POST /v1/chat/completions"),
            new Hop("APIServer", "entrypoints/openai/chat_completion/serving.py", "render chat template, call engine_client.generate"),
            new Hop("APIServer", "v1/engine/async_llm.py", "AsyncLLM.add_request"),
            new Hop("APIServer", "v1/engine/input_processor.py", "build EngineCoreRequest (token ids, SamplingParams)"),
            new Hop("APIServer", "v1/engine/core_client.py", "AsyncMPClient: msgpack over ZMQ to the engine"),
            new Hop("EngineCore", "v1/engine/core.py", "input thread -> input_queue -> run_busy_loop"),
            new Hop("EngineCore", "v1/core/sched/scheduler.py", "add_request -> waiting queue; schedule() each step"),
            new Hop("EngineCore", "v1/core/kv_cache_manager.py", "allocate_slots: blocks from block_pool.py"),
            new Hop("Worker", "v1/worker/gpu_model_runner.py", "execute_model: inputs, block tables, forward"),
            new Hop("Worker", "v1/sample/sampler.py", "sample next token ids"),
            new Hop("EngineCore", "v1/core/sched/scheduler.py", "update_from_output -> EngineCoreOutputs"),
            new Hop("APIServer", "v1/engine/output_processor.py", "detokenize (detokenizer.py), RequestOutput"),
            new Hop("APIServer", "entrypoints/openai/chat_completion/serving.py", "SSE chunk or JSON response"));
        for (int i = 0; i < hops.size(); i++) {
            Hop h = hops.get(i);
            System.out.printf("%2d %-10s vllm/%-48s %s%n", i + 1, h.process(), h.file(), h.what());
        }

        Status[] life = {Status.WAITING, Status.RUNNING, Status.PREEMPTED, Status.RUNNING, Status.FINISHED_LENGTH_CAPPED};
        StringBuilder path = new StringBuilder();
        for (Status s : life) path.append(path.isEmpty() ? "" : " -> ").append(s);
        System.out.println("status: " + path);
        Status last = life[life.length - 1];
        System.out.println("finished? " + (last.ordinal() > Status.PREEMPTED.ordinal())
                + ", finish_reason = " + FINISH_REASON.get(last));
    }
}

 1 APIServer  vllm/entrypoints/openai/chat_completion/api_router.py POST /v1/chat/completions
 2 APIServer  vllm/entrypoints/openai/chat_completion/serving.py    render chat template, call engine_client.generate
 3 APIServer  vllm/v1/engine/async_llm.py                           AsyncLLM.add_request
 4 APIServer  vllm/v1/engine/input_processor.py                     build EngineCoreRequest (token ids, SamplingParams)
 5 APIServer  vllm/v1/engine/core_client.py                         AsyncMPClient: msgpack over ZMQ to the engine
 6 EngineCore vllm/v1/engine/core.py                                input thread -> input_queue -> run_busy_loop
 7 EngineCore vllm/v1/core/sched/scheduler.py                       add_request -> waiting queue; schedule() each step
 8 EngineCore vllm/v1/core/kv_cache_manager.py                      allocate_slots: blocks from block_pool.py
 9 Worker     vllm/v1/worker/gpu_model_runner.py                    execute_model: inputs, block tables, forward
10 Worker     vllm/v1/sample/sampler.py                             sample next token ids
11 EngineCore vllm/v1/core/sched/scheduler.py                       update_from_output -> EngineCoreOutputs
12 APIServer  vllm/v1/engine/output_processor.py                    detokenize (detokenizer.py), RequestOutput
13 APIServer  vllm/entrypoints/openai/chat_completion/serving.py    SSE chunk or JSON response
status: WAITING -> RUNNING -> PREEMPTED -> RUNNING -> FINISHED_LENGTH_CAPPED
finished? true, finish_reason = length

The 13 hops form a loop between the two processes:

1. Hops 1 to 5 run in the API server. The router in chat_completion/api_router.py hands the request to OpenAIServingChat in serving.py, which renders the chat template and calls engine_client.generate. AsyncLLM.add_request turns the prompt into token ids and SamplingParams (the sampling settings) with the InputProcessor. It registers the request with the OutputProcessor and sends an EngineCoreRequest to the engine. The transport is a ZMQ socket (a fast message-passing library) with msgpack encoding (a compact binary format similar to JSON). It is implemented by AsyncMPClient in core_client.py, described there as "ZMQ + background proc EngineCore w/ asyncio".
2. Hops 6 to 11 run in the engine core and the workers. A background thread receives the request and puts it into input_queue. run_busy_loop drains the queue and calls step. Each step calls scheduler.schedule(), which decides how many tokens each request gets in this step and asks the KV cache manager for blocks (allocate_slots). The executor runs the model on the scheduled tokens (GPUModelRunner.execute_model) and samples the next token (vllm/v1/sample/sampler.py). scheduler.update_from_output appends the new tokens, checks the stop conditions, and produces EngineCoreOutputs.
3. Hops 12 and 13 run in the API server again. An output-handler task pulls the outputs, the OutputProcessor detokenizes incrementally (detokenizer.py), and the serving code writes an SSE chunk or, at the end, the JSON response. SSE chunks are the streaming format from The Chat Completions Protocol and Streaming in Java.

Hops 6 to 11 repeat once for every generated token. The request goes in once, and the engine core and the API server then trade outputs back and forth until the request finishes.

The status line follows RequestStatus in vllm/v1/request.py:

• A request starts WAITING.
• It becomes RUNNING when the scheduler admits it.
• It becomes PREEMPTED when the scheduler needs its KV blocks for other requests. _preempt_request in the scheduler frees the request's blocks and sets num_computed_tokens back to 0, so the request recomputes its KV cache when it runs again.
• Every status after PREEMPTED in the enum counts as finished (is_finished compares with it). _FINISHED_REASON_MAP turns FINISHED_LENGTH_CAPPED into the OpenAI finish reason length, FINISHED_STOPPED into stop, and FINISHED_ABORTED into abort.

In the program's example, the request is preempted once, runs again, and ends because it reached its length limit, so the finish reason is length.

Example 3: paged KV cache block tables

The problem: the KV cache of a request grows by one token per step, and nobody knows in advance how long a request will become. One simple approach reserves the full context length for every request at admission. That wastes most of the memory, because most requests are much shorter than the maximum. Another approach, growing one contiguous array, fragments memory as requests start and end.

vLLM instead divides the KV cache into fixed-size blocks. A request's block table lists the physical blocks that hold its tokens, in order. The idea is called PagedAttention, after the pages of virtual memory in an operating system: the table maps a request's logical positions to physical blocks that can be anywhere. The program uses blocks of 4 tokens (vLLM's default is 16, CacheConfig.DEFAULT_BLOCK_SIZE) and a pool of 10 blocks. It allocates blocks for two requests, A and B, runs decode steps, frees A, and then tries to fit two new requests, C and D.

import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

public class BlockTables {
    static final int BLOCK_SIZE = 4;                      // vLLM's default is 16 tokens per block
    static final Deque<Integer> free = new ArrayDeque<>();
    static final Map<String, List<Integer>> tables = new LinkedHashMap<>();
    static final Map<String, Integer> lengths = new LinkedHashMap<>();

    /** Makes room for `newTokens` more tokens; false when the pool is empty (the request would be preempted). */
    static boolean allocate(String req, int newTokens) {
        List<Integer> table = tables.getOrDefault(req, new ArrayList<>());
        int total = lengths.getOrDefault(req, 0) + newTokens;
        int needed = (total + BLOCK_SIZE - 1) / BLOCK_SIZE - table.size();
        if (needed > free.size()) return false;
        for (int i = 0; i < needed; i++) table.add(free.pollFirst());
        tables.put(req, table);
        lengths.put(req, total);
        return true;
    }

    static void release(String req) {
        tables.remove(req).forEach(free::addLast);          // freed blocks go to the end of the free queue
        lengths.remove(req);
    }

    static int slot(String req, int position) {             // physical slot of one token in the KV cache tensor
        return tables.get(req).get(position / BLOCK_SIZE) * BLOCK_SIZE + position % BLOCK_SIZE;
    }

    static void show(String title) {
        System.out.println(title + "  free=" + free);
        tables.forEach((r, t) -> System.out.println("  " + r + " tokens=" + lengths.get(r) + " table=" + t));
    }

    public static void main(String[] args) {
        for (int b = 1; b < 10; b++) free.addLast(b);       // block 0 is the pool's reserved null block
        allocate("A", 10);                                  // prompt of 10 tokens -> 3 blocks
        allocate("B", 5);                                   // prompt of 5 tokens  -> 2 blocks
        show("after prefill");
        for (int step = 0; step < 3; step++) { allocate("A", 1); allocate("B", 1); }
        show("after 3 decode steps");
        System.out.println("  A token 9 -> slot " + slot("A", 9) + ", A token 12 -> slot " + slot("A", 12)
                + ", B token 7 -> slot " + slot("B", 7));
        int waste = 0;
        for (String r : tables.keySet()) waste += tables.get(r).size() * BLOCK_SIZE - lengths.get(r);
        System.out.println("  reserved but unused slots: " + waste);
        release("A");
        show("after A finished");
        System.out.println("  C with 20 tokens fits: " + allocate("C", 20) + ", D with 12 tokens fits: " + allocate("D", 12));
        show("after C");
    }
}

after prefill  free=[6, 7, 8, 9]
  A tokens=10 table=[1, 2, 3]
  B tokens=5 table=[4, 5]
after 3 decode steps  free=[7, 8, 9]
  A tokens=13 table=[1, 2, 3, 6]
  B tokens=8 table=[4, 5]
  A token 9 -> slot 13, A token 12 -> slot 24, B token 7 -> slot 23
  reserved but unused slots: 3
after A finished  free=[7, 8, 9, 1, 2, 3, 6]
  B tokens=8 table=[4, 5]
  C with 20 tokens fits: true, D with 12 tokens fits: false
after C  free=[3, 6]
  B tokens=8 table=[4, 5]
  C tokens=20 table=[7, 8, 9, 1, 2]

The output, part by part:

• Block 0 is never handed out. BlockPool in vllm/v1/core/block_pool.py takes the first block of its free queue as a null_block, a placeholder for table entries that point nowhere, so a pool of N blocks serves N - 1.
• After prefill, request A (10 tokens) holds blocks 1, 2, 3 and request B (5 tokens) holds 4 and 5. Prefill is the first step, when the whole prompt is processed. Blocks are allocated in whole units, so A's third block has 2 empty slots (3 blocks x 4 slots = 12, minus 10 tokens) and B's second block has 3.
• After three decode steps, A has 13 tokens and needed a fourth block (block 6). B has 8 tokens and still fits in its two blocks. Blocks are allocated only when a token crosses a block boundary. This is why memory grows with the actual length of each sequence instead of the maximum.
• The slot of a token is table[position / 4] * 4 + position % 4. Take A's token 9: position 9 is in the third block of A's table (9 / 4 = 2, which is block 3) at offset 1 (9 % 4), so the slot is 3 x 4 + 1 = 13. A's token 12 is in block 6 at offset 0, so the slot is 24. The model runner builds this mapping (slot_mapping) for every scheduled token, so the attention kernel writes new keys and values to the right places and reads a request's history through its block table.
• Only 3 slots are reserved but unused: at most one partly filled block per request. A system that reserved the maximum context for each request at admission would leave everything beyond the actual length unused.
• When A finishes, its blocks go back to the end of the free queue, and C (20 tokens, 5 blocks) can use them. The physical blocks of C are not contiguous (7, 8, 9, 1, 2), which does not matter because every access goes through the table. D needs 3 blocks and only 2 are free, so the allocation fails. In vLLM, allocate_slots returning nothing is the signal that makes the scheduler preempt a running request, or leave a waiting one in the queue.

Prefix caching is a related feature. It is on by default (enable_prefix_caching = True in CacheConfig). Full blocks are also indexed by a hash of their tokens and everything before them, so a new request with the same prefix, such as the same system prompt, can point its table at existing blocks instead of computing them again. The free queue then doubles as an eviction order: freed blocks keep their contents until they are reused. The later lesson on the paged KV cache and prefix caching builds that part.

Example 4: continuous batching

The problem: a GPU is most efficient when it works on several sequences at once, so a server groups requests into a batch. But requests finish at different times, because they generate different numbers of tokens. In static batching, a new batch starts only when the old one is completely empty, so the places of finished requests sit idle while the longest request keeps going. In continuous batching, a free place is refilled at the next step.

The simulation runs six requests with different output lengths and at most three sequences per step. Each step, every running request produces one token. It runs first with static batching and then with continuous batching. Compare the number of steps, the slot use, and the mean latency (the average time from a request's arrival to its completion).

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class ContinuousBatching {
    record Req(String id, int arrival, int outputTokens) {}

    static final List<Req> REQS = List.of(
            new Req("A", 0, 8), new Req("B", 0, 2), new Req("C", 0, 3),
            new Req("D", 1, 2), new Req("E", 2, 6), new Req("F", 3, 1));
    static final int MAX_SEQS = 3;          // like --max-num-seqs

    /** Each step, every running request produces one token (prefill counted as the step of the first token). */
    static void run(boolean continuous) {
        List<Req> waiting = new ArrayList<>(REQS), running = new ArrayList<>();
        int[] produced = new int[REQS.size()];
        int step = 0, busySlotSteps = 0, totalLatency = 0;
        StringBuilder timeline = new StringBuilder();
        while (!waiting.isEmpty() || !running.isEmpty()) {
            boolean admit = continuous || running.isEmpty();   // static: refill only when the whole batch is done
            final int now = step;
            while (admit && running.size() < MAX_SEQS && !waiting.isEmpty() && waiting.get(0).arrival() <= now) {
                running.add(waiting.remove(0));
            }
            StringBuilder ids = new StringBuilder();
            List<Req> done = new ArrayList<>();
            for (Req r : running) {
                ids.append(r.id());
                int i = REQS.indexOf(r);
                if (++produced[i] == r.outputTokens()) { done.add(r); totalLatency += step + 1 - r.arrival(); }
            }
            busySlotSteps += running.size();
            timeline.append(String.format("%-4s", ids.isEmpty() ? "-" : ids));
            running.removeAll(done);
            step++;
        }
        System.out.printf(Locale.ROOT, "%-10s steps=%2d  slot use=%2d/%2d  mean latency=%.2f steps%n", continuous ? "continuous" : "static",
                step, busySlotSteps, step * MAX_SEQS, totalLatency / (double) REQS.size());
        System.out.println("           " + timeline.toString().trim());
    }

    public static void main(String[] args) {
        run(false);
        run(true);
    }
}

static     steps=14  slot use=22/42  mean latency=6.67 steps
           ABC ABC AC  A   A   A   A   A   DEF DE  E   E   E   E
continuous steps= 9  slot use=22/27  mean latency=4.17 steps
           ABC ABC ACD ADE AEF AE  AE  AE  E

Each column of the timeline is one step and lists the requests that produced a token in it. With static batching, A, B, and C start together. B finishes after 2 steps and C after 3, but D, E, and F wait until A finishes its 8 tokens at step 8. The batch then runs with one request for five steps. Of 42 possible request-steps (14 steps times 3 places), 22 were used.

With continuous batching, D takes B's place at step 2 and E takes C's place at step 3, then F joins at step 4, and everything is done after 9 steps. 22 of 27 request-steps were used. The work is the same (22 tokens), but the places stay full, and the mean time from arrival to completion drops from 6.67 to 4.17 steps.

In vLLM there is no separate batching phase. Every engine step calls schedule(), which looks at all running and waiting requests again. --max-num-seqs limits the number of running requests. DEFAULT_MAX_NUM_SEQS is 128 in SchedulerConfig, and EngineArgs.get_batch_defaults sets 256 for vllm serve on GPUs below 70 GiB and on A100s, and 1,024 on larger GPUs.

The simulation counts steps, not time. A real decode step with more sequences takes somewhat longer than one with fewer, but much less than proportionally. The reason is that decode is limited by reading the weights from memory, not by arithmetic, as FLOPs, Bandwidth, and the Roofline Model showed. The weights are read once per step no matter how many sequences share the step. That is the reason a full batch raises throughput.

Example 5: the token budget and chunked prefill

The problem: one step can contain two kinds of work. Decoding is cheap, one token per running request. Prefill is expensive, because a new request's whole prompt must be processed. If a 10,000-token prompt took a whole step by itself, every running request would stall until it finished.

vLLM's scheduler solves this with a token budget per step (--max-num-batched-tokens). The comment at the top of Scheduler.schedule states the model: there is no prefill phase and no decode phase. Each request has num_computed_tokens (tokens whose KV data exists) and a total token count, and each step assigns tokens so that the computed count catches up with the total. A long prompt is split across several steps (chunked prefill), and the leftover budget goes to it after the decodes are served.

The simulation uses a budget of 8 tokens and three requests: A has a 5-token prompt, B has a 13-token prompt, and C arrives at step 1 with a 3-token prompt. In the output, A:5+tok means 5 tokens were processed for A and the step produced one new output token for A.

import java.util.ArrayList;
import java.util.List;

public class ChunkedPrefill {
    static final class Req {
        final String id; final int arrival, prompt, maxOutput;
        int computed, output;                                   // num_computed_tokens, generated tokens
        Req(String id, int arrival, int prompt, int maxOutput) {
            this.id = id; this.arrival = arrival; this.prompt = prompt; this.maxOutput = maxOutput;
        }
        int numTokens() { return prompt + output; }             // prompt plus output tokens so far
    }

    static final int TOKEN_BUDGET = 8;   // like --max-num-batched-tokens (default 2048 on most GPUs)
    static final int MAX_SEQS = 4;       // like --max-num-seqs

    public static void main(String[] args) {
        List<Req> waiting = new ArrayList<>(List.of(
                new Req("A", 0, 5, 3), new Req("B", 0, 13, 2), new Req("C", 1, 3, 2)));
        List<Req> running = new ArrayList<>();
        for (int step = 0; !waiting.isEmpty() || !running.isEmpty(); step++) {
            int budget = TOKEN_BUDGET;
            StringBuilder plan = new StringBuilder();
            List<Req> scheduled = new ArrayList<>();
            List<Integer> counts = new ArrayList<>();
            for (Req r : running) {                              // 1) running requests first
                int n = Math.min(r.numTokens() - r.computed, budget);
                if (n == 0) continue;
                budget -= n; scheduled.add(r); counts.add(n);
            }
            while (budget > 0 && running.size() < MAX_SEQS && !waiting.isEmpty() && waiting.get(0).arrival <= step) {
                Req r = waiting.remove(0);                       // 2) then waiting requests, first come first served
                int n = Math.min(r.numTokens() - r.computed, budget);   // chunked prefill: take what fits
                budget -= n; running.add(r); scheduled.add(r); counts.add(n);
            }
            List<Req> finished = new ArrayList<>();
            for (int i = 0; i < scheduled.size(); i++) {
                Req r = scheduled.get(i);
                r.computed += counts.get(i);
                boolean sampled = r.computed == r.numTokens();  // caught up: the step produces one new token
                if (sampled) r.output++;
                plan.append(String.format(" %s:%d%s", r.id, counts.get(i), sampled ? "+tok" : ""));
                if (r.output == r.maxOutput) finished.add(r);
            }
            running.removeAll(finished);
            System.out.printf("step %d used %d/%d:%s%s%n", step, TOKEN_BUDGET - budget, TOKEN_BUDGET, plan,
                    finished.isEmpty() ? "" : "  finished " + finished.stream().map(r -> r.id).toList());
        }
    }
}

step 0 used 8/8: A:5+tok B:3
step 1 used 8/8: A:1+tok B:7
step 2 used 7/8: A:1+tok B:3+tok C:3+tok  finished [A]
step 3 used 2/8: B:1+tok C:1+tok  finished [B, C]

The schedule follows the order of the real scheduler: running requests first, then waiting requests in arrival order (policy = "fcfs", first come first served, by default), while the budget and the sequence limit allow.

• Step 0: A's whole prompt (5 tokens) fits, and its last prompt token produces A's first output token (+tok). B gets the remaining 3 tokens of the budget: a prefill chunk of 3 of its 13 prompt tokens, with no output yet.
• Step 1: A, now running, needs 1 token (a decode step). B continues with 7 more prompt tokens and fills the budget (1 + 7 = 8). C has arrived but there is no budget left, so it waits.
• Step 2: A decodes its third and last token and finishes. B's last 3 prompt tokens complete its prompt and produce its first token. C's 3-token prompt fits in the remaining budget. 7 of 8 tokens are used.
• Step 3: B and C decode and finish.

The budget keeps every step at a bounded size, so A's decoding continued at every step while B's long prompt was processed in pieces. Without chunking, B's 13 tokens would not fit in an 8-token step at all, and a large prompt would occupy whole steps while decodes waited.

In vLLM, chunked prefill is on by default (enable_chunked_prefill = True in SchedulerConfig). EngineArgs turns it off by default only for models that do not support it. The default budget for vllm serve comes from EngineArgs.get_batch_defaults: 2,048 tokens on GPUs with less than 70 GiB (and on A100s), 8,192 on larger GPUs such as H100, and 16,384 on GPUs with 160 GiB or more. long_prefill_token_threshold (default 0, which means off) can cap how many prompt tokens one request takes per step.

Example 6: startup memory profiling and the number of KV blocks

The problem: the number of KV blocks cannot be a fixed number, because it depends on the GPU, the model, and how much memory other things use. vLLM measures it at startup. Each worker loads the weights, runs one forward pass with dummy inputs at the largest batch size, measures memory, and gives the rest of its share of the GPU to the KV cache.

The program repeats the arithmetic of determine_available_memory in vllm/v1/worker/gpu_worker.py and of the block computation in vllm/v1/core/kv_cache_utils.py. It uses a Llama-3.1-8B-shaped model on a 24 GiB GPU. A GiB (gibibyte) is 2^30 bytes. The weight size follows from the parameter count. The other three measurements (non-PyTorch memory, the transient peak, and CUDA graph memory) are assumed values, because only a run on the GPU measures them. The program prints the memory budget, then the number of blocks and the resulting capacity for two KV cache number formats and two maximum request lengths.

import java.util.Locale;

public class KvBlocks {
    static final double GIB = 1L << 30;

    static String gib(long bytes) {                      // vLLM's format_gib: round(b / GiB, 2)
        return String.format(Locale.ROOT, "%.2f", bytes / GIB);
    }

    public static void main(String[] args) {
        long total = 24L << 30;                              // a GPU reporting 24 GiB
        double utilization = 0.92;                           // --gpu-memory-utilization default in 0.30.0
        long requested = (long) Math.ceil(total * utilization);

        // Assumed measurements for a Llama-3.1-8B-shaped model in BF16 (a real start measures these)
        long weights = 8_030_261_248L * 2;
        long nonTorch = (long) (0.60 * GIB);                 // CUDA context, NCCL, allocator overhead
        long transientPeak = (long) (1.20 * GIB);            // activations of the profile run
        long cudaGraphs = (long) (0.35 * GIB);               // estimated CUDA graph memory
        long totalConsumed = weights + nonTorch;
        long nonKv = totalConsumed + transientPeak;
        long available = requested - nonKv - cudaGraphs;
        System.out.println("requested      = " + gib(requested) + " GiB (total x " + utilization + ")");
        System.out.println("weights        = " + gib(weights) + " GiB");
        System.out.println("non-KV memory  = " + gib(nonKv) + " GiB, CUDA graphs " + gib(cudaGraphs) + " GiB");
        System.out.println("Available KV cache memory: " + gib(available) + " GiB");

        int layers = 32, kvHeads = 8, headDim = 128, blockSize = 16;
        for (int bytesPerValue : new int[] {2, 1}) {          // auto (BF16) and an FP8 KV cache
            long page = (long) blockSize * kvHeads * (headDim + headDim) * bytesPerValue;   // one layer, one block
            long bytesPerBlock = page * layers;
            long numBlocks = available / bytesPerBlock;
            for (int maxModelLen : new int[] {32768, 8192}) {
                long blocksPerRequest = (maxModelLen + blockSize - 1) / blockSize;
                double concurrency = numBlocks / (double) blocksPerRequest;
                long tokens = (long) (concurrency * maxModelLen);
                System.out.printf(Locale.ROOT, "kv bytes/value=%d block=%d KiB blocks=%d | GPU KV cache size: %,d tokens, "
                        + "Maximum concurrency for %,d tokens per request: %.2fx%n",
                        bytesPerValue, bytesPerBlock / 1024, numBlocks, tokens, maxModelLen, concurrency);
            }
        }
    }
}

requested      = 22.08 GiB (total x 0.92)
weights        = 14.96 GiB
non-KV memory  = 16.76 GiB, CUDA graphs 0.35 GiB
Available KV cache memory: 4.97 GiB
kv bytes/value=2 block=2048 KiB blocks=2545 | GPU KV cache size: 40,720 tokens, Maximum concurrency for 32,768 tokens per request: 1.24x
kv bytes/value=2 block=2048 KiB blocks=2545 | GPU KV cache size: 40,720 tokens, Maximum concurrency for 8,192 tokens per request: 4.97x
kv bytes/value=1 block=1024 KiB blocks=5091 | GPU KV cache size: 81,456 tokens, Maximum concurrency for 32,768 tokens per request: 2.49x
kv bytes/value=1 block=1024 KiB blocks=5091 | GPU KV cache size: 81,456 tokens, Maximum concurrency for 8,192 tokens per request: 9.94x

The calculation, line by line:

1. request_memory in vllm/v1/worker/utils.py takes ceil(total * gpu_memory_utilization): 22.08 GiB of 24. The default utilization in 0.30.0 is 0.92 (CacheConfig.gpu_memory_utilization). The check next to it raises an error at startup when the GPU has less free memory than this, for example because another process uses the GPU.
2. The weights take 14.96 GiB: 8,030,261,248 parameters at 2 bytes each (BF16, a 16-bit number format).
3. The profiler (memory_profiling in vllm/utils/mem_utils.py) computes non_kv_cache_memory. That is everything consumed since the worker started (the weights plus non-PyTorch memory such as the CUDA context) plus the transient peak of the profiling run (the activations). With the assumed 0.60 and 1.20 GiB, the total is 16.76 GiB.
4. The estimated CUDA graph memory (Example 7) is subtracted too, because VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS defaults to 1. What remains, 22.08 - 16.76 - 0.35 = 4.97 GiB, is logged as Available KV cache memory.
5. One block holds 16 tokens. Per layer, the page size is block size times KV heads times (key dimension plus value dimension) times bytes per value: 16 x 8 x 256 x 2 = 65,536 bytes. For 32 layers, a block takes 2,048 KiB. available_memory // bytes_per_block gives 2,545 blocks.
6. The second log line is computed by update_kv_cache_capacity. A request at the maximum length of 32,768 tokens needs 2,048 blocks, so the cache holds 2,545 / 2,048 = 1.24 such requests. The token capacity is reported as 1.24 times 32,768 = 40,720 tokens (2,545 blocks of 16).

The other lines show the two levers that change the result:

• With --max-model-len 8192, the same memory serves 4.97 maximum-length requests. The token capacity does not change, only the number of full-length requests that fit.
• With an FP8 KV cache (--kv-cache-dtype fp8, an 8-bit number format), values take 1 byte, blocks halve to 1,024 KiB, and the capacity doubles to 81,456 tokens.

When the available memory cannot hold even one request of max_model_len, startup fails with the error in kv_cache_utils.py that begins "To serve at least one request with the model's max seq len". The message suggests raising gpu_memory_utilization or lowering max_model_len. The lesson on serving with vLLM shows that error.

Example 7: CUDA graphs, compilation, and eager mode

The problem: during a decode step, the GPU runs hundreds of small operations called kernels, and the CPU must launch each one. For small batches, the launching cost can take longer than the work itself. A CUDA graph is a recording of all the GPU kernel launches of one forward pass at one fixed batch size. Replaying it launches the whole pass with one call, which removes the per-kernel launch cost.

Because a recorded graph only works for its exact size, vLLM captures graphs for a list of batch sizes (counted in tokens) and pads each batch up to the next captured size. Padding means adding unused filler tokens. The program builds the default list with the rule in vllm/config/vllm.py, and dispatches a few batch sizes the way CudagraphDispatcher in vllm/v1/cudagraph_dispatcher.py does. Look at how much padding each batch size needs and which sizes have no graph.

import java.util.ArrayList;
import java.util.List;
import java.util.TreeSet;

public class CaptureSizes {
    /** Default capture sizes of vLLM 0.30.0 (config/vllm.py) for one-token decode steps. */
    static List<Integer> captureSizes(int maxNumSeqs, int maxNumBatchedTokens, boolean blackwell) {
        int ceiling = blackwell ? 1024 : 512;
        int max = Math.min(maxNumBatchedTokens, Math.min(maxNumSeqs * 2, ceiling));
        TreeSet<Integer> sizes = new TreeSet<>();
        for (int s : new int[] {1, 2, 4}) if (s <= max) sizes.add(s);
        if (max >= 8) for (int s = 8; s < Math.min(max + 1, 256); s += 8) sizes.add(s);
        if (max >= 256) for (int s = 256; s <= max; s += 16) sizes.add(s);
        if (maxNumBatchedTokens <= max) sizes.add(maxNumBatchedTokens);
        return new ArrayList<>(sizes);
    }

    /** A batch is padded up to the next captured size; above the largest size it runs without a graph. */
    static String dispatch(List<Integer> sizes, int numTokens) {
        if (numTokens > sizes.get(sizes.size() - 1)) return "no CUDA graph (eager/compiled)";
        for (int s : sizes) if (s >= numTokens) return "graph " + s + " (+" + (s - numTokens) + " padding)";
        throw new IllegalStateException();
    }

    public static void main(String[] args) {
        List<Integer> sizes = captureSizes(256, 2048, false);
        System.out.println(sizes.size() + " sizes: " + sizes.subList(0, 6) + " ... " + sizes.subList(31, 37)
                + " ... " + sizes.get(sizes.size() - 1));
        for (int n : new int[] {1, 3, 37, 255, 300, 512, 513, 2048}) {
            System.out.printf("%5d tokens -> %s%n", n, dispatch(sizes, n));
        }
        System.out.println("max_num_seqs=64: " + captureSizes(64, 2048, false).size() + " sizes up to "
                + captureSizes(64, 2048, false).get(captureSizes(64, 2048, false).size() - 1));
    }
}

51 sizes: [1, 2, 4, 8, 16, 24] ... [232, 240, 248, 256, 272, 288] ... 512
    1 tokens -> graph 1 (+0 padding)
    3 tokens -> graph 4 (+1 padding)
   37 tokens -> graph 40 (+3 padding)
  255 tokens -> graph 256 (+1 padding)
  300 tokens -> graph 304 (+4 padding)
  512 tokens -> graph 512 (+0 padding)
  513 tokens -> no CUDA graph (eager/compiled)
 2048 tokens -> no CUDA graph (eager/compiled)
max_num_seqs=64: 19 sizes up to 128

With max_num_seqs = 256 and a 2,048-token budget, the largest captured size is min(2048, min(256 x 2, 512)) = 512 (the ceiling is 1,024 on data-center Blackwell GPUs). The list is 1, 2, 4, then every multiple of 8 below 256, then every multiple of 16 from 256 to 512, which gives 51 sizes. A batch of 3 tokens runs as the graph for 4 with one padding token, 37 runs as 40, and 300 runs as 304. Above 512 tokens (a step with a large prefill chunk), the dispatcher returns CUDAGraphMode.NONE, and the step runs without a graph. With max_num_seqs = 64, the largest size drops to 128 and the list to 19 sizes, which shortens startup and saves graph memory.

The default mode is FULL_AND_PIECEWISE. The documentation string in config/compilation.py calls it the default and the most performant mode for most models. It has two parts. Uniform decode batches replay a full graph of the whole model. Batches that mix prefill and decode replay piecewise graphs, in which the attention operation runs outside the graph. Piecewise graphs need the model to be compiled, which is the job of torch.compile (PyTorch's compiler) with vLLM's own backend (CompilationMode.VLLM_COMPILE, mode 3, the default for the V1 engine). Compilation produces the Dynamo bytecode transform time and torch.compile took ... s in total log lines, and it caches its result on disk for later starts.

The optimization level selects these defaults:

• -O0 turns off compilation and CUDA graphs.
• -O1 compiles and uses piecewise graphs.
• -O2 (the default) adds full graphs.
• -O3 is currently the same as -O2 (OptimizationLevel in config/vllm.py).

--enforce-eager disables both compilation and graphs. The source logs "Enforce eager set, disabling torch.compile and CUDAGraphs" when it is set. Eager mode starts faster and saves the graph memory, but each step then pays the full kernel launch overhead, which matters most for small decode batches.

Example 8: installation with uv and with the Docker image

The problem: before vllm serve works, the machine must match what the published package supports. The PyPI page of vLLM 0.30.0 (PyPI is the Python package index) lists three files and the supported Python range. The program checks five machines against them. A wheel is a prebuilt Python package that installs without compiling. Its file name encodes what it needs: the Python version, the operating system, and the CPU architecture.

import java.util.List;

public class WheelCheck {
    // Files published on PyPI for vllm 0.30.0, and its Requires-Python (from the PyPI JSON metadata)
    static final List<String> FILES = List.of(
            "vllm-0.30.0-cp38-abi3-manylinux_2_28_aarch64.whl",
            "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
            "vllm-0.30.0.tar.gz");
    static final int MIN_PY = 10, MAX_PY_EXCLUSIVE = 15;          // ">=3.10,<3.15"

    record Host(String name, String os, String arch, int pythonMinor, int glibcMinor) {}

    static String check(Host h) {
        if (h.pythonMinor() < MIN_PY || h.pythonMinor() >= MAX_PY_EXCLUSIVE) return "no: Python 3." + h.pythonMinor() + " is outside >=3.10,<3.15";
        if (!h.os().equals("linux")) return "no prebuilt wheel: only the source archive, a build from source";
        for (String f : FILES) {
            if (f.endsWith(h.arch() + ".whl")) {
                return h.glibcMinor() >= 28 ? "wheel " + f : "no: manylinux_2_28 needs glibc 2.28 or newer";
            }
        }
        return "no prebuilt wheel for " + h.arch();
    }

    public static void main(String[] args) {
        List<Host> hosts = List.of(
                new Host("Ubuntu 24.04, x86_64, Python 3.12", "linux", "x86_64", 12, 39),
                new Host("Ubuntu 22.04, aarch64, Python 3.12", "linux", "aarch64", 12, 35),
                new Host("CentOS 7, x86_64, Python 3.10", "linux", "x86_64", 10, 17),
                new Host("Ubuntu 24.04, Python 3.9", "linux", "x86_64", 9, 39),
                new Host("macOS 15, arm64, Python 3.12", "macos", "arm64", 12, 0));
        for (Host h : hosts) System.out.printf("%-36s -> %s%n", h.name(), check(h));
    }
}

Ubuntu 24.04, x86_64, Python 3.12    -> wheel vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl
Ubuntu 22.04, aarch64, Python 3.12   -> wheel vllm-0.30.0-cp38-abi3-manylinux_2_28_aarch64.whl
CentOS 7, x86_64, Python 3.10        -> no: manylinux_2_28 needs glibc 2.28 or newer
Ubuntu 24.04, Python 3.9             -> no: Python 3.9 is outside >=3.10,<3.15
macOS 15, arm64, Python 3.12         -> no prebuilt wheel: only the source archive, a build from source

The published wheels are Linux-only, for x86_64 and aarch64, tagged manylinux_2_28. That tag means they were built against glibc 2.28 (the standard C library on Linux), so older distributions such as CentOS 7 with glibc 2.17 cannot use them. The tag cp38-abi3 means one wheel works with every CPython version from 3.8 on through the stable ABI (a promise that compiled code keeps working across Python versions), and Requires-Python narrows that to >=3.10,<3.15. Other systems get only the source archive, which needs a full build. The wheel's dependencies pin torch==2.13.0 and flashinfer-python==0.6.18.post1 (from the same metadata). That is why vLLM is installed into its own virtual environment instead of next to other PyTorch projects.

The project description on PyPI gives uv pip install vllm as the installation command (uv is a fast Python package manager). A complete sequence with a pinned version follows. It is representative and was not run in this course's build environment, which has no GPU. It creates a virtual environment, installs vLLM, checks the version, and starts a server:

uv venv --python 3.12 vllm-env
source vllm-env/bin/activate
uv pip install vllm==0.30.0
vllm --version
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --port 8000

vllm --version prints the package version (0.30.0), from an argparse version action in vllm/entrypoints/cli/main.py. The vllm command has the subcommands serve, chat, complete, run-batch, collect-env, launch, and the benchmark commands. vllm collect-env prints the versions of PyTorch, CUDA, drivers, and GPUs, which belongs in every bug report. vllm serve listens on port 8000 by default (port: int = 8000 in entrypoints/launchers/cli_args.py). With no --host, it binds all interfaces and logs 0.0.0.0, which means other machines on the network can reach it. This differs from llama-server's 127.0.0.1 default. A client on the same machine can use --host 127.0.0.1 to keep the server private.

The source mentions "official vllm-openai images" in a comment in envs.py. The vLLM documentation publishes them on Docker Hub as vllm/vllm-openai, and the image's entry point runs the OpenAI-compatible server. A Docker image packages the program together with its dependencies. The following command follows that documentation and was not run here:

docker run --gpus all -p 8000:8000 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.30.0 --model Qwen/Qwen2.5-Coder-7B-Instruct

--gpus all needs the NVIDIA Container Toolkit on the host, which lets containers use the GPUs. The volume mount shares the Hugging Face cache, so the weights are downloaded once, in the cache layout described in The Hub HTTP API and Cache Layout in Java. The vLLM documentation adds --ipc=host (or a larger --shm-size) so that the container can use the host's shared memory. PyTorch uses shared memory to share data between processes, in particular for tensor-parallel inference.

Example 9: supported hardware and platform detection

The problem: vLLM runs on several kinds of hardware, and it must pick the right backend without being told. A platform is vLLM's abstraction for one kind of device. PlatformEnum in vllm/platforms/interface.py lists CUDA (NVIDIA), ROCm (AMD), TPU (Google), XPU (Intel), CPU, and out-of-tree (OOT) plugins, which are platforms added by separately installed packages. The program applies the selection rules of resolve_current_platform_cls_qualname in vllm/platforms/__init__.py to seven hosts. Compare each host's setup with the result.

import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

public class PlatformDetect {
    record Host(String name, boolean targetCpu, Map<String, Boolean> probes, List<String> outOfTreePlugins) {}

    /** The selection rules of resolve_current_platform_cls_qualname() in vllm/platforms/__init__.py. */
    static String resolve(Host h) {
        if (h.targetCpu()) return "CpuPlatform (VLLM_TARGET_DEVICE=cpu)";
        List<String> builtin = new ArrayList<>();
        h.probes().forEach((name, found) -> { if (found) builtin.add(name); });
        if (h.outOfTreePlugins().size() >= 2) return "error: more than one platform plugin " + h.outOfTreePlugins();
        if (h.outOfTreePlugins().size() == 1) return "plugin " + h.outOfTreePlugins().get(0);
        if (builtin.size() >= 2) return "error: more than one platform plugin " + builtin;
        if (builtin.size() == 1) return builtin.get(0);
        return "UnspecifiedPlatform";
    }

    static Map<String, Boolean> probes(boolean tpu, boolean cuda, boolean rocm, boolean xpu, boolean cpu) {
        Map<String, Boolean> m = new LinkedHashMap<>();           // probe order: tpu, cuda, rocm, xpu, cpu
        m.put("tpu", tpu); m.put("cuda", cuda); m.put("rocm", rocm); m.put("xpu", xpu); m.put("cpu", cpu);
        return m;
    }

    public static void main(String[] args) {
        List<Host> hosts = List.of(
            new Host("Linux + NVIDIA GPU, PyPI wheel", false, probes(false, true, false, false, false), List.of()),
            new Host("Linux + AMD GPU, ROCm build", false, probes(false, false, true, false, false), List.of()),
            new Host("macOS laptop", false, probes(false, false, false, false, true), List.of()),
            new Host("Linux, no GPU, CPU build", false, probes(false, false, false, false, true), List.of()),
            new Host("NVIDIA GPU host, forced CPU", true, probes(false, true, false, false, false), List.of()),
            new Host("Ascend NPU with plugin", false, probes(false, false, false, false, false), List.of("ascend")),
            new Host("Linux, no GPU, CUDA wheel", false, probes(false, false, false, false, false), List.of()));
        for (Host h : hosts) System.out.printf("%-34s -> %s%n", h.name(), resolve(h));
    }
}

Linux + NVIDIA GPU, PyPI wheel     -> cuda
Linux + AMD GPU, ROCm build        -> rocm
macOS laptop                       -> cpu
Linux, no GPU, CPU build           -> cpu
NVIDIA GPU host, forced CPU        -> CpuPlatform (VLLM_TARGET_DEVICE=cpu)
Ascend NPU with plugin             -> plugin ascend
Linux, no GPU, CUDA wheel          -> UnspecifiedPlatform

At import time, vLLM calls a probe function for each built-in platform. A probe is a quick check of whether that kind of hardware is present. The CUDA probe asks NVML (the NVIDIA management library) for the device count, and it accepts CUDA when the count is above 0 and the installed vLLM is not a CPU build. ROCm, TPU, and XPU have their own probes. The CPU probe succeeds when the version string marks a CPU build, or on macOS.

Exactly one platform must be found. Two built-in matches raise an error, and no match gives UnspecifiedPlatform, on which serving fails (the last host in the output). VLLM_TARGET_DEVICE=cpu overrides the probes (the "forced CPU" host). A package registered as a platform plugin takes precedence over the built-in ones. The PyPI description names Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Apple Silicon, and others as plugin hardware, and the plugin name in the program is invented.

Hardware facts that follow from the source:

For this course's hardware tiers, vLLM fits tiers 3 and 4 (Linux with NVIDIA or AMD GPUs). A tier 1 or 2 machine runs llama.cpp, and the vLLM lessons give commands and representative output for those readers.

Example 10: reading the startup log

The problem: starting vLLM takes more than a minute, and the log is long. A reader needs to know what each part says, and a supervisor program (like the loop serve command built later in the course) needs to pull out the important numbers. The log below is representative. Every message text, file name, and line number comes from the 0.30.0 source (the log format is INFO <month-day> <time> [<file>:<line>] <message>, from vllm/logger.py, with the (name pid=N) prefix added by decorate_logs). The numbers are the assumed values of the 8B model from Example 6. The timestamps, durations, and process ids are illustrative, and the log assumes VLLM_DISABLE_LOG_LOGO=1.

The program parses the log with regular expressions, which are text patterns that match parts of a string. It counts the lines per process and prints the key numbers: the weights size, the KV memory, the token capacity, the concurrency, the compile time, and the time from the first line to readiness.

import java.time.Duration;
import java.time.LocalTime;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class StartupLog {
    // Representative `vllm serve` log for vLLM 0.30.0: message texts, file names, and line numbers from the source;
    // values for a Llama-3.1-8B-shaped BF16 model on a 24 GiB GPU (the numbers of the KV-block example)
    static final String LOG = """
        (APIServer pid=1000) INFO 10-03 12:00:01 [api_utils.py:346] vLLM server version 0.30.0, serving model example-org/Coder-8B-Instruct
        (APIServer pid=1000) INFO 10-03 12:00:01 [api_utils.py:286] non-default args: {'model_tag': 'example-org/Coder-8B-Instruct', 'max_model_len': 32768, 'api_key': '***'}
        (APIServer pid=1000) INFO 10-03 12:00:04 [model.py:692] Resolved architecture: LlamaForCausalLM
        (APIServer pid=1000) INFO 10-03 12:00:04 [model.py:2030] Using max model len 32768
        (APIServer pid=1000) INFO 10-03 12:00:04 [scheduler.py:288] Chunked prefill is enabled with max_num_batched_tokens=2048.
        (EngineCore pid=1100) INFO 10-03 12:00:09 [core.py:123] Initializing a V1 LLM engine (v0.30.0) with config: model='example-org/Coder-8B-Instruct', ..., max_seq_len=32768, ..., enforce_eager=False, kv_cache_dtype=auto, ...
        (EngineCore pid=1100) INFO 10-03 12:00:12 [gpu_model_runner.py:5317] Starting to load model example-org/Coder-8B-Instruct...
        (EngineCore pid=1100) INFO 10-03 12:00:13 [cuda.py:538] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
        (EngineCore pid=1100) Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:16<00:00,  4.10s/it]
        (EngineCore pid=1100) INFO 10-03 12:00:30 [default_loader.py:430] Loading weights took 16.42 seconds
        (EngineCore pid=1100) INFO 10-03 12:00:31 [gpu_model_runner.py:5416] Model loading took 14.96 GiB memory and 18.530412 seconds
        (EngineCore pid=1100) INFO 10-03 12:00:38 [backends.py:1155] Dynamo bytecode transform time: 6.41 s
        (EngineCore pid=1100) INFO 10-03 12:00:56 [backends.py:393] Compiling a graph for compile range (1, 2048) takes 17.38 s
        (EngineCore pid=1100) INFO 10-03 12:00:56 [monitor.py:53] torch.compile took 24.61 s in total
        (EngineCore pid=1100) INFO 10-03 12:00:59 [gpu_worker.py:640] Available KV cache memory: 4.97 GiB
        (EngineCore pid=1100) INFO 10-03 12:00:59 [kv_cache_utils.py:2395] GPU KV cache size: 40,720 tokens, Maximum concurrency for 32,768 tokens per request: 1.24x
        (EngineCore pid=1100) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|##########| 51/51 [00:05<00:00,  9.52it/s]
        (EngineCore pid=1100) Capturing CUDA graphs (decode, FULL): 100%|##########| 35/35 [00:03<00:00, 10.11it/s]
        (EngineCore pid=1100) INFO 10-03 12:01:08 [gpu_model_runner.py:6929] Graph capturing finished in 9 secs, took 0.31 GiB
        (EngineCore pid=1100) INFO 10-03 12:01:08 [core.py:372] init engine (profile, create kv cache, warmup model) took 37.80 s (compilation: 24.61 s)
        (APIServer pid=1000) INFO 10-03 12:01:10 [entry.py:135] Supported tasks: ('generate',)
        (APIServer pid=1000) INFO 10-03 12:01:10 [entry.py:139] Starting vLLM server on http://0.0.0.0:8000
        (APIServer pid=1000) INFO 10-03 12:01:10 [launcher.py:61] Available routes are:
        (APIServer pid=1000) INFO 10-03 12:01:10 [launcher.py:70] Route: /health, Methods: GET
        (APIServer pid=1000) INFO 10-03 12:01:10 [launcher.py:70] Route: /v1/models, Methods: GET
        (APIServer pid=1000) INFO 10-03 12:01:10 [launcher.py:70] Route: /v1/chat/completions, Methods: POST
        """;

    static String find(String regex) {
        Matcher m = Pattern.compile(regex).matcher(LOG);
        return m.find() ? m.group(1) : "?";
    }

    public static void main(String[] args) {
        Pattern line = Pattern.compile("^\\((\\w+) pid=(\\d+)\\) (?:(INFO|WARNING|ERROR) \\S+ \\S+ \\[([\\w.]+):(\\d+)\\] )?(.*)$",
                Pattern.MULTILINE);
        Matcher m = line.matcher(LOG);
        int lines = 0, api = 0, core = 0, bars = 0;
        while (m.find()) {
            lines++;
            if (m.group(1).equals("APIServer")) api++; else core++;
            if (m.group(3) == null) bars++;
        }
        System.out.printf("%d lines: %d APIServer, %d EngineCore, %d without a level (progress bars)%n", lines, api, core, bars);
        System.out.println("weights        " + find("Model loading took ([\\d.]+) GiB") + " GiB");
        System.out.println("KV memory      " + find("Available KV cache memory: ([\\d.]+) GiB") + " GiB");
        long kvTokens = Long.parseLong(find("KV cache size: ([\\d,]+) tokens").replace(",", ""));
        System.out.println("KV tokens      " + kvTokens + " = " + kvTokens / 16 + " blocks of 16");
        System.out.println("concurrency    " + find("per request: ([\\d.]+)x") + "x at max length " + find("Using max model len (\\d+)"));
        System.out.println("compile        " + find("torch.compile took ([\\d.]+) s") + " s, graphs " + find("took ([\\d.]+) GiB\\n") + " GiB");
        System.out.println("engine init    " + find("warmup model\\) took ([\\d.]+) s") + " s");
        System.out.println("ready at       " + find("Starting vLLM server on (\\S+)"));
        Matcher t = Pattern.compile("INFO \\d\\d-\\d\\d (\\d\\d:\\d\\d:\\d\\d)").matcher(LOG);
        LocalTime first = null, last = null;
        while (t.find()) { last = LocalTime.parse(t.group(1)); if (first == null) first = last; }
        System.out.println("first to ready " + Duration.between(first, last).toSeconds() + " s");
    }
}

26 lines: 11 APIServer, 15 EngineCore, 3 without a level (progress bars)
weights        14.96 GiB
KV memory      4.97 GiB
KV tokens      40720 = 2545 blocks of 16
concurrency    1.24x at max length 32768
compile        24.61 s, graphs 0.31 GiB
engine init    37.80 s
ready at       http://0.0.0.0:8000
first to ready 69 s

The 26 lines fall into five phases:

1. Configuration, in the API server (APIServer pid=1000). The lines show the version and model, the arguments that differ from defaults (the API key is printed as *** by redact_sensitive_args), the architecture from config.json (LlamaForCausalLM), the maximum length, and the chunked-prefill budget. Without VLLM_DISABLE_LOG_LOGO=1, the first message is a four-line text logo that contains the same version and model.
2. Engine start and weight loading, in the engine core (EngineCore pid=1100). The lines show the full engine configuration on one long line, the attention backend chosen from the candidates in priority order (FLASH_ATTN first on GPUs other than Blackwell), a progress bar over the safetensors shards (the files that hold the weights), and the memory the weights took (14.96 GiB).
3. Compilation. The lines show Dynamo tracing (the first step of torch.compile), compiling the graph for the token range 1 to 2,048 (the budget), and the total. A second start with the same model and settings reads the compiled artifacts from the cache and is faster.
4. Memory and graphs. The lines are the two KV cache lines of Example 6, two progress bars for capturing piecewise graphs (51 sizes) and full decode graphs, and the graph memory. init engine ... took 37.80 s sums profiling, KV cache creation, and warm-up.
5. Serving, in the API server again. The lines show the supported tasks, the listen address, and the route list. The port is bound early in the start (a comment in launcher.py says this avoids races with Ray), but HTTP requests are answered only once the server starts right after Starting vLLM server on.

The progress bars have the process prefix but no level and file, so a parser must accept lines without them. The regular expression makes that part optional and counts 3 such lines. For the loop serve supervisor in a later workshop lesson, three numbers matter: the KV token capacity (to warn when a requested context does not fit), the maximum concurrency, and the readiness line. Health polling of /health is the reliable readiness signal, and the log explains a slow or failed start. In this example, 69 seconds pass from the first line to readiness, and most of that time is weight loading and compilation.

Example 11: vLLM or llama.cpp

The problem: given a machine and a workload, which engine should serve the model? There is no single best answer. The answer depends on the hardware, the model file format, and the number of concurrent users. The comparison below uses facts from this section of the course. The program turns them into rules and applies them to six situations, and prints the decision with its reason.

import java.util.ArrayList;
import java.util.List;

public class EngineChoice {
    record Need(String name, boolean nvidiaOrAmdGpu, boolean linux, int concurrentUsers,
                boolean ggufOnly, boolean cpuOrAppleOnly, boolean multiGpu) {}

    /** Criteria from this section: platforms, file formats, concurrency, and parallelism. */
    static String choose(Need n, List<String> reasons) {
        if (n.cpuOrAppleOnly()) { reasons.add("CPU or Apple GPU: llama.cpp builds for both; the vLLM PyPI wheel is the CUDA build"); return "llama.cpp"; }
        if (!n.linux()) { reasons.add("vLLM wheels are manylinux only"); return "llama.cpp"; }
        if (n.ggufOnly()) { reasons.add("GGUF is llama.cpp's native format; vLLM 0.30.0 has no GGUF loader"); return "llama.cpp"; }
        if (n.multiGpu()) { reasons.add("tensor and pipeline parallelism across GPUs"); return "vLLM"; }
        if (n.nvidiaOrAmdGpu() && n.concurrentUsers() >= 8) {           // 8 is this example's threshold
            reasons.add("many concurrent requests: paged KV blocks, token-budget scheduler, CUDA graphs");
            return "vLLM";
        }
        reasons.add("few concurrent requests: slots and quantized GGUF weights fit a single user");
        return "llama.cpp";
    }

    public static void main(String[] args) {
        List<Need> needs = List.of(
            new Need("laptop, 16 GB RAM, one developer", false, true, 1, false, true, false),
            new Need("MacBook with 36 GB unified memory", false, false, 1, false, true, false),
            new Need("RTX 4090 box, one coding agent", true, true, 2, false, false, false),
            new Need("RTX 4090 box, team of 20", true, true, 20, false, false, false),
            new Need("4 x H100, 70B model", true, true, 50, false, false, true),
            new Need("GPU server, GGUF-only model", true, true, 20, true, false, false));
        for (Need n : needs) {
            List<String> reasons = new ArrayList<>();
            String engine = choose(n, reasons);
            System.out.printf("%-36s -> %-9s %s%n", n.name(), engine, reasons);
        }
    }
}

laptop, 16 GB RAM, one developer     -> llama.cpp [CPU or Apple GPU: llama.cpp builds for both; the vLLM PyPI wheel is the CUDA build]
MacBook with 36 GB unified memory    -> llama.cpp [CPU or Apple GPU: llama.cpp builds for both; the vLLM PyPI wheel is the CUDA build]
RTX 4090 box, one coding agent       -> llama.cpp [few concurrent requests: slots and quantized GGUF weights fit a single user]
RTX 4090 box, team of 20             -> vLLM      [many concurrent requests: paged KV blocks, token-budget scheduler, CUDA graphs]
4 x H100, 70B model                  -> vLLM      [tensor and pipeline parallelism across GPUs]
GPU server, GGUF-only model          -> llama.cpp [GGUF is llama.cpp's native format; vLLM 0.30.0 has no GGUF loader]

Each decision follows from one row of the table:

• The laptop and the MacBook have no supported GPU for vLLM, so they use llama.cpp. The vLLM wheel requires CUDA on Linux.
• On the same RTX 4090 workstation, one coding agent sends few requests at a time. llama.cpp's slots and a quantized GGUF are enough, and it starts in seconds. A team of 20 benefits from vLLM's shared block pool and token-budget scheduler, which keep the GPU busy with many sequences.
• Four H100 GPUs serving a 70B model need tensor parallelism, where vLLM's process layout from Example 1 applies.
• A model distributed only as GGUF points to llama.cpp. The PyPI description of vLLM 0.30.0 still lists GGUF among its quantization formats. However, the 0.30.0 source has no gguf entry in QuantizationMethods (model_executor/layers/quantization/__init__.py) and no GGUF model loader, so the package cannot load such a file.

The threshold of 8 concurrent users is a parameter of this example, not a measured boundary. A later lesson on benchmarking measures the crossover for a given model and GPU. loop supports both engines through the same OpenAI-compatible client from The Chat Completions Protocol and Streaming in Java, so the choice can change without changing the client.

Interview points

• vllm serve on one GPU runs two processes. The API server handles HTTP, templates, the tokenizer, the detokenizer, and AsyncLLM. The engine core holds the scheduler, the KV cache manager, and the model (through UniProcExecutor). With tensor or pipeline parallelism, MultiprocExecutor adds one worker process per GPU.
• The API server and the engine core talk over ZMQ sockets with msgpack-encoded EngineCoreRequest and EngineCoreOutputs. The engine core runs a busy loop of schedule, execute, and update_from_output.
• The V1 scheduler has no separate prefill and decode phases. Each step assigns tokens so that num_computed_tokens catches up with the request's token count. Running requests go first, within the limits of max_num_batched_tokens and max_num_seqs.
• Chunked prefill is on by default. Long prompts are split across steps so that decodes keep running.
• KV memory is managed in blocks of 16 tokens by default. Block tables map logical positions to physical blocks, and block 0 is a reserved null block. Preemption frees a request's blocks and resets its computed tokens to 0, so the request recomputes them later.
• Available KV memory is ceil(total x gpu_memory_utilization) (default 0.92) minus the weights, other consumed memory, peak activations, and the CUDA graph estimate. The number of blocks is available memory divided by (page size x layers).
• The startup log reports Available KV cache memory and GPU KV cache size: N tokens, Maximum concurrency for L tokens per request: Xx, where X = blocks / ceil(L / 16).
• CUDA graphs are captured for 1, 2, 4, multiples of 8 below 256, and multiples of 16 up to a default maximum of 512 tokens. Batches are padded to the next size, and larger ones run without a graph. --enforce-eager or -O0 turns graphs and compilation off.
• vLLM 0.30.0 wheels are manylinux_2_28 for x86_64 and aarch64, for Python 3.10 to 3.14, with PyTorch pinned to 2.13.0.
• Choose vLLM for many concurrent requests on supported GPUs and for multi-GPU serving. Choose llama.cpp for CPUs, Apple machines, GGUF models, single users, and fast starts.

Summary

• vLLM's V1 engine separates the API server process from the EngineCore process, and it adds worker processes when a model spans several GPUs.
• A request flows from api_router.py and serving.py through AsyncLLM, InputProcessor, and AsyncMPClient to the engine core, the scheduler, the KV cache manager, the GPU model runner, and the sampler. It then comes back through the OutputProcessor and the detokenizer.
• RequestStatus goes from WAITING to RUNNING, possibly to PREEMPTED, and then to a finished status that maps to stop, length, or abort.
• The block pool hands out fixed-size KV blocks, keeps freed blocks in a queue for reuse and prefix-cache eviction, and reserves block 0 as a null block.
• Continuous batching refills free places at every step. In the simulation it finished 6 requests in 9 steps instead of 14.
• The token budget (--max-num-batched-tokens, 2,048 by default on GPUs under 70 GiB) bounds each step, and prompt chunks fill what decodes leave.
• gpu_memory_utilization defaults to 0.92 in 0.30.0. The KV cache gets what remains after the weights, the measured peak activations, and the CUDA graphs.
• For the Llama 3.1 8B shape, a 16-token block is 2 MiB in BF16 and 1 MiB in FP8.
• The default CUDA graph mode is FULL_AND_PIECEWISE with torch.compile mode VLLM_COMPILE, and the default optimization level is -O2.
• vllm serve listens on port 8000 on all interfaces by default, and the PyPI description recommends uv pip install vllm.
• Platform detection probes CUDA, ROCm, TPU, XPU, and CPU and requires exactly one match, with plugins taking precedence.
• The startup log's file and line fields point to the source of each message, which makes the log a map of the startup sequence.