Lesson 73 of 73
Reading Inference Engine Codebases
An inference engine is the program that loads a model file and turns a prompt into generated text, one token at a time. Earlier lessons used engines from the outside: you started them, sent HTTP requests, and read their logs. This lesson opens four of them and reads their source code. The goal is not to understand every line. The goal is to learn where each important mechanism lives, so that when an engine behaves in a surprising way, you know which file to open.
The lesson follows a request through each engine. It asks five questions: where the request enters, how the model's computation is described (the model graph, a list of math operations to run in order), where the KV cache lives (the memory that stores the keys and values the model computed for earlier tokens, so they are not computed again), how the next token is chosen (sampling), and how many requests share the hardware (scheduling).
After the lesson, you can find these parts in the source of llama.cpp, vLLM, Ollama, and SGLang, explain how each is built, and compare them with the Java engine that section 7 of this course builds. Source code in C++, Go, and Python is shown only in short excerpts. The main material is a set of Java programs, each of which rebuilds one mechanism on small inputs so that you can run it and watch it work.
| Project | Version read here | Language | Role |
|---|---|---|---|
| llama.cpp and ggml | commit 0c1e57098 (ggml 0.25.3) |
C and C++ | Engine library, command-line tools, and llama-server |
| vLLM | 0.30.0 | Python with CUDA kernels | Serving engine for GPUs with continuous batching and a paged KV cache |
| Ollama | v0.35.1 | Go | Model manager and API server that runs llama-server as a child process |
| SGLang | 0.5.21 | Python with CUDA kernels | Serving engine with a radix-tree prefix cache |
The table gives one line per project. llama.cpp is a C and C++ library plus tools, and ggml is the tensor library underneath it. vLLM and SGLang are Python servers built for GPUs that serve many users at once. Ollama is a Go program that downloads models and exposes an API, and it starts llama-server from llama.cpp to do the actual token generation.
Every fact about a project in this lesson comes from its source at the version in the table. Source code changes quickly, so a file name or a number may differ in a newer release. Paths are relative to each project's root: the repository root for llama.cpp, the vllm/ package for vLLM, the module root for Ollama, and the sglang/ package for SGLang. The companion lesson Reading Coding Harness Codebases uses the same method on coding agents. Earlier lessons cover the outside of these engines: llama.cpp Architecture (ggml tensors, graphs, backends, the startup log), llama-server: Endpoints, Slots, and Prompt Cache, vLLM Architecture and Installation, and The Engine Landscape and How Ollama Works. This lesson reads the inside. Each Java program reproduces one mechanism on small inputs, is shorter than the original, and names the function it follows.
Example 1: a method for reading an engine
Source trees hold thousands of files, and reading them from the top does not work. A better method is to ask which part of the engine answers your question, and open only the files for that part. Every engine has the same five parts, and a request touches them in this order:
- Request path. The HTTP request arrives and its text becomes a list of token ids.
- Model graph. Code describes one forward pass, which is one run of the model that produces scores for the next token.
- KV cache. Keys and values of past tokens are stored, and the engine finds free space for new ones.
- Sampler. The scores (called logits) become one chosen token.
- Scheduler. When several requests are active, something decides who runs in the next step.
The table lists where each part lives in each project. "Delegates" means Ollama does not implement that part itself and relies on llama-server.
| Part | Question to answer | llama.cpp | vLLM | Ollama | SGLang |
|---|---|---|---|---|---|
| Request path | Where does an HTTP request become a sequence of tokens to run? | tools/server/ |
entrypoints/openai/, v1/engine/ |
server/routes.go, llm/ |
srt/entrypoints/, srt/managers/ |
| Model graph | Where is one forward pass described? | src/models/<arch>.cpp |
model_executor/models/<arch>.py |
delegates to llama-server |
srt/models/ |
| KV cache | Where are keys and values stored, and how is space found? | src/llama-kv-cache.cpp, src/llama-kv-cells.h |
v1/core/kv_cache_manager.py, v1/core/block_pool.py |
delegates | srt/mem_cache/ |
| Sampler | Where are logits turned into a token? | src/llama-sampler.cpp, common/sampling.cpp |
v1/sample/ |
delegates | srt/sampling/ |
| Scheduler | Who decides which requests run in the next step? | tools/server/server-context.cpp (slots) |
v1/core/sched/scheduler.py |
server/sched.go (models, not tokens) |
srt/managers/scheduler.py |
The first step is to measure. Line counts show where the code is and which files are worth opening first. The command below uses wc -l (word count, lines only) on the files named in the table. It runs from a directory that holds the source trees, and the output is a line count per file, with a total after each group.
wc -l llama.cpp/tools/server/server-context.cpp llama.cpp/src/models/llama.cpp llama.cpp/src/llama-kv-cache.cpp \
llama.cpp/src/llama-sampler.cpp llama.cpp/ggml/src/ggml-cpu/ops.cpp
wc -l vllm/v1/engine/core.py vllm/v1/core/sched/scheduler.py vllm/v1/core/kv_cache_manager.py \
vllm/v1/core/block_pool.py sglang/srt/mem_cache/radix_cache.py ollama/server/sched.go ollama/server/routes.goOutput from the reference source trees (paths shortened to the path inside each project):
5559 tools/server/server-context.cpp
250 src/models/llama.cpp
2934 src/llama-kv-cache.cpp
4385 src/llama-sampler.cpp
12435 ggml/src/ggml-cpu/ops.cpp
25563 total
2578 vllm/v1/engine/core.py
3250 vllm/v1/core/sched/scheduler.py
941 vllm/v1/core/kv_cache_manager.py
892 vllm/v1/core/block_pool.py
823 sglang/srt/mem_cache/radix_cache.py
1800 server/sched.go
3277 server/routes.go
13561 totalThe whole Llama architecture in llama.cpp is 250 lines, because that file only describes the graph: which operations run, in which order. It does not contain any arithmetic. The CPU operators take 12,435 lines in one file, because every operator needs a kernel (the low-level loop that does the arithmetic) for each data type and shape. The server file is 5,559 lines because it handles HTTP, slots, templates, and caching together.
On the Python side, the vLLM scheduler is 3,250 lines and the KV cache manager is 941. SGLang's radix cache, which is a whole prefix-caching data structure, fits in 823 lines. Reading starts with the small files that describe structure and moves to the large ones only for the question at hand.
Example 2: llama.cpp, model loading and per-architecture graph building
The question is how llama.cpp goes from a GGUF file to a graph for one architecture. An architecture is the design of a model family, such as Llama or Qwen2: how many layers it has, how attention is wired, and which normalization it uses. A GGUF file (the model file format covered in earlier lessons) stores the architecture name as text, and the engine must pick the code that matches. Open src/llama.cpp. llama_model_load creates a loader, then a model object:
ml.print_info();
std::unique_ptr<llama_model> model_ptr(llama_model_create(ml, params));Three loading steps follow in this order: model->load_hparams(ml), model->load_vocab(ml), and model->load_tensors(ml). In plain words: read the model's settings (hyperparameters such as layer count), read the vocabulary (the table of tokens), then read the weights (the big arrays of numbers). The first two are wrapped so that an error names the step ("error loading model hyperparameters: ..." or "error loading model vocabulary: ..."). With vocab_only set, the function returns after the vocabulary, which is how llama-tokenize can work on a vocabulary-only GGUF.
llama_model_create in src/llama-model.cpp reads the architecture from the loader. When the general.architecture string is not a known name, it throws unknown model architecture: '<name>'. For a known architecture, llama_model_mapping is a switch with one case per architecture (152 case labels at this commit) that returns a new object of that architecture's class, for example new llama_model_llama(params).
Each class lives in src/models/, which holds models.h and 156 .cpp files, most of them one architecture each. The Llama file has three parts. load_arch_hparams reads hyperparameters and sets a size label. load_arch_tensors creates every weight tensor with its shape, for example tok_embd as {n_embd, n_vocab}; when the output tensor is missing, it reuses the token embedding (tied weights). The graph constructor builds one forward pass, layer by layer:
// norm
cur = build_norm(inpL,
model.layers[il].attn_norm, NULL,
LLM_NORM_RMS, il);
cb(cur, "attn_norm", il);The rest of a layer follows the same pattern: build_qkv, ggml_rope_ext on Q and K, build_attn, a residual ggml_add named ffn_inp, build_norm again, then build_ffn with LLM_FFN_SILU and LLM_FFN_PAR (SwiGLU) or build_moe_ffn when the layer has an expert router, and a second residual add. Each cb(cur, "name", il) call names a tensor (attn_norm-0, ffn_out-3) for debugging tools such as llama-eval-callback.
The graph constructor uses a few terms from earlier lessons. RMS normalization (LLM_NORM_RMS) rescales a vector so its values have a stable size. RoPE (ggml_rope_ext) encodes token position by rotating pairs of values in Q and K. A residual add (ggml_add) adds a layer's input back to its output. SwiGLU is the feed-forward variant that Llama uses, and an MoE (mixture of experts) layer routes each token to a few expert networks.
The program below reproduces two things. The first is the size label of llama_model_llama::load_arch_hparams, which guesses "7B" or "70B" from the layer count. The second is the architecture lookup of llama_model_create, which maps a name to a class. Look at the last line of the output, where the unknown name fails, and at the 4-layer model that gets ?B.
import java.util.List;
import java.util.Map;
import java.util.function.Function;
public class ArchDispatch {
record HParams(int nLayer, int nVocab, int nHead, int nHeadKv, int nExpert) {}
// load_arch_hparams of llama_model_llama in src/models/llama.cpp: the size label from the layer count.
static String llamaType(HParams h) {
if (h.nExpert() == 8) return switch (h.nLayer()) { case 32 -> "8x7B"; case 56 -> "8x22B"; default -> "?B"; };
return switch (h.nLayer()) {
case 16, 22 -> "1B";
case 26, 28 -> "3B";
case 30 -> "256M";
case 32 -> h.nVocab() == 49152 ? "3B" : (h.nVocab() < 40000 ? "7B" : "8B");
case 36 -> "8B";
case 40 -> "13B";
case 48 -> "34B";
case 60 -> "30B";
case 80 -> h.nHead() == h.nHeadKv() ? "65B" : "70B";
default -> "?B";
};
}
// llama_model_create and llama_model_mapping in src/llama-model.cpp: one class per architecture.
static final Map<String, Function<HParams, String>> MAPPING = Map.of(
"llama", h -> "llama_model_llama, model type = " + llamaType(h),
"qwen2", h -> "llama_model_qwen2");
static String create(String arch, HParams h) {
Function<HParams, String> f = MAPPING.get(arch);
if (f == null) return "error: unknown model architecture: '" + arch + "'";
return f.apply(h);
}
public static void main(String[] args) {
List<Object[]> models = List.of(
new Object[] {"llama", new HParams(16, 128256, 32, 8, 0)},
new Object[] {"llama", new HParams(32, 32000, 32, 32, 0)},
new Object[] {"llama", new HParams(32, 128256, 32, 8, 0)},
new Object[] {"llama", new HParams(80, 32000, 64, 8, 0)},
new Object[] {"llama", new HParams(32, 32000, 32, 8, 8)},
new Object[] {"llama", new HParams(4, 151936, 8, 2, 0)},
new Object[] {"mamba9", new HParams(24, 50280, 0, 0, 0)});
for (Object[] m : models) {
HParams h = (HParams) m[1];
System.out.printf("%-6s n_layer=%-2d n_vocab=%-6d -> %s%n", m[0], h.nLayer(), h.nVocab(), create((String) m[0], h));
}
}
}llama n_layer=16 n_vocab=128256 -> llama_model_llama, model type = 1B
llama n_layer=32 n_vocab=32000 -> llama_model_llama, model type = 7B
llama n_layer=32 n_vocab=128256 -> llama_model_llama, model type = 8B
llama n_layer=80 n_vocab=32000 -> llama_model_llama, model type = 70B
llama n_layer=32 n_vocab=32000 -> llama_model_llama, model type = 8x7B
llama n_layer=4 n_vocab=151936 -> llama_model_llama, model type = ?B
mamba9 n_layer=24 n_vocab=50280 -> error: unknown model architecture: 'mamba9'The label is a lookup from layer count to a familiar model size, because the file does not store the size in billions as a number. Some layer counts are shared by two models, so tie-breakers are needed. For example, 32 layers is "7B" with a vocabulary under 40,000 tokens and "8B" above it. The details follow. 16 layers is "1B". 32 layers is "7B" with a vocabulary under 40,000 tokens and "8B" above it; the code uses the vocabulary size to tell two generations of Llama apart. 80 layers is "65B" when every query head has its own key and value head and "70B" when key and value heads are shared. A model with 8 experts gets "8x7B" at 32 layers. The course's tiny test model, with 4 layers, gets "?B", which matches the real log line of the pinned build for that model: print_info: model type = ?B. The label is only printed; it changes nothing in the computation. The last line shows the error for an architecture string that llama.cpp does not know; llama_model_load catches it and logs error loading model: unknown model architecture: 'mamba9'.
The two pairing styles described next are two ways to apply the same rotation idea, and a model must be run with the style it was trained with. If an engine picks the wrong style, the output degrades even though no error appears. Two more facts from src/llama-model.cpp matter for the Java engine of section 7. llama_model_rope_type returns LLAMA_ROPE_TYPE_NORM for LLM_ARCH_LLAMA (the tiny model's log prints rope type = 0), meaning rotations act on pairs of consecutive values in each head, while LLM_ARCH_QWEN2 and others return LLAMA_ROPE_TYPE_NEOX, which pairs value i with value i + n_rot/2. And the Llama graph uses an attention scale of 1/sqrt(n_embd_head) unless the GGUF sets f_attention_scale.
Example 3: llama.cpp, the KV cache and memory modules
The question is how llama.cpp stores keys and values for several sequences in one cache. Recall why the cache exists: when the model generates token number 100, it needs the keys and values of tokens 1 to 99. Computing them again every step would be wasteful, so they are stored once. With a server, many conversations are active at once, so one cache must hold the data of all of them without mixing them up. Open src/llama-memory.h. llama_memory_i is an interface whose operations work on sequences:
virtual bool seq_rm (llama_seq_id seq_id, llama_pos p0, llama_pos p1) = 0;
virtual void seq_cp (llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) = 0;
virtual void seq_keep(llama_seq_id seq_id) = 0;
virtual void seq_add (llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos shift) = 0;A sequence is one independent token stream with an id, for example one server slot. seq_rm removes a range of positions, seq_cp makes a range belong to a second sequence too, seq_keep drops every other sequence, and seq_add shifts positions (used when the context is full and old tokens are discarded). The same interface has init_batch, which splits a batch into micro-batches and checks that they fit, and state_write/state_read, which save and restore the cache.
Several classes implement the interface. llama_model::create_memory picks one when the context is created (src/llama-context.cpp calls it). Its default branch chooses llama_memory_recurrent for recurrent architectures (Mamba-style layers that keep a fixed-size state instead of keys and values), one of the llama_memory_hybrid classes for architectures that mix recurrent and attention layers, llama_kv_cache_iswa when the model uses sliding-window attention (two caches: a small one for sliding-window layers and a full one for the others), and llama_kv_cache otherwise. Embedding models such as BERT get no memory at all (res = nullptr).
The bookkeeping of llama_kv_cache is in src/llama-kv-cells.h. For each cell (one token's slot in the K and V tensors) it keeps a position and a set of sequence ids:
class llama_kv_cells {
public:
using seq_set_t = std::bitset<LLAMA_MAX_SEQ>;A bitset is a row of on/off flags, one per possible sequence id. If a cell's bitset has flag 3 on, the cell belongs to sequence 3. LLAMA_MAX_SEQ is 256 (src/llama-cparams.h), so each cell carries 256 flags. A position of -1 means the cell is empty (is_empty returns pos[i] == -1). A cell can belong to several sequences at once, which is how a shared prompt prefix is stored once. seq_rm on one cell clears one sequence's bit; when no bit is left, it sets the position to -1 and the cell is free again.
When new tokens arrive, the engine must decide which cells to put them in. A picture helps. The cache is a row of numbered boxes, and a marker called head points to the place where the search for free boxes begins.
cell: 0 1 2 3 4 5 6 7 8 9 10 11
state: A A A A A . . . . . . . A = used by a sequence, . = free
^
head (search starts here, free cells are taken one by one)find_slot in src/llama-kv-cache.cpp chooses cells for a new micro-batch of tokens. A micro-batch is a slice of the batch small enough to be processed at once. It starts at a moving head. If many cells before the head are free, it starts from 0 instead, "hoping to fill it":
// if we have enough unused cells before the current head ->
// better to start searching from the beginning of the cache, hoping to fill it
if (head_cur > cells.get_used() + 2*n_tokens) {
head_cur = 0;
}From there it wraps around at the end and takes empty cells one by one (or a contiguous run when cont is set). With sliding-window attention, a cell whose token has left the window counts as free too. After the batch is stored, apply_ubatch moves head to one past the last cell it used. The real run of the tiny model on the pinned build reports the size of this structure: llama_kv_cache: size = 4.00 MiB (4096 cells, 4 layers, 1/1 seqs), K (f16): 2.00 MiB, V (f16): 2.00 MiB. 4,096 cells times 4 layers times 64 values per cell (2 KV heads of 32 values) times 2 bytes is 2 MiB for K, and the same for V. By hand: 4,096 x 4 x 64 x 2 bytes = 2,097,152 bytes, which is exactly 2 MiB. K and V together give the 4.00 MiB in the log line.
Example 4: llama.cpp's cell search in Java
The question is what the cache looks like after a few requests come and go. Reading the C++ is slow, so this program reproduces the idea on 12 cells that fit on one output line. Watch how two sequences end up interleaved, how a freed sequence leaves holes, and how the head wraps around. The program reproduces find_slot with cont = false, apply_ubatch, seq_rm, and seq_cp on a 12-cell cache, and prints the same view that find_slot writes to the debug log when debugging is on: . for an empty cell, a digit for a cell with one sequence, M for a cell shared by several.
import java.util.ArrayList;
import java.util.BitSet;
import java.util.List;
public class KvCells {
// llama_kv_cells (src/llama-kv-cells.h): per cell a position (-1 = empty) and a bitset of sequence ids.
final int[] pos;
final BitSet[] seq;
int head;
KvCells(int size) {
pos = new int[size];
seq = new BitSet[size];
for (int i = 0; i < size; i++) { pos[i] = -1; seq[i] = new BitSet(); }
}
int used() { int n = 0; for (int p : pos) if (p != -1) n++; return n; }
// find_slot with cont = false, then apply_ubatch: test cells one by one from head, wrapping around.
List<Integer> add(int seqId, int firstPos, int nTokens) {
int h = head > used() + 2 * nTokens ? 0 : head % pos.length; // many free cells before head: restart at 0
List<Integer> idxs = new ArrayList<>();
for (int tested = 0; idxs.size() < nTokens && tested < pos.length; tested++, h = (h + 1) % pos.length)
if (pos[h] == -1) idxs.add(h);
if (idxs.size() < nTokens) throw new IllegalStateException("no slot for " + nTokens + " tokens");
for (int i = 0; i < nTokens; i++) { pos[idxs.get(i)] = firstPos + i; seq[idxs.get(i)].set(seqId); }
head = idxs.getLast() + 1;
return idxs;
}
void seqRm(int seqId) { // a cell becomes empty when its last sequence leaves
for (int i = 0; i < pos.length; i++)
if (seq[i].get(seqId)) { seq[i].clear(seqId); if (seq[i].isEmpty()) pos[i] = -1; }
}
void seqCp(int src, int dst) { for (BitSet s : seq) if (s.get(src)) s.set(dst); }
String map() { // the debug view printed by find_slot
StringBuilder sb = new StringBuilder();
for (int i = 0; i < pos.length; i++)
sb.append(pos[i] == -1 ? '.' : seq[i].cardinality() == 1 ? (char) ('0' + seq[i].nextSetBit(0)) : 'M');
return sb.toString();
}
static void show(String what, List<Integer> cells, KvCells kv) {
System.out.printf("%-24s cells %-16s %s head=%-2d used=%d%n", what, cells, kv.map(), kv.head, kv.used());
}
public static void main(String[] args) {
KvCells kv = new KvCells(12);
show("seq 0 prompt, 5 tokens", kv.add(0, 0, 5), kv);
show("seq 1 prompt, 4 tokens", kv.add(1, 0, 4), kv);
show("seq 0 decode at pos 5", kv.add(0, 5, 1), kv);
show("seq 1 decode at pos 4", kv.add(1, 4, 1), kv);
kv.seqRm(0);
show("seq_rm(0)", List.of(), kv);
show("seq 1 adds pos 5..7", kv.add(1, 5, 3), kv);
kv.seqCp(1, 2);
show("seq_cp(1 -> 2)", List.of(), kv);
}
}seq 0 prompt, 5 tokens cells [0, 1, 2, 3, 4] 00000....... head=5 used=5
seq 1 prompt, 4 tokens cells [5, 6, 7, 8] 000001111... head=9 used=9
seq 0 decode at pos 5 cells [9] 0000011110.. head=10 used=10
seq 1 decode at pos 4 cells [10] 00000111101. head=11 used=11
seq_rm(0) cells [] .....1111.1. head=11 used=5
seq 1 adds pos 5..7 cells [11, 0, 1] 11...1111.11 head=2 used=8
seq_cp(1 -> 2) cells [] MM...MMMM.MM head=2 used=8Each line shows the cache as 12 characters, one per cell, after one operation. The first two prompts fill cells 0 to 4 and 5 to 8 in order, and each decode step takes the next free cell, so after four operations the two sequences are interleaved (00000111101.). Cells do not have to be contiguous per sequence: attention reads each cell's position, and the attention mask hides the cells of other sequences.
seq_rm(0) empties six cells and leaves the head at 11. When sequence 1 adds three tokens, head (11) is not greater than the used cells plus twice the batch (5 + 6 = 11), so the search starts at the head: it takes cell 11, wraps to 0, and takes cells 0 and 1. The new head is 2.
seq_cp(1 -> 2) adds sequence 2 to every cell of sequence 1 without copying any K or V data, so all eight used cells show M. llama-server uses this operation when one request asks for several completions of the same prompt: the prompt is computed once and shared.
Example 5: llama.cpp, sampler implementations
The question is how llama.cpp structures its samplers. Recall that the model's last step produces one score (a logit) for every token in the vocabulary, and that sampling turns those scores into one chosen token. Real engines apply several small steps in a row: remove unlikely tokens, change how sharp the distribution is, then draw one token at random. llama.cpp builds each step as a separate sampler object so they can be combined in any order. The interface is a C struct of function pointers in include/llama.h:
struct llama_sampler_i {
const char * (*name) (const struct llama_sampler * smpl); // can be NULL
void (*accept)( struct llama_sampler * smpl, llama_token token); // can be NULL
void (*apply) ( struct llama_sampler * smpl, llama_token_data_array * cur_p); // requiredapply edits a candidate array of (token, logit, probability) in place: it may sort, rescale, or shrink it. accept tells the sampler which token was finally chosen, which repetition penalties need. A chain is itself a sampler whose apply calls each member in order (llama_sampler_chain_apply in src/llama-sampler.cpp). Most samplers are short. This is the body of top-k:
k = std::min(k, (int) cur_p->size);
// Sort scores in descending order
if (!cur_p->sorted) {
llama_token_data_array_partial_sort_inplace(cur_p, k);
}
cur_p->size = k;Top-k keeps only the k highest-scoring tokens. With k = 40, 40 candidates stay and the rest are dropped. Min-p does not compute probabilities at all. It keeps a token when its logit is at least max_logit + logf(p), which is the same as a probability of at least p times the top probability, because the softmax divides every probability by the same sum. Top-p calls llama_sampler_softmax_impl on the current candidates and keeps the shortest sorted prefix whose cumulative probability reaches p.
The default order is in common/common.h, and --help prints it: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature. common/sampling.cpp adds a logit-bias sampler first when there are biases, then the samplers in that order, and appends dist, which draws the token from the remaining candidates with a seeded random generator. A sampler whose parameters make it a no-op is replaced by an empty sampler whose name starts with ?: llama_sampler_init_top_k returns llama_sampler_init_empty("?top-k") when k <= 0. Real output of the tiny model (random weights) on the pinned build with default settings:
sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> distWith defaults, the repeat penalty (1.0), DRY (multiplier 0), top-n-sigma (-1), typical-p (1.0), and XTC (probability 0) are off, so five members print with ?. Active are top-k (40), top-p (0.95), min-p (0.05), and temperature (0.8, implemented by the sampler named temp-ext), followed by dist. The chain keeps its length and order whatever the parameters, so the printed line is a direct record of which samplers were active.
Example 6: the default sampler order in Java
The question is what it means that temperature comes last in the default order. Temperature divides every logit by a number: above 1 it flattens the distribution so unlikely tokens get more chance, and below 1 it sharpens it. Filters like top-k and top-p remove tokens. Because both kinds of step change the candidate list, the order in which they run changes the result. The program follows llama_sampler_top_k_impl, llama_sampler_top_p_apply, llama_sampler_min_p_apply, and llama_sampler_temp_impl on six candidates, prints the probabilities after each step, and then runs the same samplers with temperature moved to the front (as --samplers "temperature;top_k;top_p;min_p" would). The sampler chain of Penalties, Logit Bias, and Sampler Chains has the same shape; this version only adds the per-step print. In the output, each entry reads id:probability, so 3:0.439 means token 3 has probability 0.439 among the current candidates.
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Locale;
public class ChainOrder {
record Cand(int id, double logit) {}
// Each sampler edits the candidate list, as apply() edits llama_token_data_array in src/llama-sampler.cpp.
interface Sampler { String name(); List<Cand> apply(List<Cand> cur); }
static double[] softmax(List<Cand> cur) { // llama_sampler_softmax_impl: exp(logit - max) / sum
double max = cur.stream().mapToDouble(Cand::logit).max().orElseThrow(), sum = 0;
double[] p = new double[cur.size()];
for (int i = 0; i < p.length; i++) sum += p[i] = Math.exp(cur.get(i).logit() - max);
for (int i = 0; i < p.length; i++) p[i] /= sum;
return p;
}
static Sampler topK(int k) { // sort descending, keep the first k
return new Sampler() {
public String name() { return "top-k"; }
public List<Cand> apply(List<Cand> cur) {
return cur.stream().sorted(Comparator.comparingDouble(Cand::logit).reversed()).limit(k).toList();
}
};
}
static Sampler topP(double p) { // keep the smallest prefix whose probability sum >= p
return new Sampler() {
public String name() { return "top-p"; }
public List<Cand> apply(List<Cand> cur) {
List<Cand> s = cur.stream().sorted(Comparator.comparingDouble(Cand::logit).reversed()).toList();
double[] prob = softmax(s);
double cum = 0;
for (int i = 0; i < s.size(); i++) if ((cum += prob[i]) >= p) return s.subList(0, i + 1);
return s;
}
};
}
static Sampler minP(double p) { // keep logit >= max_logit + ln(p), that is prob >= p * max_prob
return new Sampler() {
public String name() { return "min-p"; }
public List<Cand> apply(List<Cand> cur) {
double min = cur.stream().mapToDouble(Cand::logit).max().orElseThrow() + Math.log(p);
return cur.stream().filter(c -> c.logit() >= min).toList();
}
};
}
static Sampler temp(double t) { // divide every logit by t
return new Sampler() {
public String name() { return "temp"; }
public List<Cand> apply(List<Cand> cur) { return cur.stream().map(c -> new Cand(c.id(), c.logit() / t)).toList(); }
};
}
static void run(String title, List<Sampler> chain, List<Cand> logits) {
System.out.println(title);
List<Cand> cur = logits;
for (Sampler s : chain) {
cur = s.apply(cur);
double[] p = softmax(cur);
List<String> shown = new ArrayList<>();
for (int i = 0; i < cur.size(); i++) shown.add(String.format(Locale.ROOT, "%d:%.3f", cur.get(i).id(), p[i]));
System.out.printf(" after %-6s %s%n", s.name(), shown);
}
}
public static void main(String[] args) {
double[] values = {2.0, 1.5, 0.3, 3.1, -1.0, 2.8};
List<Cand> logits = new ArrayList<>();
for (int i = 0; i < values.length; i++) logits.add(new Cand(i, values[i]));
run("default order: top_k;top_p;min_p;temperature (k=4, p=0.90, min_p=0.20, temp=2.0)",
List.of(topK(4), topP(0.90), minP(0.20), temp(2.0)), logits);
run("temperature first: temperature;top_k;top_p;min_p (same values)",
List.of(temp(2.0), topK(4), topP(0.90), minP(0.20)), logits);
}
}default order: top_k;top_p;min_p;temperature (k=4, p=0.90, min_p=0.20, temp=2.0)
after top-k [3:0.439, 5:0.326, 0:0.146, 1:0.089]
after top-p [3:0.482, 5:0.357, 0:0.161]
after min-p [3:0.482, 5:0.357, 0:0.161]
after temp [3:0.410, 5:0.353, 0:0.237]
temperature first: temperature;top_k;top_p;min_p (same values)
after temp [0:0.177, 1:0.138, 2:0.076, 3:0.307, 4:0.039, 5:0.264]
after top-k [3:0.346, 5:0.298, 0:0.200, 1:0.156]
after top-p [3:0.346, 5:0.298, 0:0.200, 1:0.156]
after min-p [3:0.346, 5:0.298, 0:0.200, 1:0.156]In the default order, top-k keeps the four largest logits (ids 3, 5, 0, 1). Top-p computes probabilities over those four with the original logits: 0.439, then 0.765 after two, then 0.911 after three, which reaches 0.90, so token 1 is dropped. Min-p keeps every token with at least 0.20 times the top probability (0.2 x 0.482 = 0.096), and all three pass. Only then does temperature 2.0 divide the logits, which flattens the distribution over the three survivors: token 0 rises from 0.161 to 0.237.
With temperature first, the logits are halved before any filter runs. Top-k still keeps the same four ids, but their probabilities are now flatter (0.346, 0.298, 0.200, 0.156), and the cumulative sum reaches 0.90 only with all four, so top-p keeps token 1 too. The same parameters give four candidates instead of three. With temperature last, a high temperature reweights the surviving tokens but cannot bring back tokens that the filters removed.
Example 7: llama.cpp, server slots and the task queue
The question is how llama-server runs several requests on one model. Two users may send prompts at the same moment, and the model's weights should be loaded only once. The server therefore keeps one model in memory and divides its context among several independent workspaces. Open tools/server/server-queue.cpp. server_queue::start_loop is the main loop:
while (true) {
QUE_DBG("%s", "processing new tasks\n");
if (process_new_tasks(false)) {
break; // terminate
}
// all tasks in the current loop is processed, slots data is now ready
QUE_DBG("%s", "update slots\n");
// this will run the main inference process for all slotsHTTP handler threads turn each request into a task and post it to this queue. One loop thread assigns each new task to a slot (a share of the context with its own sequence id; --parallel sets the number of slots), then calls update_slots() in server-context.cpp. update_slots first adds the last sampled token of every generating slot to the batch, then fills the rest of the batch, up to n_batch tokens, with prompt tokens of slots that are still reading their prompt. It calls llama_decode and samples one token per slot whose prompt is done. When no task is waiting, the loop waits on a condition variable.
Slot choice is in get_available_slot. If the request names a slot (id_slot), that slot is used. Otherwise each idle slot is scored by how much of the new prompt matches the start of the tokens it already holds:
// fraction of the Longest Common Prefix length with respect to the input prompt length
const size_t lcp_len = tokens.get_common_prefix(task.tokens);
const float f_sim_cur = float(lcp_len) / task.tokens.size();A slot qualifies when this fraction is above slot_prompt_similarity (default 0.10, --slot-prompt-similarity). With no qualifying slot, the least recently used idle slot is taken. When no slot is idle, process_single_task calls queue_tasks.defer(...), and the task waits until a slot frees. The reason to score slots by shared prefix is speed. If a new prompt starts with the same tokens that a slot already holds, those tokens are already in the KV cache, and only the new tokens need computing. The program reproduces these rules with four requests. Each output line names the chosen slot and the reason. f_sim_best is the match fraction, and f_keep is the share of the slot's cached tokens that stay useful.
import java.util.List;
import java.util.Locale;
public class SlotSelect {
static class Slot {
final int id; int[] tokens; long tLast; boolean processing;
Slot(int id, int[] tokens, long tLast) { this.id = id; this.tokens = tokens; this.tLast = tLast; }
}
static int commonPrefix(int[] a, int[] b) {
int n = 0;
while (n < a.length && n < b.length && a[n] == b[n]) n++;
return n;
}
// get_available_slot in tools/server/server-context.cpp, without the explicit id_slot case.
static String choose(List<Slot> slots, int[] prompt, float similarity) {
Slot best = null;
float fBest = 0;
for (Slot s : slots) {
if (s.processing || s.tokens.length == 0) continue;
float f = (float) commonPrefix(s.tokens, prompt) / prompt.length;
if (f > fBest && f > similarity) { fBest = f; best = s; }
}
if (best != null) {
float fKeep = fBest * prompt.length / best.tokens.length;
return String.format(Locale.ROOT, "slot %d by LCP similarity, f_sim_best = %.3f, f_keep = %.3f%s",
best.id, fBest, fKeep, fKeep < 0.5f ? ", update_cache" : "");
}
Slot lru = null;
for (Slot s : slots) if (!s.processing && (lru == null || s.tLast <= lru.tLast)) lru = s;
return lru == null ? "no idle slot: the task waits in the queue" : "slot " + lru.id + " by LRU, update_cache";
}
static int[] seq(int start, int len) { int[] t = new int[len]; for (int i = 0; i < len; i++) t[i] = start + i; return t; }
static int[] concat(int[] a, int[] b) {
int[] r = java.util.Arrays.copyOf(a, a.length + b.length);
System.arraycopy(b, 0, r, a.length, b.length);
return r;
}
public static void main(String[] args) {
int[] system = seq(1000, 80); // a shared 80-token system prompt
List<Slot> slots = List.of(new Slot(0, concat(system, seq(7000, 300)), 10),
new Slot(1, concat(system, seq(5000, 40)), 30),
new Slot(2, new int[0], 5));
System.out.println("turn 2 of chat A: " + choose(slots, concat(concat(system, seq(5000, 40)), seq(9000, 20)), 0.1f));
System.out.println("new chat, same sys: " + choose(slots, concat(system, seq(8000, 10)), 0.1f));
System.out.println("unrelated prompt: " + choose(slots, seq(1, 50), 0.1f));
slots.forEach(s -> s.processing = true);
System.out.println("all slots busy: " + choose(slots, seq(1, 50), 0.1f));
}
}turn 2 of chat A: slot 1 by LCP similarity, f_sim_best = 0.857, f_keep = 1.000
new chat, same sys: slot 0 by LCP similarity, f_sim_best = 0.889, f_keep = 0.211, update_cache
unrelated prompt: slot 2 by LRU, update_cache
all slots busy: no idle slot: the task waits in the queueTurn 2 of chat A repeats chat A's 120 tokens and adds 20. Slot 1 holds exactly those 120 tokens, so its fraction is 120/140 = 0.857, higher than slot 0's 80/140, and f_keep (the share of the slot's cached tokens that stay useful) is 1.000: only the 20 new tokens need computing.
A new chat with the same 80-token system prompt matches slots 0 and 1 equally (80/90 = 0.889). The comparison is strict (f_sim_cur > f_sim_best), so the first slot in the list wins. Slot 0 holds 380 tokens, and only 80 of them stay useful, so f_keep is 0.211. Below 0.5 the server sets update_cache: before the slot's old prompt is overwritten, it is saved to the RAM prompt cache. That cache is on by default (--cache-ram, 8192 MiB) and is used only for completion tasks.
An unrelated prompt matches nothing, so the LRU rule picks slot 2, which has the oldest t_last_used (5). An LRU pick also sets update_cache. When every slot is busy, the task returns to the queue and waits; llama-server: Endpoints, Slots, and Prompt Cache shows this in the metrics as requests_deferred.
Example 8: llama.cpp, chat and tool-call parsing
The question is how llama-server knows where a tool call starts in a model's output for a chat template it has no code for. A chat template is the recipe that turns a list of messages into the exact text a model was trained on, with special markers around each role. Every model family writes tool calls in its own format, for example inside <tool_call> tags or after [TOOL_CALLS]. To return a clean tool_calls field to the client, the server must parse that format back out of the model's text. At this commit, common/chat.cpp handles a few template families with dedicated code. common_chat_try_specialized_template checks the template source for markers, such as <|channel|> for GPT-OSS and [TOOL_CALLS] with [ARGS] (and no [CALL_ID]) for Ministral. Every other template is analyzed automatically by common/chat-diff-analyzer.cpp. The analyzer renders the template several times with sentinel strings that cannot occur in real text:
static const std::string FUN_FIRST = "FFF_FIRST_FUN_F";
static const std::string FUN_SECOND = "SSS_SECOND_FUN_S";
static const std::string ARG_FIRST = "AA_ARG_FST_AA";
static const std::string ARG_SECOND = "BB_ARG_SND_BB";
static const std::string USER_MSG = "U_USER_MSG Hello END_U";analyze_tool_calls renders a user message followed by an assistant message with plain content (A_ASST_MSG I can help END_A). Then it renders the same conversation with an assistant message whose content is empty and which holds one tool call to FFF_FIRST_FUN_F with the arguments AA_ARG_FST_AA: "XXXX" and BB_ARG_SND_BB: "YYYY". calculate_diff_split in common/chat-auto-parser-helpers.cpp splits the two outputs into a common prefix, a common suffix, and the two differing middles. The middle of the second rendering is this template's tool-call syntax. Further comparisons (two calls versus one, other argument values, another call id) find the markers around names, arguments, and repeated calls, and autoparser::peg_generator::generate_parser builds a PEG parser from them.
The idea works like a spot-the-difference puzzle. Render the same conversation twice, once with plain text and once with a tool call, and whatever differs between the two outputs is the tool-call syntax. Sentinels make the differing text easy to find, because they cannot appear anywhere else. The program reproduces the first comparison for the assistant part of models/templates/Qwen-Qwen2.5-7B-Instruct.jinja, written out in Java, with a character-level diff (the real function first splits the text into marker segments and compares those). The output prints the four parts of the diff (common prefix, common suffix, and the two different middles), then cuts the tool-call middle into three pieces.
public class TemplateDiff {
// Sentinel strings from common/chat-diff-analyzer.cpp; they cannot occur in a real template.
static final String ASSISTANT_MSG = "A_ASST_MSG I can help END_A";
static final String FUN_FIRST = "FFF_FIRST_FUN_F", ARG_FIRST = "AA_ARG_FST_AA", ARG_SECOND = "BB_ARG_SND_BB";
// The assistant turn of models/templates/Qwen-Qwen2.5-7B-Instruct.jinja, written out by hand.
static String renderAssistant(String content, String toolName, String argsJson) {
StringBuilder sb = new StringBuilder("<|im_start|>assistant");
if (toolName == null) return sb.append("\n").append(content).append("<|im_end|>\n").toString();
if (!content.isEmpty()) sb.append("\n").append(content);
sb.append("\n<tool_call>\n{\"name\": \"").append(toolName).append("\", \"arguments\": ")
.append(argsJson).append("}\n</tool_call>");
return sb.append("<|im_end|>\n").toString();
}
record Diff(String prefix, String suffix, String left, String right) {}
// A character-level version of calculate_diff_split (common/chat-auto-parser-helpers.cpp).
static Diff diff(String l, String r) {
int p = 0;
while (p < l.length() && p < r.length() && l.charAt(p) == r.charAt(p)) p++;
int s = 0;
while (s < l.length() - p && s < r.length() - p
&& l.charAt(l.length() - 1 - s) == r.charAt(r.length() - 1 - s)) s++;
return new Diff(l.substring(0, p), l.substring(l.length() - s), l.substring(p, l.length() - s),
r.substring(p, r.length() - s));
}
static String show(String s) { return "[" + s.replace("\n", "\\n") + "]"; }
public static void main(String[] args) {
String withoutTools = renderAssistant(ASSISTANT_MSG, null, null);
String withTools = renderAssistant("", FUN_FIRST,
"{\"" + ARG_FIRST + "\": \"XXXX\", \"" + ARG_SECOND + "\": \"YYYY\"}");
Diff d = diff(withoutTools, withTools);
System.out.println("prefix: " + show(d.prefix()));
System.out.println("suffix: " + show(d.suffix()));
System.out.println("left: " + show(d.left()));
System.out.println("right: " + show(d.right()));
String right = d.right();
System.out.println("before name: " + show(right.substring(0, right.indexOf(FUN_FIRST))));
System.out.println("between: " + show(right.substring(right.indexOf(FUN_FIRST) + FUN_FIRST.length(),
right.indexOf("{\"" + ARG_FIRST))));
System.out.println("after args: " + show(right.substring(right.lastIndexOf("\"YYYY\"}") + 7)));
}
}prefix: [<|im_start|>assistant\n]
suffix: [<|im_end|>\n]
left: [A_ASST_MSG I can help END_A]
right: [<tool_call>\n{"name": "FFF_FIRST_FUN_F", "arguments": {"AA_ARG_FST_AA": "XXXX", "BB_ARG_SND_BB": "YYYY"}}\n</tool_call>]
before name: [<tool_call>\n{"name": "]
between: [", "arguments": ]
after args: [}\n</tool_call>]The common prefix is <|im_start|>assistant\n, the start of every assistant turn in this template. The common suffix is <|im_end|>\n. The left middle is the sentinel content, and the right middle is the whole tool-call section. Because the sentinels are unique strings, finding them in the right middle gives the exact text before the name (<tool_call>\n{"name": "), between the name and the arguments (", "arguments": ), and after the arguments (}\n</tool_call>). These three strings are what a parser for this template needs, and they were found without any code written for Qwen.
A new template therefore works without a code change, as long as it renders tool calls from structured messages; a template that ignores tool_calls produces no difference, and analyze_tool_calls finds nothing. The list of workarounds at the top of chat-diff-analyzer.cpp (patches for old Qwen and DeepSeek reasoning templates, Granite 3.3, DeepSeek-R1-Distill-Qwen, and others) shows the cases where the analysis alone was not enough.
Example 9: ggml, one operator from definition to CPU kernel
The question is what code runs between ggml_rms_norm in a graph and the floating-point loop on the CPU. RMS normalization (RMSNorm) divides each row of numbers by the row's root mean square, so that the values keep their direction but have a stable size. The formula is y = x / sqrt(mean(x^2) + eps), where eps is a tiny constant that prevents division by zero. For the row (1, 2, 3, 4): the squares are 1, 4, 9, 16, their mean is 7.5, and 1/sqrt(7.5) is about 0.3651, so the row becomes (0.3651, 0.7303, 1.0954, 1.4606). The operator is a good sample because it is small, and every ggml operator takes the same path. It goes through four files.
The declaration is in ggml/include/ggml.h: ggml_rms_norm(ctx, a, eps). The definition in ggml/src/ggml.c computes nothing; it creates the result tensor and records the operation:
struct ggml_tensor * result = inplace ? ggml_view_tensor(ctx, a) : ggml_dup_tensor(ctx, a);
ggml_set_op_params(result, &eps, sizeof(eps));
result->op = GGML_OP_RMS_NORM;
result->src[0] = a;The caller in llama.cpp is build_norm in src/llama-graph.cpp, which passes hparams.f_norm_rms_eps (read from the GGUF key <arch>.attention.layer_norm_rms_epsilon) and then multiplies by the norm weight with ggml_mul.
When the graph runs on the CPU backend, ggml/src/ggml-cpu/ggml-cpu.c handles the node in two places. While planning, ggml_get_n_tasks sets the number of tasks: GGML_OP_RMS_NORM is in a group that gets n_tasks = n_threads. While computing, the graph walker first calls ggml_cpu_try_fuse_ops. When the next node is a GGML_OP_MUL by an F32 weight of the same width, and fusion is not disabled, it calls ggml_compute_forward_rms_norm_mul_fused and skips the multiply node. Otherwise ggml_compute_forward switches on tensor->op and calls ggml_compute_forward_rms_norm(params, tensor). Since build_norm creates exactly that pair (ggml_rms_norm, then ggml_mul), an RMSNorm with its weight can run on the CPU as one pass over the data.
The kernel is ggml_compute_forward_rms_norm_f32 in ggml/src/ggml-cpu/ops.cpp. Each thread gets its index ith and the thread count nth:
for (int64_t i01 = ith; i01 < ne01; i01 += nth) {
const float * x = (float *) ((char *) src0->data + i01*nb01 + i02*nb02 + i03*nb03);
ggml_float sum = 0.0;ggml_float is double (typedef double ggml_float; in ggml/src/ggml-cpu/vec.h), so the sum of squares is accumulated in double precision, while the mean and the scale are float. The program follows this kernel without the fused weight: two threads, five rows, rows split by index stride. Each log line says which thread handled which row and the scale it used. After the log, the program prints the normalized rows.
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
public class RmsNormThreads {
// ggml_compute_forward_rms_norm_f32 (ggml/src/ggml-cpu/ops.cpp): thread ith of nth takes rows ith, ith+nth, ...
// The sum of squares is a double (ggml_float); mean and scale are float.
static void rmsNorm(float[][] x, float[][] y, float eps, int ith, int nth, List<String> log) {
for (int i01 = ith; i01 < x.length; i01 += nth) {
double sum = 0.0;
for (float v : x[i01]) sum += (double) (v * v);
float mean = (float) (sum / x[i01].length);
float scale = (float) (1.0 / Math.sqrt(mean + eps));
for (int i00 = 0; i00 < x[i01].length; i00++) y[i01][i00] = x[i01][i00] * scale;
synchronized (log) { log.add(String.format(Locale.ROOT, "row %d by thread %d, scale %.4f", i01, ith, scale)); }
}
}
public static void main(String[] args) throws InterruptedException {
float[][] x = { {1, 2, 3, 4}, {0.5f, 0.5f, 0.5f, 0.5f}, {-2, 0, 2, 0}, {10, 0, 0, 0}, {3, 3, 3, 3} };
float[][] y = new float[x.length][x[0].length];
float eps = 1e-5f; // f_norm_rms_eps of the tiny model
int nth = 2; // n_tasks = n_threads for GGML_OP_RMS_NORM
List<String> log = new ArrayList<>();
List<Thread> threads = new ArrayList<>();
for (int ith = 0; ith < nth; ith++) {
int t = ith;
threads.add(Thread.ofPlatform().start(() -> rmsNorm(x, y, eps, t, nth, log)));
}
for (Thread t : threads) t.join();
log.stream().sorted().forEach(System.out::println);
for (float[] row : y) {
StringBuilder sb = new StringBuilder();
for (float v : row) sb.append(String.format(Locale.ROOT, "%8.4f", v));
System.out.println(sb);
}
}
}row 0 by thread 0, scale 0.3651
row 1 by thread 1, scale 2.0000
row 2 by thread 0, scale 0.7071
row 3 by thread 1, scale 0.2000
row 4 by thread 0, scale 0.3333
0.3651 0.7303 1.0954 1.4606
1.0000 1.0000 1.0000 1.0000
-1.4142 0.0000 1.4142 0.0000
2.0000 0.0000 0.0000 0.0000
1.0000 1.0000 1.0000 1.0000Thread 0 takes rows 0, 2, and 4, and thread 1 takes rows 1 and 3: the loop starts at ith and steps by nth, so rows are interleaved rather than split into halves. No locks are needed for the data, because each thread writes only its own rows (the program locks only its log list). The log is sorted before printing because the two threads finish in any order.
Each scale is 1/sqrt(mean of squares + eps). Row 0 (1, 2, 3, 4) has a mean of squares of 7.5, so the scale is 0.3651. Row 1 has a mean of 0.25 and a scale of 2.0000 (eps changes only the fifth decimal). Row 3 has one value of 10, a mean of 25, and a scale of 0.2. Rows 1 and 4 both normalize to all ones, since RMSNorm removes the overall size of a vector and keeps its direction.
During decoding a batch often has one row, so only thread 0 has RMSNorm work; the split pays off during prompt processing. The same path (declaration, graph node, CPU dispatch, kernel per type) holds for every ggml operator. ggml_mul_mat adds one more step: it chooses a dot-product function by weight type through type_traits_cpu (Example 18), and it splits rows into chunks that threads claim from a shared counter, as Quantized Kernels, the Vector API, and Threads reproduced.
Example 10: vLLM, engine core and scheduler
The question is what one vLLM engine step does. vLLM serves many users on GPUs, so its main loop repeats one cycle: choose which requests to advance, run the model once for all of them, and record the new tokens. This is called continuous batching, because requests join and leave the batch between steps instead of waiting for a whole batch to finish. vLLM 0.30.0 runs the HTTP server (entrypoints/openai/api_server.py) in one process. It turns requests into engine requests (v1/engine/async_llm.py, v1/engine/input_processor.py) and sends them over ZeroMQ (v1/engine/core_client.py imports zmq) to an engine core process. That process runs EngineCoreProc.run_busy_loop in v1/engine/core.py: read the input queue until there is work, then step. The step is short:
scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
future = self.model_executor.execute_model(scheduler_output, non_block=True)
grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)Then it waits for the model output, calls sample_tokens when the model returned no tokens yet, processes aborts that arrived during the step, and calls self.scheduler.update_from_output(...), which appends the new tokens, checks stop conditions, and frees finished requests. The executor sends the work to GPU workers (v1/worker/gpu_worker.py, v1/worker/gpu_model_runner.py), which run the model code in model_executor/models/.
The scheduler is Scheduler.schedule in v1/core/sched/scheduler.py. Its opening comment states the idea:
# NOTE(woosuk) on the scheduling algorithm:
# There's no "decoding phase" nor "prefill phase" in the scheduler.
# Each request just has the num_computed_tokens and
# num_tokens_with_spec. num_tokens_with_spec =
# len(prompt_token_ids) + len(output_token_ids) + len(spec_token_ids).The comment says that the scheduler does not treat prefill (reading the prompt) and decode (generating tokens) as separate phases. Each request only records how many tokens it has computed and how many it has in total. The scheduler's job is to close that gap, a few tokens per step. A new request has a large gap (its whole prompt), and a generating request has a gap of one token.
Each step has a token budget (max_num_batched_tokens), which limits how many tokens one model run may process. Running requests are served first, in order. Each gets min(tokens it is behind, remaining budget), so a long prompt is processed over several steps (chunked prefill, taught in Batching and Scheduling). For each one, kv_cache_manager.allocate_slots must find KV blocks. When it cannot, the scheduler preempts a request: with the default first-come-first-served policy it pops the last running request. _preempt_request frees that request's blocks, sets request.num_computed_tokens = 0, and calls self.waiting.prepend_request(request), which puts it at the front of the waiting queue. Waiting requests are admitted next, but only under this condition:
# Next, schedule the WAITING requests.
if not preempted_reqs and self._pause_state == PauseState.UNPAUSED:A step that preempted anything admits nobody new. When a waiting request cannot get blocks, the loop stops at it, so later waiting requests wait too. The reasoning is that memory is short: admitting a new request in a step that just evicted another would repeat the same problem. The cost is that a request can wait behind one that does not fit yet.
Example 11: vLLM's scheduler step in Java
The question is how a step-by-step schedule looks when the KV cache runs out. To preempt a request means to pause it and take its memory back. The paused request keeps its place in line, but its KV blocks are gone, so later it must recompute everything from the start. A block is a fixed-size chunk of KV cache (here 4 tokens), and a request owns as many blocks as it needs. The program reproduces the running-first loop, chunked prefill, preemption of the last running request, and the "no admission after preemption" rule, with a 16-token budget and a cache of six 4-token blocks. Prefix caching is left out here (Example 12 reads it). A request samples a token in a step that brings its computed tokens up to its total tokens. Each output line shows one step: A+10 means request A computed 10 tokens in that step, and the line also shows free blocks and the waiting queue.
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
public class VllmStep {
static final int BLOCK = 4, BUDGET = 16; // block_size, max_num_batched_tokens
static int freeBlocks = 6; // a tiny KV cache: 6 blocks of 4 tokens
static class Req {
final String id; final int prompt, maxTokens; int output, computed, blocks;
Req(String id, int prompt, int maxTokens) { this.id = id; this.prompt = prompt; this.maxTokens = maxTokens; }
int tokens() { return prompt + output; }
}
static boolean allocate(Req r, int newTokens) { // kv_cache_manager.allocate_slots, reduced
int need = (r.computed + newTokens + BLOCK - 1) / BLOCK - r.blocks;
if (need > freeBlocks) return false;
freeBlocks -= Math.max(need, 0); r.blocks += Math.max(need, 0);
return true;
}
static void preempt(Req r, List<Req> running, Deque<Req> waiting) {
running.remove(r); freeBlocks += r.blocks; r.blocks = 0; r.computed = 0; // recompute from scratch later
waiting.addFirst(r);
}
public static void main(String[] args) {
List<Req> running = new ArrayList<>();
Deque<Req> waiting = new ArrayDeque<>(List.of(new Req("A", 10, 6), new Req("B", 9, 4), new Req("C", 6, 2)));
for (int step = 1; !running.isEmpty() || !waiting.isEmpty(); step++) {
int budget = BUDGET;
StringBuilder line = new StringBuilder();
boolean preempted = false;
for (int i = 0; i < running.size() && budget > 0; i++) { // 1) RUNNING requests first
Req r = running.get(i);
int n = Math.min(r.tokens() - r.computed, budget);
while (!allocate(r, n)) {
Req victim = running.getLast(); // FCFS: preempt the newest
preempt(victim, running, waiting); preempted = true;
line.append("preempt ").append(victim.id).append("; ");
if (victim == r) break;
}
if (!running.contains(r)) break;
r.computed += n; budget -= n; line.append(r.id).append('+').append(n).append(' ');
}
while (!preempted && budget > 0 && !waiting.isEmpty()) { // 2) WAITING, only if nobody was preempted
Req r = waiting.peekFirst();
int n = Math.min(r.tokens() - r.computed, budget);
if (!allocate(r, n)) break;
waiting.pollFirst(); running.add(r);
r.computed += n; budget -= n; line.append(r.id).append('+').append(n).append(' ');
}
for (Req r : running) if (r.computed == r.tokens()) r.output++; // a full prefill or decode samples a token
for (Req r : List.copyOf(running))
if (r.output == r.maxTokens) { running.remove(r); freeBlocks += r.blocks; line.append("(").append(r.id).append(" done) "); }
System.out.printf("step %d: %-28s free blocks %d, waiting %s%n", step, line.toString().strip(), freeBlocks,
waiting.stream().map(r -> r.id).toList());
}
}
}step 1: A+10 B+6 free blocks 1, waiting [C]
step 2: A+1 B+3 free blocks 0, waiting [C]
step 3: A+1 B+1 free blocks 0, waiting [C]
step 4: preempt B; A+1 free blocks 2, waiting [B, C]
step 5: A+1 free blocks 2, waiting [B, C]
step 6: A+1 (A done) free blocks 6, waiting [B, C]
step 7: B+11 C+5 free blocks 1, waiting []
step 8: B+1 C+1 (B done) free blocks 4, waiting []
step 9: C+1 (C done) free blocks 6, waiting []Step 1 serves the waiting queue: A's whole 10-token prompt (3 blocks) and the first 6 of B's 9 prompt tokens (2 blocks), which uses the 16-token budget, so C waits. One block is left. Step 2 gives A one decode token and B the rest of its prompt (3 tokens, one more block). C needs 2 blocks for its 6-token prompt, none are free, so it keeps waiting while 12 tokens of budget stay unused.
Step 4 is the preemption. A's next token is its 13th, which needs a fourth block, and no block is free. The scheduler preempts the last running request, B, frees its 3 blocks, and gives A one. Since something was preempted, no waiting request is admitted in this step. B is now first in the waiting queue, ahead of C. It needs 3 blocks for its 11 tokens (9 prompt and 2 already generated), and only 2 are free until A finishes in step 6. C cannot pass B, because the waiting loop stops at the first request that does not fit.
In step 7, B is admitted with all 11 tokens computed again from scratch, and C gets the remaining 5 tokens of budget as a chunk; C's last prompt token runs in step 8. Preemption by recomputation costs exactly this: B's prompt and its generated tokens are processed twice. In the real engine, prefix caching can return some of B's freed blocks from the cache, if no other request reused them in between.
Example 12: vLLM, the KV cache manager and prefix caching
The question is how vLLM reuses KV blocks across requests and what it evicts first. Many requests start with the same text, such as a shared system prompt. If the blocks that hold the KV data for that text are kept after the first request finishes, the next request can use them and skip the computation. This is prefix caching. It needs two things: a way to recognize that a block holds the same prefix (a hash), and a rule for which unused block to throw away when space runs out (an eviction order). Open v1/core/kv_cache_utils.py and v1/core/block_pool.py. Every block is a KVCacheBlock with a block_id, a reference count ref_cnt, and, once it is full and cached, a block hash. hash_block_tokens computes that hash from the parent block's hash, the block's tokens, and extra keys (LoRA adapter names, multimodal inputs, cache salt):
if not parent_block_hash:
parent_block_hash = NONE_HASH
curr_block_token_ids_tuple = tuple(curr_block_token_ids)
return BlockHash(
hash_function((parent_block_hash, curr_block_token_ids_tuple, extra_keys))
)Because each hash includes its parent's, a hash identifies the whole prefix up to the end of its block. This matters because the same four tokens can appear after different earlier text, and their KV data would differ. Two blocks match only when everything before them matches too. Paged KV Cache and Prefix Caching built this chain.
Free blocks are kept in FreeKVCacheBlockQueue, a doubly linked list. Its docstring states the eviction order:
The queue is ordered by block ID in the beginning. When a block is allocated
and then freed, it will be appended back with the eviction order:
1. The least recent used block is at the front (LRU).A freed block keeps its hash and stays findable until it is reused. Four functions do the work.
get_computed_blocksinkv_cache_manager.pyfinds the longest run of cached full blocks for a new request, capped atrequest.num_tokens - 1.touchinblock_pool.pyraises a hit block'sref_cntand removes it from the free queue if it was there.get_new_blockspops blocks from the front of the queue and evicts each one's old hash (_maybe_evict_cached_block).free_blockslowersref_cnt; a block that reaches 0 goes to the front of the queue if it has no hash ("LIFO reuse of non-cached blocks") and to the back if it has one ("FIFO reuse of cached blocks for LRU eviction behavior"). The caller passes a request's blocks in reverse order, so the tail of a prefix sits before its head in the queue.
The cap has a comment that explains its cost:
# NOTE: When all tokens hit the cache, we must recompute the last token
# to obtain logits. Thus, set max_cache_hit_length to prompt_length - 1.
# This can trigger recomputation of an entire block, rather than just
# the single last token, because allocate_slots() requires
# num_computed_tokens to be block-size aligned.Example 13: vLLM's block pool in Java
The question is which blocks a sequence of requests hits, recomputes, and evicts. A hit is a block found in the cache and reused. Read the four requests in the output as a story: A fills the cache, B reuses part of A's prefix, C reuses a smaller part than you might expect, and D forces an eviction. The program reproduces the four functions of Example 12 with 4-token blocks and an 8-block pool. The hash chain uses Java's String.hashCode of the parent hash and the block's tokens instead of vLLM's hash function; only equality matters.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
public class BlockPool {
static final int BLOCK = 4;
static class Block { final int id; int refCnt; String hash; Block(int id) { this.id = id; } }
final LinkedList<Block> freeQueue = new LinkedList<>(); // FreeKVCacheBlockQueue: front = evict first
final Map<String, List<Block>> cached = new HashMap<>(); // block hash -> blocks (no de-duplication)
BlockPool(int n) { for (int i = 0; i < n; i++) freeQueue.add(new Block(i)); }
static List<String> hashes(int[] tokens) { // hash_block_tokens: parent hash + block tokens
List<String> out = new ArrayList<>();
String parent = "none";
for (int s = 0; s + BLOCK <= tokens.length; s += BLOCK)
out.add(parent = Integer.toHexString((parent + Arrays.toString(Arrays.copyOfRange(tokens, s, s + BLOCK))).hashCode()));
return out;
}
List<Block> allocate(String name, int[] tokens) {
List<String> hs = hashes(tokens);
List<Block> blocks = new ArrayList<>();
int maxHit = (tokens.length - 1) / BLOCK; // the last token is always recomputed
for (int i = 0; i < Math.min(hs.size(), maxHit) && cached.containsKey(hs.get(i)); i++) {
Block b = cached.get(hs.get(i)).getFirst();
if (b.refCnt == 0) freeQueue.remove(b); // touch(): a hit leaves the free queue
b.refCnt++; blocks.add(b);
}
int hit = blocks.size();
for (int i = hit; i < (tokens.length + BLOCK - 1) / BLOCK; i++) {
Block b = freeQueue.removeFirst(); // get_new_blocks: pop the LRU block
if (b.hash != null) { // _maybe_evict_cached_block
cached.get(b.hash).remove(b);
if (cached.get(b.hash).isEmpty()) cached.remove(b.hash);
System.out.println(" evict block " + b.id); b.hash = null;
}
b.refCnt = 1;
if (i < hs.size()) { b.hash = hs.get(i); cached.computeIfAbsent(b.hash, k -> new ArrayList<>()).add(b); }
blocks.add(b);
}
System.out.printf("%s: %d tokens, cached %d, computed %d, blocks %s, free %d%n", name, tokens.length,
hit * BLOCK, tokens.length - hit * BLOCK, blocks.stream().map(b -> b.id).toList(), freeQueue.size());
return blocks;
}
void free(List<Block> blocks) { // free_blocks: tail block first in the queue
for (Block b : blocks.reversed()) if (--b.refCnt == 0) {
if (b.hash == null) freeQueue.addFirst(b); else freeQueue.addLast(b);
}
}
static int[] tokens(int shared, int own, int seed) {
int[] t = new int[shared + own];
for (int i = 0; i < t.length; i++) t[i] = i < shared ? 100 + i : seed + i;
return t;
}
public static void main(String[] args) {
BlockPool pool = new BlockPool(8);
List<Block> a = pool.allocate("req A", tokens(8, 5, 500));
pool.free(a);
List<Block> b = pool.allocate("req B", tokens(8, 3, 900));
pool.allocate("req C", tokens(8, 0, 0));
pool.allocate("req D", tokens(0, 15, 300));
}
}req A: 13 tokens, cached 0, computed 13, blocks [0, 1, 2, 3], free 4
req B: 11 tokens, cached 8, computed 3, blocks [0, 1, 3], free 5
req C: 8 tokens, cached 4, computed 4, blocks [0, 4], free 4
evict block 2
req D: 15 tokens, cached 0, computed 15, blocks [5, 6, 7, 2], free 0Request A has 13 tokens: 3 full blocks (0, 1, 2) that get hashes and a partial block (3) that does not. When A finishes, free walks the blocks in reverse: block 3 goes to the front of the free queue, and blocks 2, 1, 0 go to the back in that order, so the tail of A's prefix will be evicted before its head.
Request B shares A's first 8 tokens. Blocks 0 and 1 are hits, so 8 tokens are cached and only 3 are computed, in one new block. The new block is block 3, from the front of the queue: the unhashed block is reused first, which keeps cached blocks alive longer.
Request C is exactly those 8 shared tokens. Both blocks are in the cache, but the cap allows hits on at most 7 tokens, and only full blocks count, so only block 0 is used; the second block is computed again into new block 4. This is the "recomputation of an entire block" case from the source comment. vLLM does not de-duplicate equal blocks (BlockHashToBlockMap, "NOTE #1": "We currently don't de-duplicate the blocks in the cache"), so the hash of the second block now maps to two blocks. The reason given in the source is that block ids must not change, so block tables stay append-only.
Request D shares nothing and needs 4 blocks. The queue holds blocks 5, 6, 7 (never used) and 2 (A's third block, still hashed). Block 2 is popped last, and since it has a hash, its cache entry is evicted. The free queue's order is the eviction policy.
Example 14: vLLM, attention backends
The question is how vLLM picks the attention kernel for a GPU. Attention is the most expensive step of the model, and several libraries implement it (FlashAttention, FlashInfer, and others). They differ in which GPUs, data types, and head sizes they support, and in speed. vLLM therefore keeps a ranked list of candidates and takes the first one that is valid for the current hardware and model. The compute capability of an NVIDIA GPU is its generation number, such as 8.x for the A100 and 9.x for the H100; newer numbers mean newer hardware features. Open v1/attention/backends/registry.py. AttentionBackendEnum lists every backend with its class path:
FLASH_ATTN = "vllm.v1.attention.backends.flash_attn.FlashAttentionBackend"
FLASH_ATTN_DIFFKV = (
"vllm.v1.attention.backends.flash_attn_diffkv.FlashAttentionDiffKVBackend"
)
TRITON_ATTN = "vllm.v1.attention.backends.triton_attn.TritonAttentionBackend"Each backend class (for example FlashAttentionBackend in v1/attention/backends/flash_attn.py) answers questions about itself: get_name, get_impl_cls (the class with the attention forward), get_builder_cls (the class that prepares per-step metadata such as block tables), and supports_* checks for head size, data type, KV cache data type, sliding window, non-causal attention, compute capability, and more. FlashAttentionBackend.supports_head_size starts by rejecting head sizes that are not multiples of 8.
The choice is made per platform. For NVIDIA GPUs, _get_backend_priorities in platforms/cuda.py returns an ordered list based on the compute capability and the model's needs. get_valid_backends calls validate_configuration on each candidate and keeps those with no reasons against them, and the candidate with the lowest priority index wins. When none is valid, the error lists every backend with its reasons. A backend named with --attention-backend is validated alone; if it fails, startup stops with Selected backend ... is not valid for this configuration. The program reproduces the branch for models without MLA (multi-head latent attention, the compressed-KV attention of DeepSeek models, which has its own list).
import java.util.ArrayList;
import java.util.List;
public class AttnPriorities {
// The non-MLA branch of _get_backend_priorities in vllm/platforms/cuda.py (vLLM 0.30.0).
static List<String> priorities(int ccMajor, boolean nonCausal, boolean mmPrefix) {
List<String> out = new ArrayList<>();
if (ccMajor == 10 && !nonCausal) {
if (mmPrefix) out.add("TRITON_FLASHINFER");
out.addAll(List.of("FLASHINFER", "FLASH_ATTN", "TRITON_ATTN", "FLEX_ATTENTION", "TURBOQUANT"));
} else {
if (ccMajor == 9 && mmPrefix) out.add("TRITON_FLASH_ATTN");
out.addAll(List.of("FLASH_ATTN", "FLASHINFER", "TRITON_ATTN", "FLEX_ATTENTION", "TURBOQUANT"));
}
return out;
}
// One of the checks a candidate must pass: FlashAttentionBackend.supports_head_size starts with head_size % 8.
static String firstUsable(List<String> candidates, int headSize) {
for (String c : candidates) {
if (c.equals("FLASH_ATTN") && headSize % 8 != 0) continue;
return c;
}
return "none";
}
public static void main(String[] args) {
System.out.println("cc 8.x: " + priorities(8, false, false));
System.out.println("cc 9.x, multimodal prefix: " + priorities(9, false, true));
System.out.println("cc 10.x, causal: " + priorities(10, false, false));
System.out.println("cc 10.x, non-causal: " + priorities(10, true, false));
System.out.println("cc 8.x, head size 128 -> " + firstUsable(priorities(8, false, false), 128));
System.out.println("cc 8.x, head size 100 -> " + firstUsable(priorities(8, false, false), 100)
+ " (if it passes its own checks)");
}
}cc 8.x: [FLASH_ATTN, FLASHINFER, TRITON_ATTN, FLEX_ATTENTION, TURBOQUANT]
cc 9.x, multimodal prefix: [TRITON_FLASH_ATTN, FLASH_ATTN, FLASHINFER, TRITON_ATTN, FLEX_ATTENTION, TURBOQUANT]
cc 10.x, causal: [FLASHINFER, FLASH_ATTN, TRITON_ATTN, FLEX_ATTENTION, TURBOQUANT]
cc 10.x, non-causal: [FLASH_ATTN, FLASHINFER, TRITON_ATTN, FLEX_ATTENTION, TURBOQUANT]
cc 8.x, head size 128 -> FLASH_ATTN
cc 8.x, head size 100 -> FLASHINFER (if it passes its own checks)On compute capability 8.x, FlashAttention comes first and FlashInfer second. On 9.x, a model that needs a multimodal prefix gets TRITON_FLASH_ATTN in front. On 10.x with causal attention, the order flips and FlashInfer leads. The source comment gives the reason for the exception: "SM100f defaults to FlashInfer for TRTLLM causal attention, but its non-causal cutlass path (used for dflash attention) is known to have problems", so non-causal attention on 10.x uses the 8.x order. The last two lines apply one validity check: head size 128 passes FlashAttention's multiple-of-8 rule, while 100 does not, so the next candidate is tried.
Example 15: vLLM, tool parsers
The question is how vLLM turns model text into tool_calls for many model families. The problem is the same as in Example 8: each model family writes tool calls in its own text format, and the API must return structured data. vLLM solves it with one small parser class per format. A request for streaming makes the task harder, because the text arrives in small pieces, and a piece may end in the middle of a tag or a name. Open tool_parsers/abstract_tool_parser.py. A parser has two methods. extract_tool_calls(model_output, request) handles a complete response. extract_tool_calls_streaming handles a stream and receives the text so far, the new text, and the token ids of both:
def extract_tool_calls_streaming(
self,
previous_text: str,
current_text: str,
delta_text: str,
previous_token_ids: Sequence[int],
current_token_ids: Sequence[int],
delta_token_ids: Sequence[int],
request: ChatCompletionRequest,
) -> DeltaMessage | None:ToolParserManager keeps a registry. tool_parsers/__init__.py registers parsers lazily: a name maps to a module and a class, and the module is imported on first use. The registered names at this version:
grep -oE '^ "[a-z0-9_.-]+": \(' vllm/tool_parsers/__init__.py | sed -E 's/^ "([^"]+)".*/\1/' | tr '\n' ' '
grep -cE '^ "[a-z0-9_.-]+": \(' vllm/tool_parsers/__init__.pyOutput from the reference source tree:
dots deepseek_v3 deepseek_v31 deepseek_v32 deepseek_v4 deepseek_v41 cohere_command3 cohere_command4 ernie45 glm45 glm47 ling3 granite-20b-fc granite granite4 hermes poolside_v1 hunyuan_a13b hy_v3 hy_v4 internlm jamba lfm2 kimi_k2 kimi_k3 k2_horizon llama3_json llama4_json llama4_pythonic longcat mimo minimax_m2 minimax_m3 minicpm5 mistral olmo3 muse_glimmer openai phi4_mini_json pythonic qwen3_coder qwen3_xml seed_oss step3 step3p5 inkling xlam gigachat3 functiongemma gemma4 apertus
51The name is what --tool-call-parser takes. The approach is the opposite of llama.cpp's template analysis (Example 8): vLLM has one hand-written parser per format, chosen by the user. Serving with vLLM used the hermes parser. The streaming side is the harder one. The docstring of extract_tool_calls_streaming in hermes_tool_parser.py says that on each call it "re-parses the full current_text to find <tool_call> regions, then diffs against previously sent state to emit only new content, tool names, or argument fragments". _extract_content holds back any suffix that could be the start of <tool_call> (partial_tag_overlap), _extract_tool_name sends a name only once its closing quote has arrived, and _compute_args_diff sends only argument characters not sent before. The program follows that design. It feeds seven chunks to the parser one at a time. For each chunk, the output shows the chunk in brackets and what the parser sent to the client in response. Look for the chunks that produce nothing.
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class StreamingToolParser {
// Modeled on Hermes2ProToolParser.extract_tool_calls_streaming (vllm/tool_parsers/hermes_tool_parser.py):
// re-parse the whole current text on every chunk, then send only what was not sent before.
static final String START = "<tool_call>", END = "</tool_call>";
static final Pattern NAME = Pattern.compile("\"name\":\\s*\"([^\"]+)\"");
static final Pattern ARGS = Pattern.compile("\"arguments\":\\s*");
int sentContentIdx, sentArgs;
boolean nameSent;
static int partialTagOverlap(String text, String tag) { // longest text suffix that is a tag prefix
for (int n = Math.min(text.length(), tag.length() - 1); n > 0; n--)
if (text.endsWith(tag.substring(0, n))) return n;
return 0;
}
List<String> extract(String current) {
List<String> out = new ArrayList<>();
int start = current.indexOf(START);
int sendable = start < 0 ? current.length() - partialTagOverlap(current, START) : start;
if (sendable > sentContentIdx) { out.add("content " + current.substring(sentContentIdx, sendable)); sentContentIdx = sendable; }
if (start < 0) return out;
String json = current.substring(start + START.length());
int end = json.indexOf(END);
json = (end < 0 ? json : json.substring(0, end)).strip();
Matcher name = NAME.matcher(json), args = ARGS.matcher(json);
if (!nameSent && name.find()) { nameSent = true; out.add("name " + name.group(1)); }
if (nameSent && args.find()) {
String a = json.substring(args.end());
if (end >= 0 && a.endsWith("}")) a = a.substring(0, a.length() - 1).strip(); // complete: drop outer }
if (a.length() > sentArgs) { out.add("arguments+ " + a.substring(sentArgs)); sentArgs = a.length(); }
}
return out;
}
public static void main(String[] args) {
List<String> chunks = List.of("Running the", " tests.\n<tool", "_call>\n{\"name\": \"ba", "sh\", \"arguments\": ",
"{\"command\": ", "\"mvn -q test\"", "}}\n</tool_call>");
StreamingToolParser parser = new StreamingToolParser();
String current = "";
for (String delta : chunks) {
current += delta;
String sent = parser.extract(current).toString().replace("\n", "\\n");
System.out.printf("%-26s -> %s%n", "[" + delta.replace("\n", "\\n") + "]", sent);
}
}
}[Running the] -> [content Running the]
[ tests.\n<tool] -> [content tests.\n]
[_call>\n{"name": "ba] -> []
[sh", "arguments": ] -> [name bash]
[{"command": ] -> [arguments+ {"command":]
["mvn -q test"] -> [arguments+ "mvn -q test"]
[}}\n</tool_call>] -> [arguments+ }]The first chunk is plain text and goes out as content. The second ends with <tool, which could be the start of <tool_call>, so only tests.\n is sent and <tool is held back. With the third chunk the tag is complete, but the name is not (the closing quote of "bash" is missing), so nothing is sent: a client never sees a partial function name.
The fourth chunk completes the name, which goes out once. From then on, argument text is sent as it arrives, as raw partial JSON; the trailing space of {"command": is held until the next chunk because the parser strips the JSON text. The last chunk closes the tag, so the JSON is complete: the outer } belongs to the wrapper object, not to the arguments, and is dropped, leaving one } to send. Joined, the argument deltas are {"command": "mvn -q test"}, which is what a client accumulates from streamed tool_calls[0].function.arguments, as The Chat Completions Protocol and Streaming in Java showed from the client side.
Example 16: Ollama, routes, scheduler, runner
The question is how Ollama routes a request to a model process. Ollama does not run the model inside its own process. It keeps a child process per loaded model, so its job is to receive requests, find or start the right child, and pass the request on. Open server/routes.go (Ollama v0.35.1). The route table registers Ollama's own API and the compatible ones on the same handlers:
r.POST("/api/generate", s.withInferenceRequestLogging("/api/generate", s.GenerateHandler)...)
r.POST("/api/chat", s.withInferenceRequestLogging("/api/chat", s.ChatHandler)...)/v1/chat/completions, /v1/responses, and /v1/messages (Anthropic's format) also end in s.ChatHandler, after middleware that converts the request and response formats (middleware.ChatMiddleware, middleware.ResponsesMiddleware, middleware.AnthropicMessagesMiddleware). Handlers call scheduleRunner, which asks the scheduler for a runner with s.sched.getRunner(...). A runner here is a loaded model in a child process.
The scheduler is in server/sched.go. getRunner has a fast path and a queue:
if runner != nil && !runner.needsReload(c, req) {
req.useLoadedRunner(runner, s.finishedReqCh)
} else {
select {
case s.pendingReqCh <- req:
default:
req.errCh <- ErrMaxQueue
}
}pendingReqCh is a buffered channel of size OLLAMA_MAX_QUEUE (default 512, envconfig/config.go). The select with default is a non-blocking send: when the buffer is full, the request fails at once with ErrMaxQueue, which handleScheduleError in routes.go maps to HTTP 503 (http.StatusServiceUnavailable). A single goroutine, processPending, takes pending requests one at a time, loads or reloads models, and unloads others when memory or OLLAMA_MAX_LOADED_MODELS requires it. The program reproduces the queue with a capacity of 2, so overflow shows up after a few requests. A buffered channel is Go's bounded queue, and ArrayBlockingQueue is the Java equivalent. The output shows, per request, whether it used a loaded model, entered the queue, or was rejected.
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public class OllamaQueue {
record LlmRequest(int n, String model) {}
final BlockingQueue<LlmRequest> pendingReqCh; // make(chan *LlmRequest, maxQueue)
final Map<String, Integer> loaded = new HashMap<>(); // model -> number of requests using its runner
OllamaQueue(int maxQueue) { pendingReqCh = new ArrayBlockingQueue<>(maxQueue); }
// getRunner in server/sched.go: a loaded runner is used at once; otherwise a non-blocking send to the queue.
String getRunner(LlmRequest req) {
if (loaded.containsKey(req.model())) {
loaded.merge(req.model(), 1, Integer::sum);
return "uses loaded runner for " + req.model();
}
if (!pendingReqCh.offer(req)) // select { case ch <- req: default: errCh <- ErrMaxQueue }
return "ErrMaxQueue: server busy, please try again. maximum pending requests exceeded";
return "queued for the scheduler (" + pendingReqCh.size() + " pending)";
}
// processPending: one request at a time; loading a model blocks the requests behind it.
void processOne() {
LlmRequest req = pendingReqCh.poll();
if (req == null) return;
loaded.merge(req.model(), 1, Integer::sum);
System.out.println(" scheduler: loaded " + req.model() + " for request " + req.n());
}
public static void main(String[] args) {
OllamaQueue sched = new OllamaQueue(2); // OLLAMA_MAX_QUEUE=2 (the default is 512)
List<String> models = List.of("qwen2.5-coder:7b", "llama3.2:3b", "qwen2.5-coder:7b", "gemma3:4b", "phi4");
for (int i = 0; i < models.size(); i++)
System.out.println("request " + i + " " + models.get(i) + ": " + sched.getRunner(new LlmRequest(i, models.get(i))));
sched.processOne();
sched.processOne();
System.out.println("request 5 qwen2.5-coder:7b: " + sched.getRunner(new LlmRequest(5, "qwen2.5-coder:7b")));
}
}request 0 qwen2.5-coder:7b: queued for the scheduler (1 pending)
request 1 llama3.2:3b: queued for the scheduler (2 pending)
request 2 qwen2.5-coder:7b: ErrMaxQueue: server busy, please try again. maximum pending requests exceeded
request 3 gemma3:4b: ErrMaxQueue: server busy, please try again. maximum pending requests exceeded
request 4 phi4: ErrMaxQueue: server busy, please try again. maximum pending requests exceeded
scheduler: loaded qwen2.5-coder:7b for request 0
scheduler: loaded llama3.2:3b for request 1
request 5 qwen2.5-coder:7b: uses loaded runner for qwen2.5-coder:7bRequests 0 and 1 name models that are not loaded and fill the two queue places. Request 2 names the same model as request 0, but that model is not loaded yet, so it also needs the queue and is rejected: the fast path depends on the runner being loaded, not on a load being in progress. After the scheduler loads two models, request 5 finds its model loaded and skips the queue entirely. The comment on the activeLoading field states the rule: "We can only load one model at a time but new requests to models that already loaded can happen in parallel".
The runner is in llm/llama_server.go. Its header comment states the split: "Models with explicit Ollama renderers/parsers, Harmony handling, MLX, or an enabled Go TEMPLATE layer still render prompts in Go and call /completion. Other GGUF chat models use llama-server's chat_template handling through /v1/chat/completions." startLlamaServer starts the binary with exec.Command, and getServerStatus checks its /health endpoint. The LlamaServer interface in llm/server.go lists what the scheduler needs from a runner, from Load and Completion to MemorySize and Pid. Ollama's token-level batching is therefore llama.cpp's (Example 7); its own scheduler works at the level of whole models.
Example 17: SGLang, the radix cache
The question is how SGLang stores shared prefixes as a tree. vLLM matches prefixes block by block. SGLang uses another structure: a tree in which each path from the root spells out a token sequence. Sequences that start the same way share the start of the path, so the shared part is stored once. Open srt/mem_cache/radix_cache.py (SGLang 0.5.21). Each TreeNode has children (keyed by the first token, or the first page, of the child's key), parent, key (a run of token ids), value (the KV cache indices for those tokens), lock_ref, and last_access_time. A node's key can hold many tokens; this makes the tree a radix tree, a trie in which chains of single children are merged into one node.
When a new key matches only part of a node's key, the node is split. _split_node creates a new node for the shared part and makes the old node its child:
new_node.children = {key[split_len:].child_key(self.page_size): child}
new_node.parent = child.parent
new_node.lock_ref = child.lock_ref
new_node.key = child.key[:split_len]
new_node.value = child.value[:split_len].clone()
child.parent = new_node
child.key = child.key[split_len:]A running request protects its path with inc_lock_ref, which walks from its node to the root and increments lock_ref on each node; locked nodes are never evicted. evict builds a heap of evictable leaves ordered by the eviction strategy (--radix-eviction-policy, default lru), pops leaves until enough tokens are freed, and pushes a parent onto the heap when it becomes an unlocked leaf. The program reproduces insertion with splits, path locking, and leaf eviction by oldest access, with a counter in place of the clock. It inserts three sequences, locks one path, prints the tree (indentation shows depth, and t is the last access time), evicts 4 tokens, and prints the tree again.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.TreeMap;
public class RadixSplit {
static int clock; // stands in for time.monotonic()
static class Node {
final TreeMap<Integer, Node> children = new TreeMap<>(); // keyed by the first token of the child's key
Node parent; int[] key = new int[0]; int lockRef; int lastAccess = clock;
}
final Node root = new Node();
static int match(int[] a, int[] b) { int n = 0; while (n < a.length && n < b.length && a[n] == b[n]) n++; return n; }
Node split(Node child, int len) { // _split_node: new_node takes the first len tokens
Node mid = new Node();
mid.parent = child.parent; mid.key = Arrays.copyOf(child.key, len); mid.lockRef = child.lockRef;
child.parent.children.put(mid.key[0], mid);
child.key = Arrays.copyOfRange(child.key, len, child.key.length);
child.parent = mid; mid.children.put(child.key[0], child);
return mid;
}
int insert(int[] key) { // _insert_helper; returns the matched prefix length
Node node = root; int matched = 0; clock++;
while (key.length > 0 && node.children.containsKey(key[0])) {
node = node.children.get(key[0]); node.lastAccess = clock;
int p = match(node.key, key);
if (p < node.key.length) node = split(node, p);
matched += p; key = Arrays.copyOfRange(key, p, key.length);
}
if (key.length > 0) { Node n = new Node(); n.parent = node; n.key = key; node.children.put(key[0], n); }
return matched;
}
int evict(int tokens) { // evict: leaves with lock_ref 0, oldest access first
int evicted = 0;
while (evicted < tokens) {
List<Node> leaves = new ArrayList<>();
collect(root, leaves);
Node x = leaves.stream().filter(n -> n.lockRef == 0).min(Comparator.comparingInt(n -> n.lastAccess)).orElse(null);
if (x == null) break;
evicted += x.key.length; x.parent.children.remove(x.key[0]);
}
return evicted;
}
void collect(Node n, List<Node> out) {
if (n != root && n.children.isEmpty()) out.add(n);
for (Node c : n.children.values()) collect(c, out);
}
void print(Node n, String indent) {
for (Node c : n.children.values()) {
System.out.println(indent + Arrays.toString(c.key) + (c.lockRef > 0 ? " locked" : "") + " t=" + c.lastAccess);
print(c, indent + " ");
}
}
public static void main(String[] args) {
RadixSplit tree = new RadixSplit();
System.out.println("insert A, matched " + tree.insert(new int[] {1, 2, 3, 4, 5, 6}));
System.out.println("insert B, matched " + tree.insert(new int[] {1, 2, 3, 9, 9}));
System.out.println("insert C, matched " + tree.insert(new int[] {7, 8}));
for (Node n = tree.root.children.get(1).children.get(9); n != tree.root; n = n.parent)
n.lockRef++; // inc_lock_ref: a running request locks B's path
tree.print(tree.root, " ");
System.out.println("evict 4 tokens -> evicted " + tree.evict(4));
tree.print(tree.root, " ");
}
}insert A, matched 0
insert B, matched 3
insert C, matched 0
[1, 2, 3] locked t=2
[4, 5, 6] t=2
[9, 9] locked t=2
[7, 8] t=3
evict 4 tokens -> evicted 5
[1, 2, 3] locked t=2
[9, 9] locked t=2The tree before eviction looks like this:
root
+-- [1, 2, 3] (locked)
| +-- [4, 5, 6] rest of A
| +-- [9, 9] (locked) rest of B
+-- [7, 8] CInsert A creates one node holding all six tokens. Insert B shares [1, 2, 3], so the matched length is 3 and A's node is split: a new node [1, 2, 3] with two children, [4, 5, 6] (the rest of A) and [9, 9] (the rest of B). Insert C shares nothing and becomes a second child of the root. The tree stores 10 tokens for three sequences of 13 tokens in total.
Locking B's path marks [9, 9] and [1, 2, 3]. Evicting 4 tokens removes the oldest unlocked leaf, [4, 5, 6] (3 tokens, last accessed at time 2), then the next, [7, 8] (2 tokens, time 3), for a total of 5. Eviction removes whole leaves, so it can free more than asked. [1, 2, 3] survives both because it is locked and because it is not a leaf.
Compared with vLLM's blocks (Example 12), the tree matches at token granularity (with a page size of 1) and evicts whole suffixes, while vLLM matches whole blocks and evicts block by block. The Engine Landscape and How Ollama Works showed the matching side with a plain trie; this program adds the splits and eviction that a real cache needs. SGLang's scheduler can also use the tree to order waiting requests (--schedule-policy lpm, longest prefix match); the default policy is fcfs.
Example 18: the section 7 Java engine compared with llama.cpp
The question is which llama.cpp files do the work that each part of the course's own Java engine does. This gives a map from what you built to where the production version lives, and shows what a teaching engine leaves out. Section 7 builds a GGUF loader with the FFM API (Loading GGUF with the FFM API), a tokenizer from GGUF metadata (A Real Tokenizer from GGUF Metadata), an FP32 forward pass (The Forward Pass in Java), and quantized kernels with the Vector API (Quantized Kernels, the Vector API, and Threads); a later lesson assembles them into a chat engine. The command counts the lines of the matching llama.cpp files.
cd llama.cpp
wc -l ggml/src/gguf.cpp src/llama-vocab.cpp src/models/llama.cpp src/llama-graph.cpp \
ggml/src/ggml-cpu/quants.c ggml/src/ggml-cpu/arch/x86/quants.c src/llama-sampler.cpp common/chat.cppOutput from the pinned checkout:
1731 ggml/src/gguf.cpp
4531 src/llama-vocab.cpp
250 src/models/llama.cpp
3949 src/llama-graph.cpp
1339 ggml/src/ggml-cpu/quants.c
4108 ggml/src/ggml-cpu/arch/x86/quants.c
4385 src/llama-sampler.cpp
1533 common/chat.cpp
21826 total| Java engine part (section 7) | llama.cpp counterpart | Difference to look for |
|---|---|---|
GGUF loader with MemorySegment views |
ggml/src/gguf.cpp, src/llama-model-loader.cpp, src/llama-mmap.cpp |
llama.cpp also handles split files, tensor overrides, and placement on several devices |
Tokenizer from tokenizer.ggml.* keys |
src/llama-vocab.cpp (llm_tokenizer_bpe_session and others) |
One file covers BPE, SentencePiece, WordPiece, and more, with per-model pre-tokenizer regexes |
Config record with a neox flag |
load_arch_hparams and llama_model_rope_type (Example 2) |
llama.cpp reads one class per architecture; the Java record covers llama and qwen2 with a switch |
| Forward pass computed operation by operation | src/models/llama.cpp plus build_* helpers in src/llama-graph.cpp |
llama.cpp builds a graph once and runs it on any backend; the Java engine computes as it goes |
| Q8_0 and Q4_0 dot products | ggml_vec_dot_q8_0_q8_0_generic in ggml/src/ggml-cpu/quants.c; SIMD versions in arch/*/quants.c |
type_traits_cpu sets vec_dot_type = GGML_TYPE_Q8_0 for Q4_0 and Q8_0 weights, so activations are quantized to Q8_0 before the dot product |
| Threads over matrix rows | chunked rows in ggml_compute_forward_mul_mat, ith/nth loops in ops.cpp (Example 9) |
ggml splits work per operator; one thread pool runs the whole graph |
| Sampler chain | src/llama-sampler.cpp (Examples 5 and 6) |
Same chain idea; more samplers, empty ? members, and a grammar sampler |
| Chat template | common/chat.cpp with its Jinja engine |
Templates come from the GGUF's tokenizer.chat_template, plus tool-call analysis (Example 8) |
KV cache as one float[] per layer, indexed by position |
src/llama-kv-cache.cpp, src/llama-kv-cells.h (Examples 3 and 4) |
Cells indexed by cell number, with positions and sequence sets stored separately; slot search, shifting, and sliding-window variants |
The generic Q8_0 dot product shows the arithmetic that the Java kernels reproduce:
for (; ib < nb; ++ib) {
int sumi = 0;
for (int j = 0; j < qk; j++) {
sumi += x[ib].qs[j]*y[ib].qs[j];
}
sumf += sumi*(GGML_CPU_FP16_TO_FP32(x[ib].d)*GGML_CPU_FP16_TO_FP32(y[ib].d));
}The loop does three things. Inside each 32-value block, it multiplies the small integers of the two vectors and adds them (sumi). Per block, it multiplies that integer sum by the two block scales (d), which turns it back into a real number. Across blocks, it adds the results in a float (sumf). A beginner can read this as the whole trick of quantized dot products: do the cheap integer arithmetic, and apply the scales once per block. The Java engine's KV cache is the simplest case of llama.cpp's: one sequence, so the cell number equals the position, and no sequence sets are needed. The size gap in the table is mostly breadth: llama.cpp supports over 150 architectures, several tokenizer types, and many backends, while a teaching engine supports one or two of each.
Example 19: mapping table, engine concepts to source files and loop
The question is where each serving concept from this course lives in each engine and in loop (the command-line coding agent this course builds). Use the table as a reference when you need to find code quickly. The loop column names what loop uses or builds: loop serve starts an engine as a child process (Workshop: loop serve, run, and chat), loop bench measures it, and section 7's engine runs in process.
| Concept | llama.cpp 0c1e57098 |
vLLM 0.30.0 | Ollama v0.35.1 | SGLang 0.5.21 | loop |
|---|---|---|---|---|---|
| HTTP entry | tools/server/server.cpp, server-http.cpp |
entrypoints/openai/api_server.py |
server/routes.go |
srt/entrypoints/ |
loop serve talks to the engine's HTTP API |
| Request queue | server_queue (server-queue.cpp), deferred tasks |
EngineCoreProc input queue, Scheduler.waiting |
pendingReqCh |
waiting queue in srt/managers/scheduler.py |
one request at a time per chat |
| Batching | update_slots, one llama_decode per step |
Scheduler.schedule token budget, chunked prefill |
delegates to llama-server |
scheduler with --schedule-policy |
loop bench measures it from outside |
| Architecture code | src/models/*.cpp |
model_executor/models/*.py |
delegates (GGUF), mlxrunner/ (MLX) |
srt/models/ |
section 7's Llama forward pass |
| KV cache | llama_kv_cache, llama_kv_cells |
KVCacheManager, BlockPool |
delegates | RadixCache and token pools |
one float[] per layer in the Java engine |
| Prefix reuse | slot LCP similarity and RAM prompt cache (--cache-ram) |
block hash chain, get_computed_blocks |
delegates | radix tree match_prefix |
prompt layout that keeps prefixes stable |
| Sampling | llama-sampler.cpp chain |
v1/sample/ |
delegates | srt/sampling/ |
sampler chain (Project 2) |
| Tool-call parsing | common/chat.cpp, chat-diff-analyzer.cpp |
tool_parsers/ |
model/parsers/ and model/renderers/, or llama-server |
srt/function_call/ |
loop's parser for text tool calls |
| Process model | one process | API server process, engine core process over ZeroMQ, GPU workers | server process plus one llama-server child per model |
server process with the tokenizer manager, plus scheduler and detokenizer processes | loop plus one engine child process |
Read by row, the table shows the main design split: Ollama owns models and processes but not tokens, while llama.cpp, vLLM, and SGLang own tokens. Read by column, it shows a design choice: llama.cpp runs in one process, while vLLM and SGLang split into several processes so that the HTTP handling, scheduling, and GPU work do not block each other.
Example 20: reading exercises
These exercises are for practice on the real source trees, since reading code is a skill that improves by doing it. Each exercise names a question, the files to open, and what a complete answer contains. The method is the one from Example 1: find the file by its part, measure it, read the smallest function that answers the question, and check the answer with a small program or a real log line.
| # | Question | Files | A complete answer names |
|---|---|---|---|
| 1 | How does llama-server shift the KV cache when a slot's context is full? |
tools/server/server-context.cpp (n_discard), src/llama-kv-cache.cpp (seq_add, build_graph_shift) |
Which positions are removed, how RoPE is reapplied to the kept keys, and which flag enables shifting |
| 2 | How does llama_kv_cache_iswa decide which layers use the sliding-window cache? |
src/llama-kv-cache-iswa.cpp, src/llama-hparams.cpp |
The per-layer flag and the two cache sizes |
| 3 | What does the grammar sampler do to a candidate array? | src/llama-grammar.cpp, src/llama-sampler.cpp |
How rejected tokens are marked and when the grammar state advances |
| 4 | How does ggml split GGML_OP_MUL_MAT across threads? |
ggml/src/ggml-cpu/ggml-cpu.c |
The chunk size, the shared counter, and the fallback to one part per thread |
| 5 | When does vLLM's scheduler stop admitting waiting requests, besides the token budget? | vllm/v1/core/sched/scheduler.py |
max_num_running_reqs, preemption in the same step, and paused states |
| 6 | How does vLLM's request_block_hasher handle extra keys such as LoRA adapters? |
vllm/v1/core/kv_cache_utils.py (get_request_block_hasher, _gen_lora_extra_hash_keys) |
Which values enter extra_keys and why two adapters never share blocks |
| 7 | How does Ollama decide that a loaded runner needs a reload? | server/sched.go (needsReload) |
The option and adapter comparisons that force a new process |
| 8 | How does SGLang's lpm policy reorder the waiting queue? |
srt/managers/schedule_policy.py |
How prefix match lengths are computed and when the policy falls back to fcfs |
Interview points
- An engine has five parts on a request's path: request handling, model graph, KV cache, sampler, and scheduler; Ollama delegates all but the first and model-level scheduling to
llama-server. - llama.cpp creates one model class per architecture (
llama_model_mapping), loads hyperparameters, vocabulary, and tensors in that order, and builds each forward pass as a ggml graph whose tensors are named withcb(). - llama.cpp's KV cache stores a position and a bitset of sequence ids per cell;
find_slottakes free cells from a moving head, so sequences interleave, andseq_cpshares cells between sequences without copying data. - A llama.cpp sampler is a struct of function pointers (
name,accept,apply); the chain applies members in a fixed order, members with no effect print with?, and temperature comes after the filters by default. llama-serverruns one loop thread that assigns tasks to slots by longest-common-prefix similarity (above 0.10 by default) or LRU, and batches the next tokens of all busy slots into onellama_decodecall.- llama.cpp finds tool-call syntax for unknown templates by rendering the template with sentinel strings and diffing the outputs; vLLM has 51 hand-written parsers chosen with
--tool-call-parser. - A ggml operator call only creates a graph node; the CPU backend dispatches on
op, splits rows across threads by stride, and can fuse RMSNorm with the following multiply. - vLLM's scheduler has no prefill or decode phase: each request is behind by some tokens, running requests are served first within a token budget, and when blocks run out the last running request is preempted and later recomputed.
- vLLM's block pool keeps freed blocks with their hashes in an LRU queue, puts unhashed blocks at the front, caps a prefix hit at
num_tokens - 1, and does not de-duplicate equal blocks. - SGLang's radix cache splits nodes on partial matches, locks a running request's path up to the root, and evicts whole unlocked leaves by access time.
- Ollama queues model loads in a bounded channel (
OLLAMA_MAX_QUEUE, 512) with a non-blocking send and returns 503 when it is full; requests to loaded models skip the queue.
Summary
- llama.cpp at
0c1e57098maps architectures to classes insrc/llama-model.cpp(152caselabels) and keeps architecture code insrc/models/; the Llama file is 250 lines. - An unknown
general.architecturefails withunknown model architecture: '<name>'; themodel typelabel comes from the layer count and prints?Bfor unlisted counts. llama_memory_ihasseq_rm,seq_cp,seq_keep,seq_add, andseq_div; implementations includellama_kv_cache,llama_kv_cache_iswa,llama_memory_recurrent, and the hybrid memories;LLAMA_MAX_SEQis 256.- The default sampler order ends with temperature, then
dist; defaults are top-k 40, top-p 0.95, min-p 0.05, temperature 0.8. llama-serverpicks slots byid_slot, then by prompt similarity above 0.10, then by LRU, and saves displaced prompts to the RAM cache (--cache-ram, 8192 MiB by default).- llama.cpp's automatic chat parser diffs template renders that use sentinels such as
FFF_FIRST_FUN_F; GPT-OSS, Ministral, and a few others have dedicated handlers. - The ggml RMSNorm CPU kernel accumulates in
doubleand gives rowito threadi mod nth. - vLLM 0.30.0 runs an API server process and an engine core process connected by ZeroMQ; a step is schedule, execute, sample,
update_from_output. - vLLM preempts the last running request (FCFS), resets its computed tokens to 0, puts it first in the waiting queue, and admits nobody in that step.
- vLLM block hashes chain the parent hash, the block's tokens, and extra keys; freed hashed blocks go to the back of the free queue, unhashed ones to the front.
- vLLM's CUDA attention priorities for non-MLA models start with FlashAttention on 8.x and 9.x and with FlashInfer on 10.x for causal attention.
- vLLM registers 51 tool parsers lazily; the Hermes parser re-parses the full text per chunk and sends only new content, names, and argument text.
- Ollama v0.35.1 queues model loads in
pendingReqCh(default 512) and runs GGUF models inllama-serverchild processes. - SGLang 0.5.21's
RadixCachesplits nodes with_split_node, protects paths withlock_ref, and evicts leaves bylruby default.