Inference Engines: vLLM, SGLang, TensorRT-LLM, Dynamo

Running a model with PyTorch alone means you write the serving logic yourself: how requests are grouped, how memory for the KV cache is managed, how the work is split across GPUs. An inference engine is a ready-made server program that does all of that and exposes it through settings. You point it at a model, choose options, and get a fast API. Engines are the level where most inference engineering happens today.

This lesson compares the three competitive engines, vLLM, SGLang, and TensorRT-LLM, and then covers NVIDIA Dynamo, which sits above them for very large deployments. Afterwards you can say which engine fits a given situation, read the launch commands, and explain when Dynamo is worth its extra complexity. The lesson uses terms from earlier lessons: GPUs and bandwidth from Inside a GPU, and the stack levels from CUDA, kernels, and kernel fusion.

Example 1: the three engines at a glance

There are three competitive inference engines: vLLM, SGLang, and TensorRT-LLM. They give good performance out of the box for LLMs and for other modalities with similar architectures (Vision language models and embedding models covers some). In late 2025, vLLM and SGLang also started to support some image and video generation models, through vLLM Omni and SGLang Diffusion. TensorRT-LLM does not support image or video generation models. For those models, engineers can use TensorRT or PyTorch directly (Serving image and video generation models).

Engines are valuable because they are configurable. With pre-optimized components at a higher level of abstraction, engineers spend their time testing combinations of techniques instead of reimplementing routine parts. At a high level, vLLM and SGLang are more general, simpler to adopt, and have day-zero support for more models. TensorRT-LLM has a steeper learning curve and usually reaches the best performance.

All three run out of the box with core features such as continuous batching and support the main performance techniques: post-training quantization (Quantization), speculative decoding (Speculative decoding), prefix caching (KV caching), parallelism (Model parallelism), and disaggregation (Prefill-decode disaggregation). Baseten, an inference provider, uses all three engines, with TensorRT-LLM used the most. Inference engineers should know all three and choose per deployment.

A general principle explains the table: the more constraints you add, the better the performance you can reach. A broad engine has to handle many models and hardware targets. A narrow one can specialize.

Example 2: why continuous batching matters

Continuous batching is the feature that all three engines share, so it is worth seeing in numbers. A batch is a group of requests that the GPU processes together, because decode is memory-bound and extra requests share the cost of reading the weights. Each step of decode produces one token for every request in the batch.

With static batching, a batch starts together and ends when its slowest request finishes. With continuous batching (called in-flight batching in TensorRT-LLM, where it works at the level of tokens), a finished request leaves immediately and a waiting request takes its place at the next step. A worked case has 6 requests with these output lengths in tokens: 2, 8, 3, 5, 3, 4, and a GPU that handles 4 requests at once. Each step makes one token per request.

Static batching:
  batch 1 = requests with 2, 8, 3, 5 tokens -> runs 8 steps (waits for the 8-token request)
  batch 2 = requests with 3, 4 tokens       -> runs 4 steps
  total: 12 steps

Continuous batching (4 slots):
  slot 1: 2-token request (steps 1-2), then 3-token request (steps 3-5)
  slot 2: 8-token request (steps 1-8)
  slot 3: 3-token request (steps 1-3), then 4-token request (steps 4-7)
  slot 4: 5-token request (steps 1-5)
  total: 8 steps

Useful slot-steps: 2 + 8 + 3 + 5 + 3 + 4 = 25
Static: 25 / (12 steps x 4 slots = 48) = 52 percent utilization
Continuous: 25 / (8 steps x 4 slots = 32) = 78 percent utilization

The same work finishes in two thirds of the steps, so throughput rises by 12 / 8 = 1.5 times. Real engines do this at every decode step across hundreds of requests.

Example 3: vLLM

vLLM has the largest market share among engines. GitHub stars are a rough measure of popularity, and at the time of writing vLLM had twice as many stars as SGLang and TensorRT-LLM combined. It was first released in the summer of 2023, a few months before the others. It started at UC Berkeley and is now hosted by The PyTorch Project in The Linux Foundation.

Its best selling point is broad support. It supports the most hardware (NVIDIA, AMD, and Intel GPUs, plus Google TPUs) and the most models and architectures. Nearly every open LLM is integrated with vLLM from day zero. Through vLLM Omni it also supports multimodal inference, with image, audio, and video inputs and outputs. Broad engines can reach impressive performance when tuned well, and narrow ones such as TensorRT-LLM can still go higher.

The developer experience is built around the vllm serve command, with settings passed as flags. vLLM installs with pip and has official Docker images with bundled dependencies for various hardware. This command, an illustration of the style, serves DeepSeek V3.1 across eight GPUs:

vllm serve deepseek-ai/DeepSeek-V3.1 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --served-model-name ds31

--tensor-parallel-size 8 splits each layer across eight GPUs. --enable-expert-parallel places different experts of a Mixture of Experts model on different GPUs (Model parallelism). --served-model-name is the name that clients use in requests.

Use vLLM when:

• You want to stand up a model server quickly with solid performance for almost any open model.
• You want to run an "Omni" model with several input and output modalities.
• You use a smaller GPU or older architecture where TensorRT-LLM gives few benefits.

Example 4: SGLang

SGLang is the other major community-driven fast engine. It was first released in December 2023, grew with Chinese open models such as DeepSeek and Qwen, and is the engine of choice for inference at xAI.

Its distinct idea is in the developer experience: a fast backend runtime paired with a flexible frontend language. In practice, you can choose individual components of the engine for deep customization without rewriting everything else. SGLang supports NVIDIA and AMD GPUs and has strong day-zero support for many models. It works closely with labs such as DeepSeek, Qwen, Kimi, and Z AI to release optimized implementations of new architecture features, for example DeepSeek's Multi-Latent Attention.

SGLang has invested heavily in large-scale deployments of MoE LLMs, especially multi-node deployments on systems such as the GB200 NVL72 (Instances, multi-GPU, MIG, and other accelerators), for high throughput. Such systems give very cost-efficient inference for large models with heavy traffic. The developer experience is built around the sglang.launch_server command, again with flags:

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2 \
  --trust-remote-code \
  --tp 8

The --speculative-* flags turn on EAGLE speculative decoding with a short draft (Speculative decoding). --tp 8 is tensor parallelism over eight GPUs. --trust-remote-code allows the model's own code from its repository to run.

SGLang Diffusion supports image and video generation. It introduces a pipeline abstraction that orchestrates several stages, which maps closely to the architecture of those models. For speed it adds diffusion-specific parallelism methods and reuses the scheduler and optimized kernels of the main SGLang package.

Use SGLang when:

• You want excellent out-of-the-box throughput with decent latency on large MoE models such as DeepSeek and Kimi.
• You want the engine experience for image and video generation models.
• You want control and customization, and want to take part in the SGLang community.

Example 5: TensorRT-LLM

TensorRT-LLM is NVIDIA's open-source engine. Of the three, it offers the highest performance and the most flexibility to expert users. Its name needs a warning, because there are two major versions and only the older one is really related to TensorRT.

V1 was released in the summer of 2025. Deployments of the older major version are still common, so always check which version is in use.

TensorRT-LLM gets the best performance largely because it has access to kernels written by NVIDIA engineers, some of them closed source. These handwritten and manually fused kernels support the latest architectures such as Hopper and Blackwell and NVIDIA-specific number formats such as NVFP4 (GPU generations). It also has a full implementation of in-flight batching, which helps throughput, and supports nearly every performance setting: quantization, speculation algorithms, prefix caching, chunked prefill (splitting a long prompt into pieces so that it does not block other requests), flexible parallelism, and disaggregation.

V1 has a developer experience like vLLM and SGLang. Besides flags to the trtllm-serve command, it expects a config.yaml file for deeper customization. An illustration:

trtllm-serve deepseek-ai/DeepSeek-R1-0528 \
  --host 0.0.0.0 \
  --port 8000 \
  --backend pytorch \
  --max_batch_size 1024 \
  --max_num_tokens 3200 \
  --max_seq_len 2048 \
  --kv_cache_free_gpu_memory_fraction 0.8 \
  --tp_size 8 \
  --ep_size 8 \
  --trust_remote_code \
  --extra_llm_api_options ${EXTRA_LLM_API_FILE}

--max_batch_size caps requests per step, --max_num_tokens caps tokens per step, and --max_seq_len caps the length of one sequence. --kv_cache_free_gpu_memory_fraction 0.8 lets the KV cache use 80 percent of the GPU memory left after loading the weights. --tp_size and --ep_size set tensor and expert parallelism. --extra_llm_api_options points to the YAML file. The best way to install TensorRT-LLM is to run one of NVIDIA's official Docker containers.

Use TensorRT-LLM when:

• You run a well-supported model architecture on a Hopper or newer GPU.
• You accept extra engineering work in return for the best performance.
• Optionally, you plan to serve with NVIDIA Dynamo and want the most closely integrated engine.

flowchart TD
  S["Pick an engine"] --> Q1{"Hopper or newer GPU, supported model, and you want top performance?"}
  Q1 -- "yes" --> T["TensorRT-LLM"]
  Q1 -- "no" --> Q2{"Large MoE model, image/video engine, or deep customization?"}
  Q2 -- "yes" --> G["SGLang"]
  Q2 -- "no" --> V["vLLM: broad support, quick start, Omni models, smaller or older GPUs"]

The diagram condenses the three "use when" lists. Real choices can differ, and the guidance is to select per deployment.

Example 6: NVIDIA Dynamo

NVIDIA Dynamo is a distributed system for model serving, announced at NVIDIA GTC in March 2025. A distributed system is a program that runs across many machines and coordinates them. Dynamo works with every engine (vLLM, SGLang, and TensorRT-LLM) as a backend, and it provides an orchestration layer for large-scale deployments.

flowchart TD
  C["Clients"] --> D["Dynamo: KV-aware router, planner"]
  D --> P["Prefill workers (engine)"]
  D --> Dec["Decode workers (engine)"]
  P -- "KV cache" --> Dec

Dynamo supports three techniques (Prefill-decode disaggregation, KV caching, and Model parallelism explain them).

As with the engines, configuring Dynamo for a use case takes real work. Its abstractions for distributed KV routing, disaggregation, and multi-node parallelism collect information at runtime, so configuration can change as traffic changes. For example, an SLA-based planner can scale the number of prefill and decode workers up and down to meet user-defined limits on TTFT (time to first token) and TPS (tokens per second), as Measuring latency and throughput defines them.

Two worked cases with made-up numbers show the ideas.

KV-aware routing: a request shares a 1,000-token system prompt and adds 200 new tokens.
  Worker A has that prompt cached: computes 200 tokens of prefill
  Worker B has nothing cached:     computes 1,200 tokens of prefill
  Routing to A saves 1,000 / 1,200 = 83 percent of the prefill work

SLA planner: 30 requests/s, 2,000 input tokens and 500 output tokens each
  prefill demand = 30 x 2,000 = 60,000 tokens/s; one prefill worker does 20,000 -> 60,000 / 20,000 = 3 workers
  decode demand  = 30 x 500   = 15,000 tokens/s; one decode worker does 5,000   -> 15,000 / 5,000   = 3 workers
  traffic doubles to 60 requests/s -> 6 prefill workers and 6 decode workers

As a general principle, the more scale you have, the more optimization techniques are available. Dynamo is built for scale: big models and big traffic. It excels at serving foundation models such as the trillion-parameter Kimi family to many concurrent users. For smaller models it can still give moderate gains in large deployments. It is a good choice when building an inference API for a foundation model built from scratch, or when serving an open model in a high-usage product.

Many deployments do not need it. Unless the volume is high enough for disaggregation and KV-aware routing to matter, Dynamo is unnecessary work and overhead, and the engines can be used directly. Dynamo is the newest project of the chapter, so features are still being added. It is open source under Apache 2.0, with an active community, public CI, and support from NVIDIA engineers.

Interview points

• Three competitive engines: vLLM, SGLang, TensorRT-LLM. All support continuous batching, quantization, speculative decoding, prefix caching, parallelism, and disaggregation.
• vLLM: UC Berkeley origin, summer 2023, hosted by The PyTorch Project in The Linux Foundation, largest market share (twice the stars of the other two combined), most hardware (NVIDIA, AMD, Intel GPUs, TPU) and models, vllm serve, vLLM Omni.
• SGLang: December 2023, used by xAI, fast backend runtime with flexible frontend language, strong on MoE and multi-node GB200 NVL72, SGLang Diffusion, sglang.launch_server, NVIDIA and AMD.
• TensorRT-LLM: NVIDIA's engine, best performance, hardest to use, NVIDIA only, no image or video models. V0 is a TensorRT plugin and V1 (summer 2025) is standalone on PyTorch. It uses handwritten and some closed-source kernels and trtllm-serve with a config.yaml.
• Use vLLM for quick, broad, Omni, and smaller or older GPUs. Use SGLang for large MoE models, diffusion, and customization. Use TensorRT-LLM for supported models on Hopper or newer GPUs when extra effort buys top performance.
• Dynamo orchestrates vLLM, SGLang, and TensorRT-LLM, with KV cache re-use and routing, disaggregation, and multi-node parallelism, and an SLA-based planner on TTFT and TPS limits.
• Dynamo suits big models and traffic. Without enough volume it is unnecessary overhead.

Summary

• Engines package continuous batching and the main optimization techniques behind a few settings.
• Comparison: performance (good, good, best), ease (straightforward, straightforward, hard), model support (most, most, some), hardware (GPU and TPU, NVIDIA and AMD, NVIDIA only), license (Apache 2.0 for all three).
• Commands: vllm serve, python3 -m sglang.launch_server, trtllm-serve.
• Continuous batching removes idle slots: in the worked case 8 steps instead of 12.
• Dynamo (GTC, March 2025) adds KV cache re-use, disaggregation, and multi-node parallelism over any of the engines.
• More constraints give more performance, and more scale makes more techniques available.