Inference Engineering Glossary

Inference engineering has its own vocabulary: terms from machine learning, GPU hardware, networking, and cloud operations. This page lists 213 terms in alphabetical order. Each entry gives a short plain-language definition and a link to the lesson that explains the idea in full, with examples. Use it as a lookup while reading the other lessons, or to check a word met in an interview or a paper.

A few conventions apply. Terms with a qualifier in parentheses, such as "Core (CUDA)", name the area in which the term has that meaning. Numbers for GPU models come from the lessons that teach them. Acronyms are spelled out in the term itself where the source defines one. If two terms are the same thing under different names, the entry says so, for example "In-flight batching" and "Continuous batching".

A diagram of how the main groups of terms relate:

flowchart LR
    M["Models: token, attention, KV cache"] --> H["Hardware: GPU, HBM, NVLink"]
    H --> S["Software: CUDA, engines, quantization"]
    S --> P["Production: replicas, autoscaling, routing"]
    P --> C["Clients: streaming, async, SLAs"]

Models run on hardware, software makes the hardware fast, production practice scales it, and clients consume the result. The sections below group the terms by first letter.

A

Activation function: A nonlinear function, such as ReLU, placed between the linear layers of a network. Without it, stacking many linear layers would collapse into one matrix multiplication and the network could not learn anything more complex than a single layer. See Neural Networks for Inference Engineers.
Active-active: A high-availability design in which several regions or clusters serve live traffic at the same time. If one fails, the others keep serving without a break. See Multi-Cloud Capacity.
Active-passive: A failover design in which a standby cluster or region is kept ready but idle. If the active one fails, traffic is cut over to the standby. See Multi-Cloud Capacity.
Ada Lovelace (architecture): NVIDIA's graphics-oriented GPU generation, released alongside Hopper in 2022. It suits small models and cost-sensitive work and is not suited to large-scale LLM inference. See GPU Generations.
Agent: An AI application that takes actions instead of only giving information. One agent task usually needs several inference calls, often to several models and modalities, plus access to tools. See Scale, Specialization, and Your App.
AI-native application: A product whose main experience and value depend on generative models. Inference choices follow from the product's limits: the modality, the latency budget, the unit economics, and the usage pattern. See Scale, Specialization, and Your App.
Ampere (architecture): An older NVIDIA GPU generation still found in legacy or small deployments. Hopper and Blackwell generally beat it on both speed and cost at scale. See GPU Generations.
Application Programming Interface (API): A structured way to send requests and receive responses. Inference engines expose an API so that other programs can query models. See What Is Inference Engineering.
Arithmetic intensity: The number of operations an algorithm performs for each byte it moves. Compared with a GPU's ops:byte ratio, it shows whether a kernel is compute bound or memory bound. See Finding the Bottleneck.
Attention: The core transformer mechanism that relates each token to earlier tokens through query, key, and value projections and a softmax. Its compute and memory cost make it a main target for optimization. See Attention and Mixture of Experts.
Automatic Speech Recognition (ASR): Models that take audio in and give text out, such as Whisper. The decoder part dominates the run time, so LLM-style optimizations and in-flight batching help. See Speech Models.
Autoregressive token generation: Generating tokens one after another, where each token depends on all earlier ones. It has two phases: prefill, which processes the input tokens, and decode, which produces the output tokens. See How an LLM Generates Text.
Autoscaling: Automatically raising and lowering the number of replicas serving a model, based on traffic or utilization. It matches capacity to demand to keep latency SLAs while limiting wasted spend. See Autoscaling.
Autoscaling window: The rolling time span used to decide whether to scale up or down. Longer windows keep the replica count steady, and shorter windows react faster to spikes. See Autoscaling.

B

B200: An NVIDIA Blackwell data center GPU with 192 GB of VRAM, 8 TB/s of memory bandwidth, and 5 petaFLOPS of FP8 compute. See GPU Generations.
B300: An NVIDIA Blackwell data center GPU with 288 GB of VRAM, 8 TB/s of memory bandwidth, and 5 petaFLOPS of FP8 compute. See GPU Generations.
Bandwidth: The amount of data per second that can pass through a memory, such as VRAM, or through an interconnect, such as NVLink. See Inside a GPU.
Baselines: Carefully recorded measurements of speed and quality taken before any optimization. They make it possible to tell which change caused a gain or a regression. See Benchmarking and Profiling.
Basic Linear Algebra Subprograms (BLAS): A standard interface for the basic operations of linear algebra, such as matrix multiplication. See CUDA, Kernels, and Kernel Fusion.
Batch: Processing several inputs at the same time, which is common in LLM inference. See Autoscaling.
Batch sizing: Choosing how many requests run together. It is a main lever between latency and throughput: larger batches raise total throughput and worsen latency for each user. See Autoscaling.
Benchmark (intelligence): A measurement of how well a model answers questions or takes appropriate actions, for example MMLU. See Choosing a Model.
Benchmark (performance): A measurement of the latency and throughput of an inference service for a given model under a defined workload. See Benchmarking and Profiling.
BF16: A 16-bit floating-point format with a larger exponent than FP16. Its wider range of values helps preserve outliers. It is used in training and sometimes in inference. See Quantization.
Bin packing (multi-cloud): Treating different pools of GPUs across clouds, regions, and clusters as one schedulable resource. Multi-cloud capacity management makes this possible. See Multi-Cloud Capacity.
Blackwell (architecture): NVIDIA's GPU generation from late 2024. It adds FP4 support, microscaling formats (MXFP8, MXFP4, NVFP4), and high memory bandwidth. See GPU Generations.
Blue-green deployment: Two parallel production environments, blue and green. Traffic is shifted from one to the other for deployments without downtime and for quick rollback. See Testing and Deployment.

C

Cache-aware routing: Sending requests to replicas that already hold a matching prefix or the needed LoRA. A higher cache hit rate gives a lower time to first token. See Autoscaling.
Canary deployment: A new deployment that first receives a small share of live traffic to check stability and speed. Over time it takes over all production traffic. See Testing and Deployment.
Causal language model (CLM): A decoder-only transformer that predicts the next token from the earlier context. All generative LLMs covered in this course are of this kind. See How an LLM Generates Text.
Central Processing Unit (CPU): A general-purpose processor tuned for sequential work. It handles orchestration, scheduling, networking, and preprocessing, and it rarely runs generative AI inference itself. See Inside a GPU.
Chat template: The model-specific way of formatting and serializing messages: roles, separators, and the beginning and end of sequence tokens. See How an LLM Generates Text.
Chunked prefill: Splitting a long input into chunks and overlapping prefill with decode or other work, so that one long sequence does not monopolize the resources. See KV Caching.
Classifier-free guidance: Balancing an unconditional denoising pass and a prompt-conditioned one at each step of image generation. Lower guidance gives more creativity, and higher guidance follows the prompt more closely. See How Image and Video Models Generate.
CLIP (text encoder): A text and image encoder used in earlier image pipelines such as SDXL. Modern systems often replace it with full LLMs for better prompt understanding. See How Image and Video Models Generate.
Closed model: A proprietary model whose weights are not available, such as GPT-5, Claude Sonnet, or Google Gemini. See Choosing a Model.
Cold start: The time from starting a new replica to its first successful response. The steps are GPU provisioning, container startup, model load, and inference engine compilation. See Autoscaling.
ComfyUI: A workflow tool for assembling image pipelines from a base model, refiner, LoRAs, and ControlNets. It encourages modular, swappable parts. See Serving Image and Video Models.
Compute-bound: Limited by available FLOPS instead of memory bandwidth. LLM prefill and image and video generation are usually compute bound. See Finding the Bottleneck.
Context Parallelism (CP): Copying the weights to every GPU and splitting the attention context across them. It is important for video models, where attention covers a very large latent space. See Model Parallelism.
Context window: The maximum number of tokens a model can process in one request, counting input, reasoning, and output. See KV Caching.
Continuous batching (in-flight): Interleaving requests at the token level so that GPU slots are always busy. It keeps the latency cost of batching small. See Autoscaling.
Control plane (multi-cloud): The global coordinator that deploys models and allocates resources across clusters. See Multi-Cloud Capacity.
Core (CUDA): A general-purpose arithmetic unit that runs a wide range of scalar and element-wise operations. See Inside a GPU.
Core (Tensor): A specialized unit for mixed-precision matrix multiply-accumulate operations. Tensor Cores are the most important kind of compute for inference. See Inside a GPU.
Cross-attention: Attention in which one sequence supplies the queries and another sequence supplies the keys and values, for example text conditioning an image. It is common in multimodal and denoising pipelines. See Attention and Mixture of Experts.
cuBLAS: CUDA's implementation of BLAS, with high-quality matrix multiplication (GEMM) and related building blocks. See CUDA, Kernels, and Kernel Fusion.
CUDA: NVIDIA's programming model and platform for GPU kernels, graphs, memory, and execution. See CUDA, Kernels, and Kernel Fusion.
CUDA driver: A low-level interface between an application and the GPU hardware, which manages memory and execution. See CUDA, Kernels, and Kernel Fusion.
CUDA graph: A directed acyclic graph of kernels and other GPU operations, used to optimize workflows that repeat. See CUDA, Kernels, and Kernel Fusion.
CUDA kernel: A function written by a developer that runs in parallel on the GPU. See CUDA, Kernels, and Kernel Fusion.
CUDA runtime: The API aimed at developers for launching kernels and managing memory. See CUDA, Kernels, and Kernel Fusion.
cuDNN: A set of building blocks for deep neural networks in CUDA. See CUDA, Kernels, and Kernel Fusion.
CuTe: A C++ template library for domain-specific work that abstracts tiled tensor operations. It helps developers write precision-aware, hardware-tuned GEMM and fused kernels. See CUDA, Kernels, and Kernel Fusion.
CUTLASS: A CUDA C++ template library with building blocks for fast, architecture-tuned GEMM and related kernels. See CUDA, Kernels, and Kernel Fusion.

D

Data sovereignty: Legal limits on the geographic place where model inputs and outputs may be processed and stored. See Multi-Cloud Capacity.
Decode: The memory-bound phase of LLM inference. The generation loop emits one token per forward pass. See How an LLM Generates Text.
DeepGEMM: A library of clean and efficient GEMM kernels from the DeepSeek AI team, with strong FP8 performance. See CUDA, Kernels, and Kernel Fusion.
Denoising model: The central part of a diffusion pipeline. It repeatedly refines random noise in latent space into an image or video. See How Image and Video Models Generate.
Diarization: Splitting audio by speaker, which answers who spoke when. It is often paired with voice activity detection in speech recognition pipelines. See Speech Models.
Diffusers (library): Reference implementations of image and video generation pipelines. See Frameworks and Model Formats.
Disaggregation: Separating prefill and decode onto engines that scale independently on separate hardware. See Disaggregation.
Distillation: Training a smaller student model to imitate a larger teacher using the teacher's probability distributions, not only its final outputs, so that the student keeps the teacher's behavior with fewer parameters. See Choosing a Model.
Docker: Container technology for building standard packages of inference services together with their dependencies. See Containerization.
Dockerfile: A human-readable file of well-specified, machine-interpretable instructions for creating an image. See Containerization.
Dynamic batching: Starting a batch when it is full or when a short timer ends, whichever comes first. It balances stable latency with utilization and is superseded by continuous batching for LLMs. See Autoscaling.
Dynamic range (quantization): The range of absolute values a number format can represent. Floating-point formats have a higher dynamic range than integers of the same size because of their exponent and mantissa structure. See Quantization.

E to G

EAGLE (speculation): A small, purpose-built draft model trained to read hidden states and propose several tokens, aiming for high acceptance rates in speculative decoding. See Speculative Decoding.
Elo (quality meta-metric): A score from head-to-head win rates used to compare model quality. It is a useful directional signal beyond intelligence benchmarks. See Choosing a Model.
Embedding model: A model that turns text or image input into vectors of fixed size, used for semantic similarity in RAG and agent memory. Modern ones often have LLM backbones and Matryoshka representations. See Vision Language and Embedding Models.
Encoder: A network that converts raw input into internal representations, such as audio features in Whisper. In encoder-decoder models it is paired with a decoder. See Speech Models.
Evals: Task-specific tests that mirror real use of a model, used to measure model intelligence for a particular product. See Choosing a Model.
Expert Parallelism (EP): Spreading the experts of a mixture-of-experts model across GPUs, with several whole experts per GPU. It raises total throughput with low communication overhead between GPUs. See Model Parallelism.
Few-step image generation: Models that give usable images in eight steps or fewer. They are 80 to 90 percent faster with a noticeable quality cost and suit real-time applications. See Serving Image and Video Models.
Feynman (architecture): A future NVIDIA generation after Rubin. Few details are known. Expect continued emphasis on low precision and memory bandwidth. See GPU Generations.
Fine-tuning: Adapting a pretrained base model to a domain, which often lets a much smaller model meet quality needs. See Choosing a Model.
FlashAttention: A series of attention kernels that minimize memory traffic. FlashAttention 3 is written for Hopper, and FlashAttention 4 targets Blackwell. See Optimizing Attention.
Floating-point data formats: Number precisions such as FP16, FP8, and FP4 used in inference. They have a high dynamic range through an exponent and mantissa structure. See Quantization.
FLOPS: Floating-point operations per second, usually measured on Tensor Cores. See Finding the Bottleneck.
Foundation model: A model trained on broad data that serves as the base for many tasks. Examples are GPT, Claude, and Llama. They are fine-tuned or used directly through prompting. See Choosing a Model.
Function calling: Also called tool calling or tool use. A model receives a set of available functions with a prompt and returns structured output that names the chosen functions and their arguments. See How an LLM Generates Text.
GB200: An NVIDIA superchip that joins a Grace CPU and a B200 GPU through a fast NVLink chip-to-chip link. It is used in rack-scale NVLink systems such as NVL72 and helps with KV cache offloading, LoRA swapping, and other techniques that benefit from NVLink-C2C. See GPU Generations.
General matrix-matrix multiplication (GEMM): A BLAS algorithm that multiplies two matrices. It is the key operation for inference. See CUDA, Kernels, and Kernel Fusion.
Generative AI: Models that create new content in many modalities (text, images, audio, video, code) by learning patterns in training data, unlike predictive ML models. See What Is Inference Engineering.
Generative Pretrained Transformer (GPT): A family of text-generation large language models created by OpenAI. See How an LLM Generates Text.
GH200: An NVIDIA superchip that joins a Grace CPU and an H200 GPU through a fast NVLink chip-to-chip link. It is used in rack-scale NVLink systems and helps with KV cache offloading, LoRA swapping, and other techniques that benefit from NVLink-C2C. See GPU Generations.
Goodhart's Law: The saying "When a measure becomes a target, it ceases to be a good measure." See Benchmarking and Profiling.
GPU node: A standard chassis of 8 GPUs connected by NVLink and NVSwitch. See Instances, Multi-GPU, MIG, and Other Accelerators.
Grace CPU: An ARM-based NVIDIA CPU with fast chip-to-chip links to the GPU. It is used alongside Hopper and Blackwell GPUs. See GPU Generations.
Graphics Processing Unit (GPU): A highly parallel processor, first built for graphics and now widely used to train and run generative AI models. See Inside a GPU.
gRPC: A schema-first protocol with bidirectional streaming for structured data. See Client Code.

H to K

Head (attention): One independent attention computation inside a layer. See Attention and Mixture of Experts.
High-Bandwidth Memory (HBM): The memory used as VRAM on data center GPUs. Recent generations are HBM3, HBM3e, and HBM4. See Inside a GPU.
Hopper (architecture): NVIDIA's 2022 GPU generation, with FP8 support and asynchronous programming features. See GPU Generations.
Hyperscaler: A large general cloud provider, such as AWS or GCP. See Multi-Cloud Capacity.
Image generation pipeline: A foundation model for images is a pipeline of several models: a text encoder, an iterative denoiser, and a VAE. See How Image and Video Models Generate.
Inference: Serving AI models in production. See What Is Inference Engineering.
Inference engine: A high-performance runtime (vLLM, SGLang, TensorRT-LLM) that supports techniques such as batching, caching, quantization, and speculation. See Inference Engines.
InfiniBand: An interconnect between nodes for scaling inference and training across several nodes. Its bandwidth is higher than alternatives such as Ethernet and far lower than NVLink. See Instances, Multi-GPU, MIG, and Other Accelerators.
In-flight batching: Another name for continuous batching: token-level interleaving of requests for high utilization with stable latency. See Autoscaling.
Input sequence: The tokens a request gives to a model. They are processed during prefill. See How an LLM Generates Text.
Input Sequence Length (ISL): The number of tokens in the input sequence of a request. See Measuring Latency and Throughput.
Instance (cloud): The provisioned virtual machine, including GPUs, CPU and RAM, storage, networking, and interconnect. See Instances, Multi-GPU, MIG, and Other Accelerators.
Integer data formats: Number formats such as INT8 and INT4, which have a limited dynamic range. See Quantization.
Inter-token latency (ITL): The time between generated tokens during decode. It converts to perceived tokens per second: 2 milliseconds of ITL equals 500 tokens per second. See Measuring Latency and Throughput.
Iterative denoising (diffusion): Starting from noise and refining it step by step into an image or video in latent space. See How Image and Video Models Generate.
Jitter traffic (bench): Adding randomness to arrival times and sequence shapes in a benchmark, so the test looks more like real traffic than uniform or bursty synthetic loads. See Benchmarking and Profiling.
Kernel fusion: Re-implementing two or more kernels as one kernel that does both jobs, which avoids needless trips through memory. See CUDA, Kernels, and Kernel Fusion.
KV cache: Stored key and value tensors for each token, so attention does not recompute them. It turns the attention work from quadratic in sequence length to linear. See KV Caching.

L to M

L0/L1/L2 caches (GPU): The cache levels on the chip: for instructions, for shared memory, and for global data. See Inside a GPU.
Large Language Model (LLM): A generative model that takes a text prompt and returns new text. GPT, Claude, Llama, and DeepSeek are well-known LLM families. See What Is Inference Engineering.
Latency percentiles: Latency reported as P50, P90, P95, and P99, which shows both the typical and the worst-case user experience. See Measuring Latency and Throughput.
Latent consistency: A few-step method that predicts the target latents directly, sometimes repeated for refinement. It is very fast, with lower fidelity than full diffusion. See Serving Image and Video Models.
Latent space (images/videos): A smaller representation, for example 128x128, in which denoising takes place. See How Image and Video Models Generate.
LLM: Short for large language model, such as GPT-5, Llama, or DeepSeek. See What Is Inference Engineering.
Load testing: Sending sustained heavy traffic to find throughput limits and to study queue and autoscaling behavior. See Testing and Deployment.
Local (edge) inference: Running inference on end-user devices such as phones and computers. See Local Inference.
Logit biasing: Nudging or limiting token probabilities to steer structured output, such as JSON or tool calls. It is applied after the logits are computed and before sampling. See How an LLM Generates Text.
Logits: A vector with one raw (not yet normalized) score for each token in the vocabulary, produced by every forward pass during decode. See How an LLM Generates Text.
Lookahead decoding: Building n-grams during inference so that draft tokens can be predicted without a separate model. See Speculative Decoding.
LoRA: Low-rank adaptation, a lightweight fine-tuning method that makes small changes to a model. Inference services often swap between thousands of LoRAs on one foundation model. See Choosing a Model.
Machine learning (ML): Predictive modeling for tasks such as classification and forecasting, as opposed to generative AI, which creates new output. See What Is Inference Engineering.
Matmul: Matrix multiplication. See Neural Networks for Inference Engineers.
Matryoshka representations (embeddings): Nested vectors that allow variable size, where the early part of the vector carries more of the meaning. They allow a trade-off between vector size and embedding quality. See Vision Language and Embedding Models.
Medusa (speculation): Adding extra decoder heads through fine-tuning, so that one pass produces several draft tokens. See Speculative Decoding.
Microscaling formats: Floating-point formats such as MXFP8, MXFP4, and NVFP4 that use blockwise quantization with small blocks that each have a scale factor (for example every 32 elements) to improve accuracy. See Quantization.
Mixture of Experts (MoE): A model design in which the linear layers are split into sparse experts. A router activates a subset of the experts for each forward pass. See Attention and Mixture of Experts.
Model parallelism (overview): Splitting work across GPUs with tensor, expert, or pipeline parallelism. The choice depends on model size, topology, and the goal of latency or throughput. See Model Parallelism.
Multi-cloud capacity management: A global scheduler that places workloads across providers and regions. See Multi-Cloud Capacity.
Multi-Instance GPU (MIG): A feature of larger Ampere, Hopper, Blackwell, and Rubin GPUs that carves a GPU into up to eight slices of memory and seven slices of compute. See Instances, Multi-GPU, MIG, and Other Accelerators.
Multi-node inference: Scaling across two or more nodes with InfiniBand when one 8-GPU node lacks the VRAM for weights, activations, and KV cache. It needs pipeline or expert parallelism between nodes, because tensor parallelism uses too much all-to-all communication for InfiniBand. See Model Parallelism.

N to O

Neocloud: A specialized cloud provider focused on GPUs, such as Coreweave and Nebius. See Multi-Cloud Capacity.
Neural audio codec: A learned encoder that compresses audio into tokens, paired with a decoder that turns tokens back into audio. See Speech Models.
N-gram speculation: Using n-grams seen during prefill to propose long draft sequences during decode. It is extremely effective for code completion. See Speculative Decoding.
NIM: An NVIDIA Inference Microservice: a pre-packaged, containerized microservice for a specific model. See Containerization.
Node: The physical 8-GPU base unit with NVLink and NVSwitch. Multi-node setups add InfiniBand between nodes. See Instances, Multi-GPU, MIG, and Other Accelerators.
NVFP4: NVIDIA's 4-bit floating-point microscaling format with two scale factors and blockwise quantization with a block size of 16. See Quantization.
NVIDIA Dynamo: An open-source distributed serving platform for KV reuse, disaggregation, and multi-GPU and multi-node orchestration. See Inference Engines.
NVL72: A rack-scale Blackwell system that connects 72 GPUs and 36 CPUs, built to serve very large models at very high throughput. See Instances, Multi-GPU, MIG, and Other Accelerators.
NVLink: A one-to-one communication layer between GPUs, up to 1800 GB/s on Blackwell and 900 GB/s on Hopper. See Inside a GPU.
NVSwitch: An all-to-all communication layer on top of NVLink that coordinates all the GPUs in a node. See Inside a GPU.
Offline inference: Asynchronous batch processing of large jobs, tuned for throughput and cost instead of per-request latency. See Scale, Specialization, and Your App.
Omni-modal: Models that accept several input modalities (text, images, video, audio) and produce several output modalities. See Vision Language and Embedding Models.
Online inference: Real-time serving of requests under tight latency budgets. See Scale, Specialization, and Your App.
ONNX: An intermediate representation and runtime for models. See Frameworks and Model Formats.
Open model: A model whose weights are freely available, such as Llama, DeepSeek, or Whisper. See Choosing a Model.
Ops:byte ratio (GPU): The peak operations per byte of memory bandwidth for a GPU at a given precision. Comparing it with arithmetic intensity shows the bottleneck. See Finding the Bottleneck.
Out-of-memory error (OOM): A common failure in which the GPU runs out of VRAM while loading weights or running inference. See Inside a GPU.
Output sequence: The tokens a model generates during decode. See How an LLM Generates Text.
Output Sequence Length (OSL): The number of tokens in the output sequence of a request. See Measuring Latency and Throughput.

P to Q

PagedAttention: An attention optimization that stores KV blocks in fixed-size pages, which improves performance, especially with long context. See KV Caching.
PCIe (GPU form factor): A data center GPU form factor that connects through standard PCI Express slots. PCIe GPUs often have lower base specifications and fewer interconnect options than the SXM version of the same GPU. See Instances, Multi-GPU, MIG, and Other Accelerators.
Perceived TPS: Tokens per second as seen by one user during streaming. It is the more exact name for what people usually mean by TPS. See Measuring Latency and Throughput.
Pipeline Parallelism (PP): Splitting the layers into stages across GPUs. It is acceptable for dense models across nodes, but it creates bubbles, where some GPUs sit idle waiting for other steps. See Model Parallelism.
Prefill: The compute-bound phase of LLM inference in which the input sequence is processed and the KV cache is built. See How an LLM Generates Text.
Prefix caching: Reusing the KV cache of shared prefixes across requests to skip prefill. It greatly improves time to first token for code completion, multi-turn chat, and agents. See KV Caching.
Pretraining: Large-scale training, usually self-supervised, on broad data to create a base model. See Choosing a Model.
Prompt: The instruction given to a model. For diffusion models it also includes a negative prompt and step and guidance parameters. See How an LLM Generates Text.
PyTorch compile (torch.compile): Capturing a graph and choosing and fusing kernels for a specific GPU. Compiled engines should be cached to shorten cold starts. See Frameworks and Model Formats.
PyTorch Profiler: A developer tool that measures CPU time, GPU time, and memory for each operation. See Benchmarking and Profiling.
Quantization-aware training: A training method that computes the quantization scales and optimizes the weights together, so the final model is already calibrated for low-precision deployment. See Quantization.
Quantization (post-training): Lowering the precision of weights, activations, and possibly the KV cache to cut compute and memory bandwidth needs. See Quantization.
Queue (request): Holds extra traffic while autoscaling brings replicas online. See Autoscaling.

R to S

Real-time factor (RTF): A measure of how fast a speech recognition model transcribes audio. Transcribing one hour of audio in six seconds gives an RTF of 600. See Speech Models.
Retrieval-augmented generation (RAG): An application pattern that fetches extra context for the LLM beyond the prompt. See Vision Language and Embedding Models.
Ring attention: A context parallelism method in which GPUs pass partial attention results around a ring. It lowers all-to-all pressure for very large contexts. See Model Parallelism.
Roofline model: A plot of arithmetic intensity against bandwidth and compute limits, which shows visually whether to optimize memory or compute. See Finding the Bottleneck.
Rotary positional embeddings (RoPE): A positional encoding that represents positions as learned rotations. It improves extrapolation to long context at the cost of more memory use in attention during inference. See KV Caching.
Routing (inference): Placing requests on replicas by load, KV cache, available LoRAs, and sequence shapes, to improve speed and utilization. See Autoscaling.
Rubin (architecture): The next NVIDIA generation (2026), which brings HBM4 and CPX for compute-bound workloads. See GPU Generations.
Sampling (decode): Choosing an output token from the logits. Common strategies are greedy (argmax), temperature sampling, top-k, and top-p (nucleus) sampling. See How an LLM Generates Text.
Scale factor (quantization): A multiplier used to map low-precision values back to their original number format. See Quantization.
Scale to zero: Turning off all replicas when idle and starting them on demand. It needs fast cold starts and reliable queueing and suits predictable or development workloads. See Autoscaling.
SDXL: An earlier diffusion image pipeline (base, refiner, and CLIP) that is good for study. Modern systems keep the structure with larger and more capable parts. See How Image and Video Models Generate.
Service Level Agreement (SLA): A contractual promise about latency, throughput, uptime, or another measure of a system. See Measuring Latency and Throughput.
Service Level Objective (SLO): An internal target set to meet or beat the SLA for a system. See Measuring Latency and Throughput.
SGLang: A fast inference engine with flexible frontends and backends and strong mixture-of-experts support. See Inference Engines.
Shadow traffic: Mirroring real production requests to a candidate deployment. See Testing and Deployment.
SNAC (audio decoder): A high-performance audio decoder path often paired with text-to-speech token streams. See Speech Models.
Softmax: A function that turns scores into probabilities. It is used in attention and to normalize logits into a probability distribution during decode. See Attention and Mixture of Experts.
Sparsity (FLOPS): In tensors with 2:4 structured sparsity, 50 percent of the values are 0, and Tensor Cores can skip multiplying by 0. Most inference is dense, not sparse. See Inside a GPU.
Special Function Unit (SFU): A dedicated unit that speeds up certain math operations, such as sine and cosine, so they stay off the CUDA Cores. See Inside a GPU.
Speculative decoding: A family of methods that generate and check draft tokens, so decode yields several tokens per forward pass. See Speculative Decoding.
Streaming Multiprocessor (SM): The GPU compute unit that holds cores and cache. See Inside a GPU.
Structured output: LLM output that follows a given schema. It is produced by limiting generation to the schema through logit biasing, instead of asking in the prompt. See How an LLM Generates Text.
SXM (GPU form factor): A socketed GPU module with higher-bandwidth connections and more power than PCIe. SXM GPUs often have higher base specifications and are the standard for inference. See Instances, Multi-GPU, MIG, and Other Accelerators.

T to Z

Temperature: Controls randomness in token choice. Low values such as 0.1 make output more deterministic, and high values such as 1.5 make it more varied. See How an LLM Generates Text.
Tensor Parallelism (TP): Splitting tensor operations across the GPUs of one node. It gives the best latency for each user and needs frequent all-reduce synchronization. See Model Parallelism.
TensorRT: NVIDIA's optimized runtime for high-performance inference, with fused kernels, quantization, and other optimizations. See Frameworks and Model Formats.
TensorRT-LLM: An NVIDIA inference engine with a Python API and both TensorRT-engine and PyTorch-backend execution paths, with fused kernels, quantization, and speculative decoding. See Inference Engines.
Thread: The smallest unit of execution on a GPU. Kernels launch many threads to get massive parallelism. See CUDA, Kernels, and Kernel Fusion.
Throughput: Total work per unit of time, for example total tokens per second. See Measuring Latency and Throughput.
Time to first byte (TTFB): The time until the first byte of output is returned. It is a latency metric. See Measuring Latency and Throughput.
Time to first token (TTFT): The time until the first token of output is returned. It is a latency metric. See Measuring Latency and Throughput.
Token: The basic unit of text for an LLM: an integer that stands for a string of characters. In English the ratio is about 4 tokens for every 3 words for most tokenizers. See How an LLM Generates Text.
Tokenizer: A component that converts strings into token sequences, and back, in a fixed way. Models have different tokenizers, and more efficient ones improve end-to-end latency. See How an LLM Generates Text.
Tokens per second (TPS): See perceived TPS. A latency metric: the number of tokens streamed to the end user each second. See Measuring Latency and Throughput.
Training: Learning model weights from data with backpropagation and optimization. It is compute-heavy, usually runs on large GPU clusters, and produces the weights used in inference. See Neural Networks for Inference Engineers.
Transformer: The architecture behind generative AI models. See Attention and Mixture of Experts.
Transformers (library): Reference implementations of LLMs and other transformer-based models. See Frameworks and Model Formats.
Triton Inference Server: An NVIDIA production serving framework that supports several backends. See Frameworks and Model Formats.
VAE (variational autoencoder): Used in inference to decode from latent space to pixel space for image and video models. During training it also encodes from pixels to latent space. See How Image and Video Models Generate.
Vector database: A database for storing and searching the semantic vectors made by embedding models. See Vision Language and Embedding Models.
Vector similarity: A check of how close two vectors are, using a formula such as cosine similarity. Vectors with high similarity carry similar meaning. See Vision Language and Embedding Models.
Vera CPU: An ARM-based NVIDIA CPU with fast chip-to-chip links to the GPU. It succeeds the Grace CPU alongside the Rubin GPU generation. See GPU Generations.
Vision-language model (VLM): A model that accepts images and video plus text prompts and outputs text. See Vision Language and Embedding Models.
vLLM: A widely used inference engine with broad model and hardware support and strong defaults. See Inference Engines.
Vocabulary: The full set of tokens, usually more than 100,000, that an LLM uses to represent data. See How an LLM Generates Text.
Voice activity detection (VAD): A lightweight model that splits streams or files into chunks that contain speech, for speech recognition. See Speech Models.
VRAM (device memory): The memory on the GPU, used for weights, KV cache, and activations. Total VRAM limits model size and KV headroom, and bandwidth limits decode tokens per second. See Inside a GPU.
WebSocket: A lightweight bidirectional streaming transport, well suited to unstructured audio chunks and real-time experiences. See Client Code.
Weights-only quantization: Lowering the precision of the weights in linear layers while keeping other parts, such as the KV cache and attention, at higher precision. It is the conservative approach: best quality preservation and smallest performance gain. See Quantization.
Workload plane (multi-cloud): One cluster with compute resources that runs inference and handles requests. See Multi-Cloud Capacity.

Summary

The glossary defines 213 terms from the course, in alphabetical order.
Every entry links to the lesson that teaches the idea.
Terms with a qualifier in parentheses name the area where they apply.
In-flight batching and continuous batching, and TPS and perceived TPS, are different names for the same ideas.