codeloopback
Guide contents

Inference Engineering

Serve generative AI models fast, cheaply, and reliably: LLM and image model mechanics, compute and memory bottlenecks, GPUs, CUDA and inference engines, quantization, speculative decoding, KV caching, parallelism, disaggregation, speech and vision models, autoscaling, multi-cloud production, and interview questions.

35 lessons in 9 sections

Start with What Is Inference Engineering →

Foundations

What inference is, the kinds of AI apps and workloads, choosing a model, and measuring latency and throughput.

  1. 1.What Is Inference Engineering
  2. 2.Scale, Specialization, and Your App
  3. 3.Choosing a Model: Evaluation, Fine-Tuning, Distillation
  4. 4.Measuring Latency and Throughput

Models

How neural networks, LLMs, attention, mixture of experts, and image and video models run, and how to find their bottleneck.

  1. 5.Neural Networks for Inference Engineers
  2. 6.How an LLM Generates Text
  3. 7.Attention and Mixture of Experts
  4. 8.How Image and Video Models Generate
  5. 9.Finding the Bottleneck: Compute or Memory
  6. 10.Optimizing Attention

Hardware

GPU compute, memory, and caches, GPU generations, instances, other accelerators, and local inference.

  1. 11.Inside a GPU: Compute, Memory, and Caches
  2. 12.GPU Generations: Hopper, Ada, Blackwell, Rubin, Grace
  3. 13.Instances, Multi-GPU, MIG, and Other Accelerators
  4. 14.Local Inference on Desktops and Phones

Software

CUDA and kernels, PyTorch and model formats, inference engines, NVIDIA Dynamo, and benchmarking.

  1. 15.CUDA, Kernels, and Kernel Fusion
  2. 16.PyTorch, Model Formats, ONNX, TensorRT, Transformers
  3. 17.Inference Engines: vLLM, SGLang, TensorRT-LLM, Dynamo
  4. 18.Benchmarking, Load Testing, and Profiling

Techniques

Quantization, speculative decoding, KV caching, model parallelism, and disaggregation.

  1. 19.Quantization: Number Formats and Approaches
  2. 20.Speculative Decoding: Draft-Target, Medusa, EAGLE, N-grams
  3. 21.KV Caching, Prefix Reuse, and Long Context
  4. 22.Model Parallelism: Tensor, Expert, Multi-Node
  5. 23.Prefill-Decode Disaggregation

Modalities

Serving vision language, embedding, speech, image, and video generation models.

  1. 24.Vision Language Models and Embedding Models
  2. 25.Speech: ASR, Diarization, TTS, Speech-to-Speech
  3. 26.Serving Image and Video Generation Models

Production

Containers, autoscaling, multi-cloud capacity, deployment, cost, observability, and client code.

  1. 27.Containerization, Dependencies, and NIMs
  2. 28.Autoscaling, Batching, Cold Starts, and Routing
  3. 29.Multi-Cloud Capacity, Reliability, and Security
  4. 30.Testing, Deployment, Cost, and Observability
  5. 31.Client Code: Latency, Async Inference, Streaming

Reference

Glossary of inference terms and further reading.

  1. 32.Inference Engineering Glossary
  2. 33.Further Reading

Interview Practice

Interview questions for senior and lead engineers covering the whole course.

  1. 34.Interview Questions: Models, Hardware, Software
  2. 35.Interview Questions: Techniques, Modalities, Production