codeloopback
Guide contents

Inference Engineering

FlashAttention and long context, batching and scheduling, paged KV cache and prefix caching, speculative decoding, constrained decoding, multi-GPU serving, and benchmarking.

7 lessons

  1. 42.Attention Optimizations and Long Context
  2. 43.Batching and Scheduling
  3. 44.Paged KV Cache and Prefix Caching
  4. 45.Speculative Decoding
  5. 46.Constrained Decoding and Structured Output
  6. 47.Multi-GPU Inference and Serving at Scale
  7. 48.Benchmarking Inference