Home
Data Structures and Algorithms
About

Guide contents

Inference Engineering

Foundations

1.What Is Inference Engineering
2.Scale, Specialization, and Your App
3.Choosing a Model: Evaluation, Fine-Tuning, Distillation
4.Measuring Latency and Throughput

Models

5.Neural Networks for Inference Engineers
6.How an LLM Generates Text
7.Attention and Mixture of Experts
8.How Image and Video Models Generate
9.Finding the Bottleneck: Compute or Memory
10.Optimizing Attention

Hardware

11.Inside a GPU: Compute, Memory, and Caches
12.GPU Generations: Hopper, Ada, Blackwell, Rubin, Grace
13.Instances, Multi-GPU, MIG, and Other Accelerators
14.Local Inference on Desktops and Phones

Software

15.CUDA, Kernels, and Kernel Fusion
16.PyTorch, Model Formats, ONNX, TensorRT, Transformers
17.Inference Engines: vLLM, SGLang, TensorRT-LLM, Dynamo
18.Benchmarking, Load Testing, and Profiling

Techniques

19.Quantization: Number Formats and Approaches
20.Speculative Decoding: Draft-Target, Medusa, EAGLE, N-grams
21.KV Caching, Prefix Reuse, and Long Context
22.Model Parallelism: Tensor, Expert, Multi-Node
23.Prefill-Decode Disaggregation

Modalities

24.Vision Language Models and Embedding Models
25.Speech: ASR, Diarization, TTS, Speech-to-Speech
26.Serving Image and Video Generation Models

Production

27.Containerization, Dependencies, and NIMs
28.Autoscaling, Batching, Cold Starts, and Routing
29.Multi-Cloud Capacity, Reliability, and Security
30.Testing, Deployment, Cost, and Observability
31.Client Code: Latency, Async Inference, Streaming

Reference

32.Inference Engineering Glossary
33.Further Reading

Interview Practice

34.Interview Questions: Models, Hardware, Software
35.Interview Questions: Techniques, Modalities, Production

Inference Engineering

Foundations

1.What Is Inference Engineering
2.Scale, Specialization, and Your App
3.Choosing a Model: Evaluation, Fine-Tuning, Distillation
4.Measuring Latency and Throughput

Models

5.Neural Networks for Inference Engineers
6.How an LLM Generates Text
7.Attention and Mixture of Experts
8.How Image and Video Models Generate
9.Finding the Bottleneck: Compute or Memory
10.Optimizing Attention

Hardware

11.Inside a GPU: Compute, Memory, and Caches
12.GPU Generations: Hopper, Ada, Blackwell, Rubin, Grace
13.Instances, Multi-GPU, MIG, and Other Accelerators
14.Local Inference on Desktops and Phones

Software

15.CUDA, Kernels, and Kernel Fusion
16.PyTorch, Model Formats, ONNX, TensorRT, Transformers
17.Inference Engines: vLLM, SGLang, TensorRT-LLM, Dynamo
18.Benchmarking, Load Testing, and Profiling

Techniques

19.Quantization: Number Formats and Approaches
20.Speculative Decoding: Draft-Target, Medusa, EAGLE, N-grams
21.KV Caching, Prefix Reuse, and Long Context
22.Model Parallelism: Tensor, Expert, Multi-Node
23.Prefill-Decode Disaggregation

Modalities

24.Vision Language Models and Embedding Models
25.Speech: ASR, Diarization, TTS, Speech-to-Speech
26.Serving Image and Video Generation Models

Production

27.Containerization, Dependencies, and NIMs
28.Autoscaling, Batching, Cold Starts, and Routing
29.Multi-Cloud Capacity, Reliability, and Security
30.Testing, Deployment, Cost, and Observability
31.Client Code: Latency, Async Inference, Streaming

Reference

32.Inference Engineering Glossary
33.Further Reading

Interview Practice

34.Interview Questions: Models, Hardware, Software
35.Interview Questions: Techniques, Modalities, Production

Home/Inference Engineering/Foundations

Foundations

What inference is, the kinds of AI apps and workloads, choosing a model, and measuring latency and throughput.

4 lessons

1.What Is Inference Engineering
2.Scale, Specialization, and Your App
3.Choosing a Model: Evaluation, Fine-Tuning, Distillation
4.Measuring Latency and Throughput

© 2026 codeloopback