codeloopback
Guide contents

Hardware, Numbers, and Quantization

CPUs, GPUs, memory bandwidth and the roofline model, number formats from FP32 to FP4, memory math for weights and KV cache, and quantization from first principles to k-quants, i-quants, GPTQ, AWQ, and making your own GGUF.

9 lessons

  1. 16.CPUs, GPUs, and Memory for Inference
  2. 17.FLOPs, Bandwidth, and the Roofline Model
  3. 18.Number Formats: FP32, FP16, BF16, FP8, INT8, INT4, and FP4
  4. 19.Memory Math: Weights, KV Cache, and What Fits
  5. 20.Quantization Fundamentals
  6. 21.Building Q8_0 and Q4_0 Like ggml
  7. 22.k-quants, i-quants, and the Importance Matrix
  8. 23.GPTQ, AWQ, EXL3, NF4, and FP8 Serving
  9. 24.Making, Quantizing, and Testing Your Own GGUF