Inference Engines
llama.cpp architecture, flags, tuning, and llama-server; the chat completions protocol and streaming in Java; vLLM; Ollama and the other engines; and the loop serve and chat commands.
- 33.llama.cpp Architecture
- 34.Running llama.cpp: Core Flags
- 35.Tuning llama.cpp
- 36.llama-server: Endpoints, Slots, and Prompt Cache
- 37.The Chat Completions Protocol and Streaming in Java
- 38.vLLM Architecture and Installation
- 39.Serving with vLLM
- 40.The Engine Landscape and How Ollama Works
- 41.Workshop: loop serve, run, and chat