codeloopback
Guide contents

Local LLMs and Coding Agents in Java

Run language models on your own machine and build a coding agent in Java: tokenizers, transformers, sampling, quantization, GGUF and Hugging Face, llama.cpp and vLLM, inference engineering, an inference engine in pure Java, and an Ollama-style CLI with picocli that downloads models, chats, and writes, runs, and fixes code with tools.

73 lessons in 10 sections

Start with What Is a Large Language Model? →

How LLMs Work

What an LLM is, prompts and context, the vocabulary of agents and LLM engineering, a first run, next-token prediction, tokenizers and BPE, the math of transformers, self-attention, the transformer block, prefill, decode and the KV cache, sampling, training stages, and chat templates.

  1. 1.What Is a Large Language Model?
  2. 2.Talking to a Model: Prompts, Context, and Limits
  3. 3.Agents and LLM Engineering: The Vocabulary
  4. 4.Introduction and First Run
  5. 5.What a Language Model Computes
  6. 6.Tokens, Vocabularies, and Special Tokens
  7. 7.Building a BPE Tokenizer
  8. 8.The Math Behind Transformers, in Java
  9. 9.Self-Attention Step by Step
  10. 10.The Transformer Block and the Full Model
  11. 11.Generation, Prefill, Decode, and the KV Cache
  12. 12.Sampling: Greedy, Temperature, Top-k, Top-p, and Min-p
  13. 13.Penalties, Logit Bias, and Sampler Chains
  14. 14.How Models Are Trained, and the Variants You Download
  15. 15.Chat Templates, Roles, and Special Tokens

Hardware, Numbers, and Quantization

CPUs, GPUs, memory bandwidth and the roofline model, number formats from FP32 to FP4, memory math for weights and KV cache, and quantization from first principles to k-quants, i-quants, GPTQ, AWQ, and making your own GGUF.

  1. 16.CPUs, GPUs, and Memory for Inference
  2. 17.FLOPs, Bandwidth, and the Roofline Model
  3. 18.Number Formats: FP32, FP16, BF16, FP8, INT8, INT4, and FP4
  4. 19.Memory Math: Weights, KV Cache, and What Fits
  5. 20.Quantization Fundamentals
  6. 21.Building Q8_0 and Q4_0 Like ggml
  7. 22.k-quants, i-quants, and the Importance Matrix
  8. 23.GPTQ, AWQ, EXL3, NF4, and FP8 Serving
  9. 24.Making, Quantizing, and Testing Your Own GGUF

Model Files and Hugging Face

The Hugging Face Hub, its HTTP API and cache layout, and the safetensors and GGUF file formats, with a Java reader for each.

  1. 25.The Hugging Face Hub
  2. 26.The Hub HTTP API and Cache Layout in Java
  3. 27.The safetensors Format
  4. 28.The GGUF Format

The loop CLI and Model Management

Build the loop command-line tool with picocli: options, subcommands, configuration, packaging, reliable large downloads, and the pull, list, show, rm, and fit commands.

  1. 29.picocli Basics
  2. 30.Subcommands, Configuration, and Packaging
  3. 31.Downloading Large Files Reliably in Java
  4. 32.Workshop: loop pull, list, show, rm, and fit

Inference Engines

llama.cpp architecture, flags, tuning, and llama-server; the chat completions protocol and streaming in Java; vLLM; Ollama and the other engines; and the loop serve and chat commands.

  1. 33.llama.cpp Architecture
  2. 34.Running llama.cpp: Core Flags
  3. 35.Tuning llama.cpp
  4. 36.llama-server: Endpoints, Slots, and Prompt Cache
  5. 37.The Chat Completions Protocol and Streaming in Java
  6. 38.vLLM Architecture and Installation
  7. 39.Serving with vLLM
  8. 40.The Engine Landscape and How Ollama Works
  9. 41.Workshop: loop serve, run, and chat

Inference Engineering

FlashAttention and long context, batching and scheduling, paged KV cache and prefix caching, speculative decoding, constrained decoding, multi-GPU serving, and benchmarking.

  1. 42.Attention Optimizations and Long Context
  2. 43.Batching and Scheduling
  3. 44.Paged KV Cache and Prefix Caching
  4. 45.Speculative Decoding
  5. 46.Constrained Decoding and Structured Output
  6. 47.Multi-GPU Inference and Serving at Scale
  7. 48.Benchmarking Inference

Build an Inference Engine in Java

A transformer inference engine in plain Java: GGUF loading with the FFM API, a tokenizer from GGUF metadata, the forward pass, quantized kernels with the Vector API, and a chat engine.

  1. 49.Loading GGUF with the FFM API
  2. 50.A Real Tokenizer from GGUF Metadata
  3. 51.The Forward Pass in Java
  4. 52.Quantized Kernels, the Vector API, and Threads
  5. 53.Capstone: A Chat Engine in Pure Java

Tool Calling and Coding Agents

Function calling, the agent loop, file, shell, and web tools, permissions and sandboxing, context engineering, edit formats, small-model reliability, MCP, evaluation, and the loop agent command.

  1. 54.Function Calling from First Principles
  2. 55.The Agent Loop and a Tool Framework
  3. 56.File Tools
  4. 57.The Shell Tool
  5. 58.Web Search and Fetch Tools
  6. 59.Permissions, Sandboxing, and Prompt Injection
  7. 60.Context Engineering for Coding Agents
  8. 61.Edit Formats
  9. 62.Making Small Local Models Reliable
  10. 63.Workshop: loop agent
  11. 64.The Model Context Protocol
  12. 65.Evaluating Models and Agents

An HTTP Gateway with Spring Boot

An Ollama-style HTTP layer in front of llama.cpp and vLLM with Spring Boot: an OpenAI-compatible API, streaming proxy, model loading and scheduling, security, metrics, and the loop gateway workshop.

  1. 66.Spring Boot Essentials for an LLM Gateway
  2. 67.An OpenAI-Compatible API in Front of llama-server
  3. 68.Streaming Responses Through the Gateway
  4. 69.Loading, Unloading, and Scheduling Models
  5. 70.Securing and Observing the Gateway
  6. 71.Workshop: loop gateway

Reading Real Codebases

Guided reading of open-source coding harnesses (Pi, Aider, mini-swe-agent, OpenHands, Crush) and inference engines (llama.cpp, vLLM, Ollama, SGLang).

  1. 72.Reading Coding Harness Codebases
  2. 73.Reading Inference Engine Codebases