What Is Inference Engineering

When an application calls a language model, the model has already been trained. Someone spent months and a large budget to learn its numbers. The application only needs answers, and it needs them quickly, cheaply, and every time it asks. Producing those answers in production is called inference, and the craft of making it fast, inexpensive, and reliable is inference engineering.

This lesson explains why the field exists, what it covers, and how the rest of the course is organized. It introduces the vocabulary used later: weights, open and closed models, runtime, infrastructure, and tooling. Afterwards you can say what an inference engineer does, name the three layers of an inference stack, and place any later lesson on the map.

Inference is the most valuable category in the AI industry, while inference engineering itself is still young. Inference engineers work across the whole stack, from CUDA (the GPU programming platform) at the bottom to Kubernetes (software that manages many servers) at the top, to make serving of generative AI models faster, less expensive, and more reliable.

No machine learning background is assumed. If you have called an LLM API from your own code, you already know the user's side of inference. This course covers the other side: what happens behind that API.

Example 1: training versus inference

A model is a large collection of numbers called weights. The weights decide how the model turns an input into an output. A model has two phases in its life:

A small analogy: training is like writing and printing a cookbook, and inference is like cooking a meal from it for a customer. The analogy stops at one point. Printing a book happens once, but a served model answers millions of requests, so the cost of inference grows with usage while the cost of training does not.

During the last decade's machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers learned both phases for classic models. Inference for classic models is relatively direct. In the early days of Baseten (an inference provider), the company ran models built with tools such as XGBoost on lightweight CPUs with a simple software stack.

Generative AI models are different. They are far larger, and the hardware they need is a GPU (graphics processing unit, a chip with thousands of small arithmetic units). Taking the weights, renting some GPUs, and starting a server does not give speed and reliability that are good enough for large-scale production use. Doing it well takes three layers of work, described in Example 4.

Example 2: why every team can now run a model

Inference engineering used to be rare. On November 30, 2022, the day ChatGPT launched, there were perhaps a few hundred inference engineers in the world, although they did not use that title yet. They worked mostly at frontier labs such as OpenAI, Midjourney, and Anthropic, or at large technology companies such as Google and NVIDIA.

At that time one possible future looked likely: training would be so hard and expensive that only a handful of companies would build closed models. Everyone else would consume intelligence through APIs, paying a token at a time. A token is a small piece of text, often a short word or part of a word, and API prices are quoted per million tokens.

Three years later, training turned out to be hard and expensive, but not limited to a handful of companies. Open models spread widely: more than two million of them were on Hugging Face (the largest public repository of models) in late 2025, about 25 times more than five years earlier. Research labs around the world release open models of every kind regularly:

The two kinds of model differ in whether the weights are public:

• A closed model is proprietary, and its weights are not available to the public. Examples are GPT-5 and Claude Sonnet.
• An open model has publicly available weights. Examples are Llama and DeepSeek. Open models are usually released under the MIT license or a similar permissive license. Some restrict commercial use, so the license terms must always be checked.

Until December 2024 there was a meaningful gap in intelligence between closed and open models. That gap disappeared when DeepSeek V3 and R1 were released. Today a new closed model is matched by an open model within months, sometimes within weeks. Occasionally an open model such as Kimi K2 Thinking even exceeds closed models for a short window.

Closed models keep getting smarter and cheaper, yet the movement toward open models is still accelerating. Even though open models chase closed ones on benchmarks, they change the choices of product builders. Both kinds keep improving, and each time they cross a capability threshold a new class of product becomes possible. In 2022 the AI-native products of today could not be built. Later, closed models got smart enough for categories such as customer service voice agents and AI-powered IDEs (integrated development environments, the editors programmers use). Those early models were slow, expensive, and unreliable, but the capability was there and companies formed around it. When open models crossed the same thresholds, builders began replacing closed models with them. Many also began fine-tuning open models, which means adapting a pre-trained model with new data, to cross thresholds sooner and even to beat closed models on their own product and domain.

Example 3: what open models change, and a calculation on "nines"

Moving to open models lets a team use inference engineering to improve the model behind a product in three new directions:

Latency is the time a user waits. Throughput is how much work the system finishes per second. The measuring lesson defines both exactly.

"Nines" is a way to quote availability, the share of time a service works. Two nines means 99 percent, three nines means 99.9 percent, and so on. The numbers look close together, but the allowed downtime differs enormously. The calculation below converts nines into downtime per year. A year has 365 x 24 x 60 = 525,600 minutes, and the downtime is the failing fraction of that.

minutes per year = 365 x 24 x 60 = 525,600
downtime per year = 525,600 x (1 / 10^nines)

2 nines (99%)      525,600 x 0.01    = 5,256 min     = 87.6 hours
3 nines (99.9%)    525,600 x 0.001   = 525.6 min     = 8.76 hours
4 nines (99.99%)   525,600 x 0.0001  = 52.56 min
5 nines (99.999%)  525,600 x 0.00001 = 5.26 min

80 percent less on a 100,000 bill: 100,000 x (1 - 0.80) = 20,000

Two nines allows 87.6 hours of downtime per year, more than three and a half days. Four nines allows about 52.56 minutes. Each extra nine divides the allowed downtime by ten. That is why the difference between "two nines" and "four nines" matters for a product that sits in the revenue path of a business. The last calculation is an illustration, not a measurement: "at least 80 percent less" means a bill of 100,000 would fall to 20,000 or less.

The dependence on an outside provider is the reason availability is capped. A product that calls one API cannot be more available than that API. A dedicated deployment is a service the team operates itself, so its uptime is a design decision.

Three years ago inference engineering looked like a niche. Now every company that wants differentiated, competitive AI products needs an inference strategy. The market shows it:

• AI-native startups such as Cursor, Clay, Gamma, and Mercor build products on open and in-house models and grow very fast.
• Digital-native companies such as Notion and Superhuman deeply integrate AI into products that hundreds of millions of people use.
• Teams that blend research and engineering, such as World Labs, Writer, Mirage, and many others, train and productize their own foundation models (large general-purpose models that other work builds on).
• Enterprise and regulated industries, which usually adopt technology slowly, are moving too. OpenEvidence, Abridge, and Ambience bring generative AI to healthcare. At the world's largest companies, AI initiatives are moving past the pilot stage into large-scale user adoption.

Baseten powers inference for all the companies listed above. The space is young, relatively few people work on inference, and newcomers can become experts quickly. Problems at every level of the stack are new and technically deep.

Example 4: the three layers of an inference stack

A complete inference system has three layers that must work together:

• Runtime: makes one model run as fast and efficiently as possible on one GPU-backed instance (an instance is one rented or owned machine, possibly with several GPUs).
• Infrastructure: scales across clusters, regions, and clouds without creating silos, while keeping uptime excellent.
• Tooling: gives engineers the right level of abstraction, so they have control and still stay productive.

flowchart TB
    T["Tooling: the abstraction engineers work with"]
    I["Infrastructure: autoscaling, capacity across regions and clouds, reliability"]
    R["Runtime: one model on one instance (CUDA, PyTorch, inference engines, kernels, techniques)"]
    T --> I --> R

The runtime layer makes sure that a model on a GPU, or across several GPUs in one instance, performs as well as it can. It rests on a software stack: CUDA (NVIDIA's programming platform for GPUs), PyTorch (a machine learning library), and inference engines such as vLLM, SGLang, and TensorRT-LLM (programs that load a model and serve requests). Low-level optimization matters here. A kernel (a small program that runs on the GPU) such as FlashAttention gives large gains.

On top of that stack, the runtime uses six model performance techniques that apply new research to the specific problems of inference on generative models:

These techniques work for models of every modality, not only LLMs (large language models). Other modalities are vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation. Each extends what AI systems can do and each needs its own optimizations. The speculative decoding lesson and the rest of the techniques section go deeper.

Runtime work is not enough, because any single model server eventually receives more traffic than it can handle. That is not a CUDA problem or a PyTorch problem. It is a systems problem that the infrastructure layer solves. The kind of infrastructure problem changes with scale:

Multi-cloud infrastructure, designed with care, also improves reliability, because downtime in one region or one provider does not stop the service. For global applications, running inference near end users improves end-to-end latency. One way to picture this is a request flowing through speculation (a runtime optimization for latency) and a figure of capacity unified across cloud providers. The production lessons cover both.

flowchart LR
    U[Users] --> G[Global routing]
    G --> A[Cloud A, region 1]
    G --> B[Cloud B, region 2]
    G --> C[Cloud C, region 3]
    A --- P[(One pool of compute)]
    B --- P
    C --- P

The third layer is tooling. Both inference providers such as Baseten and internal teams that build inference must decide what developer experience to offer. Developer experience is subjective, and the possible choices form a range:

The middle position is where this course aims. Its map covers technologies and techniques across runtime, infrastructure, and tooling.

Example 5: a map of the course

The course is organized into sections. Each row says what a section covers.

The sequence mirrors the layers. Models, hardware, and software build the runtime layer. Production builds the infrastructure and client side.

This course reflects the state of the field in early 2026. Like LLMs, courses have knowledge cutoffs: product names and numbers will change, while the principles, concepts, and foundational technologies stay useful for years.

Interview points

• Inference is the phase where a trained model serves production requests. Training learns the weights from data.
• Inference for generative AI is complex because the models are large and need GPUs. Taking weights and renting GPUs does not give fast, reliable production serving.
• A complete stack has three layers: runtime (one model on one instance), infrastructure (scaling across clusters, regions, and clouds), and tooling (the abstraction level for engineers).
• The six runtime techniques are batching, caching, quantization, speculation, parallelism, and disaggregation.
• Infrastructure problems change with scale: autoscaling first, capacity past a few hundred GPUs, then silos, then a global pool of compute.
• A closed model has no public weights. An open model does, usually under MIT or a similar license, and the license terms must be checked.
• Open models matter because they permit dedicated deployments that optimize latency, reach four nines of availability or better, and often cost at least 80 percent less at scale.
• Each extra nine of availability divides the allowed downtime by ten. Four nines is about 52.56 minutes per year.

Summary

• Inference engineering makes generative AI serving faster, less expensive, and more reliable.
• After ChatGPT launched in November 2022, a few hundred specialists did this work. Open models, with more than two million on Hugging Face, spread it to every engineering team.
• The gap between closed and open model intelligence disappeared in December 2024 with DeepSeek V3 and R1.
• Dedicated open-model deployments can improve latency, availability (four nines or better against two nines), and cost (often at least 80 percent less at scale).
• The stack has runtime, infrastructure, and tooling layers, which must work together.
• Runtime techniques: batching, caching, quantization, speculation, parallelism, disaggregation. They apply to every modality.
• Tooling should offer enough control for mission-critical work and enough abstraction for productivity.
• The course moves from foundations to models, hardware, software, techniques, modalities, and production.