Containerization, Dependencies, and NIMs

The earlier lessons in this course made a single model run fast on a single GPU. That work only pays off when it reaches real users and keeps working as traffic grows. This chapter, which covers lessons 27 to 31, is about that step: production. Production is the live environment where real traffic arrives and mistakes cost money or users.

Production tests every assumption made during tuning. The length of prompts, the daily pattern of traffic, and even the topic a user decides to chat about change the observed performance. Keeping infrastructure secure and reliable is also a different skill from tuning a model on a GPU. If one instance serves a model very efficiently, enough traffic will still overwhelm it. That is not a PyTorch or CUDA problem. It is an infrastructure problem, and it needs different tools and a different way of thinking.

Scaling adds new questions: where to get GPUs, how to balance traffic across them, and how to prevent downtime. Cost accounting also gets harder, because the bill changes from "price per million tokens" to "pay directly for the machines". Latency in production comes from more than prefill and decode. The server, the network, and sometimes the client all add time, so the whole path has to be measured and tuned. The original notes behind this course come from Baseten, an inference provider, and end the chapter by pointing to it for mission-critical inference workloads.

This first lesson covers the packaging step. After it, a reader can explain what a container, an image, a Dockerfile, and a registry are, can describe why inference images need pinned dependencies, and can decide when to use an NVIDIA Inference Microservice (NIM) instead of building a container by hand.

Example 1: the four container words

A server program needs a specific operating system setup, specific library versions, and specific files. A developer laptop and a production machine rarely match. The classic result is "it works on my machine". Containerization is the practice of packaging an application together with its dependencies so that it is deployed in a standard way. A container turns a program into a packaged artifact that runs anywhere.

A smaller version of this idea is a program zipped together with its libraries. The zip still needs the right language runtime and operating system on the target machine. A container goes one step further and also bundles the operating system files and the runtime.

Containers are lightweight because they share the kernel of the host operating system. The word kernel here means the Linux kernel, not a CUDA kernel. A virtual machine carries its own full operating system, and a container does not, so a container starts faster and uses less space. This makes containers a good fit for inference services.

For most developers, containers mean Docker. Four terms come up all the time:

The relation is like class and object in object-oriented languages. An image is the fixed, built package (like a class), and a container is a running instance of it (like an object). The analogy stops at one point: many containers can start from one image, but a container's changes never flow back into the image, as the next example shows.

NVIDIA, several cloud providers, and Docker itself all run container registries. A popular registry for AI is Docker Hub. Docker Hub is to images what Hugging Face is to model weights and what PyPI is to Python packages: the place where people publish and download the packaged thing.

flowchart LR
    DF[Dockerfile] -->|build| IMG[Image]
    IMG -->|push| REG[("Registry")]
    REG -->|pull| HOST[GPU instance]
    HOST -->|run| CON[Container]

Reading the diagram from left to right: a Dockerfile builds an image, the image is pushed to a registry, a GPU instance pulls it, and the instance runs it as a container.

Example 2: layers and the container that forgets

Docker images are built from layers. A layer is a set of filesystem changes stacked on top of the layers below it. A developer can take a pre-configured base image and add more layers on top. There are three kinds:

flowchart BT
    A["Base image: OS and CUDA"] --> B["Additional layer: Python packages"]
    B --> C["Additional layer: inference engine"]
    C --> D["Additional layer: application code and config"]
    D --> E["Container layer: temporary, writable, lost on exit"]

The diagram stacks the layers from bottom to top. Everything except the top layer is read-only and shared. The top layer is the only one that a running program can change.

A worked case shows two facts: the image size is the sum of its permanent layers, and what a container writes at run time disappears when it ends. The sizes are made-up round numbers to keep the arithmetic readable. They are not measurements.

layer                            size
base: OS plus CUDA libraries     4.0 GB
python packages                  3.0 GB
inference engine                 2.0 GB
application code and config      0.1 GB
image size = 4.0 + 3.0 + 2.0 + 0.1 = 9.1 GB

container starts and writes /tmp/scratch.txt -> container layer holds 1 file
container terminates                          -> container layer discarded, 0 files
image size afterwards                         -> still 9.1 GB

The four permanent layers add up to 9.1 GB. The file created at run time lived only in the container layer, so it vanished at exit while the image stayed at 9.1 GB. This is why a service must not keep anything it needs later (for example uploaded files or results) on the container's own filesystem. Anything that must survive goes to storage outside the container.

The sizes also show why images matter for speed. A lesson later in this chapter, on cold starts, shows that loading a multi-gigabyte image onto a fresh instance is part of the waiting time for every new replica.

Inference engines such as vLLM and SGLang publish official base images for their active releases. Starting from one of these proven images is generally better than building one from scratch, because someone has already solved the hard matching of CUDA, drivers, and engine version.

Example 3: what an inference image contains

Dependency chains for inference are long and fragile. Reaching a working build is hard, and breaking changes are common in this ecosystem. A container preserves a known good build. An image is built for a specific GPU architecture and a specific model. It holds several kinds of runtime parts:

These parts must fit together. A PyTorch build made for one CUDA version may not run on a machine with a different one, and an engine release may need a specific torch version. Because each part constrains the others, the whole set is a "stack" that is validated together.

Two habits keep the stack healthy.

Pack light. A hiker on a backpacking trip carries only what is needed. Images built for inference are often many gigabytes. Fast deployment and efficient operation need a small image, so only strictly necessary dependencies go in.

Pin versions. A pinned dependency tree names the exact version of every dependency. Pinning keeps runtime behavior consistent across environments and lets the same image be rebuilt later with the same result. Tools such as uv, poetry, and pip flag version conflicts and fail the image build. After one successful build with pinned versions, every later build resolves to the same versions, which protects the service against breaking changes.

An illustration of a pinned requirements file follows. The version slots are placeholders, not real recommendations.

# requirements.txt (illustration)
torch==<exact version>
transformers==<exact version>
vllm==<exact version>

A worked case shows why pinning matters. Imagine a package index that publishes a new release of one library every month. The service code was written for the 1.x interface, and release 2.0 changed it. A build that asks for "latest" gets a different version each month. A build that pins an exact version gets the same result every time. The release numbers and the break are invented for the demonstration.

In months 1 and 2 the "latest" build works, so nothing looks wrong. In month 3 release 2.0 arrives with a changed interface, and every fresh build that asks for "latest" breaks. The build that pinned 1.1 keeps working in all four months. Pinning does not make the library better. It makes the build repeatable, so a change in the outside world cannot silently change the service.

Pinning also has a cost: the pinned versions fall behind. Updating them becomes a deliberate step, where the team raises the pin, rebuilds, tests, and deploys. Lesson testing and deployment covers how to roll such a change out safely.

New models and day-zero support

Breaking changes are most common around newly released models. When a new version of a model such as DeepSeek is announced, the whole inference ecosystem races to offer day-zero support, which means working support on the day the model is published. To get there, inference engineers often build images from overnight builds or other developer pre-releases instead of stable releases. These early versions have more bugs. The images are often rebuilt on stable releases in the days and weeks after the model drops.

Example 4: NVIDIA Inference Microservices (NIMs)

A NIM (NVIDIA Inference Microservice) is a pre-built Docker container for a popular open model. It packages the engine, the model setup, and the NVIDIA libraries so the service runs without assembling those parts by hand. Because it is a container, the inference service is portable. NVIDIA created two types:

NIMs exist for many common combinations of model, GPU architecture, GPU count, and configuration.

A NIM is like any other container, so there are three ways to use it:

1. As a starting point, with extra layers added on top.
2. As a reference architecture, to read and learn how NVIDIA configures a model.
3. As an out-of-the-box inference service.

The trade-off is control. A NIM is a done-for-you configuration. Someone who needs maximum control over the engine, the flags, and the dependency set is generally better off building a container from a less opinionated base image than adapting a NIM. The choice follows from the criterion: choose a NIM when a ready configuration for the exact model and GPU saves time, and choose a custom container when every layer needs to be controlled.

Interview points

• A container packages an application with its dependencies and shares the host Linux kernel, so it is lighter than a virtual machine. That kernel is unrelated to a CUDA kernel.
• Image is the built package, container is the running environment, Dockerfile is the build recipe, and registry is the store (Docker Hub for images, comparable to Hugging Face for weights and PyPI for Python packages).
• An image has base layers, additional layers from Dockerfile steps, and a thin ephemeral container layer that is discarded when the container ends.
• An inference image carries the CUDA toolkit version (CUDA, cuDNN, drivers), Python packages such as torch, transformers, and diffusers, the inference engine version, and system packages such as ffmpeg.
• Pin every dependency to an exact version so that a rebuild resolves identically, and keep images small because they are often many gigabytes.
• Day-zero support for a new model often uses pre-release builds, which are later rebuilt on stable releases.
• vLLM and SGLang publish official base images, and starting from one is generally better than building from scratch.
• A multi-LLM NIM serves a model family on a supported architecture. An LLM-specific NIM is tuned for one model on one GPU setup. Build a custom container when maximum control matters more than a ready configuration.

Summary

• Production work covers where to get GPUs, how to balance traffic, how to prevent downtime, how to account for cost, and how to remove latency from the server, network, and client.
• Containerization standardizes deployment and ends "it works on my machine".
• Docker images are layered, and the writable container layer is lost when the container terminates.
• Inference images are tied to a GPU architecture and a model, and contain CUDA components, Python packages, an engine, and system packages.
• Pinned dependencies preserve a known good build, and lean images deploy faster.
• NIMs are NVIDIA's pre-built containers for popular open models, in a multi-LLM form and an LLM-specific form, usable as a base, a reference, or a finished service.