Instances, Multi-GPU, MIG, and Other Accelerators

Nobody rents a bare GPU. In the cloud, GPUs come inside a package called an instance, and the quality of the other parts of that package decides whether the GPU can do its job. A model that is too large for one GPU needs several GPUs linked by fast connections. A model that is too small for one GPU wastes most of a large chip. This lesson covers the cases of both sizes: spreading one model across many GPUs, and slicing one GPU into several smaller ones. It ends with the other companies that build chips for AI inference.

Afterwards you can list what an instance contains, explain what NVLink, NVSwitch, and InfiniBand do and how their speeds compare, work out how many GPUs a model needs, describe how a multi-instance GPU is cut into slices, and name the competing accelerators and what each is betting on. The GPU names and memory sizes come from GPU generations.

Example 1: what an instance contains

The unit of GPU allocation in the cloud is the instance. An instance is a virtual machine, a software-defined computer that runs on a physical server. It includes six parts.

"Device" and "host" are the terms used throughout GPU programming: the GPU is the device and the CPU side of the machine is the host.

Not all GPUs and not all instances are alike. NVIDIA publishes reference architectures, but each cloud provider builds its systems to its own preferences, so instances differ in compute, memory, storage, and interconnect. Even the same GPU model can differ from one instance to another. The NVIDIA A100 comes in two form factors (physical designs): PCIe, which plugs into a standard slot, and SXM, which is mounted on a board built for GPUs. Most A100 inference runs on SXM because it has five percent higher memory bandwidth than the PCIe version.

When you provision an instance, check what you are actually getting. Any component, not only the GPU, can be a bottleneck or a point of failure during inference. A slow disk delays loading weights, and a slow network or interconnect delays everything that crosses it.

Example 2: eight GPUs in a node

A model is often too big for one GPU, or engineers want to use several GPUs together to improve performance. Two, four, eight, or more GPUs are common for large models such as DeepSeek and for demanding modalities such as video generation. The standard unit is the node, a server that contains eight GPUs. A B200 node contains eight B200 GPUs. Inside a node, two systems connect the GPUs.

• NVLink is a direct, one-to-one communication layer between GPUs. It reaches up to 1800 GB/s on Blackwell and 900 GB/s on Hopper.
• NVSwitch is an all-to-all layer on top of NVLink that coordinates all the GPUs in a node, so any GPU can reach any other.

These fast links make it possible to spread one model across up to a full 8-GPU node. When you size a deployment, the first question is how many GPUs a model needs. The calculation uses the rule from Inside a GPU: memory for the weights plus 50 percent headroom. The two example models are hypothetical.

needed = parameters x bytes per parameter x 1.5     GPUs = ceiling(needed / GPU memory)

70B parameters at 1 byte: needed = 70 x 1 x 1.5 = 105 GB
  H100 (80 GB):  105 / 80  = 1.31  -> 2 GPUs
  H200 (141 GB): 105 / 141 = 0.74  -> 1 GPU
  B200 (192 GB): 105 / 192 = 0.55  -> 1 GPU

400B parameters at 1 byte: needed = 400 x 1 x 1.5 = 600 GB
  H100 (80 GB):  600 / 80  = 7.5   -> 8 GPUs (one full node)
  H200 (141 GB): 600 / 141 = 4.26  -> 5 GPUs
  B200 (192 GB): 600 / 192 = 3.13  -> 4 GPUs

The 70B model at 8 bits needs 105 GB, which is two H100s, one H200, or one B200. The 400B model needs 600 GB. Eight H100s hold 640 GB, so one node is enough, but there is little room to spare. These are counts of memory only, and real deployments often round the count up to 2, 4, or 8 GPUs because parallelism methods split a model evenly. Speed also matters, and it depends on how the GPUs talk to each other, which is where the interconnect speeds come in.

Example 3: between nodes, InfiniBand

Sometimes one node is not enough. Running a very demanding workload on more than eight GPUs requires a fast link between nodes. The standard node-to-node interconnect for NVIDIA GPUs is InfiniBand. It competes with networking technologies such as Ethernet, and NVIDIA bought Mellanox, the maker of InfiniBand, in 2019.

InfiniBand is much slower than NVLink, with specifications up to 400 Gb/s per Network Interface Controller (NIC, the network card). It is still the fastest node-to-node interconnect on the market, since Ethernet tops out at 100 Gb/s per NIC. Note the units: NVLink is quoted in gigabytes per second (GB/s), while network links are quoted in gigabits per second (Gb/s). One byte is 8 bits, so 400 Gb/s is 50 GB/s.

Not every cloud provider uses InfiniBand. Some offer their own interconnects, and others have InfiniBand on some GPUs but not all. When you provision GPUs, double-check which interconnect is provided and what bandwidth it delivers.

flowchart LR
  subgraph NodeA["Node A (8 GPUs)"]
    A1["GPU"] --- SW1["NVSwitch (NVLink between all GPUs)"]
    A2["GPU"] --- SW1
    A3["... 6 more GPUs"] --- SW1
  end
  subgraph NodeB["Node B (8 GPUs)"]
    B1["GPU"] --- SW2["NVSwitch (NVLink between all GPUs)"]
    B2["GPU"] --- SW2
    B3["... 6 more GPUs"] --- SW2
  end
  SW1 -- "InfiniBand, up to 400 Gb/s per NIC" --- SW2

The calculation converts all the links to GB/s and computes how long each one needs to move 10 GB of data, a size picked to make the gaps visible for the exchanges of a heavy multi-GPU workload (the methods are in Model parallelism).

bytes per second = bits per second / 8
InfiniBand: 400 Gb/s / 8 = 50 GB/s        Ethernet: 100 Gb/s / 8 = 12.5 GB/s

time to move 10 GB = 10 / bandwidth
  NVLink Blackwell  1,800 GB/s: 10 / 1,800 = 5.6 ms     (1x)
  NVLink Hopper       900 GB/s: 10 / 900   = 11.1 ms    (2x slower than Blackwell NVLink)
  InfiniBand           50 GB/s: 10 / 50    = 200 ms     (36x slower)
  Ethernet           12.5 GB/s: 10 / 12.5  = 800 ms     (144x slower)

NVLink on Hopper / InfiniBand = 900 / 50 = 18x

NVLink on Hopper is 18 times as fast as InfiniBand, which is more than an order of magnitude. A faster link can carry far more data before it becomes the bottleneck. This is why techniques for using several GPUs are designed around the topology: heavy traffic stays on NVLink inside a node, and only lighter traffic crosses InfiniBand. Model parallelism and Prefill-decode disaggregation show the methods that respect this.

Example 4: NVL72 and the next rack-scale system

For NVLink connections among more than eight GPUs, NVIDIA offers a high-end rack-scale system. The NVL72 GB200 combines 72 Blackwell GPUs and 36 Grace CPUs in one full rack. It gives very high throughput for serving the largest models under heavy traffic. The next generation is the NVIDIA Vera Rubin NVL 144 CPX, which pairs the updated Vera CPUs and Rubin GPUs with the new Rubin CPX chip (the compute-bound prefill chip from GPU generations).

The calculation totals the memory of the standard nodes and the NVL72 rack, using the GPU memory sizes from the previous lesson. The GB200 memory per GPU is an assumption (the B200 figure of 192 GB), used only to show the scale.

8 x H100 node: 8 x 80  =    640 GB
8 x H200 node: 8 x 141 =  1,128 GB
8 x B200 node: 8 x 192 =  1,536 GB
NVL72 (72 Blackwell GPUs, assuming 192 GB each): 72 x 192 = 13,824 GB
NVL72 / one 8-GPU node: 72 / 8 = 9x the GPUs on NVLink

An NVL72 rack keeps 72 GPUs on NVLink, nine times as many as one 8-GPU node, so a model that spans many GPUs does not have to leave NVLink for the slower InfiniBand.

Example 5: when the GPU is too big, multi-instance GPUs

The opposite problem also occurs. The newest architectures need big GPUs such as the H100 or B200, with large amounts of compute and memory. A model with a couple of billion parameters or fewer cannot use that much. Even with large batch sizes, valuable GPU resources sit idle. Moving the small model to an older, slower GPU is one option. A better one is to run it on a fraction of a new GPU.

Multi-instance GPU (MIG) is a hardware-level capability of larger GPUs, including the A100, H100, H200, and B200. It divides one GPU into as many as seven pieces. Each piece gets its own slice of CPU, RAM, storage, and the other resources needed to form an instance. An H100 has eight memory slices and seven compute slices, and a MIG instance is built from some of each.

flowchart LR
  H["One H100: 7 compute slices, 8 memory slices"] --> I1["MIG instance A: 3 compute slices, 4 memory slices (40 GB)"]
  H --> I2["MIG instance B: 3 compute slices, 4 memory slices (40 GB)"]
  H --> I3["1 compute slice left over"]

The diagram is one possible split. Each instance is a separate small GPU with its own share of the other resources.

For example, an H100 MIG instance with three slices has about 3/7 of the compute and can use as much as half of the VRAM, which is 40 GB. It also receives about half of the CPU cores, CPU memory, storage, and network bandwidth of the H100 underneath.

Seven slices look odd, because software engineers think in powers of two. Compute slices are made of Streaming Multiprocessors, and GPUs rarely have a tidy count of them. An SXM H100 has 132 SMs. The GPU is cut into seven equal compute slices and the leftover SMs stay idle. The calculation shows the numbers.

Illustration of an even split: 132 / 7 = 18 per slice (remainder 6)   -> 7 x 18 = 126 SMs used, 6 SMs idle

compute of a k-slice instance = k/7 x 1,979 teraFLOPS (FP8 dense)
  1 slice : 1/7 x 1,979 = about 283 teraFLOPS
  2 slices: 2/7 x 1,979 = about 565 teraFLOPS
  3 slices: 3/7 x 1,979 = about 848 teraFLOPS
  7 slices: 7/7 x 1,979 = 1,979 teraFLOPS

3-slice instance: up to half the VRAM = 80 / 2 = 40 GB
3B-parameter model at 16 bits: 3 x 2 = 6 GB of weights

In this even-split illustration each compute slice has 18 SMs and 6 of the 132 SMs are left idle. The exact SM count per slice in a real MIG profile is set by NVIDIA and can be lower, so the number of idle SMs can be higher than 6. The point is that seven equal slices rarely use every SM. A 3-slice instance has about 848 teraFLOPS, still more than a whole L40 (362 teraFLOPS), and has 40 GB of memory. For a small model such as Orpheus TTS, a 3B-parameter text-to-speech model with about 6 GB of weights at 16 bits, running two MIG instances is sometimes a more efficient use of resources than giving the full GPU to a single instance. Each instance serves the model independently, so two instances can handle up to about twice the traffic of one on the same GPU.

Example 6: other datacenter accelerators

NVIDIA's lead in AI hardware has made it the world's most valuable company. It is far from the only company that builds hardware for AI inference. Large companies such as Amazon and Google and a large group of startups are putting billions of dollars into alternatives. The table lists the notable options. A public company has publicly traded stock, and a startup does not.

An ASIC (application-specific integrated circuit) is a chip designed for one job. It can be more efficient than a GPU at that job and is less flexible, because a GPU is a fairly general-purpose accelerator. Every company competing to win inference work from NVIDIA bets on a specific edge.

These options share three challenges.

• Software: without CUDA, hardware providers have to rebuild the whole inference stack for their accelerators (CUDA, kernels, and kernel fusion explains what CUDA provides).
• Manufacturing: companies have to build what is described as the most complex object humankind has ever created.
• Distribution: after a chip is made, someone must install it and bring it online so that capacity reaches the market.

Competition speeds up innovation. A market with more hardware options in the datacenter is good for every inference engineer, and competition will grow as inference workloads become more valuable. Exploration outside the datacenter, such as local inference, will grow too.

Interview points

• An instance is a virtual machine with GPUs (device), CPUs (host), host memory, storage, networking, and interconnect. Any part can be a bottleneck or failure point, so check them all when provisioning.
• The same GPU can differ between instances: the A100 comes as PCIe and SXM, and SXM has five percent more memory bandwidth.
• A node has eight GPUs. NVLink links GPUs one to one (up to 1800 GB/s on Blackwell and 900 GB/s on Hopper), and NVSwitch makes the links all-to-all inside the node.
• InfiniBand connects nodes, with up to 400 Gb/s per NIC against 100 Gb/s for Ethernet. NVIDIA bought Mellanox, its maker, in 2019. NVLink is about an order of magnitude faster, so topology decides which traffic goes where.
• The NVL72 GB200 joins 72 Blackwell GPUs and 36 Grace CPUs on NVLink in one rack. The Vera Rubin NVL 144 CPX is the next generation.
• MIG splits an A100, H100, H200, or B200 into up to seven slices. An H100 has seven compute slices and eight memory slices. A 3-slice instance has about 3/7 of the compute and up to half of the 80 GB VRAM. An SXM H100 has 132 SMs, so some SMs sit idle.
• MIG suits small models, such as Orpheus TTS (3B parameters), better than a whole big GPU.
• Alternatives to NVIDIA win on memory bandwidth, power efficiency, or platform integration, and face the software, manufacturing, and distribution challenges.

Summary

• A cloud instance bundles GPUs, CPUs, host memory, storage, networking, and interconnect.
• Large models need several GPUs. A node holds eight, linked by NVLink and NVSwitch. Beyond eight GPUs, InfiniBand (up to 400 Gb/s per NIC) links nodes, and the NVL72 GB200 rack links 72 GPUs on NVLink.
• Link speeds: NVLink 1800 GB/s on Blackwell and 900 GB/s on Hopper, InfiniBand 400 Gb/s (50 GB/s), Ethernet at most 100 Gb/s (12.5 GB/s).
• MIG fits small models on fractions of the newest GPUs, up to seven slices per GPU.
• Other accelerators: AMD MI350, AWS Inferentia and Trainium, Cerebras WSE-3, Etched Sohu, Furiosa RNGD, Google TPU, Groq LPU, Qualcomm Cloud AI 100 Ultra, and Sambanova RDU.
• Their edges are memory bandwidth, power efficiency, and platform integration. Their shared challenges are software, manufacturing, and distribution.