Model Parallelism: Tensor, Expert, Multi-Node

Every frontier language model today is too large to serve from a single GPU. Models grew, GPUs grew, and the models grew faster. When one GPU cannot hold a model plus the cache for its requests, the work has to be split across several GPUs. This is model parallelism: several GPUs cooperate on one copy of a model.

Splitting has a price. GPUs must exchange data while they work, and that exchange is slower than reading a GPU's own memory. This lesson explains how to estimate how many GPUs a model needs, the three ways to split a model (pipeline, tensor, and expert parallelism), which one fits which goal, and what changes when more than one machine is involved. Afterwards you can choose a split for a given model and explain why.

Terms from earlier lessons. VRAM is a GPU's own memory. An OOM (out-of-memory) error happens when a program asks for more VRAM than exists. The KV cache is the stored key and value vectors of the tokens seen so far (KV Caching). A node is one machine with several GPUs, often eight. NVLink and NVSwitch are the fast links between GPUs inside a node, and InfiniBand is the network between nodes (Instances, Multi-GPU, MIG, and Other Accelerators). A Mixture of Experts (MoE) model contains many small expert sub-networks, and each token uses only a few (Attention and Mixture of Experts).

Example 1: how many GPUs does a model need

At 8-bit precision (FP8), one billion parameters take about one gigabyte of VRAM. DeepSeek-V3.1 has 671 billion parameters, so its weights alone are about 671 GB. A single B200 GPU has 180 GB, so loading the weights on one GPU fails immediately with an OOM error.

Fitting the weights is not enough. Four B200 GPUs have 4 x 180 = 720 GB. In theory 720 GB can hold 671 GB of weights. But the KV cache needs space too, and it often takes 80 percent or more of the VRAM that remains after the weights. With 720 - 671 = 49 GB left, four GPUs could not serve DeepSeek with a reasonable sequence length or batch size. A full node of eight B200 GPUs is needed for real production traffic on a model of this size.

The minimum GPU count can be estimated by multiplying precision, parameter count, and the expected KV cache allocation, then rounding up to the next available instance size. This is the calculation, with DeepSeek in FP8:

bits_precision    = 8        (FP8 quantization)
params            = 671      (billions of parameters)
kv_cache_alloc    = 1.8      (multiplier: weights plus room for the cache)

vram_required = (bits_precision / 8) x params x kv_cache_alloc
              = (8 / 8) x 671 x 1.8
              = 1207.8 GB      (about 1200 GB)

B200 instance sizes in GB: 180 (1 GPU), 360 (2 GPUs), 720 (4 GPUs), 1440 (8 GPUs)

round up 1207.8 to the next size: 1440 GB
=> serving DeepSeek in FP8 needs 8 x B200

Reading it: the multiplier 1.8 says that the weights are only a little over half of the memory budget (1 / 1.8 = 56 percent), and the rest is reserved for the cache and buffers. The multiplier is a planning input, not a fixed constant: a deployment that needs a larger cache for long sequences or big batches uses a larger value. In words: pick the smallest instance whose VRAM covers weights plus cache. Here 720 GB is too small, and 1440 GB is the next size up.

Using more than the minimum is often useful. Even for a mid-size model such as GPT OSS, extra GPUs allow larger KV caches and better per-user latency. But all of this assumes that inference scales well from one GPU to several, and the limit is the communication between GPUs.

Example 2: the communication limit

NVLink and NVSwitch connect GPUs within a node, and InfiniBand connects nodes (see GPU Architecture). They have high bandwidth, but it is only a fraction of the speed of VRAM. Decode is memory-bandwidth-bound, so every byte that must cross a slower link risks becoming the new bottleneck. Designing multi-GPU inference to avoid those bottlenecks is called topology-aware parallelism.

A worked comparison with round numbers (illustration only; real figures depend on the hardware): if VRAM reads at about 8 TB/s and a link carries 0.9 TB/s, a transfer of the same size takes about 8 / 0.9 = 9 times longer over the link than from local memory. So the amount of data exchanged per layer has to be small compared with the work done, or the GPUs wait.

There are three main forms of model parallelism in inference:

Rules of thumb that follow: Tensor Parallelism is generally best for low-latency inference within a single node. Expert Parallelism raises throughput for MoE LLMs. Pipeline Parallelism is used only for multi-node inference.

graph TD
    M["Model too big for one GPU"] --> PP["Pipeline: GPU 1 has layers 1-20, GPU 2 has layers 21-40"]
    M --> TP["Tensor: every GPU holds a slice of every layer"]
    M --> EP["Expert: every GPU holds whole experts"]

There is also a family of data-parallel strategies such as Context Parallelism, which splits the computation itself across devices. They are rare in LLM inference but needed for video generation (Serving Image and Video Generation Models).

Example 3: Tensor Parallelism for lower latency

Tensor Parallelism should be the default for multi-GPU inference. It supports dense models such as Llama 405B (where all weights are used for every token) and the MoE models that dominate the open model landscape today.

TP splits each layer (unlike Pipeline Parallelism, which keeps layers whole) and spreads the pieces over the allocated GPUs. For each layer, the cost of reading weights from memory and running the matrix multiplication is shared. In a MoE model, each expert runs across multiple GPUs.

A small case: a weight matrix for one layer takes 8 GB at FP8. With TP8 (tensor parallelism over 8 GPUs) every GPU holds 8 / 8 = 1 GB of it and reads only 1 GB per token, so reading takes about one eighth of the time. For the whole of DeepSeek: 671 GB / 8 = about 84 GB of weights per GPU, which leaves most of each 180 GB GPU for the cache.

The cost is communication. After each layer, the partial results from the GPUs must be combined into one output before the next layer can start. This combining step is called an all-reduce. Inside a node with fast NVLink and NVSwitch, its overhead is small.

graph LR
    I["Layer input"] --> G1["GPU 1: slice 1"]
    I --> G2["GPU 2: slice 2"]
    I --> G3["GPU 3: slice 3"]
    I --> G4["GPU 4: slice 4"]
    G1 --> R["all-reduce: combine into one output"]
    G2 --> R
    G3 --> R
    G4 --> R
    R --> N["Next layer"]

Raising the TP degree improves tokens per second per user, as long as the model is big enough and the sequences are long enough that the communication does not cancel the speedup of the faster forward pass. That is the case for most frontier models.

Example 4: Expert Parallelism for higher throughput

Expert Parallelism divides the experts across GPUs. A model with 128 experts served as EP8 over eight GPUs puts 128 / 8 = 16 full experts on each GPU.

EP raises total system throughput, which makes serving more scalable and cheaper per token. Each token takes just as long as before, because an expert still runs on one GPU. But since experts process their tokens separately, the whole system handles more tokens at once.

EP also needs less communication than TP. The expert router, the small component that decides which experts a token uses, is copied to every GPU because it is small. GPUs only exchange tokens being sent to experts on other GPUs. Unlike TP, they do not have to gather the results of each layer.

Many deployments mix both to get both benefits. A common layout uses TP for the attention layers and EP for the sparse MoE layers.

graph LR
    A["Attention layers: TP across 8 GPUs"] --> E["MoE layers: EP across the same 8 GPUs"]

Because EP needs less communication, it scales well to multi-node deployments and to machines with limited interconnect bandwidth.

Example 5: multi-node inference

Serving a huge model at high precision, supporting inputs of millions of tokens, or simply wanting the fastest inference can require more than eight GPUs, which means several nodes. GPUs are designed to work across nodes, and multi-node training has been standard for years in building frontier models. Multi-node inference adds two challenges:

• Infrastructure: reliably provisioning two or more interconnected GPU nodes and building abstractions across cloud providers (see Multi-Cloud Capacity, Reliability, and Security).
• Parallelism: communicating effectively over InfiniBand, which is much slower than NVLink.

InfiniBand adds a wrinkle to topology-aware parallelism. TP needs too much communication between GPUs to suit multi-node use. Two options work well over InfiniBand:

1. For dense models, use TP inside each node and PP between nodes. A name such as TP8PP2 means TP over 8 GPUs in each node and 2 pipeline stages (2 nodes), for 16 GPUs.
2. For MoE models, also try EP across all GPUs (for example EP16), which needs less communication than TP.

For MoE models, TP8PP2 generally gives lower latency per user, and EP16 gives higher overall system throughput.

TP8PP2:  node 1 = pipeline stage 1 (layers 1..N/2), TP across its 8 GPUs
         node 2 = pipeline stage 2 (layers N/2+1..N), TP across its 8 GPUs
         traffic between nodes: only the stage boundary, over InfiniBand
EP16:    16 GPUs over 2 nodes, 128 experts / 16 = 8 experts per GPU

Unless the model and KV cache really need more than one node, multi-node inference is probably not the best use of the extra hardware. The extra nodes are often better used for horizontal scaling (more replicas of the same deployment) or for disaggregated serving (Prefill-Decode Disaggregation).

Interview points

• Minimum GPUs: precision in bytes x parameters (in billions) x a KV-cache multiplier, rounded up to the next instance size. DeepSeek-V3.1 in FP8 gives 1 x 671 x 1.8 = about 1,200 GB, which rounds up to 1,440 GB, eight B200 GPUs. Four B200 GPUs (720 GB) hold the weights but leave too little for the cache.
• Communication between GPUs is the limit on scaling, because NVLink and InfiniBand are slower than VRAM. Designing around that is topology-aware parallelism.
• Pipeline Parallelism splits layers and has poor latency and utilization, so it is used only across nodes. Tensor Parallelism splits tensors in each layer, needs an all-reduce after each layer, and is best for low latency inside one node. Expert Parallelism puts whole experts on GPUs, needs less communication, and raises throughput of MoE models.
• A 128-expert model at EP8 has 16 experts per GPU. The expert router is replicated on each GPU because it is small.
• Many deployments use TP for attention and EP for MoE layers.
• Multi-node over InfiniBand: TP inside nodes with PP between them (TP8PP2) for dense models, or EP (EP16) for MoE models. TP8PP2 favors latency, EP16 favors throughput.
• Use multiple nodes only when needed. Otherwise extra nodes are better as replicas or for disaggregation.

Summary

• Frontier models do not fit on one GPU, and the weights must leave room for the KV cache, so the GPU count is estimated from precision, parameters, and a cache multiplier.
• The three splits are pipeline (layers), tensor (inside layers), and expert (whole experts).
• TP is the default for low latency within a node. EP raises throughput and needs less communication. Mixing them is common.
• Context Parallelism is a data-parallel style used mainly in video generation.
• Over InfiniBand, use TP8PP2 for dense models or EP16 for MoE models. Prefer replicas or disaggregation over extra nodes unless the model requires them.