Multi-Cloud Capacity, Reliability, and Security

Autoscaling inside one cluster works up to a point. A product with a global user base and heavy traffic needs thousands of GPUs spread around the world. No single cluster, and often no single cloud provider, can supply that many GPUs in the right places, keep them running through failures, and meet every legal requirement.

This lesson covers the next level up: managing GPU capacity across many clusters, regions, and providers. It starts with the architecture that coordinates them, then covers where GPUs come from, how traffic is sent to the nearest cluster, how often GPUs fail, which designs survive failures, and what security and compliance ask of an inference system. After it, a reader can explain why true multi-cloud is more than a collection of separate clouds, estimate failure counts for a GPU fleet, and compare active-active with active-passive designs.

Example 1: from siloed clouds to one pool

The simplest way to use several clouds is to treat each as a silo: some compute at one provider, some at another, with separate tooling for each. In that setup, compute cannot flow between clouds, and moving a workload from one cloud to another is a slow, error-prone, manual process.

True multi-cloud inference needs a multi-region, multi-provider bin packing tool. Bin packing means placing workloads into limited pools of capacity (the bins) so that the space is used well. The tool must treat distinct pools of compute as fungible, which means interchangeable: a GPU in one provider can stand in for a GPU in another. Kubernetes does this within a single cluster. Multi-cloud capacity management does it at global scale. It must take a global view, which allows self-healing (the system repairs itself when something fails) and global scheduling.

Running true multi-cloud inference gives four benefits:

Going from one cluster in one cloud to many clusters in many clouds needs a new coordination layer. It has two parts:

flowchart TD
    CP["Global control plane: deploys models, global scaling, receives events"]
    CP --- W1["Workload plane: cluster in provider A, region 1"]
    CP --- W2["Workload plane: cluster in provider B, region 2"]
    CP --- W3["Workload plane: cluster in provider C, region 3"]
    U1[Users] --> W1
    U2[Users] --> W2
    U3[Users] --> W3

This repeats the control and worker split of a single Kubernetes cluster (autoscaling, Example 1) at the level of many clusters. User traffic goes straight to the workload planes. The control plane is not in the request path.

The separation of duties is deliberate. Each workload plane can serve traffic on its own. If something happens to the control plane or to any one workload plane, the other workloads keep running unaffected.

Example 2: where GPUs come from

Several kinds of companies sell access to GPUs. The three main types:

They differ in capacity, availability, and reliability. Hyperscalers generally cost a premium. Across all providers there is a trade-off between cost and other factors: uptime SLAs, support, regional availability, instance configuration, and cluster sizes.

The first challenge is securing capacity at all. Getting the GPUs you want is often difficult, especially the newest hardware. Large clusters are harder still, since relatively few providers offer blocks of hundreds of nodes. Many cloud providers give most of their in-demand GPUs to their largest customers on long-term reservations. A team may therefore need several providers to get the GPUs it needs in the regions it needs.

Cloud GPUs are bought through three mechanisms:

Large-scale inference generally blends them. A baseline of low-cost reserved instances carries the steady load, and a mix of on-demand and spot instances handles traffic peaks. The GPUs are spread across clusters worldwide, close to end users.

A worked case shows why. It takes a day of demand, in GPUs per hour, and prices three plans. The prices are invented round numbers that only keep the relative order (reserved cheapest per hour, spot in between, on-demand highest). They are not quotes from any provider. Reserved GPUs are paid for all 24 hours whether used or not. Spot pre-emption notices are ignored.

demand per hour (GPUs): 20 x6, 30, 40, 60, 80, 90 x5, 80, 70, 60, 50, 40, 30, 20 x3
sum of hourly demand = 1,170 GPU-hours      lowest hour = 20 GPUs (the baseline)
prices (invented): reserved $2.00, spot $2.50, on-demand $4.00 per GPU-hour

all on-demand:         1,170 x 4.00                        = $4,680.00 per day
reserve for the peak:  90 GPUs x 24 h x 2.00               = $4,320.00 per day
blend:  reserved baseline  20 x 24 x 2.00                  =   $960.00
        peak GPU-hours above baseline = 1,170 - 480        =   690
        half on spot       345 x 2.50                      =   $862.50
        half on-demand     345 x 4.00                      = $1,380.00
        blend total                                        = $3,202.50 per day

Paying on-demand rates for everything costs the most. Reserving enough for the busiest hour avoids that rate, but it pays for idle GPUs in every quiet hour. The blend, with a reserved baseline and a mix for peaks, is the cheapest of the three here, because the steady 20 GPUs are always busy and the expensive capacity exists only during the peak hours. A real plan also has to account for spot pre-emption notices, which this simple model ignores.

Example 3: geo-aware load balancing

Successful AI products have users on every continent. As one cluster has a load balancer to give each GPU the right amount of traffic, a multi-cluster system needs a global load balancer. It has two goals that pull in different directions. A request should not wait in a queue while spare capacity sits elsewhere. But sending a request from Singapore to a server in San Francisco should not become a habit.

A rule of thumb: a request needs about five milliseconds to pass through one time zone. Sending data from New York to San Francisco crosses three time zones, so it takes fifteen milliseconds one way. Latency budgets are small, so workloads should run as close to the end user as possible.

A worked case applies the rule to standard-time offsets from UTC (New York -5, San Francisco -8, Singapore +8, Toronto -5). The formula is one-way ms = |offset A - offset B| x 5. The result is an estimate built on the rule of thumb. It is not a network measurement. Two clusters exist, one in San Francisco and one in New York.

New York -> San Francisco: |-5 - (-8)| = 3 zones x 5 = 15 ms one way
user in Singapore:     to San Francisco |8 - (-8)| = 16 zones = 80 ms
                       to New York      |8 - (-5)| = 13 zones = 65 ms   -> nearest: New York
user in Toronto:       to New York      0 zones = 0 ms                  -> nearest: New York
                       to San Francisco 3 zones = 15 ms
user in San Francisco: to San Francisco 0 ms                            -> nearest: San Francisco
                       to New York      15 ms

New York to San Francisco comes out at 15 ms. A Singapore user is 80 ms from San Francisco under this rule of thumb, so a request that goes there and back adds 160 ms before any inference starts, which is large against a 300 ms budget. The New York cluster at 65 ms one way is closer, and a cluster in the user's own region would be closer still. A real global load balancer weighs distance together with queue depth and spare capacity, so a nearby but overloaded cluster may lose to a slightly farther free one.

Example 4: building for reliability

GPUs are known for failing often in production. Everyone who has done a large training run knows to plan for hardware failure. The Llama 3 paper ("The Llama 3 Herd of Models", Grattafiori and colleagues, 2024) reported a run on 16,000 GPUs for 54 days with 419 unexpected interruptions, mostly from hardware failure. The arithmetic below turns this into a rate and then compares it with inference fleets.

GPU-hours in the run   = 16,000 GPUs x 54 days x 24 h = 20,736,000
GPU-hours per failure  = 20,736,000 / 419            = about 49,490  (roughly 50,000)

one 8-GPU node for a year = 8 x 24 x 365 = 70,080 GPU-hours
expected failures per node-year = 70,080 / 50,000 = 1.4

1,000-GPU fleet for a year = 1,000 x 24 x 365 = 8,760,000 GPU-hours
expected failures per year = 8,760,000 / 50,000 = 175   (about one every two days)

The run used about 20.7 million GPU-hours and had one interruption for roughly every 49,500 GPU-hours. That rounds to one failure per 50,000 GPU-hours. 50,000 hours sounds long, but one node of eight GPUs serving inference for a year uses over 70,000 GPU-hours, so a single node would be expected to see about 1.4 failures in a year at that rate. The last line is an extra estimate that applies the same rate to a 1,000-GPU fleet: about 175 failures a year, which is roughly one every two days. Inference engineers should expect hardware failure as a normal event.

A published breakdown shows the root causes of failures in the Llama 3 training run. In that chart, GPU problems are by far the largest slice. Smaller slices are dependency, host, network, and maintenance causes. The chart is a pie without printed numbers, so only the fact that GPU is the largest cause is reliable. The sizes of the other slices are not stated.

GPU health is a node-level concern. When one GPU fails, other GPUs on the same node often fail next or need to be taken offline for maintenance. Three actions keep an individual cluster healthy: proactively noting failures, cordoning nodes (marking a node so that no new work lands on it), and cycling pods (restarting the container groups that Kubernetes runs).

GPU failures are not the only cause of downtime. Cloud providers have scheduled maintenance and their own unscheduled outages. Every layer of the infrastructure needs reinforcing for high reliability.

Multi-cloud brings two designs for this:

flowchart LR
    subgraph AA[Active-active]
        U1[Traffic] --> A1[Region 1 serving]
        U1 --> A2[Region 2 serving]
    end
    subgraph AP[Active-passive]
        U2[Traffic] --> P1[Region 1 serving]
        P1 -.cut over on failure.-> P2[Region 2 idle standby]
    end

In both designs, failing over to another workload plane when a cluster, region, or provider goes down keeps reliability high and latency low.

A worked case counts the GPUs each design needs to carry a load of 100 GPUs and survive the loss of one region. It shows why active-active spreads cost better as regions are added: each region only needs headroom for its share of the lost load. With R regions, after one fails the remaining R - 1 carry everything, so each needs ceil(100 / (R - 1)) GPUs.

active-passive (1 active + 1 idle standby):  100 x 2 = 200 GPUs provisioned
active-active, 2 regions: ceil(100 / 1) = 100 each -> 200 provisioned
active-active, 3 regions: ceil(100 / 2) =  50 each -> 150 provisioned
active-active, 4 regions: ceil(100 / 3) =  34 each -> 136 provisioned

Active-passive keeps a full copy idle, so it costs twice the load. Active-active across 2 regions costs the same, but every region is busy in normal operation. With 3 regions each needs only 50 GPUs of capacity, 150 in total, and with 4 regions the total falls to 136. The standby capacity is shared across live regions instead of sitting idle. The price is more clusters to operate.

Example 5: security and compliance

Cloud security has been a topic for security and compliance teams for over twenty years. For AI models to run mission-critical applications, inference must be both secure and compliant. The conversations usually center on three areas:

One of the simplest ways to improve security is to not store user inputs or model outputs at all. This is not always possible, because of logging requirements or user agreements to keep usage data for future model training. When retention is not needed, not storing data shrinks the attack surface, which is the set of things an attacker could reach.

Securing inference is similar to securing any other containerized workload. Data encryption, container security, network and access controls, and workload isolation remain the gold standard, validated by extensive third-party penetration testing (hired attackers who try to break in and report weaknesses).

More and more inference engineers support applications in regulated industries and compliance-heavy regions. Multi-cloud helps in two ways.

1. Certified providers. For an application to comply with a certification such as SOC 2 Type II, or a regulation such as HIPAA, its providers generally must be compliant too. Being able to move workloads to compliant providers is useful.
2. Data residency. One model can run in several regions. Some industries and countries have data residency requirements, where user data from a country cannot be processed on servers in another country. With one cluster in a provider near Toronto and another in a provider near New York, Canadian data can stay in Canada and American data in the United States, with little added latency for users across the region.

Interview points

• Siloed multi-cloud has no fluid use of compute across clouds. True multi-cloud needs a multi-region, multi-provider bin packing tool that treats pools of compute as fungible, takes a global view, and supports self-healing and global scheduling.
• The benefits are capacity, redundancy, latency, and compliance.
• The control plane handles deployment, global scaling decisions, and real-time event streams. Workload planes handle inference traffic and in-cluster scaling, and report utilization and demand. They serve independently, so one failure does not take down the rest.
• Providers come as hyperscalers (AWS, GCP), neoclouds (Coreweave, Nebius), and resellers (SF Compute Company). Hyperscalers cost a premium, and every provider trades cost against uptime SLAs, support, regions, instance types, and cluster size.
• Reserved GPUs are discounted for months or years, on-demand GPUs cost more per hour up to a quota, and spot GPUs are discounted but can be pre-empted after a notice of often minutes. Large systems use reserved for the baseline and on-demand plus spot for peaks.
• A global load balancer avoids both queues while capacity is idle elsewhere and long trips. The rule of thumb is 5 ms per time zone, so New York to San Francisco is 15 ms one way.
• Llama 3 training had 419 unexpected interruptions over 54 days on 16,000 GPUs, about one failure per 50,000 GPU-hours, and one 8-GPU node serving a year is over 70,000 GPU-hours. GPUs were the largest failure cause.
• Keep node health by noting failures, cordoning nodes, and cycling pods, because a failed GPU is often followed by failures on the same node.
• Active-active serves live traffic from several planes, and active-passive keeps an idle hot standby and cuts over on failure.
• Security covers user data, model weights, and infrastructure. Not storing inputs and outputs is one of the simplest improvements. Encryption, container security, access controls, isolation, and penetration testing are the standard. SOC 2 Type II and HIPAA need compliant providers, and data residency rules need clusters in the right countries.

Summary

• Multi-cloud inference pools capacity from many providers through a control plane and independent workload planes.
• GPUs come from hyperscalers, neoclouds, and resellers, and are bought reserved, on-demand, or spot, usually as a blend.
• Global load balancing keeps requests close to users, using about 5 ms per time zone as an estimate.
• Hardware failure is expected: about one failure per 50,000 GPU-hours in the Llama 3 run, with GPUs as the main cause.
• Active-active and active-passive are the two failover postures.
• Security centers on user data, weights, and infrastructure, and multi-region clusters support compliance and data residency.