Autoscaling, Batching, Cold Starts, and Routing

A model server on one GPU instance has a fixed capacity. Traffic does not. A chat product can be quiet at 3 a.m., busy at noon, and flooded for ten minutes when a post goes viral. If the system is sized for the quiet hour, users wait or get errors during the spike. If it is sized for the spike, expensive GPUs sit idle most of the day.

Autoscaling is the automatic adjustment of how many copies of a model are running. The goal is to always have enough resources to serve every incoming request while meeting the latency promises (called SLAs, service level agreements, covered in measuring latency and throughput), without paying for idle GPUs. Without autoscaling, a system wastes resources during traffic lulls and misses its SLAs during spikes. A good autoscaler makes the resources follow the demand curve.

This lesson builds the idea in steps. It starts with what a Kubernetes cluster is, then explains the two ways to decide when to scale and the five settings that control the decision. It then covers batching and concurrency, cold starts, routing and queues, scale to zero, and scaling the steps of a multi-model pipeline independently. After it, a reader can size a replica count from a traffic number, reason about the trade-offs in each setting, and explain why a new replica can take minutes to become useful.

Example 1: Kubernetes, replicas, and the two planes

Kubernetes is an open-source container orchestration system. It starts, stops, and monitors containers (see containerization) across many machines. Autoscaling for inference uses Kubernetes together with a cluster-level system that provisions and deallocates compute, which means that it adds and removes machines.

Kubernetes can run one or more replicas of a model container. A replica is one running copy of the model service. Each replica runs on its own instance, the machine with the GPUs and the other hardware resources that the container needs.

Kubernetes composes a group of hardware resources into a cluster. A cluster has two kinds of components:

flowchart TD
    CP["Control plane: decides routing and scaling"] --> W1["Worker: replica of model A"]
    CP --> W2["Worker: replica of model A"]
    CP --> W3["Worker: replica of model B"]

A single control plane orchestrates many workers. A cluster can run several replicas of several models at once.

The open question is how many replicas each model should have. Unless traffic is unusually steady, no single number matches the need all day. Autoscaling dynamically adjusts the number of replicas that a model gets inside the cluster.

Example 2: utilization versus traffic

There are two ways to make the decision:

The two do not always agree. In LLM prefill, a few requests with hundreds of thousands of uncached input tokens can push utilization much higher than many small requests whose prefixes are already in the cache (KV caching). Traffic counts requests, so it sees many small requests as a large load and a few large ones as a small load. Utilization sees the opposite.

Traffic-based decisions can be made ahead of time because the requests are already visible before the GPUs react. Utilization only rises after the work has started, so it is a lagging indicator. Using both together keeps resources matched with demand.

A traffic-based autoscaling system has five settings:

The exact values decide how well the system holds its SLAs without wasting money. Raising the scale down delay prevents premature scale downs when traffic comes in bursts, but it also causes unnecessary spend after traffic has truly cooled off. A longer autoscaling window keeps the replica count steady. A shorter window reacts faster to spikes.

Example 3: sizing replicas from the concurrency target

The simplest traffic rule is a division. If each replica can handle concurrencyTarget requests at once, then ceil(inFlight / concurrencyTarget) replicas are needed, kept between the minimum and the maximum. The ceiling matters: 9 requests with a target of 8 need 2 replicas, because one replica would hold 9 and exceed its target.

Worked example with a target of 8, a minimum of 1, and a maximum of 5. The formula is replicas = clamp(ceil(inFlight / 8), 1, 5).

in-flight  ceil(n / 8)  after clamp to 1..5
 0         0            1
 3         1            1
 8         1            1
 9         2            2
25         4            4
40         5            5
60         8            5   (capped: 60 - 5 x 8 = 20 requests have nowhere to run and wait in a queue, Example 6)

With 0 requests the system still keeps the minimum of 1 replica. With 8 requests one replica is exactly full. With 9 the count rises to 2. With 25 requests, 25 / 8 rounds up to 4. With 40 requests the answer is exactly 5, which is the maximum. With 60 requests the formula wants 8 replicas, but the maximum allows 5, so 20 requests are over capacity.

The next worked case adds time. Each tick is one minute. The trace shows requests in flight: 10 in most minutes, and 40 in two bursts. The target is 10 per replica, so 40 requests need 4 replicas, and the maximum is 6. The scaler scales up at once, and scales down only after the lower count has been suggested for scaleDownDelay ticks. The window is the number of recent minutes whose highest traffic counts. replica-minutes adds the replicas of every minute and serves as a proxy for GPU spend.

minute:                   0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
requests in flight:      10 10 40 40 10 10 10 10 40 40 10 10 10 10 10 10

window=1 delay=0   replicas: 1 1 4 4 1 1 1 1 4 4 1 1 1 1 1 1   replica-minutes=28
window=1 delay=3   replicas: 1 1 4 4 4 4 4 1 4 4 4 4 4 1 1 1   replica-minutes=46
window=3 delay=0   replicas: 1 1 4 4 4 4 1 1 4 4 4 4 1 1 1 1   replica-minutes=40

The first row reacts instantly in both directions. It uses the fewest replica-minutes (28), but in a real system each scale up costs a cold start (Example 5), so a system that flaps up and down often pays that delay again and again. The second row keeps the 4 replicas for three extra minutes after each spike. This delay is too short to help here: the replicas drop to 1 at minute 7, and the second spike at minute 8 still meets a cold start, while the extra minutes cost 18 more replica-minutes (46 against 28). A delay of 4 minutes would have bridged the gap, at a still higher spend. The third row uses a longer window instead of a delay. The window keeps the count at 4 for two extra minutes, and it also drops back before the second spike, at a spend of 40. Both knobs trade money for stability, and neither protects against a spike that comes after a gap longer than the knob. This is the same trade-off as in the settings table: a longer delay protects against spiky traffic and wastes money after traffic really cools.

Example 4: batching and concurrency

One replica can serve several requests at the same time by batching: the GPU processes the requests together, so the cost of reading the weights from memory is shared (see finding the bottleneck). There are three approaches:

flowchart LR
    S["Static: wait for a full batch"] --> D["Dynamic: full batch or timeout"] --> C["Continuous: fill any free slot at token level"]

Inference engines such as vLLM, SGLang, and TensorRT-LLM implement continuous batching reliably. TensorRT-LLM calls it in-flight batching. The batching happens at the token level: after every generated token, a finished request leaves and a waiting one joins. This minimizes latency compared with static batching. More on the engines is in inference engines.

The worked case compares the three on the same arrival pattern: 8 requests, one every 100 ms (at 0, 100, ..., 700 ms), with a batch size of 4. Dynamic batching has a 150 ms timeout. In the continuous case each request occupies a slot for 300 ms, and a request starts at the first free slot. The result is the average wait before a request starts work.

static (batch of 4, wait until full)
  batch 1 starts at 300 ms: waits 300, 200, 100, 0  (sum 600)
  batch 2 starts at 700 ms: waits 300, 200, 100, 0  (sum 600)
  average = 1200 / 8 = 150 ms

dynamic (start when full or 150 ms after the batch's first request)
  requests at 0, 100      -> start at 150: waits 150, 50
  requests at 200, 300    -> start at 350: waits 150, 50
  requests at 400, 500    -> start at 550: waits 150, 50
  requests at 600, 700    -> start at 750: waits 150, 50
  average = 800 / 8 = 100 ms

continuous (4 slots, 300 ms per request)
  requests at 0, 100, 200, 300 take slots 1 to 4 and wait 0
  the request at 400 finds slot 1 free since 300 ms: waits 0
  the same holds for 500, 600, 700
  average = 0 ms

Static batching makes the earliest request in each batch wait for three later ones, so the average wait is 150 ms. The time cutoff in dynamic batching limits the wait to 150 ms for the first request in a batch and less for later ones, so the average drops to 100 ms, at the price of smaller batches. Continuous batching starts every request at once because there are 4 slots and a slot is always free by the time the next request arrives. These numbers come from a toy arrival pattern. Real results depend on traffic, but the ordering of the three approaches is the point.

Batch size is a trade-off between latency and throughput. A larger batch produces more total throughput, but each user's latency gets worse. The right size depends on the model, the instance, the latency target, and the budget, so performance must be tested at several batch sizes. The setting appears in two places that must match: the concurrency target in the autoscaler, and the batch size on the replica.

The scale rule follows from this. When every active replica reaches its maximum concurrency, the system starts more replicas. When enough replicas are running half-full batches, it is time to scale back down.

Example 5: cold starts

A cold start is the time it takes to bring up a new replica. The overall quality of an autoscaler depends on it. If new replicas take a long time, scaling down feels dangerous, because the next spike would find too few replicas. The system then over-provisions, which means it keeps more replicas than it needs.

Four steps make up a cold start, and each one is optimized separately:

flowchart LR
    A[GPU procurement] --> B[Image loading] --> C[Model weight loading] --> D[Engine startup] --> E[Ready to serve]

The timeline is a sum. For large models, weight loading is often the longest step.

Procurement. Unless the system keeps a pool of warm nodes that it moves between models, procurement speed is mostly a function of the cloud provider. Node start time is one of the factors that can be negotiated in a contract (multi-cloud capacity).

Image and weight loading. Loading means writing gigabytes, often hundreds of gigabytes, onto the instance. There are two ways to make it faster: make the data smaller, or get more bandwidth. Including only necessary steps and dependencies makes images smaller. Quantizing the model weights makes them smaller too, which speeds loading in cold starts as an extra benefit.

For small models the old strategy was to bake the weights into the image, which simplified caching and loading. Models now have tens or hundreds of billions of parameters, so the weights are far larger than the image and are better loaded separately.

Where the weights come from limits the bandwidth. Loading from a third party such as Hugging Face limits the speed to their egress speed (egress is data leaving their network). Storing weights in an S3 bucket adds network latency and data transfer costs. A multi-hundred-billion-parameter model needs gigabytes per second. The best way to get that is to load over the network inside a node from a source that is cached physically near the GPU instance in the same datacenter.

Engine startup. Engines such as vLLM and SGLang start quickly. Engines such as TensorRT-LLM, and models optimized with PyTorch, have a compilation step that targets the specific hardware to build the model's inference engine, and compilation often takes several minutes. Caching built engines improves cold start times a great deal. Both TensorRT-LLM and PyTorch have caching mechanisms for this. A cached engine only runs in an instance with exactly the same GPU type, CUDA version, and software dependencies as the environment where it was built. See frameworks and model formats for compilation.

The next worked case adds up the four steps for five setups. The numbers are invented round values, chosen to show the effect of each lever. A 70-billion-parameter model has 70 x 10^9 x 2 bytes = 140 GB of weights in 16-bit format and 70 GB in 8-bit format. Loading time is size divided by bandwidth. The fixed steps are 90 s for GPU procurement, a 12 GB image at 1 GB/s (12 s), and 20 s for engine startup.

16-bit weights, 2 GB/s source:        90 + 12 + 140/2  + 20 = 90 + 12 + 70 + 20 = 192 s
8-bit weights, 2 GB/s source:         90 + 12 +  70/2  + 20 = 90 + 12 + 35 + 20 = 157 s
8-bit weights, 10 GB/s nearby cache:  90 + 12 +  70/10 + 20 = 90 + 12 +  7 + 20 = 129 s
same plus uncached engine compile:    129 + 300                                = 429 s
same with the compiled engine cached: 129 s

Halving the weight size with 8-bit weights saves 35 seconds at the same bandwidth. Moving the weights to a fast nearby cache saves another 28. Compilation without a cache adds 300 seconds, more than all the loading work combined, and a cache removes it. The other fixed costs (procurement of 90 s here) remain, which is why the contract and the warm-node pool matter.

Example 6: routing, load balancing, and queues

With several replicas online, something must decide which request goes where. Two kinds of components make that decision:

A complex system does not have only one router and one load balancer. Routing happens throughout the stack, and load balancers sit at key points to keep system-wide performance stable.

The goal is an equal split of load, but splitting by count is not enough. Three replicas and twelve requests does not mean four requests each. Requests differ in input tokens. If most requests have 100 input tokens, a single request with 10,000 will unbalance a naive system. The worked case below has one 10,000-token request and eleven 100-token requests. It compares splitting by request count (round-robin, request i goes to replica i mod 3) with splitting by token load (each request goes to the replica with the fewest tokens so far).

round-robin by count:
  replica 0 gets requests 0, 3, 6, 9 = 10,000 + 100 + 100 + 100 = 10,300 tokens
  replica 1 gets requests 1, 4, 7, 10 =  4 x 100 = 400 tokens
  replica 2 gets requests 2, 5, 8, 11 =  4 x 100 = 400 tokens

least tokens first:
  request 0 (10,000) -> replica 0
  the eleven short requests alternate between replica 1 and replica 2 (ties go to the lower number)
  replica 0 = 10,000   replica 1 = 6 x 100 = 600   replica 2 = 5 x 100 = 500

Round-robin gives every replica four requests, yet replica 0 carries 10,300 tokens of work and the others carry 400 each. The token-aware rule puts the long request alone on replica 0 and spreads the short ones over the other two, giving 10,000, 600, and 500. A single huge request cannot be split, so some imbalance remains, but the short requests no longer queue behind it.

flowchart LR
    U[Requests] --> Q["Queue: holds overflow"]
    Q --> R["Router: where should this request go"]
    R --> A["Replica A: has the prefix cached"]
    R --> B["Replica B: has the LoRA loaded"]
    R --> C["Replica C: lightest load"]

The diagram shows a router choosing among replicas using what each one holds, with a queue in front that absorbs overflow.

Some requests are better handled by particular replicas. Two examples:

• KV cache-aware routing: send a request to a replica that already holds a matching prefix in its KV cache, so the prefill work for that prefix is skipped.
• LoRA-aware routing: send a request to a replica that already has the wanted LoRA fine-tuned weights in memory. (A LoRA is a small set of fine-tuning changes layered over a base model, see model selection.)

Intelligent routing uses information from the inference engine and from orchestrators such as NVIDIA Dynamo (see inference engines) to route by sequence length, prefix, and LoRA needs.

Routing and load balancing are still not enough. When more traffic arrives than the system can handle, it must hold requests while it scales up or while existing resources free up. The infrastructure part for this is a queue. A standard queue is first-in, first-out for the excess requests. A more complex version is a priority queue, for example to serve paid users before free users when traffic is high.

When a new replica comes online, the queue must see it. Requests should not keep waiting for old replicas. Each new replica should immediately be assigned queued traffic up to its concurrency limit.

Example 7: scale to zero

Advanced autoscaling systems support scale to zero: with no traffic, the system drops to zero active replicas, and when traffic arrives it scales up automatically. It depends on two prerequisites:

Even with both, scale to zero does not fit every workload.

• It suits development, where testing comes in bursts and the first request can be slow.
• In production it suits applications that receive traffic periodically. Examples are an agent used only during business hours in one country, and an offline system built for daily batch jobs.
• It does not suit a latency-sensitive application with light, unscheduled traffic that relies on scale to zero to stay cheap. That situation is probably a sign that the application is not ready for dedicated infrastructure and should use pay-per-token APIs until it reaches greater scale.

Example 8: independent component scaling

Modern AI applications are often compound: one request passes through several models in several stages. A voice agent is an example. A voice activity detector cuts the audio into chunks, a transcription model turns them into text, and an LLM answers (see speech models). These steps have different hardware needs. The detector needs a far less powerful GPU than the transcription model it feeds. The LLM that processes the transcript may need a whole multi-GPU node. The scaling parameters of each step also differ.

flowchart LR
    A["Voice activity detector: small GPU, many replicas"] --> B["Transcription model: mid GPU"] --> C["LLM: multi-GPU node"]

For such pipelines, the autoscaling decision is split, and each step scales on its own. This gives each step the right-sized resources and avoids both bottlenecks and over-provisioning.

Every model in the pipeline should still run in the same cluster, because network hops between clusters are slow. Example numbers: 10 ms for a message inside a cluster and 50 ms between clusters, across a 5-step pipeline, against a 1,000 ms SLA.

same cluster:    5 hops x 10 ms = 50 ms
cross cluster:   5 hops x 50 ms = 250 ms
extra time:      250 - 50 = 200 ms   (5 x 40 ms)
share of SLA:    200 / 1000 = 20 percent

The 40 ms difference per hop becomes 200 ms across five hops, which is 20 percent of a one-second SLA, spent on moving messages rather than on inference.

Interview points

• Autoscaling keeps enough replicas to meet latency SLAs without paying for idle GPUs. It runs on Kubernetes, where a control plane decides routing and scaling and a worker plane runs the containers.
• Utilization is a lagging signal, and traffic can be used proactively. They can disagree, because a few huge uncached prompts load the GPU more than many small cached ones. Use both.
• The five traffic-based settings are min replicas, max replicas, autoscaling window, scale down delay, and concurrency target. A longer scale down delay protects against spiky traffic and wastes money after traffic cools.
• Static batching waits for a full batch, dynamic batching adds a timeout, and continuous (in-flight) batching swaps requests in at the token level. Bigger batches raise throughput and worsen per-user latency, and the concurrency target and the replica batch size must match.
• Cold start has four steps: GPU procurement, image loading, model loading, and engine startup. Smaller images, quantized weights, a source cached near the GPU in the same datacenter, and cached compiled engines (valid only for the same GPU type, CUDA version, and dependencies) all shorten it.
• Weights are loaded separately from the image because they are far larger than the image for current models.
• A router answers where a request should go and a load balancer answers where it could go. KV cache-aware and LoRA-aware routing send requests to replicas that already hold the prefix or the LoRA.
• A queue holds excess traffic, can be a priority queue, and must hand queued requests to new replicas up to their concurrency limit as soon as they are active.
• Scale to zero needs fast cold starts and reliable queueing, fits development and periodic workloads, and is a warning sign for latency-sensitive apps with light traffic, which should use pay-per-token APIs.
• Pipelines scale per component and run in one cluster, because 40 ms extra per hop over 5 hops is 20 percent of a 1 second SLA.

Summary

• Autoscaling adjusts the number of model replicas to demand, using Kubernetes, utilization and traffic signals, and five settings.
• Batching lets a replica serve many requests at once, and continuous batching gives the lowest latency of the three approaches.
• Concurrency target and batch size must match, and replicas are added when all are at maximum concurrency and removed when batches run half-full.
• Cold start time is the sum of procurement, image loading, model loading, and engine startup, and each step has its own fixes.
• Routers and load balancers spread requests by more than count, and queues absorb overflow.
• Scale to zero trades latency for cost and fits only some workloads.
• Compound pipelines scale each step separately inside one cluster.