Capacity Planning and Cost

Capacity planning determines how much compute, memory, storage, and network a system needs to meet demand within its latency and availability targets, now and in the future. Too little capacity causes slowdowns and outages. Too much wastes money. The work combines demand forecasts, measured per-instance performance, and rules for spare capacity.

For quick estimation techniques used in interviews see back-of-envelope estimation.

Planning process

flowchart LR
    F[Forecast demand] --> M[Measure capacity per instance]
    M --> C[Compute instances needed]
    C --> H[Add headroom and redundancy]
    H --> V[Validate with load test]
    V --> R[Monitor and revise]
    R --> F

1. Forecast demand. Use current traffic, growth trend, seasonality (daily, weekly, and annual peaks), and planned events such as launches or sales. Plan for peak, not average. The peak-to-average ratio is often 2x to 10x or more, depending on the product.
2. Measure capacity per instance. Use load tests to find the request rate one instance handles while meeting the latency target. By Little's law, the average number of requests in flight equals the arrival rate times the average latency. At 500 requests per second and 200 ms average latency, an instance holds about 100 concurrent requests, which sizes thread and connection pools.
3. Compute instances. instances = peak load / usable capacity per instance.
4. Add headroom and redundancy. Reserve spare capacity for spikes, failures, and deployments.
5. Validate with a load test at the target scale.
6. Monitor utilization and revise as the forecast changes.

Worked example

Peak traffic forecast: 6,000 requests per second. Load testing shows one instance sustains 500 requests per second at the target p99 latency at about 70 percent CPU.

• Base: 6,000 / 500 = 12 instances.
• Headroom of 30 percent for unforecast spikes: 12 x 1.3 = 15.6, rounded up to 16.
• Tolerate the loss of one availability zone, with instances spread over 3 zones. After losing a zone, 2/3 of the fleet remains, so the fleet must be 16 / (2/3) = 24 instances to still cover 16 instances' worth of capacity.
• Result: 24 instances, 8 per zone. Normal utilization is then about 12 / 24 = 50 percent of the tested capacity at peak, which is the price of surviving a zone loss. After a zone fails, the remaining 16 instances carry 6,000 requests per second, 375 each, or 75 percent of the tested capacity.

The same approach applies to databases, caches, and queues, with the relevant unit being queries per second, memory, or messages per second.

Load testing

Load testing applies simulated traffic to measure behavior. Types:

Practices:

• Use a realistic mix of requests and data sizes. Synthetic uniform traffic gives misleading cache hit ratios.
• Watch latency percentiles, error rate, and resource saturation, not just throughput.
• Find the knee: the load at which latency begins to rise sharply as queues form. Operate well below it. Latency grows non-linearly as utilization approaches 100 percent, because of queueing. In the simple M/M/1 queue model (random arrivals, one server), average response time is the service time divided by 1 - utilization. That is 2x the service time at 50 percent utilization, 5x at 80 percent, and 10x at 90 percent. Real systems deviate from the model, but the shape holds.
• Avoid coordinated omission: a load generator that waits for slow responses before sending the next request under-reports latency during stalls. Use tools that send at a fixed arrival rate.
• Test the dependencies too. The limit is often the database, a downstream service, or a connection pool, not the application instances.
• Run tests in an environment representative of production, and take care that tests do not affect real users or send real emails and payments.

Headroom

Headroom is the difference between provisioned capacity and current demand. Reasons for it:

• Demand spikes faster than new capacity can start.
• Instance or zone failures remove capacity.
• Deployments temporarily reduce the number of serving instances (rolling updates).
• Latency rises steeply at high utilization.

Common practice is to keep peak utilization of a resource well below 100 percent, with the exact target (often 50 to 70 percent for latency-sensitive services) depending on measured behavior and failure tolerance. Stateful systems such as databases need larger margins, since they cannot scale out quickly.

N+1 and N+2 redundancy: provision for the load with one or two units failed.

Autoscaling

Autoscaling adjusts the number of instances automatically. It matches capacity to demand and reduces cost, but only works when new capacity can arrive in time and the service is stateless or otherwise easy to scale out (see scalability, availability, and reliability).

Policy types

In Kubernetes, the Horizontal Pod Autoscaler applies target tracking to pod metrics. See Kubernetes basics.

Tuning

• Choose the right metric. CPU is a poor signal for services limited by I/O or connection pools. Requests per instance, queue depth, or concurrency may correlate better with load.
• Startup time matters. If an instance takes several minutes to start and warm up, reactive scaling is late. Use headroom, scheduled scaling, or predictive scaling.
• Scale out fast, scale in slowly. A cooldown period after scaling and a delay before scale-in prevent flapping (rapid oscillation).
• Set minimum and maximum bounds. The minimum guarantees a baseline and redundancy across zones. The maximum caps cost and protects downstream systems from a sudden surge in connections.
• Scaling one tier moves the bottleneck. More application instances can overload the database. Scale dependencies together or protect them with connection limits and backpressure.
• Autoscaling does not fix unbounded load. Under abuse or a retry storm, it can add cost without helping. Combine with rate limiting and load shedding.

Cost drivers

Design choices that change cost:

• Cache and CDN reduce compute, database load, and origin egress.
• Compression and efficient serialization reduce transfer and storage.
• Storage tiering moves cold data to cheaper classes (see object and blob storage). Erasure coding needs less raw storage than triple replication.
• Replication factor and multi-region multiply storage and transfer costs. See multi-region and disaster recovery.
• Cross-zone chatter between services can add transfer charges in some clouds.
• Right-sizing: choose instance sizes from measured usage. Oversized instances waste money, and undersized ones cause saturation.
• Retention limits on logs, metrics, and backups cap growth of stored data.

Pricing models

A common mix is reserved or committed capacity for the steady baseline, on-demand capacity for variable demand above it, and spot capacity for interruption-tolerant work such as batch processing, CI jobs, and stateless workers designed to checkpoint or retry. Spot is unsuitable for a database leader or for the only copy of state. Exact discounts and terms vary by provider and change over time.

Serverless (see serverless and FaaS) charges per request and execution time. It is cost-effective for spiky or low-volume traffic and can cost more than provisioned instances at steady high volume.

Cost management practice

• Tag resources by team and service so spend can be attributed.
• Track unit cost, such as cost per 1,000 requests or per active user, rather than only the total, to see whether efficiency is improving as the system grows.
• Set budgets and alerts for anomalies.
• Review idle resources, unattached volumes, and old snapshots regularly.
• Treat cost as a non-functional requirement alongside latency and availability. Trade-offs are explicit: for example, an extra region improves availability and raises cost.

Interview questions

• How many instances are needed for a given peak load? Divide peak load by the load one instance sustains within the latency target, apply headroom for spikes, and add enough redundancy to survive the loss of an instance or zone. Validate with a load test.
• Why not run servers at 100 percent utilization? Queueing makes latency rise steeply as utilization nears saturation, and there is no spare capacity for spikes, failures, or deployments. Latency-sensitive services typically run well below full utilization.
• Why can reactive autoscaling fail during a sudden spike? New instances take time to launch and warm up, and metrics are collected with delay. The spike can overload existing instances first. Headroom, scheduled or predictive scaling, and load shedding cover the gap.
• When is spot capacity appropriate? For fault-tolerant, interruptible work such as batch jobs and stateless workers, where a reclaimed instance only causes a retry. It is not appropriate for the only copy of state.
• What is coordinated omission in load testing? A load generator that waits for each response before sending the next request stops sending during a stall, so it records too few slow requests and understates latency. Fixed-rate generators avoid this.