Scalability, Availability, and Reliability

Scalability is the ability of a system to handle increased load by adding resources. Availability is the fraction of time a system is able to serve requests. Reliability is the probability that a system performs its function correctly over a period of time. The three are related but distinct: a system can scale well and still be unreliable, and a system can be highly available while returning wrong results.

Vertical and horizontal scaling

Vertical scaling (scaling up) means using a larger machine: more CPU, memory, or disk. Horizontal scaling (scaling out) means adding more machines and distributing work among them.

Most designs start vertically for simplicity, then scale out the layers that become bottlenecks.

Stateless services

A stateless service keeps no client-specific state in its own memory between requests. Every request contains, or can look up, all information needed to process it. State lives in shared stores: a database, a cache, or a session store.

Properties that follow:

• Any instance can serve any request, so a load balancer can use simple algorithms such as round robin.
• Instances can be added or removed without losing user data.
• A failed instance is replaced without recovery of its local state.
• Deployments can roll instance by instance.

Session state can be moved out of instances in two ways: stored server-side in a shared store keyed by a session ID, or stored client-side in a signed token that each request carries. The alternative, sticky sessions, pins a client to one instance, which makes that instance's failure lose the session and makes load uneven.

Stateful components (databases, caches, queues) cannot be cloned freely. They scale through replication and partitioning. See database replication and sharding and partitioning.

flowchart LR
    C["Clients"] --> LB["Load balancer"]
    LB --> A1["App server 1"]
    LB --> A2["App server 2"]
    LB --> A3["App server 3"]
    A1 --> S[("Shared state: database and cache")]
    A2 --> S
    A3 --> S

Latency versus throughput

Latency is the time one request takes. Throughput is the number of requests completed per unit time. Scaling out generally raises throughput, because more machines process more requests in parallel. It does not reduce the latency of a single request unless the work of that request is itself split across machines. Reducing latency requires other techniques: caching, faster storage, fewer network hops, data placed near users.

Percentile latency (p95, p99) is used for targets. A request that calls many backends in parallel is as slow as its slowest call, so tail latency grows with fan-out: if each of 100 parallel calls is slow on 1 percent of requests, about 63 percent of top-level requests include at least one slow call (1 - 0.99^100).

Availability and the nines

Availability = uptime / (uptime + downtime). Expressed with failure statistics, availability = MTBF / (MTBF + MTTR), where MTBF is the mean time between failures and MTTR is the mean time to repair. Availability improves either by failing less often or by recovering faster, and automated failover reduces MTTR. It is commonly quoted in nines.

Availability of composed components

Components in series (every one is required) multiply availabilities. Two components at 99.9 percent give 99.9 percent x 99.9 percent, about 99.8 percent. Chains of synchronous dependencies therefore have lower availability than any single member.

Components in parallel (any one suffices, with failures independent) fail only when all fail: availability = 1 - (1 - a)^n. Two independent replicas at 99 percent give 1 - 0.01 x 0.01 = 99.99 percent. The formula also assumes the failover between replicas is instantaneous and always succeeds, so real availability is lower. The independence assumption is important: replicas sharing a power source, network switch, deployment, or software bug do not fail independently.

public class Main {
    static double series(double... parts) {
        double product = 1.0;
        for (double part : parts) {
            product *= part;
        }
        return product;
    }

    static double parallel(double a, int n) {
        return 1.0 - Math.pow(1.0 - a, n);
    }

    public static void main(String[] args) {
        System.out.printf("series of 3 x 99.9%%: %.4f%n", series(0.999, 0.999, 0.999));
        System.out.printf("2 replicas at 99%%: %.4f%n", parallel(0.99, 2));
        System.out.printf("3 replicas at 99%%: %.6f%n", parallel(0.99, 3));
    }
}

series of 3 x 99.9%: 0.9970
2 replicas at 99%: 0.9999
3 replicas at 99%: 0.999999

SLI, SLO, and SLA

• Service level indicator (SLI). A measured quantity describing service behavior, for example the fraction of requests that succeed, or the p99 latency.
• Service level objective (SLO). A target for an SLI over a time window, for example "99.9 percent of requests succeed over 30 days".
• Service level agreement (SLA). A contract with customers that defines consequences, typically financial credits, when a target is missed. SLAs are set looser than internal SLOs so that internal alerting fires before a contract is breached.

The difference between 100 percent and the SLO is the error budget: the amount of failure allowed in the window. A 99.9 percent SLO over 30 days allows 0.1 percent failure, which is 43.2 minutes of full outage, or 1 failed request in every 1,000. When the budget is spent, teams typically slow feature releases and prioritize reliability work.

Redundancy and single points of failure

A single point of failure (SPOF) is a component whose failure stops the whole system. The remedy is redundancy: more than one instance of the component, with a mechanism to switch traffic.

Common SPOFs to check in a design: the load balancer itself, a single database primary, a single queue broker, a DNS provider, a leader in a coordination service, and shared configuration.

Redundancy needs failure detection (health checks, heartbeats), a switching mechanism (load balancer removal, leader election, DNS failover), and data replication so a standby has the state. See load balancing.

Reliability

Reliability concerns correctness under faults. A fault is a component deviating from its specification (a disk error, a crashed process, a dropped packet, a software bug). A failure is the system as a whole no longer providing its service. Fault tolerance is the design goal of preventing faults from becoming failures. Faults come from hardware, software, and human operation errors such as bad configuration changes, and the last two are correlated across replicas in ways hardware faults are not. Fault-tolerant design techniques include:

• Replication and checksums for data integrity.
• Timeouts, retries with backoff, and idempotent operations for transient failures.
• Circuit breakers and load shedding to prevent overload from spreading.
• Graceful degradation: serving reduced functionality instead of failing.
• Monitoring and alerting to detect faults early.

Interview questions

• Why must stateless services keep session data outside the instance? So any instance can handle any request and instances can be replaced or scaled without losing sessions. Session data goes to a shared store or into a signed token held by the client.
• Does horizontal scaling reduce latency? Not directly. It increases the number of requests handled concurrently. Latency of a single request changes only if its work is divided or if queuing delay was the cause.
• What is the availability of two dependent services each at 99.9 percent called in sequence? About 99.8 percent, since availabilities multiply in series.
• Why is redundancy alone not sufficient? The switch-over mechanism must work, replicas must hold current data, and failures must be independent. A shared dependency reintroduces a single point of failure.
• What is an error budget? The allowed amount of unreliability, equal to 1 minus the SLO over the window. It is used to balance feature velocity against reliability work.