Load Balancing

A load balancer distributes incoming requests or connections across a pool of backend servers. It solves three problems: no single server has to handle all traffic (horizontal scaling), failed servers can be removed from rotation without clients noticing (availability), and servers can be added or drained without changing what clients connect to. Clients see one address, the balancer's virtual IP or name.

flowchart LR
    C1["Client"] --> LB["Load balancer (virtual IP)"]
    C2["Client"] --> LB
    LB --> S1["Server 1"]
    LB --> S2["Server 2"]
    LB --> S3["Server 3"]

Balancing works best with stateless services: any server can handle any request, because state lives in a shared store (see scalability, availability, and reliability).

Layer 4 versus Layer 7

Balancers are classified by the network layer at which they make decisions.

Consequences:

• A Layer 4 balancer distributes connections. With long-lived multiplexed connections such as HTTP/2 and gRPC, one client's requests all go to one backend. Layer 7 balancing distributes per request.
• Layer 7 balancing enables content-based routing (/api to one pool, /images to another). It overlaps with an API gateway or reverse proxy.
• Layer 4 balancing can use direct server return (DSR): responses go from the backend straight to the client, bypassing the balancer, which then handles only inbound traffic.

Algorithms

Static algorithms ignore server state:

• Round robin: send requests to servers in cyclic order. Simple and even when requests cost about the same and servers are identical.
• Weighted round robin: servers with higher weight receive proportionally more requests, for machines with different capacity.
• Random: pick a server uniformly. Approximates round robin at scale.
• Hash-based (IP hash): hash the client IP (or another key such as a session ID or URL) modulo the number of servers. The same key maps to the same server, which gives affinity without stored state. Adding or removing a server remaps most keys with plain modulo hashing: going from 3 to 4 servers keeps a key on the same server only when h % 3 == h % 4, which holds for 3 of every 12 hash values, so about 75% of keys move. Large-scale systems therefore use consistent hashing, which moves only a small fraction of keys. Hashing by URL helps cache locality on cache tiers.

Dynamic algorithms use current load:

• Least connections: send to the server with the fewest active connections. With active counts of 5, 1, and 3 on servers s1, s2, and s3, the next request goes to s2. Handles requests of unequal duration better than round robin, because slow requests keep connections open.
• Weighted least connections: divide the connection count by the server's weight.
• Least response time: prefer the server with the lowest recent latency (and few connections).
• Power of two choices: pick two servers at random and send to the less loaded one. It avoids the need to track global state exactly and avoids herding many balancers onto the one least-loaded server, and gives much better balance than pure random.

Sketch in Java

The core of four selection strategies. Round robin uses an AtomicInteger so concurrent requests each get a distinct turn. The hash is FNV-1a, a simple deterministic 64-bit hash implemented here (Java's String.hashCode would also work, but its distribution on similar keys is weaker). Power of two choices takes a seeded Random, so the example is reproducible.

import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {
    record Backend(String name, int active) {}

    static class RoundRobin {
        private final AtomicInteger next = new AtomicInteger();

        int pick(List<Backend> backends) {
            return Math.floorMod(next.getAndIncrement(), backends.size());
        }
    }

    static int leastConnections(List<Backend> backends) {
        int best = 0;
        for (int i = 1; i < backends.size(); i++) {
            if (backends.get(i).active() < backends.get(best).active()) {
                best = i;
            }
        }
        return best;
    }

    static int hashPick(String key, int n) {
        long h = 0xcbf29ce484222325L; // FNV-1a offset basis
        for (byte b : key.getBytes(StandardCharsets.UTF_8)) {
            h ^= (b & 0xff);
            h *= 0x100000001b3L; // FNV prime, wraps on overflow
        }
        return (int) Long.remainderUnsigned(h, n);
    }

    /** Pick two distinct backends at random and return the less loaded one. */
    static int powerOfTwoChoices(List<Backend> backends, Random rng) {
        int first = rng.nextInt(backends.size());
        int second = rng.nextInt(backends.size() - 1);
        if (second >= first) {
            second++;
        }
        return backends.get(second).active() < backends.get(first).active() ? second : first;
    }

    public static void main(String[] args) {
        var backends = List.of(new Backend("s1", 5), new Backend("s2", 1), new Backend("s3", 3));

        var rr = new RoundRobin();
        List<String> order = new ArrayList<>();
        for (int i = 0; i < 5; i++) {
            order.add(backends.get(rr.pick(backends)).name());
        }
        System.out.println("round robin: " + order);

        System.out.println("least connections: " + backends.get(leastConnections(backends)).name());

        int a = hashPick("203.0.113.7", backends.size());
        int b = hashPick("203.0.113.7", backends.size());
        System.out.println("same key, same backend: " + (a == b));

        var rng = new Random(42);
        int[] picks = new int[backends.size()];
        for (int i = 0; i < 3000; i++) {
            picks[powerOfTwoChoices(backends, rng)]++;
        }
        System.out.println("power of two choices never picks the most loaded: " + (picks[0] == 0));
    }
}

round robin: [s1, s2, s3, s1, s2]
least connections: s2
same key, same backend: true
power of two choices never picks the most loaded: true

Power of two choices compares only two random servers per request. The most loaded server (s1, 5 active connections) loses every comparison it takes part in, so it receives no requests here. With a larger pool the effect is the same in kind: heavily loaded servers are chosen only when both random picks are heavily loaded.

Health checks

The balancer must stop sending traffic to failed servers.

• Active checks: the balancer probes each backend at an interval. A Layer 4 check opens a TCP connection. A Layer 7 check requests an endpoint such as /health and expects a 200 status.
• Passive checks: the balancer observes real traffic and marks a server unhealthy after repeated errors or timeouts (outlier detection).
• Thresholds: mark a server down after N consecutive failures and up after M consecutive successes. This avoids flapping.
• Shallow versus deep checks: a shallow check verifies the process is alive. A deep check verifies dependencies such as the database, but if a shared dependency fails, a deep check can mark every server unhealthy at once and take down the whole service. See health checks and graceful degradation.
• Connection draining: when removing a server for a deploy, the balancer stops new connections but lets in-flight requests finish, up to a timeout.
• Slow start: newly added servers receive a gradually increasing share so that cold caches and JIT warm-up are not overwhelmed.

Sticky sessions

Sticky sessions (session affinity) send all requests from one client to the same server, either by hashing the client IP, or by a cookie that the balancer sets and reads.

• Needed when a server holds in-memory session state, or local caches that are costly to rebuild, and for long-lived connections such as WebSockets (see WebSockets, SSE, and long polling).
• Costs: uneven load (a heavy user pins to one server), lost sessions when the server fails or is removed, and complicated scaling and deploys.
• IP-based affinity breaks when many clients share one address (NAT, proxies) or a client's address changes.
• The preferred alternative is stateless servers with session data in a shared store such as Redis, or in a signed token, which makes affinity unnecessary.

Avoiding the balancer as a single point of failure

A single balancer is itself a single point of failure and a capacity limit.

• Active-passive: two balancers share a virtual IP. The active one serves traffic and sends heartbeats (for example with VRRP, as implemented by keepalived). If the passive one stops receiving heartbeats, it takes over the virtual IP. Failover takes seconds, and half the capacity sits idle.
• Active-active: several balancers serve traffic at once. Traffic reaches them through DNS returning several addresses, or through ECMP (equal-cost multipath) routing in the network, which hashes each flow (by its address and port tuple) to one of the balancers announcing the same address. When a balancer is added or removed, the flow-to-balancer mapping can change, so balancers must tolerate receiving mid-connection packets. This scales capacity as well as availability.
• Connection state: if a balancer tracks connections, a failover can reset them unless state is replicated or the mapping is derived deterministically (for example with consistent hashing on the connection tuple, as in Google's Maglev).
• Cloud providers offer managed balancers that handle this redundancy internally.

Global server load balancing

Global server load balancing (GSLB) distributes traffic across data centers or regions, not servers within one site.

• DNS-based: the authoritative DNS returns the address of the best region for the client, by geography, latency, load, or health (see DNS resolution). Failover is limited by DNS TTLs and resolver caching.
• Anycast: several sites announce the same IP address using BGP, and routers deliver each packet to the topologically nearest site. If a site fails and withdraws its route, traffic shifts to another site without waiting for DNS caches. Anycast suits UDP and short-lived connections most easily, because a routing change can send later packets of a long TCP connection to a different site.
• Layered: GSLB picks a region, and a regional load balancer picks a server. CDNs use the same combination (see content delivery networks).

flowchart TD
    U["Users"] --> G["GSLB: DNS or anycast"]
    G --> R1["Region A load balancer"]
    G --> R2["Region B load balancer"]
    R1 --> A1["Servers A1..An"]
    R2 --> B1["Servers B1..Bn"]

Multi-region designs also need data replication and failover procedures (see multi-region and disaster recovery).

Interview questions

• When would you choose Layer 4 over Layer 7? Layer 4 when maximum throughput and low overhead matter, for non-HTTP protocols, or when TLS passes through untouched. Layer 7 when routing by URL or header, terminating TLS, or balancing per request.
• Why can least connections beat round robin? Requests differ in duration, so round robin can stack slow requests on one server. Least connections uses current open connections as a proxy for load.
• What is wrong with hash(ip) % N when servers change? Changing N remaps most keys, so most clients move to different servers. Consistent hashing remaps only a small fraction.
• How do you make the load balancer highly available? Run several: active-passive with a floating virtual IP and heartbeats, or active-active behind DNS or ECMP routing, and rely on health checks to remove failed members.
• How do sticky sessions hurt scalability? They create uneven load and tie a user to one server, so failures lose sessions. Externalizing session state removes the need.