Bulkhead, Fallback and Load Shedding

Three patterns limit the damage of failures and overload inside a service. A bulkhead partitions resources so that one failing dependency or workload cannot consume everything. A fallback provides an alternative result when a call fails. Load shedding rejects some requests deliberately when the service is overloaded, so the remaining requests still complete within acceptable latency.

Bulkhead

A service shares finite resources among all its work: threads, connections, memory, CPU. Without isolation, one slow dependency can take all of them. If calls to a recommendation service hang and all worker threads are waiting on it, requests that do not need recommendations cannot be served either, and the whole service appears down.

A bulkhead assigns each dependency or workload class its own bounded pool of resources. When a pool is full, additional work for that dependency is rejected or queued briefly, and other pools are unaffected.

graph TD
    R["Incoming requests"] --> S["Service"]
    S --> P1["Pool: payments, 20 slots"]
    S --> P2["Pool: recommendations, 10 slots"]
    S --> P3["Pool: search, 30 slots"]
    P1 --> D1["Payment service"]
    P2 --> D2["Recommendation service"]
    P3 --> D3["Search service"]

If the recommendation service hangs, at most 10 concurrent calls are stuck. Payments and search keep their full capacity. The cost of partitioning is lower utilization: idle slots in one pool cannot be borrowed by another pool that is full, so the pool sizes must be chosen deliberately.

Bulkhead types

Bulkheads can also separate by client or tenant: a heavy tenant gets a quota so it cannot starve others. In Kubernetes, CPU and memory limits and separate node pools provide isolation at the infrastructure level.

Sizing

Pool size follows from Little's law: the average number of concurrent requests equals arrival rate times average latency. A dependency called at 200 requests per second with an average latency of 50 ms needs about 200 x 0.05 = 10 concurrent slots on average. Size pools above that to absorb variation, but not so high that they no longer protect anything. A rejected call when the pool is full is the intended behavior, and it should trigger the fallback.

Bulkheads and circuit breakers are complementary. The bulkhead limits concurrency and protects the caller from slowness. The breaker stops calls entirely after sustained failure.

Fallback

A fallback is the behavior used when the primary call fails, times out, is rejected by a bulkhead, or is blocked by an open circuit breaker. It turns a failure into a reduced but useful response.

Design rules:

• Make fallbacks simple and independent. A fallback that calls another remote service can fail too, and fallbacks are rarely exercised, so complex ones hide bugs. Test them with fault injection (see chaos engineering).
• Do not fall back where correctness matters. Guessing a payment status or an account balance is worse than an error. Fallbacks suit reads and non-critical features.
• Expose degradation. Mark responses as degraded (a header or field), and emit metrics, so operators know a fallback is active.
• Distinguish critical from optional dependencies. A product page can render without reviews, but not without the catalog.

Load shedding

When demand exceeds capacity, a server that accepts everything queues requests. Queues increase latency for all requests. Clients time out, but the server still processes the abandoned requests, so it spends capacity on work whose results are never used. Throughput of useful work (goodput) falls while the server is busy. For example, a server that can complete 100 requests per second receives 150 per second and queues the excess. The queue grows by 50 requests every second. After 10 s, a new request waits about 5 s (500 queued requests at 100 per second). If clients time out after 2 s, nearly every request that reaches the front of the queue has already been abandoned, and goodput drops toward zero although the server is fully busy. Load shedding avoids this by rejecting excess requests early, cheaply, with a fast error such as 503 Service Unavailable or 429.

flowchart LR
    Req["Request"] --> C{"Load above limit?"}
    C -->|No| Q["Process"]
    C -->|Yes| P{"Priority?"}
    P -->|High| Q
    P -->|Low| X["Reject with 503 or 429"]

What to shed

• By priority. Classify requests: for example, user-facing checkout above background sync, above analytics. Shed the lowest class first. Priority can be carried in a header or derived from the endpoint or caller identity.
• By cost. Expensive requests (large scans) are shed before cheap ones.
• By age. Drop requests that have waited in the queue longer than the client will wait. Using a bounded queue with a short wait limit, and serving the newest requests first (LIFO) under overload, sacrifices requests that are already likely to have timed out.
• By deadline. Reject requests whose propagated deadline has already passed (see timeouts, retries, backoff and jitter).

Load signals

Shedding decisions use a measurable indicator of overload:

• Number of in-flight requests above a limit.
• Queue length or queueing delay.
• CPU utilization or memory pressure.
• Latency above a target.

A static limit needs to be tuned and becomes wrong when the workload or the hardware changes.

Adaptive concurrency limits

An adaptive concurrency limit adjusts the maximum number of in-flight requests automatically from observed latency. The approach is similar to TCP congestion control: increase the limit while latency stays close to the minimum observed value, and decrease it when latency rises, which indicates queueing. Netflix's concurrency-limits library implements algorithms of this kind (for example, based on AIMD or on Vegas-style latency gradients). Envoy has an adaptive concurrency filter based on the same idea. The result is that the limit tracks the true capacity of the service without manual tuning.

Shedding versus rate limiting

See rate limiting and backpressure between services.

Client behavior

Rejected requests should be cheap and informative. Clients should treat 503 and 429 as a signal to back off, not to retry immediately. Servers can add Retry-After. Load balancers and health checks must not treat shedding as failure of the instance, otherwise the instance is removed and the remaining ones are overloaded further.

Combining the patterns

1. Timeouts bound each call.
2. A bulkhead limits concurrent calls per dependency.
3. A circuit breaker stops calls to a dependency that keeps failing.
4. A fallback supplies a degraded response when any of the above rejects the call.
5. Load shedding at the service entrance protects the service itself when total demand is too high.

Interview questions

• What problem does a bulkhead solve? It prevents one slow or failing dependency or workload from consuming all shared resources (threads, connections), so other functions of the service continue working.
• How would you size a bulkhead for a dependency? Estimate concurrent calls with Little's law (request rate times average latency), then add headroom for variation. Calls beyond the limit are rejected or trigger the fallback.
• Why is queueing everything under overload harmful? Queues raise latency for all requests, clients time out, and the server wastes capacity processing requests whose results are discarded. Rejecting early keeps latency low for admitted requests.
• When should a fallback not be used? When a wrong substitute is worse than an error, for example payment status, balances, or authorization decisions.
• What is an adaptive concurrency limit? A limit on in-flight requests that is adjusted automatically from measured latency, increasing while latency is near its minimum and decreasing when latency indicates queueing.