Resilience Patterns for Service Calls

A call to another service can be slow, fail, or succeed twice. Resilience patterns limit how far one failing dependency spreads. This lesson covers timeouts, retries with jitter, circuit breakers, bulkheads, rate limiting, load shedding, and backpressure, and builds one service that uses them.

Pattern	Protects against	Acts on
Timeout	A call that never returns	One call
Retry with backoff and jitter	Short-lived failures	One call, several attempts
Circuit breaker	A dependency that stays broken	All calls to a dependency
Bulkhead	One dependency using all shared resources	Concurrency per dependency
Rate limiting	A caller sending too much	Requests per caller
Load shedding	Overload of this service	Incoming requests
Backpressure	A producer faster than its consumer	Queues and streams

Project setup

Cargo.toml

TOML

[package]
name = "storefront-resilience"
version = "0.1.0"
edition = "2021"
default-run = "storefront-resilience"

[dependencies]
axum = "0.8"
tokio = { version = "1", features = ["full", "test-util"] }
reqwest = { version = "0.12", default-features = false, features = ["json"] }
rand = "0.8"
serde_json = "1"

tokio needs test-util for the paused clock used in tests.

Example 1: Timeouts

A call without a timeout holds a task, a connection, and often a transaction until the peer answers. When the peer hangs, these pile up and the caller fails too. Every network call has a deadline.

tokio::time::timeout(duration, future) returns Err(Elapsed) when the time passes first and drops the inner future, which cancels it at its next await point. See Tokio time.
reqwest::Client::builder().timeout(...) sets a total limit per request. The complete example uses 800 ms.
Base the value on the dependency's observed p99 latency plus margin. A value below p99 causes needless failures.
An inner timeout is shorter than the outer deadline. With a 2 second budget, three sequential calls cannot each take 1 second.
A timeout does not undo the work. The peer may finish the operation after the caller gave up, so retried operations need idempotency (SQL transactions).

Example 2: Retries with exponential backoff and jitter

A retry sends the same request again after a failure. It helps with transient faults such as a dropped connection or a brief 503. It harms an overloaded dependency, because retries add load.

Retry only errors that can be transient: connection errors, timeouts, 502, 503, 504, and 429 (honoring Retry-After). Never retry 400, 401, 403, 404, or 422.
Retry only idempotent operations. GET, PUT, and DELETE are idempotent by definition. A POST that creates an order is idempotent only with an idempotency key.
Limit attempts (4 here) and total time.
Back off exponentially: the delay ceiling doubles per attempt up to a cap. With base 100 ms and cap 2 s the ceilings are 100, 200, 400, 800, 1600, and 2000 ms.
Add full jitter: each actual delay is a random fraction of the ceiling. Without jitter, clients that failed together retry together.

src/retry.rs in the complete example implements this with retry(policy, operation, retryable), and its tests run under Tokio's paused clock (#[tokio::test(start_paused = true)]), so sleeps finish immediately and the attempt counts are deterministic.

Retries multiply across layers. If the gateway, Orders, and Inventory each retry 3 times, one failing database call produces 27 attempts. Retry at one layer, and use a retry budget that allows retries only while they stay under a fraction of recent request volume.

Example 3: A circuit breaker

A circuit breaker stops calls to a dependency that keeps failing, so the caller frees its resources and the dependency can recover.

State	Behavior	Transition
Closed	Calls pass; failures are counted	To open after N failures
Open	Calls fail immediately	To half-open after a cool-down
Half-open	One probe call passes	Success closes the breaker; failure reopens it

src/breaker.rs holds the state in a std::sync::Mutex. The lock is held only for the state change and never across an await.
Only failures that signal an unhealthy dependency count: timeouts, connection errors, and 5xx. A 404 does not.
An open breaker needs a fallback: a cached value, a default, a degraded feature, or a 503 with Retry-After.

Example 4: A bulkhead with a semaphore

A bulkhead limits the resources one dependency can use, so its failure cannot take down the whole service. A tokio::sync::Semaphore limits concurrent calls. See semaphores.

try_acquire fails at once when no permit is free, so callers are rejected instead of queued. acquire().await with a timeout is the queueing alternative.
The permit is released when it is dropped.
With 20 permits for Payments, a hung Payments service occupies at most 20 tasks, and other routes continue.
The same idea applies to database pools: a separate pool per workload keeps a slow report from starving the API.

Example 5: Rate limiting

Rate limiting caps how many requests a caller may make in a period. A token bucket allows short bursts and a steady average: it holds up to capacity tokens, refills at a fixed rate, and each request takes one token. With capacity 3 and 1 token per second, a client can send 3 requests at once, then one per second. A rejected request gets 429 with Retry-After.

Key by API key, user, or tenant. Limit by IP address only on unauthenticated routes, because many users share an address behind a NAT.
An in-process map limits per instance. With 5 instances the effective limit is 5 times higher. A shared limit uses Redis counters (Redis caching) or the gateway.
The map in src/limits.rs grows with the number of keys. A production limiter evicts idle buckets.
Fixed windows are simple and allow double bursts at window edges. Sliding windows are smoother and keep more state.
tower::limit::RateLimitLayer applies one rate to the whole service, not per caller, and it delays requests instead of rejecting them. A per-caller limit with a 429 answer needs a custom limiter such as this one.

Example 6: Load shedding

Load shedding rejects work when the service is overloaded, so accepted work still meets its deadline. A service that accepts everything queues requests until each exceeds its client's timeout, then spends effort on responses nobody waits for.

The shed middleware in the complete example holds a semaphore with 2 permits. It calls try_acquire_owned, which fails at once when no permit is free, and answers 503 with Retry-After. The permit is released when the request finishes. Tower offers LoadShedLayer and ConcurrencyLimitLayer for the same purpose.

The middleware applies to the route it is attached to. A global cap needs one shared semaphore or a limit at the gateway.
Shedding policies drop the newest request (simple), the lowest priority, or requests that already waited too long. Rejecting must cost far less than serving.

Example 7: Backpressure

Backpressure is a signal from a slow consumer to a fast producer to slow down. Without it, the buffer between them grows until memory runs out. A bounded channel provides it. send(...).await waits when the channel is full. try_send never waits and returns Full, so the caller can drop the event, return 503, or wait with a timeout.

This program uses a channel of capacity 2 with no consumer running, so the results do not depend on timing.

src/bin/backpressure.rs

Rust

use tokio::sync::mpsc::{self, error::TrySendError};

#[tokio::main]
async fn main() {
    let (tx, mut rx) = mpsc::channel::<u32>(2);

    for event in 1..=4 {
        match tx.try_send(event) {
            Ok(()) => println!("event {event} queued"),
            Err(TrySendError::Full(_)) => println!("event {event} dropped: queue full"),
            Err(TrySendError::Closed(_)) => println!("event {event}: consumer stopped"),
        }
    }

    drop(tx);
    let mut received = Vec::new();
    while let Some(event) = rx.recv().await {
        received.push(event);
    }
    println!("consumer received {received:?}");
}

Terminal

cargo run -q --bin backpressure

Output

event 1 queued
event 2 queued
event 3 dropped: queue full
event 4 dropped: queue full
consumer received [1, 2]

Backpressure also exists in TCP flow control, HTTP/2 stream windows, Kafka consumer pause, and database pool waits. Give pool waits a timeout so waiters fail instead of accumulating. See Tokio backpressure.

Example 8: Combine the patterns

For one outgoing call the nesting from outside to inside is:

Bulkhead: reject early when the dependency already has its maximum calls in flight.
Circuit breaker: fail fast when the dependency is known to be down.
Retry with backoff, jitter, and a budget.
Timeout on each attempt.

The checkout handler in the complete example follows this order. For incoming traffic the order is: rate limit per caller, then load shedding and a concurrency limit, then handlers with their own dependency policies. Status codes: 429 means the caller exceeded its limit, and 503 means the service cannot accept work now.

Complete example

File tree:

Text

storefront-resilience/
  Cargo.toml
  src/
    main.rs
    retry.rs
    breaker.rs
    limits.rs
    client.rs
    bin/
      backpressure.rs

Cargo.toml

TOML

[package]
name = "storefront-resilience"
version = "0.1.0"
edition = "2021"
default-run = "storefront-resilience"

[dependencies]
axum = "0.8"
tokio = { version = "1", features = ["full", "test-util"] }
reqwest = { version = "0.12", default-features = false, features = ["json"] }
rand = "0.8"
serde_json = "1"

src/retry.rs

Rust

use std::{future::Future, time::Duration};

#[derive(Debug)]
pub enum CallError {
    Transient(String),
    Permanent(String),
}

impl CallError {
    pub fn is_retryable(&self) -> bool {
        matches!(self, CallError::Transient(_))
    }
}

pub struct RetryPolicy {
    pub max_attempts: u32,
    pub base: Duration,
    pub cap: Duration,
}

impl Default for RetryPolicy {
    fn default() -> Self {
        Self {
            max_attempts: 4,
            base: Duration::from_millis(100),
            cap: Duration::from_secs(2),
        }
    }
}

/// Upper bound of the delay before the next attempt: base * 2^attempt, capped.
pub fn backoff_ceiling(attempt: u32, policy: &RetryPolicy) -> Duration {
    let factor = 1u32.checked_shl(attempt).unwrap_or(u32::MAX);
    policy.base.saturating_mul(factor).min(policy.cap)
}

pub async fn retry<T, E, F, Fut>(
    policy: &RetryPolicy,
    mut operation: F,
    retryable: impl Fn(&E) -> bool,
) -> Result<T, E>
where
    F: FnMut(u32) -> Fut,
    Fut: Future<Output = Result<T, E>>,
{
    let mut attempt = 0;
    loop {
        match operation(attempt).await {
            Ok(value) => return Ok(value),
            Err(error) if retryable(&error) && attempt + 1 < policy.max_attempts => {
                // full jitter: a random delay between zero and the ceiling
                let delay = backoff_ceiling(attempt, policy).mul_f64(rand::random::<f64>());
                tokio::time::sleep(delay).await;
                attempt += 1;
            }
            Err(error) => return Err(error),
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::sync::{
        atomic::{AtomicU32, Ordering},
        Arc,
    };

    #[test]
    fn ceilings_double_up_to_the_cap() {
        let policy = RetryPolicy::default();
        let ceilings: Vec<u128> = (0..6)
            .map(|attempt| backoff_ceiling(attempt, &policy).as_millis())
            .collect();
        assert_eq!(ceilings, vec![100, 200, 400, 800, 1600, 2000]);
    }

    #[tokio::test(start_paused = true)]
    async fn retries_transient_errors_until_success() {
        let calls = Arc::new(AtomicU32::new(0));
        let counter = calls.clone();
        let result = retry(
            &RetryPolicy::default(),
            move |_| {
                let counter = counter.clone();
                async move {
                    let n = counter.fetch_add(1, Ordering::SeqCst) + 1;
                    if n < 3 {
                        Err(CallError::Transient("down".to_string()))
                    } else {
                        Ok(n)
                    }
                }
            },
            CallError::is_retryable,
        )
        .await;
        assert_eq!(result.unwrap(), 3);
        assert_eq!(calls.load(Ordering::SeqCst), 3);
    }

    #[tokio::test(start_paused = true)]
    async fn permanent_errors_are_not_retried() {
        let calls = Arc::new(AtomicU32::new(0));
        let counter = calls.clone();
        let result: Result<(), CallError> = retry(
            &RetryPolicy::default(),
            move |_| {
                let counter = counter.clone();
                async move {
                    counter.fetch_add(1, Ordering::SeqCst);
                    Err(CallError::Permanent("bad request".to_string()))
                }
            },
            CallError::is_retryable,
        )
        .await;
        assert!(result.is_err());
        assert_eq!(calls.load(Ordering::SeqCst), 1);
    }

    #[tokio::test(start_paused = true)]
    async fn gives_up_after_the_maximum_attempts() {
        let calls = Arc::new(AtomicU32::new(0));
        let counter = calls.clone();
        let result: Result<(), CallError> = retry(
            &RetryPolicy::default(),
            move |_| {
                let counter = counter.clone();
                async move {
                    counter.fetch_add(1, Ordering::SeqCst);
                    Err(CallError::Transient("down".to_string()))
                }
            },
            CallError::is_retryable,
        )
        .await;
        assert!(result.is_err());
        assert_eq!(calls.load(Ordering::SeqCst), 4);
    }
}

src/breaker.rs

Rust

use std::{
    sync::Mutex,
    time::{Duration, Instant},
};

enum State {
    Closed { failures: u32 },
    Open { since: Instant },
    HalfOpen { probing: bool },
}

pub struct Breaker {
    state: Mutex<State>,
    threshold: u32,
    cool_down: Duration,
}

impl Breaker {
    pub fn new(threshold: u32, cool_down: Duration) -> Self {
        Self {
            state: Mutex::new(State::Closed { failures: 0 }),
            threshold,
            cool_down,
        }
    }

    /// Returns true when a call may proceed. In the half-open state one probe call is admitted.
    pub fn allow(&self, now: Instant) -> bool {
        let mut state = self.state.lock().unwrap();
        match *state {
            State::Closed { .. } => true,
            State::Open { since } if now.duration_since(since) >= self.cool_down => {
                *state = State::HalfOpen { probing: true };
                true
            }
            State::Open { .. } => false,
            State::HalfOpen { probing: false } => {
                *state = State::HalfOpen { probing: true };
                true
            }
            State::HalfOpen { probing: true } => false,
        }
    }

    pub fn on_success(&self) {
        *self.state.lock().unwrap() = State::Closed { failures: 0 };
    }

    pub fn on_failure(&self, now: Instant) {
        let mut state = self.state.lock().unwrap();
        *state = match *state {
            State::Closed { failures } if failures + 1 >= self.threshold => State::Open { since: now },
            State::Closed { failures } => State::Closed { failures: failures + 1 },
            State::HalfOpen { .. } | State::Open { .. } => State::Open { since: now },
        };
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn opens_after_the_threshold_and_probes_after_the_cool_down() {
        let breaker = Breaker::new(2, Duration::from_secs(10));
        let start = Instant::now();

        assert!(breaker.allow(start));
        breaker.on_failure(start);
        assert!(breaker.allow(start));
        breaker.on_failure(start);
        assert!(!breaker.allow(start));

        let later = start + Duration::from_secs(11);
        assert!(breaker.allow(later));
        assert!(!breaker.allow(later));

        breaker.on_success();
        assert!(breaker.allow(later));
    }

    #[test]
    fn a_failed_probe_reopens_the_breaker() {
        let breaker = Breaker::new(1, Duration::from_secs(10));
        let start = Instant::now();
        breaker.on_failure(start);

        let later = start + Duration::from_secs(11);
        assert!(breaker.allow(later));
        breaker.on_failure(later);
        assert!(!breaker.allow(later));
    }
}

src/limits.rs

Rust

use std::{
    collections::HashMap,
    sync::Mutex,
    time::Instant,
};

struct Bucket {
    tokens: f64,
    last: Instant,
}

/// One token bucket per key. A bucket holds up to `capacity` tokens and refills at `rate_per_sec`.
pub struct RateLimiter {
    capacity: f64,
    rate_per_sec: f64,
    buckets: Mutex<HashMap<String, Bucket>>,
}

impl RateLimiter {
    pub fn new(capacity: u32, rate_per_sec: f64) -> Self {
        Self {
            capacity: capacity as f64,
            rate_per_sec,
            buckets: Mutex::new(HashMap::new()),
        }
    }

    pub fn check(&self, key: &str, now: Instant) -> bool {
        let mut buckets = self.buckets.lock().unwrap();
        let bucket = buckets
            .entry(key.to_string())
            .or_insert(Bucket { tokens: self.capacity, last: now });
        let elapsed = now.duration_since(bucket.last).as_secs_f64();
        bucket.tokens = (bucket.tokens + elapsed * self.rate_per_sec).min(self.capacity);
        bucket.last = now;
        if bucket.tokens >= 1.0 {
            bucket.tokens -= 1.0;
            true
        } else {
            false
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::time::Duration;

    #[test]
    fn allows_a_burst_then_refills() {
        let limiter = RateLimiter::new(3, 1.0);
        let start = Instant::now();
        assert!(limiter.check("k1", start));
        assert!(limiter.check("k1", start));
        assert!(limiter.check("k1", start));
        assert!(!limiter.check("k1", start));
        assert!(limiter.check("k2", start));
        assert!(limiter.check("k1", start + Duration::from_secs(1)));
    }
}

src/client.rs

Rust

use crate::retry::CallError;
use serde_json::Value;

pub async fn call_flaky(http: &reqwest::Client, url: &str) -> Result<Value, CallError> {
    let response = http
        .get(url)
        .send()
        .await
        .map_err(|error| CallError::Transient(error.to_string()))?;
    let status = response.status().as_u16();
    if status == 503 {
        return Err(CallError::Transient("503 from dependency".to_string()));
    }
    if !(200..300).contains(&status) {
        return Err(CallError::Permanent(format!("status {status}")));
    }
    response
        .json::<Value>()
        .await
        .map_err(|error| CallError::Permanent(error.to_string()))
}

src/main.rs

/flaky stands in for the dependency and returns 503 on two calls out of three. /checkout calls it through the bulkhead, breaker, and retry.

Rust

mod breaker;
mod client;
mod limits;
mod retry;

use axum::{
    extract::{Request, State},
    http::{HeaderMap, StatusCode},
    middleware::{from_fn_with_state, Next},
    response::{IntoResponse, Response},
    routing::get,
    Json, Router,
};
use breaker::Breaker;
use client::call_flaky;
use limits::RateLimiter;
use retry::{retry, CallError, RetryPolicy};
use serde_json::json;
use std::{
    sync::{
        atomic::{AtomicU32, Ordering},
        Arc,
    },
    time::{Duration, Instant},
};
use tokio::sync::Semaphore;

#[derive(Clone)]
struct AppState {
    http: reqwest::Client,
    flaky_url: String,
    flaky_calls: Arc<AtomicU32>,
    breaker: Arc<Breaker>,
    bulkhead: Arc<Semaphore>,
    limiter: Arc<RateLimiter>,
}

/// The dependency: fails with 503 on two calls out of three.
async fn flaky(State(state): State<AppState>) -> Response {
    let call = state.flaky_calls.fetch_add(1, Ordering::SeqCst) + 1;
    if call % 3 != 0 {
        return StatusCode::SERVICE_UNAVAILABLE.into_response();
    }
    Json(json!({ "call": call })).into_response()
}

/// Bulkhead, then circuit breaker, then retry with a per-attempt timeout (set on the client).
async fn checkout(State(state): State<AppState>) -> Response {
    let Ok(_permit) = state.bulkhead.try_acquire() else {
        return (StatusCode::SERVICE_UNAVAILABLE, "bulkhead full").into_response();
    };
    if !state.breaker.allow(Instant::now()) {
        return (StatusCode::SERVICE_UNAVAILABLE, [("retry-after", "5")], "circuit open")
            .into_response();
    }

    let http = state.http.clone();
    let url = state.flaky_url.clone();
    let result = retry(
        &RetryPolicy::default(),
        move |_attempt| {
            let http = http.clone();
            let url = url.clone();
            async move { call_flaky(&http, &url).await }
        },
        CallError::is_retryable,
    )
    .await;

    match result {
        Ok(body) => {
            state.breaker.on_success();
            Json(body).into_response()
        }
        Err(error) => {
            state.breaker.on_failure(Instant::now());
            (StatusCode::BAD_GATEWAY, format!("payment failed: {error:?}")).into_response()
        }
    }
}

async fn limited(State(state): State<AppState>, headers: HeaderMap) -> Response {
    let key = headers
        .get("x-api-key")
        .and_then(|value| value.to_str().ok())
        .unwrap_or("anonymous");
    if state.limiter.check(key, Instant::now()) {
        "ok".into_response()
    } else {
        (StatusCode::TOO_MANY_REQUESTS, [("retry-after", "1")], "rate limit exceeded")
            .into_response()
    }
}

async fn slow() -> &'static str {
    tokio::time::sleep(Duration::from_secs(2)).await;
    "done"
}

/// Load shedding: at most two requests run at once. Others get 503 immediately.
async fn shed(State(slots): State<Arc<Semaphore>>, request: Request, next: Next) -> Response {
    match slots.try_acquire_owned() {
        Ok(_permit) => next.run(request).await,
        Err(_) => (StatusCode::SERVICE_UNAVAILABLE, [("retry-after", "1")], "busy").into_response(),
    }
}

#[tokio::main]
async fn main() {
    let state = AppState {
        http: reqwest::Client::builder()
            .timeout(Duration::from_millis(800))
            .build()
            .unwrap(),
        flaky_url: "http://127.0.0.1:3000/flaky".to_string(),
        flaky_calls: Arc::new(AtomicU32::new(0)),
        breaker: Arc::new(Breaker::new(3, Duration::from_secs(10))),
        bulkhead: Arc::new(Semaphore::new(20)),
        limiter: Arc::new(RateLimiter::new(3, 1.0)),
    };

    let app = Router::new()
        .route("/flaky", get(flaky))
        .route("/checkout", get(checkout))
        .route("/limited", get(limited))
        .route(
            "/slow",
            get(slow).layer(from_fn_with_state(Arc::new(Semaphore::new(2)), shed)),
        )
        .with_state(state);

    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}

src/bin/backpressure.rs

Rust

use tokio::sync::mpsc::{self, error::TrySendError};

#[tokio::main]
async fn main() {
    let (tx, mut rx) = mpsc::channel::<u32>(2);

    for event in 1..=4 {
        match tx.try_send(event) {
            Ok(()) => println!("event {event} queued"),
            Err(TrySendError::Full(_)) => println!("event {event} dropped: queue full"),
            Err(TrySendError::Closed(_)) => println!("event {event}: consumer stopped"),
        }
    }

    drop(tx);
    let mut received = Vec::new();
    while let Some(event) = rx.recv().await {
        received.push(event);
    }
    println!("consumer received {received:?}");
}

Run the unit tests, then the service:

Terminal

cargo test
cargo run --bin storefront-resilience

cargo test runs 7 tests, all of which pass: backoff ceilings, three retry cases, two breaker cases, and the token bucket.

Requests:

Terminal

curl -s -w '\n' localhost:3000/checkout
curl -s -w '\n' localhost:3000/checkout
for i in 1 2 3 4; do curl -s -o /dev/null -w '%{http_code}\n' -H 'x-api-key: k1' localhost:3000/limited; done
for i in 1 2 3; do curl -s -w ' [%{http_code}]\n' localhost:3000/slow & done; wait

Output

{"call":3}
{"call":6}
200
200
200
429
busy [503]
done [200]
done [200]

Each /checkout makes 3 calls to /flaky (two 503 responses, then success), so the call counter advances by 3 per request.
The four /limited requests are sent within one second. The first three use the burst capacity and the fourth is rejected.
The /slow requests run at the same time. The third finds no free permit and is shed immediately, so its response prints first.

Summary

Every remote call has a timeout shorter than the caller's own deadline. A timeout does not cancel the work on the peer.
Retry only transient errors on idempotent operations, with capped exponential backoff, full jitter, and a budget. Retry at one layer.
A circuit breaker has closed, open, and half-open states. Open fails fast, and half-open admits a probe.
A bulkhead limits concurrency per dependency with a semaphore or a separate pool.
Token buckets allow bursts. Per-instance limits multiply with instance count unless state is shared.
Load shedding rejects excess work so accepted work meets its deadline. Return 503 with Retry-After.
Bounded queues give backpressure. Unbounded queues hide overload until memory runs out.
Tokio's paused clock makes backoff and cool-down tests deterministic.