Timeouts, Retries, Backoff and Jitter

A remote call can succeed, fail, or never return. Timeouts bound how long a caller waits. Retries repeat a failed call in the hope that the failure was transient. Both are required for resilient service-to-service communication, and both cause outages when configured badly: a missing timeout ties up resources indefinitely, and unrestricted retries multiply load on a dependency that is already failing.

Timeouts

A timeout is the maximum time a caller waits for a response before treating the call as failed. Without one, a slow dependency holds a thread, connection, or task in the caller for as long as the dependency stays slow. Callers then exhaust their own capacity and become slow in turn, so the failure spreads up the call chain.

Timeouts are set at several levels:

Guidelines for choosing values:

• Base the request timeout on the observed latency distribution of the dependency, for example slightly above the 99th or 99.9th percentile. A timeout below normal tail latency causes spurious failures. A timeout far above it delays failure detection.
• Connect timeouts should be much shorter than request timeouts, because connecting is normally fast.
• Every network call needs an explicit timeout. Many client libraries default to no timeout or to a very long one.
• A timeout does not cancel work on the server. The server may still finish the request, so a timed-out write can have taken effect. This is why retries require idempotent operations (see delivery semantics and idempotency).

Timeout budgets and deadline propagation

In a call chain A -> B -> C, each hop has its own timeout. If A waits 1 s for B, but B waits 5 s for C, then A gives up while B and C continue working on a request whose result nobody will read.

A deadline is an absolute point in time by which the whole request must complete. Deadline propagation passes the remaining time to each downstream call, so every hop knows how long it may spend. gRPC implements this: the client sets a deadline, it is transmitted with the call (as a timeout in the grpc-timeout header), and servers can read it and pass it on. With HTTP, the same is done with a custom header carrying the remaining budget.

sequenceDiagram
    participant Client
    participant A as Service A
    participant B as Service B
    participant C as Service C
    Client->>A: request (deadline 1000 ms)
    A->>B: call (remaining 900 ms)
    B->>C: call (remaining 700 ms)
    C-->>B: response after 200 ms
    B-->>A: response
    A-->>Client: response

Rules for a timeout budget:

• The timeout of an inner call must be smaller than the timeout of the call that contains it, leaving time for the caller's own processing and for fallback handling.
• With retries, the sum of all attempts plus waiting time must fit inside the caller's deadline.
• A service that receives a request whose deadline has already passed should reject it immediately instead of doing work.
• Example: a call with a 1 s request timeout, 3 attempts, and backoff of 100 ms and 200 ms between them can take up to 3 x 1 s + 0.3 s = 3.3 s. If the caller's own deadline is 2 s, the third attempt can never complete and only adds load. The per-attempt timeout must be lowered, or the attempts reduced, so that the total fits.

Retries

A retry repeats a failed call. Retries help with transient failures: a dropped connection, a brief overload, a leader election, a pod that was just terminated. They do not help with permanent failures such as 400 Bad Request, 404, or an authorization error, and retrying those wastes capacity.

Conditions for retrying safely:

• Retry only retryable errors. Typically connection failures, timeouts, 503 Service Unavailable, 429 Too Many Requests (respecting Retry-After), and gRPC UNAVAILABLE. Client errors in the 4xx range other than 408 and 429 are not retried.
• The operation must be idempotent, or carry an idempotency key. A request that timed out may have been processed. GET, PUT, and DELETE are idempotent by definition in HTTP. POST usually is not.
• The number of attempts must be limited, commonly 2 to 4 in total.
• Retries should happen at one layer only (see below).

Retry amplification

If each layer in a chain retries independently, the number of requests reaching the deepest dependency grows multiplicatively. Take a client that calls A, A that calls B, and B that calls C, where each of the three callers makes up to 3 attempts. In the worst case, when every attempt fails, A receives 3 requests, B receives 3 x 3 = 9, and C receives 3 x 3 x 3 = 27 for a single user request, exactly when C is struggling.

graph TD
    U["1 user request, client retries 3 times"] --> A["Service A receives 3 requests"]
    A -->|"A retries 3 times per request"| B["Service B receives 9 requests"]
    B -->|"B retries 3 times per request"| C["Service C receives 27 requests"]

A dependency that is overloaded and slow fails more requests, which causes more retries, which increases load further. This positive feedback loop is a retry storm. It can produce a metastable failure: the system stays overloaded even after the original trigger (a deploy, a brief outage) disappears, because the retry traffic alone sustains the overload.

Mitigations:

• Retry at a single layer, usually the one closest to the user or the one that knows the operation is safe to repeat. Lower layers fail fast.
• Use a retry budget: cap retries as a fraction of total traffic, for example allow retries only up to about 10% of the request rate over a window. When the budget is exhausted, failures are returned immediately. A common implementation is a token bucket: each successful call adds a fraction of a token (for example 0.1), each retry consumes one whole token, and a retry is allowed only if a token is available. Retries are then bounded at roughly the configured fraction of successful traffic. gRPC retry throttling works this way.
• Combine with a circuit breaker, which stops sending requests to a dependency that is failing consistently.
• Honor server signals such as Retry-After and 429.

Exponential backoff

Backoff inserts a delay between attempts so the dependency has time to recover. Exponential backoff doubles the delay after each failure, up to a maximum:

delay(n) = min(cap, base * 2^n)

With base = 100 ms and cap = 2 s, the delays are 100 ms, 200 ms, 400 ms, 800 ms, 1.6 s, 2 s, 2 s. The cap keeps a single wait from growing unboundedly. The total number of attempts or the overall deadline stops the sequence.

Jitter

If many clients fail at the same moment, for example when a server restarts, pure exponential backoff makes them all retry at the same instants: after 100 ms, then 200 ms, and so on. The retries arrive as synchronized bursts. Jitter adds randomness to the delay so retries spread out over time.

Common variants (the names follow the AWS Architecture Blog analysis of backoff):

Full jitter spreads load the most and is a good default. Equal jitter guarantees a minimum wait. Decorrelated jitter derives each delay from the previous one instead of the attempt number.

Java implementation

The example injects a seeded Random, so the sequence of delays is reproducible. Production code would use a shared random source such as ThreadLocalRandom. The retry loop receives a Sleeper interface instead of calling Thread.sleep, so the example records the waits instead of waiting. All three jitter variants from the table are implemented.

import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.function.IntFunction;

public class Main {
    /** Exponential value min(cap, base * 2^attempt), without overflow. */
    static long exponential(long baseMs, long capMs, int attempt) {
        long value = baseMs;
        for (int i = 0; i < attempt && value < capMs; i++) {
            value *= 2;
        }
        return Math.min(value, capMs);
    }

    /** Full jitter: random in [0, min(cap, base * 2^attempt)]. */
    static long fullJitter(Random rng, long baseMs, long capMs, int attempt) {
        long ceiling = exponential(baseMs, capMs, attempt);
        return rng.nextLong(ceiling + 1);
    }

    /** Equal jitter: half of the exponential value plus random in [0, half]. */
    static long equalJitter(Random rng, long baseMs, long capMs, int attempt) {
        long half = exponential(baseMs, capMs, attempt) / 2;
        return half + rng.nextLong(half + 1);
    }

    /** Decorrelated jitter: min(cap, random(base, previous * 3)). */
    static long decorrelatedJitter(Random rng, long baseMs, long capMs, long previousMs) {
        long upper = Math.max(baseMs, previousMs * 3);
        return Math.min(capMs, baseMs + rng.nextLong(upper - baseMs + 1));
    }

    @FunctionalInterface
    interface Sleeper {
        void sleep(long millis);
    }

    static class RetryExhaustedException extends RuntimeException {
        RetryExhaustedException(int attempts) {
            super("gave up after " + attempts + " attempts");
        }
    }

    static class TransientFailure extends RuntimeException {
        TransientFailure(String status) {
            super(status);
        }
    }

    static <T> T retry(int maxAttempts, long baseMs, long capMs, Random rng, Sleeper sleeper,
                       IntFunction<T> call) {
        for (int attempt = 0; attempt < maxAttempts; attempt++) {
            try {
                return call.apply(attempt);
            } catch (TransientFailure e) {
                System.out.println("attempt " + attempt + ": failed " + e.getMessage()
                        + ", ceiling " + exponential(baseMs, capMs, attempt) + " ms");
                if (attempt < maxAttempts - 1) {
                    sleeper.sleep(fullJitter(rng, baseMs, capMs, attempt));
                }
            }
        }
        throw new RetryExhaustedException(maxAttempts);
    }

    public static void main(String[] args) {
        var rng = new Random(42);
        List<Long> waits = new ArrayList<>();

        String status = retry(5, 100, 2000, rng, waits::add, attempt -> {
            if (attempt < 3) {
                throw new TransientFailure("503");
            }
            return "200";
        });
        System.out.println("success " + status);
        System.out.println("waits recorded: " + waits.size());
        for (int i = 0; i < waits.size(); i++) {
            long ceiling = exponential(100, 2000, i);
            System.out.println("wait " + i + " within [0, " + ceiling + "]: "
                    + (waits.get(i) >= 0 && waits.get(i) <= ceiling));
        }

        try {
            retry(3, 100, 2000, rng, waits::add, attempt -> {
                throw new TransientFailure("503");
            });
        } catch (RetryExhaustedException e) {
            System.out.println(e.getMessage());
        }

        StringBuilder ceilings = new StringBuilder();
        for (int attempt = 0; attempt < 7; attempt++) {
            ceilings.append(exponential(100, 2000, attempt)).append(' ');
        }
        System.out.println("ceilings: " + ceilings.toString().trim());
    }
}

attempt 0: failed 503, ceiling 100 ms
attempt 1: failed 503, ceiling 200 ms
attempt 2: failed 503, ceiling 400 ms
success 200
waits recorded: 3
wait 0 within [0, 100]: true
wait 1 within [0, 200]: true
wait 2 within [0, 400]: true
attempt 0: failed 503, ceiling 100 ms
attempt 1: failed 503, ceiling 200 ms
attempt 2: failed 503, ceiling 400 ms
gave up after 3 attempts
ceilings: 100 200 400 800 1600 2000 2000

The ceilings 100, 200, 400, 800, 1600, 2000, 2000 match the delays listed under exponential backoff. The delay after the last attempt is never slept, because the loop gives up instead.

Choosing a strategy

• Read operations and idempotent writes: retry with backoff and jitter, at most a few attempts, inside the deadline.
• Non-idempotent writes: retry only with an idempotency key, or do not retry.
• User-facing paths: prefer short timeouts and few retries. Background jobs can use longer backoff and more attempts.
• Latency-sensitive reads: hedged requests send a second request to another replica if the first has not answered within a high percentile (for example the 95th), and use the first response. This reduces tail latency at the cost of extra load, so it must be limited by a budget as well.

Interview questions

• Why must a timeout be set on every remote call? Without one, a slow dependency holds the caller's threads or connections indefinitely, the caller exhausts its own capacity, and the failure propagates to its callers.
• What is deadline propagation? The caller's absolute deadline, or the remaining time, is passed to every downstream call. Each service can stop working when the deadline passes and can size its own timeouts to fit inside the remaining budget.
• Why does exponential backoff need jitter? Clients that fail together compute identical delays and retry together, producing synchronized bursts. Random jitter spreads the retries over time.
• How can retries cause an outage? Layered retries multiply requests (three layers with three attempts each yield 27 requests at the bottom), and extra load on an overloaded service produces more failures and more retries. Retrying at one layer, retry budgets, and circuit breakers limit this.
• When is retrying a request unsafe? When the operation is not idempotent and has no idempotency key, since a timed-out request may already have been applied.