Backpressure, Retries, and Dead Letter Queues

An asynchronous pipeline connects components running at different speeds, and any component can fail on a given message. Two mechanisms keep the pipeline stable. Backpressure prevents a fast producer from overwhelming a slow consumer. Retries and dead letter queues handle messages whose processing fails.

Backpressure

Backpressure is a signal from a slower downstream component to the upstream one to reduce the rate. Without it, a queue between them grows without bound when the producer rate exceeds the consumer rate. Memory or disk fills, latency of queued messages grows, and eventually the system fails.

If arrival rate is λ and service rate is μ, the queue is stable only when λ < μ on average. A queue absorbs bursts, but it cannot fix a sustained λ > μ. Example: with λ = 1,200 messages/s and μ = 1,000 messages/s, the backlog grows by 200 messages/s, which is 12,000 per minute and 720,000 per hour. A message arriving after one hour of this waits about 720 s (720,000 / 1,000) before being processed.

Strategies

graph LR
    P["Producer"] -->|"send"| Q["Bounded queue"]
    Q -->|"deliver"| C["Consumer"]
    C -.->|"credit / ack: ready for more"| P
    Q -.->|"full: block, reject, or drop"| P

Log-based systems such as Kafka rely on the pull model: the log absorbs the backlog on disk, and consumer lag (latest offset minus committed offset) is the signal. A growing lag means consumers are slower than producers. Alert on lag and scale consumers, up to the partition count.

Unbounded queues hide overload. Choose a limit, and decide explicitly what happens when it is reached. Messages that waited longer than their usefulness allows should be discarded or expired using a message TTL.

Retries

Failures fall in two categories:

• Transient: timeouts, throttling, brief unavailability, deadlocks. A later attempt can succeed.
• Permanent: invalid payload, business rule violation, missing referenced data. Retrying gives the same result.

A retry policy specifies which errors to retry, how many attempts, and the delay between them.

• Fixed delay: wait the same time between attempts.
• Exponential backoff: the delay doubles each attempt (base * 2^attempt), up to a maximum. It gives the failing dependency time to recover.
• Jitter: add randomness to delays so many clients failing together do not retry in synchronized waves. See timeouts, retries, backoff, and jitter.
• Retry limit or budget: a maximum attempt count or a cap on the fraction of traffic that may be retries, to avoid amplifying an outage.

The following sketch computes a capped exponential backoff schedule and retries an operation that fails a fixed number of times. A transient failure is a dedicated exception type, so the loop retries only that type and lets other errors propagate:

import java.time.Duration;

public class Main {
    static class TransientException extends RuntimeException {
        TransientException(String message) {
            super(message);
        }
    }

    static Duration backoff(int attempt, Duration base, Duration max) {
        var delay = base;
        for (int i = 0; i < attempt; i++) {
            delay = delay.multipliedBy(2);
            if (delay.compareTo(max) >= 0) {
                return max; // stop doubling once the cap is reached, so the value cannot overflow
            }
        }
        return delay.compareTo(max) > 0 ? max : delay;
    }

    static void process(String msg, int attempt) {
        // Simulated transient failure: the first two attempts fail.
        if (attempt < 2) {
            throw new TransientException("temporary error on '" + msg + "'");
        }
    }

    public static void main(String[] args) {
        var base = Duration.ofMillis(100);
        var max = Duration.ofMillis(500);
        int maxAttempts = 5;

        for (int attempt = 0; attempt < maxAttempts; attempt++) {
            try {
                process("order-17", attempt);
                System.out.println("attempt " + attempt + ": success");
                break;
            } catch (TransientException e) {
                var delay = backoff(attempt, base, max);
                System.out.println("attempt " + attempt + ": " + e.getMessage()
                        + "; retry in " + delay.toMillis() + " ms");
            }
        }
    }
}

attempt 0: temporary error on 'order-17'; retry in 100 ms
attempt 1: temporary error on 'order-17'; retry in 200 ms
attempt 2: success

A real consumer would sleep for the delay (with jitter) and, after the last attempt, send the message to a dead letter queue.

Retrying in a message system

Consumers can retry in several ways:

• In place: retry within the handler with a short backoff. Simple, but it occupies the consumer and, on a partitioned log, blocks the partition behind it.
• Redelivery by the broker: do not ack, and the message comes back (after the visibility timeout in SQS, or on requeue in RabbitMQ). Delays are coarse, and rapid requeue loops can starve other work.
• Retry topics or delay queues: on failure, publish the message to a retry queue with a delay (for example 1 minute, then 10 minutes, then an hour), which feeds back to the main queue. This frees the main queue and implements increasing delays without blocking. Delay features vary by broker: SQS delay, RabbitMQ TTL with dead-lettering or a delay plugin, or separate Kafka retry topics that are consumed after a wait.

Every retry can duplicate a side effect, so handlers must be idempotent. See idempotent consumers and deduplication.

Poison messages and dead letter queues

A poison message is one that always fails processing, for example because it is malformed. With redelivery it fails forever, occupies a consumer, and can block the queue or partition behind it.

A dead letter queue (DLQ) is a queue where messages that cannot be processed are moved for later inspection.

graph LR
    Q["Main queue"] --> C["Consumer"]
    C -->|"success: ack"| D["Done"]
    C -->|"transient failure"| R["Retry queue with delay"]
    R --> Q
    C -->|"permanent failure or attempts exhausted"| DLQ["Dead letter queue"]
    DLQ --> O["Alert, inspect, fix, redrive"]

Typical triggers for dead-lettering:

• The delivery count exceeds a maximum (SQS maxReceiveCount, Azure Service Bus max delivery count).
• The consumer rejects the message without requeue (RabbitMQ dead letter exchange).
• The message expires (TTL) or the queue overflows its length limit (RabbitMQ).

Kafka consumers have no built-in DLQ. Applications publish failing records to a separate topic (with the error and origin metadata) and continue. Kafka Connect sink connectors can be configured to write failed records to a dead letter topic.

Operating a DLQ:

• Store the original message plus failure reason, attempt count, and source metadata (headers).
• Alert on DLQ depth. An unattended DLQ is a silent data loss.
• Fix the cause (code bug, bad data), then redrive: move messages back to the main queue. Handlers must remain idempotent since redriven messages may have partially executed.
• Set DLQ retention longer than that of the source queue, as SQS advises, because the original enqueue time is retained and a message can expire soon after being moved.

Ordering versus retry

Retrying and strict ordering conflict. Suppose messages 1, 2, 3 for the same entity arrive in order and message 1 fails.

• Block and retry message 1 until it succeeds or is dead-lettered: order is preserved, but messages 2 and 3 wait behind it (head-of-line blocking). A poison message stops the whole partition or message group.
• Move message 1 to a retry queue and continue with 2 and 3: throughput continues, but message 1 is now processed after 2 and 3, breaking order.

Options:

• Accept reordering and make handlers commutative or version-aware, ignoring an update older than the stored version.
• Preserve order per key: while a key has a failed message pending, park later messages of the same key (also to the retry path), and let other keys proceed.
• Bound blocking retries and dead-letter the message after a limited number of attempts, so one bad message cannot stall a partition forever.
• Dead-lettering a message means later messages may depend on a message that never applied. Design consumers to detect and handle gaps.

Interview questions

• What are the options when a queue's producer is faster than its consumer? Bound the queue and block, reject, or drop when it is full; make consumers pull at their pace or grant credit; scale out consumers; or limit or shed load at the edge. Unbounded queues only delay the failure.
• Why add jitter to exponential backoff? Without it, clients that fail at the same time retry at the same times and cause synchronized load spikes. Random delays spread the retries.
• What is a poison message and how is it handled? It is a message that fails every time it is processed. After a limited number of attempts it is moved to a DLQ, where it is inspected and redriven after the fix.
• Why do retries conflict with ordering? Continuing past a failed message processes it out of order, and blocking on it stalls all messages behind it. Systems choose per-key blocking, bounded retries, or order-tolerant handlers.