Idempotent Consumers and Deduplication

Brokers and retrying clients commonly guarantee at-least-once delivery: a message is delivered one or more times, never silently dropped. Duplicates arise when a consumer processes a message but crashes before acknowledging it, when an acknowledgment is lost, when a producer retries after a timeout, or when a relay republishes (outbox). An operation is idempotent if performing it several times has the same effect as performing it once. An idempotent consumer is a message handler that produces the same end state no matter how many times a message is delivered. See delivery semantics and idempotency for the general model.

Why exactly-once processing is built, not provided

Exactly-once delivery across a network is not achievable in general: a sender that gets no reply cannot tell whether the receiver processed the message. What systems provide is at-least-once delivery combined with idempotent processing, giving effectively-once results. Kafka's transactions give exactly-once semantics within Kafka read-process-write flows, but side effects in external databases or APIs still require idempotency.

Approach 1: Natural idempotency

Some operations are idempotent by their definition:

Rewrite handlers to use absolute values, upserts, and deterministic IDs derived from the message (for example, primary key = event ID or business key). Where possible, this is the simplest solution, since it needs no extra state.

Approach 2: Idempotency key with a deduplication store

Each message or request carries a unique idempotency key (event ID, or client-generated key). The consumer records processed keys and skips repeats.

Check-then-act in two steps is racy. Two concurrent deliveries can both find the key absent. The check must be atomic with the effect:

• Insert the key into a table with a unique constraint in the same database transaction as the business change. A duplicate insert fails and the transaction rolls back or is skipped.
• For non-transactional stores, use an atomic conditional write (for example, Redis SET key value NX, or a DynamoDB conditional put), but then the effect and the key record are not atomic. If the key is written first and the consumer crashes before the effect, the redelivered message is skipped and the effect is lost. If the effect is applied first and the crash occurs before the key is written, the effect repeats.
• Alternatively, INSERT ... ON CONFLICT DO NOTHING (PostgreSQL) returns the number of inserted rows. Zero rows means a duplicate, and the handler skips the business change without an error that aborts the transaction.

Approach 3: Inbox pattern

The inbox is the consumer-side counterpart of the outbox. The consumer stores each received message in an inbox table with a unique message ID, in the same transaction as the business update, and acknowledges the broker afterward.

sequenceDiagram
    participant B as Broker
    participant C as Consumer
    participant DB as Consumer DB
    B->>C: deliver message M1
    C->>DB: BEGIN
    C->>DB: INSERT INTO inbox (id) VALUES (M1)
    alt id already present
        DB-->>C: unique violation
        C->>DB: ROLLBACK
    else new
        C->>DB: apply business change
        C->>DB: COMMIT
    end
    C->>B: ack M1

If the consumer crashes after commit and before ack, the broker redelivers, the insert violates the unique constraint, and the duplicate is discarded. The ack is then sent. Effect and dedup record are atomic.

Storage management: inbox rows grow without bound. Delete entries older than the maximum time in which a duplicate can appear (redelivery window, retention of the topic, client retry limits). Choosing too short a window allows late duplicates through. Example: if a topic retains messages for 7 days and a consumer group can be reset to an earlier offset, duplicates can appear up to 7 days later, so inbox rows must be kept at least that long. Keeping the unique-ID column indexed keeps the duplicate check a single index lookup.

Approach 4: Versioning and optimistic updates

When messages carry a version or sequence number for an entity, the consumer applies an update only if it is newer than what is stored:

UPDATE product_view SET price = :price, version = :v
WHERE id = :id AND version < :v;

This makes handlers idempotent (a repeated version matches no row) and also ignores stale, out-of-order messages. Example: the stored row has version 5. A message with version 7 arrives and is applied, then a delayed message with version 6 arrives and matches no row, so the newer price is kept. If the row does not exist yet, an upsert with the same version condition creates it. The version rule assumes that each message carries the full new value of the field, not a delta. It requires a per-entity monotonic version from the producer, such as an aggregate version in event sourcing.

Idempotency keys for APIs

For synchronous APIs (payments, order creation), the client sends an Idempotency-Key header. The server stores the key with the request fingerprint and the response.

• First request: process, store the response under the key.
• Retry with the same key: return the stored response without repeating the work.
• Same key with a different payload: reject as misuse.
• A request in progress: return a conflict or wait.
• Key storage and the effect follow the same atomicity rule as above: record the key in the same transaction as the effect, or mark it in progress first and complete it afterward.

Keys expire after a retention period. A retry after expiry is processed as a new request.

Example in Java

A handler that applies each event ID at most once, with the effect and the record of the ID updated together. In a real service both happen in one database transaction. This in-memory version is not thread-safe: concurrent deliveries would need a ConcurrentHashMap.newKeySet() and a lock that covers the set insert and the balance update together.

import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class Main {
    record Delivery(String eventId, long amount) {}

    static final class Account {
        private long balance;
        private final Set<String> processed = new HashSet<>();

        /** Returns true if the event was applied, false if it was a duplicate. */
        boolean handleDeposit(String eventId, long amount) {
            if (!processed.add(eventId)) {
                return false;
            }
            balance += amount;
            return true;
        }

        long balance() {
            return balance;
        }
    }

    public static void main(String[] args) {
        var account = new Account();
        var deliveries = List.of(
                new Delivery("evt-1", 50), new Delivery("evt-2", 30), new Delivery("evt-1", 50),
                new Delivery("evt-2", 30), new Delivery("evt-3", 5));
        for (var delivery : deliveries) {
            boolean applied = account.handleDeposit(delivery.eventId(), delivery.amount());
            System.out.println(delivery.eventId() + ": applied=" + applied + " balance=" + account.balance());
        }
    }
}

evt-1: applied=true balance=50
evt-2: applied=true balance=80
evt-1: applied=false balance=80
evt-2: applied=false balance=80
evt-3: applied=true balance=85

Choosing

• Prefer natural idempotency and version checks when the domain allows.
• Use an inbox or key table when handlers have effects that are not naturally idempotent, in the same transactional store as the effect.
• For side effects on external systems (email, payment providers), pass the idempotency key downstream if the provider supports it, otherwise accept a small risk of duplicates or record intent before acting.

Interview questions

• Why can a broker not provide exactly-once delivery? A sender cannot distinguish a lost message from a lost acknowledgment, so it must resend and the receiver may see duplicates. Effectively-once comes from deduplication at the receiver.
• Why must the dedup record and the business effect be atomic? If the key is stored first and the crash occurs before the effect, the message is lost. If the effect happens first and the crash occurs before the key is stored, the effect repeats.
• How long should dedup keys be retained? At least as long as a duplicate can still arrive: the broker retention or redelivery window plus client retry horizon.
• How does versioning help beyond dedup? A version check also discards stale messages that arrive out of order, so a handler never overwrites newer data with older.