Delivery Semantics and Idempotency

When a sender transmits a message over a network, it can lose the acknowledgment, the message, or both. The sender cannot distinguish "message lost" from "message processed, acknowledgment lost". The delivery semantics of a system define how it behaves in this ambiguity.

The three guarantees

sequenceDiagram
    participant P as Producer
    participant B as Broker or service
    P->>B: message m
    B->>B: process m
    B--xP: ack lost
    Note over P: no ack, cannot tell whether m was processed
    P->>B: retry m
    B->>B: process m again (duplicate)

• At-most-once is appropriate when loss is acceptable and duplicates are harmful or costly: metrics samples, some logging, fire-and-forget notifications.
• At-least-once is the default for reliable messaging (Kafka, SQS standard queues, RabbitMQ with acknowledgments). The consumer must tolerate duplicates.
• Exactly-once is what applications usually want.

Why exactly-once delivery is not achievable

The Two Generals problem shows that two parties communicating over an unreliable channel cannot reach certainty that a message has been received. A sender that has not received an acknowledgment has only two choices: retry (risking a duplicate) or give up (risking a loss). No protocol removes both.

The practical definition is exactly-once processing (or effectively-once): the message may be delivered more than once, but the observable effect on state happens once. This is built from:

at-least-once delivery + idempotent processing (or deduplication) = effectively exactly-once

Idempotency

An operation is idempotent if applying it several times has the same effect as applying it once: f(f(x)) = f(x).

Making operations idempotent

1. Prefer naturally idempotent operations. Use absolute writes (set the state to a value) instead of relative ones (increment). Use upserts keyed by a stable business key.
2. Use an idempotency key. The client generates a unique key (a UUID) per logical operation and sends it with every retry. The server stores the key with the result. On a repeated key, it returns the stored result without re-executing.
3. Deduplicate by message ID. The consumer records processed message IDs and skips repeats.
4. Use versioning. Apply an update only if the version or sequence number is greater than the stored one (UPDATE ... WHERE version = ?), so replays and reordered messages are ignored.

Idempotency key handling

flowchart TD
    A["Request with idempotency key K"] --> B{"K already stored?"}
    B -- "no" --> C["Insert K as in progress, atomically"]
    C --> D["Execute operation"]
    D --> E["Store result under K"]
    E --> F["Return result"]
    B -- "yes, completed" --> G["Return stored result"]
    B -- "yes, in progress" --> H["Return conflict or wait"]

Requirements for correctness:

• The key check and the business write must be atomic, ideally in the same database transaction. Otherwise a crash between them leaves a processed message with no record, or a recorded key with no effect.
• The insert of the key must be protected by a unique constraint so two concurrent retries cannot both proceed.
• Keys need a retention period. It must be longer than the maximum retry window. Expiring keys too early allows late duplicates through.
• The same key with a different request payload should be rejected, as it indicates a client bug.

import java.util.HashMap;
import java.util.Map;

public class Main {
    sealed interface DebitResult permits Ok, Rejected {}

    record Ok(long balance) implements DebitResult {}

    record Rejected(String reason) implements DebitResult {}

    // The amount of the original request and the balance it produced.
    record Stored(long amount, long resultingBalance) {}

    static class PaymentService {
        private long balance;
        private final Map<String, Stored> seen = new HashMap<>();

        PaymentService(long balance) {
            this.balance = balance;
        }

        // synchronized makes the key check and the balance update one atomic step
        synchronized DebitResult debit(String key, long amount) {
            Stored original = seen.get(key);
            if (original != null) {
                if (original.amount() != amount) {
                    return new Rejected("key reused with a different payload");
                }
                return new Ok(original.resultingBalance()); // replay: stored result, no new effect
            }
            balance -= amount;
            seen.put(key, new Stored(amount, balance));
            return new Ok(balance);
        }

        synchronized long balance() {
            return balance;
        }
    }

    public static void main(String[] args) {
        var svc = new PaymentService(100);
        System.out.println("first attempt: " + svc.debit("k-1", 10));
        System.out.println("retry same key: " + svc.debit("k-1", 10));
        System.out.println("balance: " + svc.balance());
        System.out.println("same key, other amount: " + svc.debit("k-1", 25));
    }
}

first attempt: Ok[balance=90]
retry same key: Ok[balance=90]
balance: 90
same key, other amount: Rejected[reason=key reused with a different payload]

The example keeps the key store in memory for clarity, and synchronized stands in for the atomic step within one JVM. In a real service the key check and the balance update commit in one database transaction, as described above.

Payment APIs such as Stripe's use client-supplied Idempotency-Key headers this way. The HTTP methods GET, PUT, and DELETE are idempotent by definition; POST is not, which is why an idempotency key header is added to it. See API pagination, idempotency, and versioning and idempotent consumers and deduplication.

Deduplication in consumers

A consumer processing a stream at-least-once can use the inbox approach: in one local transaction, insert the message ID into a processed_messages table (unique constraint) and apply the effect. If the insert conflicts, the message is a duplicate and is skipped. Writes to an external system that is not part of the transaction (an email, a payment provider) need that system's own idempotency mechanism, because the local transaction cannot cover it.

The producer side has a matching problem: writing to a database and publishing an event are two operations that can partly fail. The transactional outbox solves it and yields at-least-once publication.

Kafka's delivery guarantees

Kafka (see Kafka internals) offers all three levels, depending on configuration.

Producer side

• acks=0: no acknowledgment, at-most-once.
• acks=1: leader acknowledges. Retries can produce duplicates.
• acks=all: all in-sync replicas acknowledge. Together with retries this gives at-least-once.
• Idempotent producer (enable.idempotence=true): the broker assigns each producer a producer ID (PID) and the producer attaches a per-partition sequence number to each record batch. The broker rejects a duplicate or out-of-order sequence, so retries do not create duplicates within one partition and one producer session. It does not deduplicate across producer restarts unless the transactional ID mechanism is used.

Consumer side

• Commit the offset before processing: at-most-once (a crash after commit loses the unprocessed message).
• Commit the offset after processing: at-least-once (a crash before commit reprocesses).

Transactions

Kafka transactions provide atomic writes to multiple partitions and atomic offset commits for a consume-process-produce loop:

1. The producer sets a transactional.id and calls initTransactions. The broker fences older producers with the same ID using an epoch.
2. It begins a transaction, produces output records, and sends the consumed offsets to the transaction (sendOffsetsToTransaction).
3. It commits or aborts. A transaction coordinator writes markers to the involved partitions.
4. Consumers set isolation.level=read_committed and see only records from committed transactions.

This gives exactly-once for pipelines whose input and output are both in Kafka (Kafka Streams enables it with processing.guarantee=exactly_once_v2). When the output goes to an external system, exactly-once requires an idempotent or transactional sink.

Interview questions

• Is exactly-once delivery possible? Not over an unreliable network, since a sender cannot tell a lost message from a lost acknowledgment. Systems achieve exactly-once effects with at-least-once delivery plus idempotent processing or deduplication.
• How does an idempotency key work, and what must be atomic? The client sends a unique key with each retry, and the server stores the key with the result and returns that result on repeats. Recording the key and applying the effect must happen in one atomic step, guarded by a unique constraint.
• Why is balance = balance + 10 unsafe to retry, and how would you fix it? Each retry adds again. Fix it by recording the transaction ID with a unique constraint and applying the change once, or by using an absolute write with a version check.
• What does Kafka's idempotent producer guarantee? No duplicates from producer retries within a single partition during one producer session, using PID and sequence numbers. Cross-session and multi-partition atomicity need transactions.