Saga Pattern

With database per service, a business operation such as placing an order updates several databases. A single ACID transaction cannot span them, and two-phase commit is often unavailable or hurts availability. A saga is a sequence of local transactions, one per service, where each step commits independently. If a step fails, the saga runs compensating transactions that semantically undo the steps already committed.

Structure

For an order:

1. Order Service creates the order in PENDING state.
2. Payment Service charges the customer.
3. Inventory Service reserves stock.
4. Order Service marks the order CONFIRMED.

If step 3 fails, the saga runs compensations in reverse order: refund the payment, then mark the order REJECTED.

sequenceDiagram
    participant O as Order
    participant P as Payment
    participant I as Inventory
    O->>O: create order PENDING
    O->>P: charge
    P-->>O: charged
    O->>I: reserve stock
    I-->>O: failed (out of stock)
    O->>P: refund (compensation)
    P-->>O: refunded
    O->>O: mark order REJECTED

Compensating transactions

A compensation does not restore the previous state by rollback, because the original transaction is committed and other actors may have observed it. It applies a new business action that offsets the first: refund instead of un-charge, cancel reservation instead of un-reserve.

Requirements:

• Compensations must be idempotent and retried until they succeed, since a saga cannot leave a step half-undone (see idempotent consumers).
• Some actions cannot be compensated, such as sending an email or shipping a parcel. Order these steps last.

Step types

• Compensable steps: can be undone by a compensating transaction. Placed first.
• Pivot step: the point of no return. It is either the last step that cannot be compensated or the first step after which no compensation is needed. If it succeeds, the saga runs to completion.
• Retriable steps: steps after the pivot. They are guaranteed to eventually succeed and are retried until they do.

In the order example, creating the order, charging the payment, and reserving stock are compensable. If the business allows no cancellation once stock is reserved, the reservation is the pivot and marking the order CONFIRMED is retriable. Placing irreversible steps (sending a confirmation email, handing a parcel to a carrier) after the pivot means a failure never requires undoing them.

Recovery from crashes and timeouts

A saga can run for seconds to days, so its progress must survive a crash of the coordinating process. The orchestrator (or each participant in choreography) persists the saga state, meaning the current step and the outcomes so far, in the same local transaction that records the next command or event. After a restart it resumes from the persisted step. A participant that does not answer within a timeout leaves the outcome unknown: the request may have been lost, or may have executed with the reply lost. The orchestrator retries the request with the same idempotency key, or compensates, and in both cases the participant must handle the repeat safely.

Coordination

The two styles are described in choreography vs orchestration.

Choreography-based saga. Each service publishes events and reacts to others: OrderCreated triggers payment, PaymentCompleted triggers inventory, StockReservationFailed triggers refund. No central component. Simple for two to three steps, hard to follow beyond that, with a risk of cyclic dependencies.

Orchestration-based saga. A saga orchestrator holds the state machine and sends commands to participants, receives replies, and issues compensations. The process is explicit and easier to test and monitor.

The saga's local commit and its message publication must be atomic, or a crash between them loses the saga's progress. Use the transactional outbox. Messages can be redelivered, so participants must deduplicate.

Lack of isolation

Sagas provide atomicity (through compensation), consistency (through local transactions and application invariants), and durability (through the local databases), but not isolation: the changes of each committed local transaction are visible to other transactions before the saga finishes. Anomalies:

• Lost update. Another saga overwrites changes made by a saga step before the saga completes or compensates. Example: a saga sets an order to CONFIRMED, another saga cancels the order, and a later compensation of the first saga restores an older state and discards the cancellation.
• Dirty read. A transaction reads data from a saga that later compensates. For example, a customer sees a PENDING order that will be rejected.
• Fuzzy (non-repeatable) read. Two steps of a saga read the same data and see different values because another saga changed it in between.

Countermeasures

• Semantic lock. A step sets a flag such as PENDING on the record. Other transactions treat the flagged record as locked or handle it specially. The flag is cleared by the saga's final step or its compensation.
• Commutative updates. Design updates so the order of execution does not matter (for example, increments and decrements instead of overwriting a value).
• Pessimistic view. Reorder steps to reduce the risk of dirty reads. For example, in a saga that cancels an order and returns credit, place the credit increase in the last step. Other transactions then never see a credit value that a compensation later has to reduce.
• Reread value. Before overwriting, reread the record and check it is unchanged, aborting the saga and restarting it if it changed (an optimistic check, similar to a version comparison).
• Version file. Record the operations performed on a record, so that operations arriving out of order can be applied in the correct sequence (for example, a cancel that arrives before the authorization it cancels).
• By value. Choose the mechanism by the risk of the operation: low-risk operations use sagas, high-risk ones (for example, large payments) use distributed transactions.

Orchestrator sketch in Java

An orchestrator is a state machine over a list of steps. Each step has an action and a compensation. A failed local transaction is reported as a SagaStepException, and the result is a sealed type with two cases, so a caller must handle both. On failure at step k, run compensations for steps k-1 down to 0. Real implementations persist state after each transition and send messages instead of calling functions.

import java.util.ArrayList;
import java.util.List;
import java.util.function.Consumer;

public class Main {
    @FunctionalInterface
    interface Action {
        /** Throws SagaStepException when the local transaction fails. */
        void run(List<String> log);
    }

    static final class SagaStepException extends RuntimeException {
        SagaStepException(String message) {
            super(message);
        }
    }

    record Step(String name, Action action, Consumer<List<String>> compensate) {}

    sealed interface SagaResult {
        record Completed() implements SagaResult {}

        record Failed(String reason) implements SagaResult {}
    }

    static void charge(List<String> log) {
        log.add("payment charged");
    }

    static void refund(List<String> log) {
        log.add("payment refunded");
    }

    static void reserve(List<String> log) {
        throw new SagaStepException("out of stock");
    }

    static void release(List<String> log) {
        log.add("stock released");
    }

    static SagaResult runSaga(List<Step> steps, List<String> log) {
        int completed = 0;
        for (var step : steps) {
            try {
                step.action().run(log);
            } catch (SagaStepException e) {
                log.add("step '" + step.name() + "' failed: " + e.getMessage());
                for (var done : steps.subList(0, completed).reversed()) {
                    done.compensate().accept(log);
                }
                return new SagaResult.Failed(e.getMessage());
            }
            completed++;
        }
        return new SagaResult.Completed();
    }

    public static void main(String[] args) {
        var steps = List.of(
                new Step("charge", Main::charge, Main::refund),
                new Step("reserve", Main::reserve, Main::release));
        var log = new ArrayList<String>();
        var result = runSaga(steps, log);
        log.forEach(System.out::println);
        System.out.println("saga result: " + result);
    }
}

payment charged
step 'reserve' failed: out of stock
payment refunded
saga result: Failed[reason=out of stock]

The failed step's own compensation is not run because its action did not commit. If a step's outcome is unknown (timeout), the orchestrator must treat it as possibly committed and compensate idempotently.

When to use

• Use sagas for multi-service business operations that tolerate temporary inconsistency.
• Avoid them when strict isolation is required across entities. Keep those entities in one service and one local transaction, which may indicate a wrong service boundary (see service granularity).

Interview questions

• How does a saga differ from 2PC? A saga commits each local transaction immediately and undoes with compensations, so it holds no cross-service locks but exposes intermediate states. 2PC holds locks until a global commit, giving isolation but blocking on coordinator failure.
• What if a compensation fails? It is retried until it succeeds, which requires idempotency. Persistent failure needs manual intervention or an alert.
• Why order the pivot step carefully? Before the pivot everything can be compensated. After it the saga must complete, so steps that cannot be undone are placed after the pivot.
• What is a semantic lock? An application-level flag on a record that marks it as being changed by an in-progress saga, so other transactions can wait, reject, or treat it specially.