Eventual Consistency and Cross-Service Queries

With database per service, no query can join tables across services. Yet screens and reports need combined data, such as orders with customer names and product titles. Options differ in freshness, availability, and complexity. Most of them are eventually consistent: if no new updates occur, all copies of the data converge to the same value, but at any moment copies can differ. See consistency models.

Why eventual consistency appears

Changing data in several services cannot be one atomic transaction (see saga pattern). Copies of data held by other services are updated through asynchronous events. Between the change and the arrival of the event, readers see old values. The delay is usually milliseconds to seconds but is unbounded during outages or consumer lag.

Approach 1: API composition

A composer calls each owning service and joins results in memory (see BFF and API composition).

• Data is as fresh as each service's own state.
• No extra storage.
• Availability is the product of the services' availabilities when all are required: three services at 99.9% each give about 99.7% (0.999^3 = 0.997). Latency is that of the slowest call when the calls run in parallel, and the sum when they run in sequence.
• Partial failure needs a defined behavior: fail the whole request, or return the data available with the missing part marked absent.
• Filtering and sorting on fields from different services is inefficient, because the composer must fetch large sets and join them itself. Example: "orders of customers in Berlin, sorted by order total" requires fetching all Berlin customers from Customer Service and then their orders from Order Service, since neither service can evaluate the whole predicate.
• Suited to lookups by key and small result sets.

Approach 2: Replicated data via events

A service keeps a local read-only copy of the fields it needs from another service, updated by that service's events.

Example: Order Service stores customerName and shippingAddress from CustomerUpdated events, so it does not call Customer Service to render an order.

flowchart LR
    CS["Customer Service"] --> CDB[("Customer DB")]
    CS -->|"CustomerUpdated"| B["Broker"]
    B --> OS["Order Service"]
    OS --> ODB[("Orders DB + customer copy")]

• Queries are local and fast, and the service works when the source is down.
• Data is stale for a short period. Choose fields where that is acceptable, or store a snapshot deliberately: an invoice should keep the address at the time of purchase, not follow later changes.
• Storage duplication and event handling code.
• Initial population and repair need a way to load full state: a snapshot event, replay from the log, or a bulk API.
• The consumer needs a copy of only the fields it uses, so it is not coupled to the owner's full model. Events should carry sufficient data (event-carried state transfer; see event-driven architecture).

Approach 3: CQRS view

A dedicated query service consumes events from several services and maintains a denormalized read model designed for specific queries, such as an order-history view with customer and product fields, or a search index (see event sourcing and CQRS).

• Supports filtering, sorting, and pagination on combined data in one store.
• Read scales independently, and the store can be chosen for the query type.
• Adds a service, and requires handling out-of-order and duplicated events, and rebuilding the view when logic changes.
• Events from different services arrive in no defined order relative to each other. A projector must tolerate an OrderPlaced event arriving before the related CustomerCreated, by buffering, by storing partial rows (the customer columns stay empty until the customer event arrives), or by using version numbers.

Approach 4: Reporting and analytics pipeline

Analytical queries scan large volumes across domains and must not run against operational databases. Stream changes to a warehouse or data lake:

flowchart LR
    S1["Service A DB"] -->|"CDC / events"| K["Event stream"]
    S2["Service B DB"] -->|"CDC / events"| K
    K --> L["Data lake / warehouse"]
    L --> BI["Reports, dashboards"]

• Sources: domain events, or CDC of tables. Domain events are a stable contract, while raw CDC exposes internal schemas.
• Loading is often delayed by minutes or done in batches (streaming loads can be faster), and the processing follows stream vs batch trade-offs.
• Each fact keeps its source service as the owner. The warehouse is a derived copy.

Comparison

Living with eventual consistency

• Read-your-writes. After a user changes something, they should see it. Return the new state in the response, or read from the source of truth for that user for a short time, or wait for the view to reach a version.
• Idempotent and order-tolerant handlers. Events can be duplicated or reordered. Use version numbers and ignore stale updates: apply an update only if its version is greater than the stored version (see idempotent consumers and deduplication).
• Reconciliation. Periodically compare copies with the source and repair drift caused by lost events or bugs.
• User experience. Show pending states (PENDING, "processing") instead of pretending the change is complete.
• Invariants. Rules that need a strongly consistent view (unique username, no overselling) must be enforced by the service that owns the data, not by a stale replica.

Interview questions

• How do you show customer names on an order list when they live in another service? Options: compose at read time, store the name locally updated by CustomerUpdated events, or serve a CQRS view. The local copy gives the fastest, most available reads with short staleness.
• Why must event handlers tolerate reordering? Events from different services or partitions have no global order, so a handler can receive a dependent event before the event it depends on. Versions or buffering resolve this.
• Why not run reports on service databases? Heavy scans harm operational latency, cross-service joins are impossible, and it couples the reporting tools to internal schemas. A derived warehouse avoids these.
• Where should uniqueness be enforced? In the service that owns the data, transactionally. Replicas cannot guarantee it.