Chapter 3: The Five Core Primitives

Consistency, Caching, Scale Breakpoints, Failure Modes, and Build vs. Buy

The Grammar of Distributed Systems

In an L6 interview, you do not win points by reciting definitions of CAP theorem or drawing an unprompted cache box. Evaluators listen for whether you understand the structural fault lines of distributed systems — the places where constraints pull in opposite directions and force an explicit architectural commitment.

Every complex system design is built from five foundational decision primitives:

1. Consistency: What happens when concurrent actors touch the same state?
2. Caching: Who owns the staleness budget, and who pays on invalidation?
3. Scale Breakpoints: At what exact throughput cliff does a naive architecture fail?
4. Failure Domains: When a dependency collapses, does the system degrade or cascade?
5. Build vs. Buy: What is the true multi-year operational maintenance tax?

Master these five primitives, and you will recognize their patterns across every one of the ten system blueprints that follow.

Primitive 1: Consistency Models & Staleness Budgets

The Illusion of "Strong Consistency Everywhere"

Candidates often claim: "We will use strong consistency for everything to be safe."

At single-node scale, strong consistency is cheap. At multi-region distributed scale, strong consistency is an aggressive tax on latency and availability. Under network partitions (which are inevitable in cloud environments), the PACELC theorem dictates that you cannot achieve linearizable reads without paying the cross-datacenter round-trip time (typically 80ms–250ms) on every operation.

The Staff engineer's job is to identify the weakest consistency model that still satisfies the business invariant.

The Three Models That Matter in Interviews

1. Linearizable (Strict Serializability): Every read sees the most recent write globally. Essential for account balances, reservation locks, and inventory deduction. Backed by Raft/Paxos or Spanner-style TrueTime consensus.
2. Read-Your-Own-Writes (Session Consistency): A user immediately sees their own updates, while other users may experience bounded eventual consistency (e.g., 500ms propagation delay). Perfect for user profiles, comments, and status updates.
3. Eventual Consistency with Bounded Staleness: Reads may return stale data up to a defined time or version window (e.g., staleness budget Delta_t < 2s). Perfect for follower feeds, view counts, and search indexing.

STAFF MOVE

Senior: "We will use strong consistency across all database replicas."

Staff: "For the payment debit path, I commit to linearizable writes via a consensus group — a double-charge destroys user trust. However, for balance display and transaction history, Read-Your-Own-Writes with a 1-second staleness budget is sufficient, protecting read availability during region partitions."

Why: L6 engineers allocate consistency budgets per data path, rather than blanketing the whole system with an unworkable latency penalty.

Primitive 2: Caching & Invalidation Ownership

The Staleness Budget Comes Before the TTL

The most dangerous sentence in a caching discussion is: "We will add Redis with a 5-minute TTL."

A TTL is an expiration timer; it is not an invalidation strategy. When you set a 5-minute TTL, you have made an implicit product decision that users are allowed to see 5-minute-old stale data.

If a customer buys a concert ticket, can the seat map show it as available 4 minutes later? No.

If an e-commerce seller changes a product price from $10 to $100, can buyers checkout at $10 for the next 5 minutes? No.

Before touching a cache configuration, the Staff engineer establishes the staleness budget with the product requirements.

The Three Cache Death Traps

1. Cache Stampede (Thundering Herd): When a high-traffic key (e.g., world cup score, celebrity profile) expires or is evicted, 20,000 concurrent requests miss the cache simultaneously and slam the database. Mitigation: Mutex locking on cache miss (SETNX), background proactive refresh, or probabilistic early expiration (XFetch algorithm).
2. Cache Penetration: Malicious queries for non-existent IDs bypass the cache entirely and hammer the storage tier. Mitigation: Bloom filters at the gateway, or caching null values with short TTLs.
3. Invalidation Contract Ambiguity: Service A writes to the database, while Service B reads from the cache. Who invalidates the key? If multiple services mutate shared state without an explicit contract, cache inconsistency becomes permanent. Mitigation: Change Data Capture (CDC via Debezium/Kafka) emitting database WAL changes to evict cache keys asynchronously.

Step-by-Step Architecture: CDC Cache Invalidation & Stampede Shield

CDC-DRIVEN CACHE INVALIDATION & STAMPEDE SHIELD FLOW

sequenceDiagram
  participant C as Client
  participant A as API Service
  participant R as Redis Cache
  participant D as Primary DB
  participant K as Debezium / Kafka
  participant W as Worker
  C->>D: 1. Write Req
  D->>K: 2. Commit WAL
  K->>W: 3. Event
  W->>R: 4. DEL key
  C->>A: 5. Read Req
  A->>R: 6. GET key
  R-->>A: 7. MISS (null)
  A->>R: 8. SETNX lock
  R-->>A: 9. LOCK ACQUIRED
  A->>D: 10. Query Row
  D-->>A: 11. Return Row
  A->>R: 12. SET + TTL
  A->>R: 13. RELEASE lock
  A-->>C: 14. 200 OK

STEP-BY-STEP EXECUTION WALKTHROUGH

1. Asynchronous Write Decoupling: The API service writes updates directly to the Primary Database without touching Redis synchronously, avoiding dual-write race conditions.
2. WAL Capture: The database engine appends the commit to its Write-Ahead Log (PostgreSQL WAL or MySQL Binlog).
3. Change Data Capture Stream: Debezium streams row mutations to a high-throughput Kafka topic db.customer.mutations.
4. Authoritative Cache Invalidation: The dedicated cache invalidation worker consumes the Kafka stream and issues a targeted Redis DEL or UNLINK command. This decouples business logic from cache invalidation ownership.
5. Concurrent Read Ingress: A client request arrives for the invalidated key.
6. Cache Probe: The service queries Redis.
7. Cache Miss: Redis returns null.
8. Distributed Lock Acquisition: Instead of allowing 5,000 concurrent threads to query the database, the service attempts to acquire an atomic distributed lock via SETNX cache:lock:{id} EX 2.
9. Single Worker Election: Only one thread wins the lock; remaining concurrent threads wait 50ms or return previous grace-period data.
10. Targeted DB Query: The winning worker executes a single indexed query against the read replica.
11. Data Return: The replica returns fresh row data in 2ms.
12. Cache Population: The worker populates Redis with the fresh value and configured TTL.
13. Lock Release: The worker deletes the lock key via Lua script.
14. Response Delivery: Fresh data is returned to the client, while all subsequent requests hit Redis at sub-millisecond speeds.

Primitive 3: Scale Breakpoints & Latency Tail Amplification

Scaling Reads vs. Scaling Writes

Scaling reads is an infrastructure problem (solved with read replicas, CDNs, and distributed caches). Scaling writes is an architectural problem (constrained by lock contention, consensus, and disk I/O).

When an interviewer asks how a system scales, candidates often say "we add more shards." A Staff engineer names the numerical cliffs:

• 0 – 10,000 writes/sec: A single properly tuned relational primary (PostgreSQL/MySQL with SSD NVMe and connection pooling like PgBouncer) handles this easily. Sharding here is premature complexity.
• 10,000 – 100,000 writes/sec: Storage write throughput and connection exhaustion hit hard limits. Read-replicas no longer help write load. Partitioning by a high-cardinality shard key (e.g., user_id or tenant_id) becomes mandatory.
• 100,000+ writes/sec: Relational joins across shards are impossible. State must be modeled as append-only immutable event streams (Kafka/LSM trees) or distributed NoSQL (Cassandra/DynamoDB).

The Math of Tail Latency Amplification

In distributed systems, the 99th percentile (p99) latency of an aggregate endpoint is governed by its downstream fan-out. If a single service fans out a request to N backend partition shards in parallel (scatter-gather), the probability that the overall request completes within the single-node p99 latency is:

P(All complete within single-node p99) = (0.99)^N

TAIL LATENCY AMPLIFICATION ACROSS SHARDS

At 100 shards, over 63% of your users experience the 99th percentile worst-case latency. Staff engineers do not design scatter-gather architectures without routing keys, bounded timeouts, and speculative hedged requests.

Primitive 4: Failure Domains & Blast Radius Containment

Fail-Open vs. Fail-Closed

Every component failure forces an intentional product decision:

The Cascading Retry Storm

When a downstream database slows down under load, naive clients with automatic retries multiply the incoming traffic:

Total Load = Base Traffic * (1 + Retries)

If 20% of requests timeout and each client retries 3 times, load surges by +60%, driving the database from slow to completely dead.

THE CASCADING RETRY STORM

flowchart LR
  A["Downstream Slows"] --> B["Requests Timeout"] --> C["Clients Retry Concurrently"]
  C -->|"Retry Storm Overwhelms"| A

Circuit Breaker State Transitions & Cascading Failure Defense

To break positive feedback loops during downstream brownouts, Staff engineers implement state-machine-driven circuit breakers coupled with client retry limits:

CIRCUIT BREAKER STATE TRANSITION & RETRY SHIELD FLOW

stateDiagram-v2
  [*] --> CLOSED: Inbound Request
  CLOSED: CLOSED (Normal Pass)
  OPEN: OPEN (Fast-Fail)
  HALF_OPEN: HALF-OPEN (5% Canary Traffic Allowed to Downstream)
  CLOSED --> OPEN: Error Rate #gt; 50% in 10s Window
  OPEN --> HALF_OPEN: Cooldown (30s Elapsed)
  HALF_OPEN --> CLOSED: Canary Success Rate #gt;= 95%
  HALF_OPEN --> OPEN: Canary Error Rate #gt; 5%

STEP-BY-STEP FAILURE ISOLATION PROTOCOL

1. Normal Operation (CLOSED State): 100% of outbound requests are routed to downstream dependencies. A rolling sliding-window counter monitors latency and HTTP 5xx return codes.

2. Trip Threshold Exceeded: If downstream error rates exceed 50% (or p99 latency breaches 2,000ms) across a 10-second evaluation window, the circuit immediately trips to OPEN.

3. Immediate Fast-Fail (OPEN State): Outbound network calls are suppressed at the client proxy layer. The client returns immediate cached fallback responses or fast-fails in <0.1ms with HTTP 503 Service Unavailable, preventing connection pool starvation.

4. Cooldown Quarantine: The breaker remains in the OPEN state for a fixed cooldown duration (e.g., 30 seconds), allowing the downstream database or service time to drain thread queues and recover.

5. Probe Canary Testing (HALF-OPEN State): Once the cooldown expires, the breaker transitions to HALF-OPEN, routing a conservative 5% canary slice of production traffic to the recovering dependency.

6. Recovery or Re-Trip:

   • If canary requests achieve ≥ 95% success rate, the circuit resets to CLOSED, safely restoring full traffic.
   • If even a single canary timeout occurs, the breaker trips back to OPEN for an extended exponential cooldown (e.g., 60 seconds).

7. Decorrelated Jitter Backoff: Client retries MUST avoid synchronized lock-step retry spikes using the decorrelated jitter algorithm:

   sleep = min(cap, random(base, sleep × 3))

8. Client-Side Retry Budgets: A client allocates a dedicated local token bucket for retries capped at 10% of total outbound requests. If the retry budget is exhausted, further retries are blocked immediately.

Primitive 5: Build vs. Buy & The Maintenance Tax

The Fallacy of "We Can Build That in a Sprint"

Engineers love building infrastructure. Designing a custom message broker, job scheduler, or authentication system sounds intellectually rewarding.

A Staff engineer evaluates the Total Cost of Ownership (TCO) across a 3-year horizon:

TCO = Implementation Cost + 3-Year Maintenance Tax + Operational Pager Burden

The maintenance tax includes security patches, zero-downtime upgrades, disaster recovery runbooks, multi-region replication, and training new engineers.

The Decision Boundary

When to deviate and build custom:

1. Cost Cliff: Managed service pricing scales linearly with usage while self-hosted scales sub-linearly. (e.g., AWS API Gateway at $3.50 per million calls becomes financially unsustainable above 500M calls/month).
2. Latency Ceiling: Off-the-shelf solutions impose network hops or GC pauses that violate hard SLAs (<5ms requirements).
3. Data Sovereignty / Compliance: Regulated customer data cannot touch third-party SaaS vendors.

STAFF CHECKLIST: The Five Primitives

When reviewing any high-level design:

• Consistency: Have I selected the weakest acceptable model for each individual path?
• Caching: Did I establish the staleness budget with product before picking a TTL?
• Scale: Can I identify the exact throughput cliff where this architecture breaks?
• Failure: Have I explicitly stated whether this service fails open or fails closed?
• Build vs Buy: Is this component a core competitive differentiator, or am I building a commodity?