Chapter 2: The Staff Decision Loop, Part 2 (Decide)

This lesson covers the second half of Chapter 2. It continues Chapter 2, Part 1 (Understand).

The DECIDE Phase

Step 4: Make the Call

You've named the intent. You've mapped the Fault Lines. You've identified who pays for each option. Now you commit.

This is where L5 and L6 engineering most visibly diverge. L5 presents options. L6 makes the call.

"It depends on your consistency requirements and access patterns" is not an answer. The interviewer knows it depends. What they want to know is: given this context, given these Fault Lines and these payers, what would you actually build? And can you defend it?

The mechanism that makes Step 4 fast and confident is the Default Staff Position — the pre-committed answer a Staff engineer gives in 30 seconds and can defend. Not because the answer is always right, but because you've already done the reasoning before the interview or the design meeting. You know your position. You know the conditions under which you'd deviate. And you know how to explain both.

Here is a table of Default Staff Positions synthesized from the 36 StaffSignal playbooks — the pre-committed answers for the most common design decisions:

Design Decision	Default Staff Position	When to deviate
Rate limiter algorithm	Token bucket	Fixed window only when simplicity is the explicit constraint and boundary exploits are acceptable
Rate limiter placement	Gateway/sidecar	SDK only when teams absolutely cannot standardize on a platform approach
Rate limiter failure mode (abuse)	Fail-open	Fail-closed only for billing/quota, never for abuse protection
Cache strategy	Cache-aside with explicit TTL	Write-through only for high write-to-read ratio; write-behind only when write latency is the primary constraint
Consistency for financial writes	Linearizable	Accept AP for balance display; never AP for debit path
Fan-out strategy	Hybrid (push for ≤10K followers, pull for celebrities)	Pure push only below 1K followers per user average; pure pull only for broadcast-style feeds
Queue delivery	At-least-once + idempotency	Exactly-once only when idempotency is genuinely impossible and cost of duplication is unacceptable
Auth enforcement	Sidecar/mesh layer	SDK only for early-stage prototyping, never in production at scale
Payment data model	Immutable append-only ledger	Status field never acceptable for financial state transitions
Retry policy	Exponential backoff with jitter	Linear backoff only for very low-volume systems; no backoff is never acceptable
Shard key selection	Evaluate 3-4 candidates against read patterns before choosing	Never pick the "obvious" primary key without validating access patterns
Message ordering	Partition by entity ID, not global ordering	Global ordering only when there is a genuine cross-entity dependency
Cache TTL	Explicit TTL per content type, documented staleness budget	Infinite TTL only for truly immutable content

These are not commandments. They are starting points with named deviation conditions. The power of a Default Staff Position is not that it's always right. It's that you can explain exactly when it's wrong, which means you can recognize the deviation conditions when you encounter them.

When an interviewer asks "why token bucket?" a L5 engineer explains the algorithm. A L6 engineer says: "Token bucket is my default for rate limiting because it smooths traffic naturally and allows controlled bursts. I'd use a fixed window only if implementation simplicity was the explicit constraint and the boundary exploit — where a client can send 2× the rate limit by sending requests at t=59s and t=61s — was acceptable to the product. Is that the constraint here?"

The Default Staff Position makes you fast. The deviation conditions make you right.

CONTROVERSIAL OPINION

"Most system design interviews are decided in the first 3 minutes. Not by what you know — by whether you commit before being asked to."

Interviewers can detect within a few minutes whether a candidate is running a process or improvising. The process looks like: clarify intent, name Fault Lines, commit to a position. Improvisation looks like: draw boxes, add complexity when pushed, say "it depends" when cornered. The candidate who commits early and has to revise once is more impressive than the candidate who never commits and hedges until the end.

Revision is not weakness. It is evidence that you're listening and updating. Perpetual hedging is evidence that you were never actually reasoning — just performing reasoning.

Step 5: Stress-Test with Failure

Every design is a theory. Failure is the experiment that tests it.

Most engineers stress-test for the obvious failures: what if the database goes down, what if the network is slow, what if the disk fills up. These are important. But they are the easy failures — the ones that everyone thought of and designed for.

The failures that cause major incidents are usually the subtle ones: the failure mode of the failure handler, the recovery process that itself becomes the attack vector, the circuit breaker that causes the cascade it was designed to prevent.

Before calling any design done, Staff engineers run through five questions:

What's the worst thing that can happen to this component? Not "what if it's slow" — what if it's completely gone? What if it returns wrong data? What if it returns errors for 10% of requests?
Who handles that failure? Not "we'll add monitoring" — which specific team, which on-call rotation, which runbook? Who has the authority to make the call about recovery?
What's the blast radius? How many users are affected? How many downstream systems? How many teams? Is the blast radius bounded or unbounded?
What's the recovery path? How long does recovery take? What manual steps are required? Who authorizes them? Can recovery be automated?
Does the recovery path have its own failure mode? This is the question most engineers skip. The recovery action that makes things worse is the source of the worst incidents.

The Blast Radius is the set of systems, teams, and users affected when a component fails. Staff engineers always have a mental model of blast radius before they commit to a design. Sanjay's question in the incident channel — what's the blast radius right now? — was Step 5 applied to a live incident. The same question belongs in every design review.

The circuit breaker cascade example. A circuit breaker is a component that monitors error rates on outbound calls and "trips" — stops sending requests — when the error rate exceeds a threshold. It's designed to prevent cascading failures. If your payment service's calls to the payment provider are failing at 40%, the circuit breaker trips and you stop hammering the provider, allowing it to recover.

Here's the failure mode: the circuit breaker trips, and 10,000 client instances immediately switch to their retry logic. All 10,000 instances retry simultaneously, generating a thundering herd of 30,000 requests in the first second. The payment provider, which was recovering from the original degradation, is now overwhelmed by a load spike 6× larger than normal traffic. The circuit breaker caused the cascade it was designed to prevent.

The stress-test question that catches this: what happens when the circuit breaker trips and all clients retry simultaneously? Answer: exponential backoff with jitter, retry budget per client, probe traffic from only 1% of clients during the half-open state. These design decisions exist specifically to prevent the cascade. But you only add them if you asked the question.

STAFF MOVE

Senior: "If Redis goes down, the rate limiter will fail gracefully and we'll degrade."

Staff: "For abuse protection, we've designed fail-open, so Redis going down means traffic flows unrestricted — that's intentional. But I need to know: is this a shared Redis cluster? If billing enforcement also uses it, billing enforcement cannot fail-open. Those need to be separate clusters with separate SLOs. Who owns the cluster separation decision — is that us or the platform team? And what's the runbook when the abuse protection cluster goes down — do we alert? How long before we escalate?"

Why: "Degrade gracefully" is not a failure plan. A failure plan names what degrades, by how much, for whom, and who decides when to stop degrading.

CONTROVERSIAL OPINION

"The most dangerous phrase in a system design interview is 'we'll add monitoring for that.'"

Monitoring is a detection strategy, not a mitigation strategy. It tells you something broke. It does not prevent the break, contain the blast radius, define the recovery path, or authorize anyone to do anything. When a candidate says "we'll add monitoring," what they mean is "I haven't thought about what happens when this fails." Staff engineers name the runbook, the on-call team, the escalation path, and the recovery action — not the dashboard. The dashboard is how you know to look at the runbook.

Running the Loop: Distributed Caching

The loop is easier to internalize when you watch it run as a coherent process. Let's apply all five steps to distributed caching — a problem we'll go deeper on in Chapter 4.

This example is not meant to be comprehensive. It's meant to show the loop working as a unified process, not as five disconnected steps.

Step 1 — Commit to Intent

"We need caching" is not an intent. It's a category. Three different caching intents produce three different architectures:

Performance caching — you're reducing latency for frequently-read data. The intent is speed. Failure mode: cache miss is just slower. This is a fail-open scenario. If the cache dies, requests fall through to the database. You want the database to handle the load spike, or you want the cache to come back quickly. The critical question here is: can the database survive direct load if the cache dies cold?

Cost reduction caching — you're reducing the number of database reads or API calls because they're expensive (monetarily or computationally). The intent is cost control. Failure mode: cache miss is expensive, not just slow. The blast radius of a cold cache is a cost spike, not a latency spike. Design accordingly — cache warming strategies matter here in a way they don't for performance caching.

Availability caching — the database is not always available (planned maintenance, regional failover), and the cache is your availability buffer. The intent is continuity. This is not a fail-open scenario. If the cache dies at the same time as the database, you have no fallback. This intent requires the cache to be more highly available than the database, which changes the SLO requirements entirely.

Name the intent before drawing a cache box. Most teams think they're doing performance caching. They're often doing all three simultaneously, with a design optimized for one and fragile in the other two.

Step 2 — Find the Fault Lines

Freshness vs. Consistency: Every cache has a staleness window — the maximum time between a data change in the database and the cache reflecting that change. Who sees stale data? What's the business impact? A cached product price that's 60 seconds stale might cause a customer to see the wrong price. A cached recommendation list that's 5 minutes stale has no meaningful impact. The staleness tolerance is a product decision that Engineering must make visible. Most teams pick a TTL without answering the business question.

Invalidation Ownership: Who is responsible for invalidating a cache entry when the underlying data changes? If two teams share a cache without an explicit invalidation contract, one team's write will silently invalidate the other team's cached data — or worse, one team's TTL will serve the other team's stale data. Shared caches without documented ownership contracts are incidents waiting to happen.

Thundering Herd: When a popular cache key expires (or is invalidated), a large number of requests simultaneously hit the database to repopulate it. At 10,000 RPS with a 1-minute TTL on a popular key, the expiry of that key generates 10,000 simultaneous database reads in the first second. Does the database survive this? Most teams don't know.

Memory vs. Latency: Cache size is bounded. Eviction is inevitable. When a key is evicted unexpectedly — because the cache is full, not because the data expired — you get a latency spike for that key. The eviction policy (LRU, LFU, TTL-based) determines which keys get evicted. Getting it wrong means your most-accessed keys get evicted during high-load periods, exactly when you need them most.

Step 3 — Name Who Pays

Design choice	Who pays	How they pay
Long TTL (stale data)	Users	Wrong prices, stale profiles, outdated content
Short TTL (more database load)	Infra	More database reads, higher cost, potential thundering herd
Complex invalidation logic	Engineering	Debugging cache coherence bugs, on-call for invalidation failures
Shared cache between teams	Both teams	Silent cache corruption, cross-team incidents, undefined blast radius
No cache warming on deployment	Users (briefly)	Cold cache spike on every deploy
Eviction policy mismatch	Users (unpredictably)	Random latency spikes for unexpected cache misses

The staleness tolerance question is a product decision. Engineering should not pick a TTL without answering: "What's the worst user experience from stale data, and what's the business impact?" Most teams pick 5 minutes because it sounds reasonable. Whether 5 minutes is acceptable depends on the data. For a product price, 5 minutes might cause a visible discrepancy. For a profile avatar, 5 minutes is imperceptible.

Step 4 — Make the Call

Default Staff Positions for caching:

Cache strategy: Cache-aside (lazy loading). Read from cache; on miss, read from DB and populate cache. This is the default because it's resilient: if the cache is unavailable, the DB still serves requests. Write-through (synchronous cache population on write) only when read-after-write consistency is required and the write latency cost is acceptable.

TTL ownership: Explicit TTL documented per cache key or key pattern, with a named owner. Never "we'll figure out the TTL later." A cache with an undocumented TTL is a future incident.

Invalidation contract: If two teams share a cache, there must be a written contract: who invalidates, on what event, with what latency guarantee. No contract means the first cross-team incident will have no clear owner.

Thundering herd mitigation: Probabilistic early expiration (jitter added to TTL so keys don't all expire simultaneously) + request coalescing (one read goes to DB; others wait for it to populate the cache). These two mechanisms together eliminate most thundering herd issues at negligible implementation cost.

Cache warming: On cold start or deploy, warm the cache from the DB before serving traffic. The cost of warming is much lower than the cost of a cold cache spike.

Step 5 — Stress-Test with Failure

What happens when the cache dies completely?

This is the first question, and most teams don't have a confident answer. The database must survive direct load from all cache-hit traffic becoming database reads simultaneously. If you have 50,000 RPS with an 80% cache hit rate, the database is handling 10,000 RPS under normal operation. When the cache dies, it suddenly receives 50,000 RPS. Does it survive? If you don't know, you haven't designed your caching layer — you've just added a component without understanding its failure mode.

What happens when you deploy a change that alters the cache key format?

Silent inconsistency. Old keys continue to be served from cache with the old format. New writes use the new format. Depending on your cache-aside logic, you may serve stale data indefinitely for any key that doesn't happen to expire before the deployment. The mitigation: cache key versioning (include a version prefix in every key; increment it on schema changes) or explicit invalidation on deploy.

What happens when two services both invalidate the same key simultaneously?

A race condition. Service A invalidates and repopulates. Service B invalidates. The repopulation from Service A is now stale. Depending on timing, this can produce a window of stale data longer than the TTL. The mitigation: CAS (compare-and-swap) operations for cache invalidation, or accepting the race and relying on TTL as the safety net.

STAFF MOVE

Senior: "We'll add a 5-minute TTL to the product cache."

Staff: "Before we set a TTL — what's the staleness budget for product data? A 5-minute-stale price shown to a customer is a product decision, not an engineering one. Who signed off on that? And does the database survive if the cache dies and 50,000 RPS hits it directly? That's the failure mode I care about first."

Why: TTL is not a caching strategy. Staleness budget is a product decision. Database survivability is a reliability requirement. Engineering's job is to implement whatever the product decides — and to make the cost of that decision visible before committing.

The Gaps: Which Steps L5 Engineers Skip

Running the loop in an interview or a design review is straightforward once you know the steps. What's less obvious is which steps L5 engineers most commonly skip — because that tells you where to focus your practice.

Step 1 (Intent) is skipped most often, by the widest margin. Engineers go directly from the problem statement to the architecture. "Design a rate limiter" → draw a token bucket. "Design a notification system" → draw a queue. The intent question feels like overhead, like you're delaying the "real" work. You're not. You're doing the most important work. The architecture that correctly serves the wrong intent is a worse outcome than no architecture.

Step 3 (Who Pays) is the second most skipped. Engineers instinctively see cost as a technical property — latency, throughput, error rate. They don't instinctively see cost as an organizational property — which team absorbs the debugging burden, which users experience the degradation, which stakeholder fields the customer calls. Making cost organizational rather than purely technical is one of the most important mindset shifts in L5-to-L6 development.

Step 5 (Stress-Test with Failure) is the most commonly done incorrectly. It gets attempted — most engineers think about failure, especially when prompted. But it usually stops at the obvious failure: database down, network slow, disk full. The failure of the failure handler, the race condition in the recovery path, the thundering herd caused by the circuit breaker — these require deliberate, systematic questioning. "What happens when the circuit breaker trips and all clients retry simultaneously?" is not a question most engineers ask unprompted. It's the question that would have prevented a major incident at three different companies.

Steps 2 (Fault Lines) and 4 (Make the Call) are attempted but imperfectly. L5 engineers find one Fault Line and solve it, without noticing that solving it constrains the solution to another Fault Line. L5 engineers present options rather than committing, which looks like thoroughness but reads as non-commitment.

The practice protocol. For the next four weeks, apply the loop to every production system you read about. Not as an exercise — as a habit.

When you read a postmortem: What was the intent? (What failure was unacceptable?) What were the Fault Lines? (Where did constraints pull apart?) Who paid? (Which team absorbed the incident cost?) What was the Default Staff Position the team violated, and why? What was the failure mode they didn't stress-test?

When you read an architecture blog post: What's the intent? What are the Fault Lines? Who pays for each design decision? What's the failure mode the author didn't mention?

When you review a teammate's design doc: Run the loop. Ask the questions. Not to be difficult — because the questions that surface issues in a design review are much cheaper than the questions that surface issues in a postmortem.

The loop is not a framework you apply to new problems. It's a habit you develop until it runs unconsciously, the way Sanjay ran it at 2:17 AM in a Slack channel while everyone else was proposing fixes.

What Comes Next

With the Staff Decision Loop established, the remainder of this book puts it to work:

Chapter 3 — The Five Core Primitives: The foundational mechanics governing all distributed systems: consistency models, caching and invalidation ownership, scale breakpoints and tail latency amplification, failure domains and retry storms, and build vs. buy maintenance taxes.

Part II — Fifteen Production System Blueprints (Chapters 4–18): Fifteen end-to-end worked masterclasses across the most frequently tested Tier-1 systems: Rate Limiting, Payment Ledgers, Flash Sales, Task Schedulers, Message Queues, Ride Dispatch, Notification Engines, Collaborative Chat, Cloud File Storage, Financial Order Matching, News Feed Fan-Out, Global Video Streaming, Search Autocomplete, Distributed URL Shortening, and Real-Time Collaborative Document Editing.

Part III — Interview Mastery & The Staff Evaluator (Chapters 19–22): How to turn this thinking into whiteboard performance. Chapter 19 provides the 45-minute pacing manual. Chapter 20 provides the complete scoring rubric and 35 follow-up probes. Chapter 21 arms you with defensible engineering positions, and Chapter 22 previews the twenty deep infrastructure playbooks in Volume 2.

Every chapter is the same process you just read. The loop doesn't change. The systems do.

STAFF CHECKLIST: The Staff Decision Loop

For any system design problem:

Have I named the intent? (Not the topic — the specific failure that's unacceptable)
Have I mapped all the Fault Lines? (Not just the obvious one — find where they interact)
For each Fault Line, have I named who pays for each side of the choice?
Have I committed to a position I can justify in 30 seconds with named deviation conditions?
Have I stress-tested with complete component failure, not just degraded performance?
Do I know who handles each failure at 3 AM, with what runbook and what authority?