News Feed Fan-Out: Resilience, Interview Questions, and Runbooks

Chapter 14 continued: sections 6 to 10, including the Staff Invariant Checklist. It continues Deep Dives, Data Models, and Schemas.

6. Failure Modes, Edge Cases & Resilience

FAILURE MODES & RECOVERY RUNBOOKS

Failure Mode	Detection Metric	Automated Recovery Action
1. Celebrity Fan-Out Spike	Kafka queue consumer lag > 10,000 msgs	Route author dynamically to Celebrity Outbox; bypass push
2. Redis Node Crash	Connection timeout to primary > 250ms	Redis Sentinel / Cluster promotes replica; cold start timeline rebuild via Kafka
3. Heavy Ranker GPU Saturation	ML inference latency p99 > 40ms	Circuit breaker trips; falls back to reverse-chron order
4. Dormant User Waste	Inactive > 30 days	Inactive user timeline TTL set to 0; rebuild on login

Failure Mode 1: The Redis Cluster Node Failure & Cold Cache Stampede

If a Redis master holding 500,000 user timelines crashes and fails over to an empty replica, what prevents a database stampede?

The Senior Mistake: Query the relational database on every cache miss to rebuild the timeline. Under 50,000 QPS, this immediately crushes the relational database.

The Staff Architecture:

Use SingleFlight / Mutex coalescing per user ID so only one thread queries storage to rebuild a missing timeline.
Implement a Staged Rebuild Pipeline: On cache miss, the gateway returns the last 20 cached posts from the local browser storage or edge CDN, while asynchronously firing a timeline.rebuild task to Kafka.
The user sees their feed immediately without database load.

Failure Mode 2: Unfollowing and Immediate Feed Invalidation

When User A unfollows Creator B, Creator B's posts must disappear from User A's feed:

The Naive Approach: Scan User A's Redis Sorted Set and delete all posts where author_id == Creator B. This requires reading all 800 items and doing 800 hash lookups - prohibitive at scale.
The Staff Approach (Lazy Invalidation): Do nothing to the Redis Sorted Set at write time! During feed hydration, the Feed Gateway checks the post's author against an in-memory Bloom filter of User A's current followings. If User A no longer follows Creator B, the item is silently dropped from the rendered batch. Cost: 0 disk operations, 0 Redis writes.

7. Ten Probing Follow-Up Questions (With Staff Answers)

Q1: "How do you handle a user who follows 5,000 people, including 200 celebrities?"

Senior Trap: "We just query all 200 celebrities in parallel and merge them with the push timeline."

Staff Answer: "Scatter-gathering across 200 distinct Redis keys violates our 120ms SLA due to tail latency amplification. We impose a strict fan-out cap: we query only the top 30 celebrities based on the user's historical engagement score (affinity rank). Posts from the remaining 170 celebrities are fetched only if the user scrolls deeply into their feed. This bounds our read-time fan-out to <= 30 parallel requests, keeping p99 latency under 25ms."

Q2: "What happens if a user's follower count crosses 25,000 while they are actively posting?"

Senior Trap: "We run a database migration to move their old posts into an outbox."

Staff Answer: "We never migrate historical posts during threshold transitions. We flip the user's is_celebrity flag in the Social Graph Cache. Future posts immediately route to their Celebrity Outbox. For past posts, they naturally decay and get pushed out of followers' inboxes as new content arrives via our 800-post capping policy. The system reaches complete steady state within 48 hours with zero data migration."

Q3: "How do you prevent duplicate posts when users rapidly pull-to-refresh?"

Senior Trap: "We check the post IDs on the client side."

Staff Answer: "We enforce deterministic keyset cursor pagination with a high-water mark timestamp. When a client pulls to refresh, it sends since_timestamp = latest_seen_timestamp. The Redis query executes ZREVRANGEBYSCORE timeline:<uid> +inf (since_timestamp. If no new posts have arrived, Redis returns an empty list in 0.1ms and the gateway returns HTTP 304 Not Modified, saving network bandwidth and payload hydration costs."

Q4: "How do you store video and image media without saturating application servers?"

Senior Trap: "The client uploads the photo to our API gateway, which writes it to S3."

Staff Answer: "Application servers should never handle binary blob transport. The client requests a presigned S3 upload URL from our Upload Service. The client uploads binary bytes directly to S3/GCS. S3 emits an asynchronous ObjectCreated notification to Kafka, which triggers our asynchronous transcoding and moderation worker pipeline. The API servers only handle lightweight JSON metadata."

Q5: "How do you handle deleted posts across millions of follower timelines?"

Senior Trap: "When a post is deleted, fan-out delete tasks to all follower Redis timelines."

Staff Answer: "Fan-out on delete suffers from the exact same write storm as fan-out on write. Instead, we use Tombstoning & Lazy Drop. The post status is updated to DELETED in the primary database and cached in a Redis Bloom filter or key-value tombstone (post:tombstone:<id>). During feed hydration, any post marked as a tombstone is dropped from the batch. The stale post ID in followers' timelines will eventually be evicted by ZREMRANGEBYRANK."

Q6: "Why use Redis Sorted Sets instead of a simple Redis List (LPUSH / LRANGE)?"

Senior Trap: "Lists are simpler and support pagination with LRANGE."

Staff Answer: "Redis Lists do not support out-of-order writes or timestamp-based deduplication. If network jitter or queue retries cause an older post to be processed after a newer post, LPUSH places the older post at the top of the user's feed, breaking chronological integrity. A Sorted Set uses the post timestamp as its score, ensuring mathematical ordering regardless of delivery arrival time, while providing O(1) deduplication by post ID."

Q7: "How do you manage feed storage for users who haven't logged in for six months?"

Senior Trap: "We keep their feeds in Redis with a long TTL."

Staff Answer: "We enforce an active timeline TTL of 30 days. Every time a user opens the app, we reset the TTL via EXPIRE timeline:<uid> 2592000. If a user is inactive for 30 days, Redis evicts their timeline automatically. Furthermore, our Social Graph worker checks an active-user Bloom filter before enqueueing fan-out tasks; inactive accounts are skipped at write time. When a dormant user returns, we perform an on-demand cold rebuild from the database."

Q8: "How do you prevent a viral post from causing hot-key memory saturation in Memcached?"

Senior Trap: "We increase the memory of our Memcached instances."

Staff Answer: "When a post goes viral, millions of feeds attempt to hydrate the exact same post ID simultaneously, creating a single-key hot spot. We apply two layers of defense: (1) Local in-process caching (LRU) on the Feed Gateway nodes with a 5-second TTL, absorbing 99% of identical hydration requests before touching the network; and (2) Key salting across Memcached: hot posts are replicated across S = 16 keys (post:<id>:salt_1, post:<id>:salt_2), and gateways query random salts."

Q9: "How do you ensure ads appear seamlessly in the feed without breaking pagination?"

Senior Trap: "The ad service inserts ads directly into the Redis timelines."

Staff Answer: "Ad insertion is strictly decoupled from timeline storage. Redis timelines contain purely organic social post IDs. The Feed Gateway fetches 20 organic posts, queries the Ad Auction Service in parallel with user targeting features, and stitches sponsored posts into fixed slots (e.g., slot 3 and slot 11) before returning the payload to the client. This allows dynamic real-time ad bidding without polluting persistent user timelines."

Q10: "If the social graph database goes down, how does feed serving degrade?"

Senior Trap: "The feed service will return an error until the graph database recovers."

Staff Answer: "Feed serving has zero synchronous dependency on the social graph database. The precomputed timeline in Redis already has the post IDs. For celebrity outbox pulling, the followed celebrity IDs are cached in the user's local session JWT or long-lived Redis cache. Even if the primary social graph database suffers a total outage, feed reading continues to operate uninterrupted for all active users."

8. Production Hardening & Operational Runbooks

PRODUCTION SLOs & ALERT THRESHOLDS

Metric	Target Threshold	Severity & Action
Read Feed p99 Latency	< 120ms (Global)	P1 Alert: Scale Gateway pods; check Memcached hit rates
Fan-Out Queue Lag	< 5 seconds backlog	P2 Alert: Autoscale fan-out workers; verify Kafka offsets
Heavy Ranker Latency	< 40ms	P2 Alert: Trip circuit breaker to fallback ranker
Redis Memory Headroom	< 75% utilization	P1 Alert: Trigger proactive cluster resharding

Incident Runbook: Mitigating Fan-Out Queue Saturation

If a sudden breaking news event triggers millions of posts simultaneously, causing fan-out queues to back up past acceptable thresholds:

Step 1 (Throttle Dormant Fans): Temporarily adjust the active-user threshold filter from 30 days down to 7 days. This immediately eliminates 40% of fan-out write tasks.
Step 2 (Lower Celebrity Threshold): Dynamically drop the celebrity cutoff K from 25,000 down to 10,000 via runtime feature flags. All accounts with > 10,000 followers immediately switch to outbox pull mode, shedding write volume by another 50%.
Step 3 (Worker Pool Autoscaling): Trigger KEDA horizontal pod autoscaling on the fan-out worker fleet, scaling worker instances to consume partition backlogs.

9. The Five Structural Fault Lines

Every design decision in feed generation balances a structural tradeoff:

THE 5 STRUCTURAL FAULT LINES

Fault Line	Option A vs Option B	Who Pays?
1. Write vs Read Optimization	Pure Push vs Pure Pull	Push: Infra pays write storage Pull: Users pay p99 latency
2. Timeline Fresh vs Compute Cost	Eager Precomputation vs On-Demand Assembly	Eager: High write waste On-Demand: High CPU on reads
3. ML Accuracy vs Read SLA	Heavy Neural Nets vs Light Heuristic Rank	Heavy: Users pay latency tax Light: Product pays engagement
4. Invalidation Cost	Eager Post Purge vs Lazy Hydration Drop	Eager: Fan-out write storm Lazy: Edge cache memory bloat
5. Pagination Semantics	OFFSET/LIMIT vs Keyset Cursor Tokens	OFFSET: DB CPU & page drift Keyset: Client state tracking

10. Chapter Summary & Staff Decision Matrix

STAFF DECISION MATRIX: NEWS FEED

Decision Point	Default Staff Position
Fan-Out Model	Hybrid (Push for < 25k followers, Pull for >= 25k)
In-Memory Cache	Redis Sorted Sets (Score: timestamp, Member: id)
Timeline Depth Cap	800 items per user (ZREMRANGEBYRANK)
Inactive User Handling	30-day TTL expiration; on-demand cold rebuild
Feed Pagination	Keyset cursor tokens based on (timestamp, post_id)
Post Hydration	Asynchronous parallel MGET from Memcached cluster
Ranking Strategy	Two-stage funnel with 40ms hard circuit breaker
Unfollow/Delete Logic	Lazy invalidation during read hydration step

The Staff Invariant Checklist

Before defending a feed generation design in an interview, verify that you have satisfied these four non-negotiable principles:

I have named the celebrity threshold: I did not blindly say "push" or "pull" and explained how both models collaborate. I specified the numerical cutoff
I have eliminated deep database scans: All reads are served from memory via opaque keyset cursors; zero OFFSET queries exist on the read path.
I have bounded timeline memory in RAM: Every timeline is strictly capped at 800 items, and inactive accounts automatically expire from memory.
I have pre-empted ranking degradation: The heavy ML ranker is protected by a circuit breaker that fails gracefully to chronological ordering on timeout.