News Feed Fan-Out: Framing, Scale Math, and Architecture

Chapter 14: News Feed & Social Timeline Fan-Out. Part II — Production System Blueprints. This lesson covers sections 1 to 3 of the chapter; it continues in Deep Dives, Data Models, and Schemas.

1. Executive Summary & The Problem Space

The Scene: The Midnight Celebrity Post

At 11:42 PM on a Friday evening, a pop icon with 120 million followers posts a 15-second video announcement to your photo-sharing platform.

Within four seconds, the write path crashes.

The background message queues backing the fan-out workers spike from an average depth of 45,000 jobs to an insurmountable 120,000,000 pending tasks. Memory utilization on the primary Redis cluster spikes from 62% to 99.4%, triggering an aggressive kernel out-of-memory (OOM) reaper. Because the background workers attempt to push the single post ID into the precomputed timeline inboxes of all 120 million followers simultaneously, the Redis replication stream lags by more than 18 minutes.

Ordinary users trying to see dinner photos posted by their friends notice their home timelines have frozen. Incoming posts from normal users are queued behind the celebrity's massive fan-out backlog. Refreshing the mobile app returns either HTTP 504 Gateway Timeouts or stale content from six hours prior.

In the emergency incident channel, a Senior engineer suggests provisioning 500 additional background worker pods and doubling the Redis cluster shards.

A Staff engineer interjects with a single question: "Why are we pushing a celebrity post to 120 million people who aren't even holding their phones right now?"

The Senior engineer looked at the problem through an implementation lens: We have a fan-out queue, so when a post arrives, we fan it out. The Staff engineer looked at the problem through an organizational and architectural lens: A social graph exhibits severe power-law skew. Treating a user with 50 followers identically to a creator with 100 million followers is architectural suicide.

The fix was not adding more Redis instances. The fix was introducing a hybrid fan-out threshold: pushing posts for normal users, but writing celebrity posts to a dedicated outbox and pulling them dynamically at read time.

Clarifying Intent: Three Incompatible Feed Systems

When an interviewer says, "Design the Instagram Home Feed" or "Design Twitter Timelines," they are not asking you to draw a generic database with a feed cache. They are testing whether you recognize that different feed intents require mutually incompatible distributed architectures:

THE THREE FEED GENERATION INTENTS

1. Intent 1: Freshness-First (Pure Reverse-Chronological): Posts must appear across follower timelines within 3 seconds of creation. The ordering key is strictly temporal (created_at). Highly vulnerable to celebrity write amplification.
2. Intent 2: Engagement-First (Algorithmic Ranking): Content is scored by affinity, expected dwell time, and interaction probability. Freshness is subordinated to relevance. Requires a multi-stage machine learning inference pipeline (Candidate Generation -> Light Ranker -> Heavy Deep Ranker).
3. Intent 3: The Hybrid Production Standard (The Staff Commitment): Reverse-chronological or lightly ranked timelines that cleanly separate normal graph fan-out from high-fanout celebrity broadcasts. It protects infrastructure stability while guaranteeing sub-150ms p99 feed delivery to clients.

STAFF MOVE

Senior: "We will use fan-out on write because reading a precomputed feed from Redis is O(1) and delivers sub-10ms response times to the mobile client."

Staff: "Pure fan-out on write collapses when users with millions of followers post. A single celebrity post generates 100 million write operations, creating massive write amplification and starving ordinary user updates. I will establish a hard threshold at K = 25,000 followers: accounts below K use fan-out on write; accounts above K use fan-out on read via a dedicated outbox, merged on the fly at read time via an in-memory k-way heap."

Why: The Staff engineer designs for the structural power-law distribution of real-world social graphs rather than assuming uniform user behavior.

Scale & Back-of-the-Envelope Economics

To make defensible architectural decisions, we anchor our design to the operational reality of a global social network (comparable to Instagram or Twitter/X):

• Daily Active Users (DAU): 500,000,000 active accounts.
• Concurrent Users (Peak): 50,000,000 simultaneous active sessions.
• Daily New Posts Created: 100,000,000 posts/day (~1,160 posts/sec average, 5,000 posts/sec peak).
• Daily Feed Refreshes (Reads): 5,000,000,000 feed requests/day (~58,000 reads/sec average, 150,000 reads/sec peak).
• Read-to-Write Ratio: 50:1. The system is overwhelmingly read-heavy.
• Average Follower Count: 200 followers per user.
• Celebrity Skew (Pareto Principle): The top 0.1% of accounts have between 100,000 and 150,000,000 followers.

SCALE & HARDWARE CAPACITY REQUIREMENTS

2. Requirements & High-Level Architecture

Functional Requirements

1. Feed Retrieval: Users can retrieve a personalized feed of the most recent and relevant posts from accounts they follow, paginated in batches of 20 items.
2. Post Publishing: A user can publish a post (text, image, video metadata) that propagates to followers' feeds within 5 seconds.
3. Social Graph Follows: Users can follow and unfollow other accounts, dynamically updating feed assembly rules.
4. Pagination: Clients can continuously scroll backward through historical feed items without encountering duplicate posts or skipping content.

Non-Functional Requirements (Staff Invariants)

1. High Availability over Strict Consistency (AP System): A user seeing a feed delayed by 3 seconds is completely acceptable; a user seeing an HTTP 500 error when opening the app is a P0 business failure.
2. Sub-120ms P99 Read Latency: Feed fetching is on the critical user interaction path. The p99 response time must not exceed 120ms globally.
3. Resilience to Celebrity Shocks: A celebrity with 100M followers posting must never degrade timeline freshness for unrelated users.
4. Bounded Storage Footprint: Memory consumption per user timeline in RAM must be strictly capped; inactive users must not consume memory.

3. End-to-End System Architecture

The architecture cleanly decouples the asynchronous fan-out write pipeline from the synchronous, low-latency feed serving and ranking path:

Figure 14.1: End-to-End Hybrid Push/Pull News Feed & Social Timeline Architecture

Architectural Components in Depth

COMPONENT 1: POST INGESTION & MEDIA INGESTION PIPELINE

The entry point for all new content created by users:

• Direct S3 Presigned Uploads: High-bandwidth media payloads (images, video reels) stream directly from the client to Amazon S3/GCS using short-lived presigned PUT URLs, completely isolating application servers from media bandwidth saturation.
• Transactional Metadata Persistence: The client submits post metadata (caption, tagged users, media URL, client idempotency token) to the Post Service. The service writes to CockroachDB/PostgreSQL with read-after-write consistency, returns an HTTP 201 Created response to the user in < 25ms, and synchronously appends a PostCreated event into Apache Kafka topic posts.raw.
• Media Preprocessing Egress: An asynchronous worker fleet triggers thumbnail generation, content moderation scanning (NSFW image detection), and copyright fingerprinting in parallel without blocking post publication.

COMPONENT 2: SOCIAL GRAPH SERVICE & FAN-OUT CLASSIFIER

The routing intelligence of the feed system:

• TAO-Style Association Graph Cache: Evaluates follower relationships using a distributed graph cache (modeled after Meta's TAO), storing bi-directional edges (User -> Follows -> Creator) in memory-mapped caches.
• Dynamic Follower Threshold Classification:
  • Standard Creators (< 25,000 followers): The post is routed to the Push Fan-Out Pipeline. The service divides the follower list into batches of 1,000 follower IDs and emits tasks to Kafka topic fanout.push.tasks.
  • Celebrity Creators (≥ 25,000 followers): The post is written once to the creator's Celebrity Outbox (a Redis Sorted Set keyed as outbox:<creator_id>). Zero follower inboxes are written, eliminating millions of write operations in a single architectural decision.
• Active User Bloom Filter: For push tasks, workers consult an in-memory Counting Bloom filter tracking active users (opened the app within 30 days). If a follower is dormant, their timeline inbox is skipped entirely, eliminating 65% of useless write operations.

COMPONENT 3: ASYNC FAN-OUT WORKER FLEET

A horizontally scalable fleet of stateless Go/Rust consumers:

• Pipelined Redis Writes: Consumes batches of 1,000 follower IDs from fanout.push.tasks. For each active follower, the worker issues pipelined Redis commands: ZADD timeline:<user_id> <timestamp> <post_id> and ZREMRANGEBYRANK timeline:<user_id> 0 -801
• Bound RAM Enforcement: The ZREMRANGEBYRANK call caps the timeline at 800 items in RAM, preventing long-tail memory leakage.
• Idempotency & Deduplication: Redis Sorted Set members are 64-bit post_ids. If a worker retries a batch after a network timeout, ZADD is natively idempotent, guaranteeing zero duplicate posts in the user's timeline.

COMPONENT 4: REDIS IN-MEMORY TIMELINE CLUSTER

The low-latency storage tier holding precomputed feeds for all 50M Daily Active Users:

• Consistent Hashing by User ID: The cluster is sharded across 16 primary-replica nodes partitioned by hash(user_id) % num_nodes.
• Memory-Optimized Storage Format: Timelines store only 8-byte post IDs and 8-byte timestamps as 64-bit floats. Full post bodies and author profiles are strictly excluded.
• Memory Sizing Physics: 800 posts × 16 bytes = 12.8 KB per active user. For 50 million DAUs, the working set is ~640 GB of RAM—easily hosted in a compact, cost-effective Redis cluster.
• LRU Inactive Eviction: Users who remain inactive for 30 days are evicted to cold storage (Cassandra/ScyllaDB), freeing cluster RAM for active sessions.

COMPONENT 5: CELEBRITY OUTBOX & K-WAY HEAP AGGREGATOR

The dynamic assembly engine executed at read time:

• Celebrity Fan-In: When a user opens their feed, the Feed Serving Gateway queries the Social Graph Service to retrieve the list of celebrities followed by that user (typically ≤ 50 celebrities).
• Parallel Fan-In Queries: Issues asynchronous non-blocking queries against outbox:<celebrity_id> keys in Redis to fetch their recent posts.
• In-Memory K-Way Min-Heap Merge: Merges the user's precomputed timeline with the celebrity outbox posts using an in-memory Min-Heap. Combining K sorted streams of size N runs in O(N log K) time, completing in under 2.5 milliseconds for a batch of 50 items.

COMPONENT 6: TWO-STAGE FEED RANKING ENGINE

The machine learning ranking tier that maximizes user engagement:

• Stage 1 (Light Heuristic Ranker): Filters the top 200 candidates down to 50 items in < 8ms using rule-based heuristics: author affinity, post freshness half-life decay, and media diversity penalties (preventing 5 consecutive video posts).

• Stage 2 (Heavy Neural Ranker): Runs deep learning models (DLRM) predicting user interaction probabilities:

  Score = w_1 P(like) + w_2 P(comment) + w_3 P(share) + w_4 P(watch_time)

• Strict 40ms Circuit Breaker SLA: The ML ranker must return within 40ms. If inference times out, the gateway immediately falls back to pure reverse-chronological ordering, ensuring zero UI stutter.

COMPONENT 7: FEED SERVING GATEWAY & POST HYDRATION

The user-facing aggregation and payload assembly gateway:

• Batched Post Hydration (MGET): Once the final 20 post IDs are ranked, the gateway issues a single batched MGET against a distributed Memcached cluster to retrieve full post payloads (author name, avatar URL, caption, media URL, like count).
• 99.4% Cache Hit Ratio: Because post payloads are immutable once created, Memcached absorbs almost all read traffic. Cache misses fall back to PostgreSQL read replicas and asynchronously repopulate Memcached.
• Keyset Cursor Pagination: Encodes the bottom post's timestamp and ID into an opaque cursor (cursor=eyJ0cyI6MTY...), preventing duplicate posts during infinite scrolling.

ByteByteGo-Style Step-by-Step Hybrid Timeline Assembly Flow

To ace the Staff interview, present the end-to-end timeline journey showing both the write fan-out path and the read assembly path:

STEP-BY-STEP HYBRID PUSH/PULL FEED ASSEMBLY FLOW

flowchart TD
    subgraph W["[CREATOR PUBLISH PATH (WRITE)]"]
        U["User<br/>(1) POST /v1/posts"] --> PS["[Post Service]<br/>(2) Commits to DB & emits to Kafka: posts.raw"]
        PS --> SG["[Social Graph & Classifier]"]
        SG -->|"If Celebrity (#gt;=25k)"| CO["(3A) Writes 1 record to Celebrity Outbox [outbox:celebrity_id]"]
        SG -->|"If Normal (#lt;25k)"| FW["(3B) Enqueues push tasks<br/>[Async Fan-Out Workers]<br/>(4) Filters inactive via Bloom filter<br/>(5) Pipelined ZADD to timeline:user_id"]
        FW --> RT["[Redis In-Memory Timelines]"]
    end
    subgraph R["[VIEWER FEED RETRIEVAL PATH (READ) (#lt;80ms)]"]
        V["Viewer<br/>(6) GET /v1/feed"] --> FG["[Feed Serving Gateway]"]
        FG --> F7["(7) Fetches precomputed inbox (Redis)"]
        FG --> F8["(8) Fetches followed celebrity outboxes (Redis)"]
        F7 --> KH["[In-Memory K-Way Heap Aggregator]<br/>(9) Merges streams in O(N log K) time (#lt;3ms)"]
        F8 --> KH
        KH --> ML["[Two-Stage ML Ranking Engine]<br/>(10) Scores engagement (40ms circuit breaker)"]
        ML --> MC["[Memcached Post Hydration]<br/>(11) Batched MGET post payloads (99.4% hit)"]
        MC --> VR["Viewer receives rendered 20-post feed<br/>(Sub-80ms total p99 response)"]
    end

STEP 1: CLIENT POST INGRESS

The user uploads post metadata (caption, media S3 URL, location) to the Post Service via POST /v1/posts.

STEP 2: DATABASE COMMITMENT & KAFKA TOPIC EMISSION

The Post Service commits the record to the persistent database (CockroachDB / PostgreSQL) and synchronously emits a PostCreated event to Apache Kafka topic posts.raw before returning HTTP 201 Created to the client in < 25ms.

STEP 3: SOCIAL GRAPH EVALUATION & DUAL-PATH CLASSIFICATION

The Social Graph Service consumes the event and queries the author's follower count:

• (3A) Celebrity Path (≥ 25,000 followers): The post ID is appended once to the author's dedicated Celebrity Outbox in Redis (ZADD outbox:<celebrity_id> <timestamp> <post_id>). No follower inboxes are written.
• (3B) Normal Path (< 25,000 followers): Followers are sliced into chunks of 1,000 user IDs and enqueued as parallel fan-out tasks to Kafka topic fanout.push.tasks.

STEP 4: ACTIVE FOLLOWER SCREENING VIA BLOOM FILTER

Async fan-out workers consume batches from fanout.push.tasks. For each follower ID, the worker queries an in-memory Counting Bloom filter to verify whether the follower has opened the platform within the past 30 days, dropping dormant accounts to eliminate 65% of write amplification.

STEP 5: PIPELINED REDIS TIMELINE INSERTION

For active followers, workers issue pipelined Redis commands: ZADD timeline:<user_id> <timestamp> <post_id> followed by ZREMRANGEBYRANK timeline:<user_id> 0 -801, enforcing an 800-post cap per timeline in RAM.

STEP 6: VIEWER FEED REQUEST INGRESS

A viewer opens the mobile application, initiating GET /v1/feed?limit=20. The request hits the Feed Serving Gateway via Anycast Layer-7 routing.

STEP 7: PRECOMPUTED PERSONAL TIMELINE FETCH

The gateway queries the viewer's precomputed Redis Sorted Set: ZREVRANGEBYSCORE timeline:<viewer_id> +inf -inf LIMIT 0 50, returning the top 50 recent post IDs in under 1.5ms.

STEP 8: FOLLOWED CELEBRITY OUTBOX PARALLEL FETCH

Simultaneously, the gateway queries the Social Graph cache for celebrities followed by the viewer (typically ≤ 50) and issues non-blocking parallel ZREVRANGEBYSCORE queries against their outbox:<celebrity_id> sets.

STEP 9: IN-MEMORY K-WAY MIN-HEAP TIMELINE MERGE

The K-Way Heap Aggregator merges the viewer's precomputed timeline stream with the followed celebrity outbox streams using an in-memory Min-Heap. Merging K streams of size N completes in O(N log K) time (< 3ms).

STEP 10: TWO-STAGE MACHINE LEARNING RANKING

The candidate post IDs pass through the Two-Stage ML Ranking Engine: a light heuristic filter selects the top 50 candidates, followed by a heavy deep neural net (DLRM) scoring engagement probability. A strict 40ms circuit breaker falls back to pure reverse-chronological order if inference times out.

STEP 11: BATCHED MEMCACHED POST PAYLOAD HYDRATION

The gateway issues a single batched MGET against Memcached to fetch post bodies, media URLs, author avatars, and like counts for the winning 20 post IDs. The fully hydrated feed payload is returned to the viewer with an opaque keyset cursor in sub-80ms total p99 response time.