Notification Fan-Out: Suppression, Rate Shaping, and Deep Dives

This lesson continues Chapter 10 and covers sections 4 to 6. It follows Notification Fan-Out: Framing, Scale Math, and Architecture and continues in Notification Fan-Out: Interview Calibration, Opinions, and Runbooks.

4. Multi-Stage Suppression, Quiet Hours & Production Schemas

The hallmark of a Staff-level notification system is deciding what NOT to send:

Figure 10.2: Multi-Stage Notification Suppression & Fatigue Defense Pipeline

The 5-Stage Suppression Pipeline

flowchart TB
  IN["Incoming Notification Event"]
  S1["STAGE 1: Legal Opt-Out & Unsubscribe<br/>• Checks global opt-out flag and channel-specific subscription settings.<br/>• If unsubscribed -> DROP immediately. (Zero marketing to opted-out users)."]
  S2["STAGE 2: Timezone-Aware Quiet Hours (Do Not Disturb)<br/>• Converts UTC epoch to user local time via IANA timezone string.<br/>• If local time in [10:00 PM, 08:00 AM] -> DEFER or DROP.<br/>• If Deferrable -> Push to Redis ZSET delayed queue for 8:05 AM morning drain."]
  S3["STAGE 3: Per-User Sliding-Window Fatigue Rate Limiter<br/>• Evaluates Redis token bucket: max 3 pushes / hour, max 1 promo email / day.<br/>• If limit breached -> SUPPRESS alert or DOWNGRADE to silent In-App Inbox."]
  S4["STAGE 4: Smart Digest Aggregation & Event Bundling<br/>• Buffers social events in 5-minute aggregation window.<br/>• Collapses 12 #quot;like#quot; events into: #quot;Alice and 11 others liked your post#quot;."]
  S5["STAGE 5: The Priority Emergency Override Gate<br/>• If Priority == P0_CRITICAL (2FA OTP, Fraud Alert, Ride Arrival):<br/>• OVERRIDES STAGES 2, 3, AND 4! Dispatches immediately to user's device!"]
  IN --> S1
  S1 -->|"Passed"| S2
  S2 -->|"Passed"| S3
  S3 -->|"Passed"| S4
  S4 -->|"Passed"| S5

TIMEZONE-AWARE QUIET HOURS CALCULATION:

To prevent waking a user in London at 3:00 AM with a marketing blast triggered by a marketing manager in San Francisco at 7:00 PM:

1. The user profile stores their IANA Timezone String (e.g., Europe/London, Asia/Tokyo), updated automatically on mobile app launch.

2. The policy engine converts NOW() to the recipient's local time:

   user_local_time = datetime.now(pytz.timezone(user.timezone))
   if user.quiet_hours_enabled and (user_local_time.hour >= 22 or user_local_time.hour < 8):
       return Action.DEFER_TO_MORNING

3. Deferred notifications are pushed into a Redis Sorted Set (ZSET) scored by release epoch timestamp (tomorrow_08_05_am). A background scheduler pulls and delivers them as a consolidated morning digest.

Production Relational Schemas & Invariant DDL (PostgreSQL)

-- 1. User Notification Preferences & Channel Configuration
CREATE TABLE user_notification_preferences (
    user_id UUID PRIMARY KEY,
    push_enabled BOOLEAN NOT NULL DEFAULT TRUE,
    email_enabled BOOLEAN NOT NULL DEFAULT TRUE,
    sms_enabled BOOLEAN NOT NULL DEFAULT TRUE,
    quiet_hours_enabled BOOLEAN NOT NULL DEFAULT TRUE,
    quiet_hours_start_time TIME NOT NULL DEFAULT '22:00:00',
    quiet_hours_end_time TIME NOT NULL DEFAULT '08:00:00',
    user_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC', -- IANA format: 'America/New_York'
    marketing_opt_out BOOLEAN NOT NULL DEFAULT FALSE,
    max_pushes_per_hour SMALLINT NOT NULL DEFAULT 3,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- 2. Registered Mobile Device Tokens (Push Targets)
CREATE TABLE user_device_tokens (
    device_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL,
    device_platform VARCHAR(16) NOT NULL, -- 'IOS', 'ANDROID', 'WEB'
    token TEXT NOT NULL UNIQUE,
    app_version VARCHAR(32) NOT NULL,
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    last_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- 3. Persistent In-App Notification Center (The Bell Inbox)
CREATE TABLE user_notifications_inbox (
    notification_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL,
    category VARCHAR(32) NOT NULL, -- 'SECURITY', 'TRANSACTIONAL', 'SOCIAL', 'PROMO'
    title VARCHAR(256) NOT NULL,
    body TEXT NOT NULL,
    action_url TEXT,
    is_read BOOLEAN NOT NULL DEFAULT FALSE,
    read_at TIMESTAMP WITH TIME ZONE,
    expires_at TIMESTAMP WITH TIME ZONE DEFAULT (NOW() + INTERVAL '30 days'),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
) PARTITION BY HASH (user_id);

-- Create 16 hash partitions for horizontal database scalability
CREATE TABLE user_notifications_inbox_0 PARTITION OF user_notifications_inbox
    FOR VALUES WITH (MODULUS 16, REMAINDER 0);
-- (... partitions 1 through 15)

-- Production Performance Partial Indexes
CREATE INDEX idx_inbox_unread_user ON user_notifications_inbox (user_id, created_at DESC)
    WHERE is_read = FALSE;

CREATE INDEX idx_active_devices ON user_device_tokens (user_id, platform)
    WHERE is_active = TRUE;

5. Priority Partitioning & Downstream Provider Rate Shaping

The most critical operational failure in large notification systems is downstream provider choking:

Figure 10.3: Priority-Based Queue Partitioning & Provider Rate Shaping

The Multi-Priority Queue Topology

To guarantee that high-volume bulk marketing campaigns never introduce latency into transactional paths, the architecture enforces physical queue isolation:

WHY DYNAMIC PRIORITY IN-MEMORY QUEUES FAIL AT SCALE

Junior engineers often attempt to implement priority using a single Redis Priority Queue or RabbitMQ priority argument. At 50,000 events/second, evaluating priority across millions of in-flight messages consumes massive CPU memory and triggers severe sorting latency. Physical queue separation into distinct Kafka topics guarantees O(1) constant-time enqueue and dequeue performance with absolute zero cross-priority head-of-line blocking.

Egress Leaky-Bucket Rate Shapers

Every third-party delivery provider has hard capacity ceilings. Sending traffic faster than these ceilings does not result in faster delivery; it results in HTTP 429 Too Many Requests, dropped connections, and degraded provider sender reputation.

SIZING PROVIDER RATE SHAPERS:

1. Apple Push Notification service (APNs):

   • Protocol: HTTP/2 over TLS with JSON Web Tokens (JWT).
   • Strategy: Workers maintain 50 persistent HTTP/2 connections. Each connection multiplexes up to 500 concurrent request streams. Outbound dispatchers shape traffic to a maximum of 25,000 pushes/second fleet-wide, smoothing bursts into a sustained line-rate flow.

2. Amazon Simple Email Service (SES):

   • Protocol: HTTPS API / SMTP.
   • Strategy: SES enforces a strict account sending quota (e.g., 1,500 emails/second). The email dispatch fleet runs a Distributed Leaky Bucket in Redis:

   -- Atomic Leaky Bucket Check in Redis
   local current = redis.call('INCR', 'egress:ses:rate')
   if current == 1 then
       redis.call('EXPIRE', 'egress:ses:rate', 1)
   end
   if current > 1500 then
       return 0 -- Reject / Defer to queue
   end
   return 1 -- Permit sending

   If the bucket is full, email dispatch workers throttle consumption from Kafka notif.p2.marketing, holding messages in the queue rather than triggering SES 429 drops.

3. Twilio SMS Gateway:

   • Protocol: REST API.
   • Strategy: Telephony carriers throttle individual shortcodes to 100 SMS/second. To support peak bursts, the system binds a pool of 10 shortcodes, distributing messages via round-robin to achieve 1,000 SMS/second.

6. Deep Dives

Deep Dive 1: APNs HTTP/2 Multiplexing, Token Churn & Feedback Service (410 Gone)

Apple Push Notification service (APNs) enforces strict communication semantics that differ fundamentally from standard REST APIs.

THE ARCHITECTURE OF HIGH-THROUGHPUT APNS DISPATCH:

1. HTTP/2 Multiplexing: APNs prohibits opening a new TCP connection per notification. Doing so triggers IP rate-limiting and handshake bans. Dispatch workers maintain a long-lived pool of TCP sockets over HTTP/2, streaming requests asynchronously with header :method = POST and path /3/device/<device_token>.

2. JWT Provider Authentication: Rather than old-style mutual TLS certificates that expire annually, workers use Token-Based Connection (.p8 private keys). The worker generates an ES256 signed JSON Web Token (JWT) cached for 50 minutes, eliminating cryptographic handshake overhead.

3. The 410 Unregistered Feedback Loop (The Critical Optimization):

   • When a user uninstalls your mobile app, APNs invalidates their device token.
   • On the next push attempt, APNs returns HTTP 410 Unregistered with JSON {"reason": "BadDeviceToken"}.
   • The Hazard: If your system continues sending pushes to dead tokens, Apple flags your account for spamming and throttles your entire app's delivery pipeline!
   • The Staff Solution: The push worker intercepts HTTP 410 responses and publishes an invalidation event to device-token.invalidated. A consumer worker immediately executes:

   UPDATE user_device_tokens SET is_active = FALSE WHERE device_token = :token;

   This immediately cleans the database, slashing outbound push volume by 15% within weeks of major app releases.

Deep Dive 2: Dynamic Channel Fallback Chains (Push -> SMS -> Email)

In enterprise workflows, critical messages must be delivered even if the user's primary device is offline or unreachable.

THE FALLBACK STATE MACHINE:

flowchart TB
  E["DYNAMIC CHANNEL FALLBACK PIPELINE<br/>Event: 2FA Login Code Requested -> Target: User 102"]
  S1["Step 1: Primary Dispatch -> Mobile Push (APNs / FCM)<br/>• Cost: $0.00001. Latency: 200ms.<br/>• Start 15-Second Delivery Acknowledgment Timer in Redis."]
  S2["Step 2: Verification Check"]
  A["Case A: User's smartphone receives push and client sends ACK webhook -><br/>Timer deleted. Workflow completed successfully! Total cost: $0."]
  B["Case B: 15-Second Timer Expires (Device is offline / in airplane mode) ->"]
  S3["Step 3: Secondary Fallback -> Telephony SMS (Twilio)<br/>• Cost: $0.015. Latency: 3-5s.<br/>• Dispatches SMS directly to user's registered phone number.<br/>• Start 45-Second Delivery Timer."]
  S4["Step 4: Tertiary Fallback -> Transactional Email (Amazon SES)<br/>• Dispatched only if SMS carrier returns permanent delivery failure."]
  E --> S1 --> S2
  S2 --> A
  S2 --> B
  B --> S3 --> S4

This multi-stage fallback guarantees 99.999% delivery reachability while minimizing expensive SMS carrier billing for 95% of connected smartphone users.

Deep Dive 3: Smart Batch Digest Aggregation ("Alice and 11 Others Liked Your Post")

In social networks and collaborative tools, viral activity triggers catastrophic notification storms. If an influencer with 10,000,000 followers tweets, sending 10,000,000 individual push notifications as comments roll in overwhelms mobile devices and irritates users.

THE 5-MINUTE SLIDING AGGREGATION BUFFER:

1. When a social event (e.g., user.liked_photo) occurs, the policy engine does not emit a push immediately.
2. It appends the event to a Redis List keyed by recipient and entity: digest:<user_id>:<photo_id>.
3. It sets a 5-Minute Aggregation Timer using Redis key expiry or a scheduled delayed task.
4. Subsequent likes within that 5-minute window simply increment the counter: RPUSH digest:102:photo_99 "user_55".
5. When the 5-minute timer fires:
   • If count == 1: Format as "Bob liked your photo."
   • If count == 2: Format as "Bob and Alice liked your photo."
   • If count > 2: Format as "Bob, Alice, and 10 others liked your photo."
6. Result: Consolidates 100 individual push alerts into a single elegant notification, slashing push traffic by over 85% on hot entities!

Deep Dive 4: Per-User Sliding-Window Fatigue Throttling in Redis

Notification fatigue is the number-one driver of application uninstalls. A robust platform must enforce strict Frequency Capping across rolling time windows.

THE REDIS SLIDING WINDOW LOG ALGORITHM:

To enforce "Maximum 3 push notifications per rolling 60-minute window":

-- KEYS[1]: user fatigue key (e.g., fatigue:push:user_102)
-- ARGV[1]: current epoch timestamp in milliseconds
-- ARGV[2]: 60-minute window in milliseconds (3,600,000)
-- ARGV[3]: max allowed notifications (3)
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local clear_before = now - window

-- 1. Purge events older than 60 minutes
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', clear_before)

-- 2. Count active events in rolling window
local count = redis.call('ZCARD', KEYS[1])
if count < limit then
    -- 3. Record current event and refresh TTL
    redis.call('ZADD', KEYS[1], now, now)
    redis.call('EXPIRE', KEYS[1], 3600)
    return 1 -- PERMIT
else
    return 0 -- SUPPRESS
end

Executed atomically in Redis in under 0.3 milliseconds.

If suppressed, the notification is not lost: it is routed silently to the user's In-App Notification Center without making the user's physical phone vibrate.

Deep Dive 5: Deliverability Reputation Defense for Email (DKIM, SPF, Bounce Circuit Breakers)

Sending 150,000,000 emails per day requires managing Internet Service Provider (ISP) Sender Reputation with Google Mail, Microsoft Outlook, and Yahoo. If your hard bounce rate exceeds 5.0% or spam complaints exceed 0.1%, Gmail automatically routes 100% of your company's emails (including password resets) directly to the Spam folder.

THE THREE DEFENSIVE SUBSYSTEMS:

1. Cryptographic Identity Verification: Every outgoing email is signed on the fly with DKIM (DomainKeys Identified Mail) 2048-bit RSA keys, matching DNS TXT records for SPF (Sender Policy Framework) and DMARC alignment.
2. Asynchronous Bounce Handling: When an email bounces, Amazon SES publishes an SNS notification to ses-feedback. If the bounce is classified as Hard Bounce (e.g., mailbox does not exist), the user's email is immediately added to an in-memory Global Suppression Denylist. Future emails to that address are dropped at the API gateway before touching SES.
3. The Automated Blast Circuit Breaker: During marketing campaigns, real-time analytics monitors bounce velocity. If the bounce rate for a specific campaign exceeds 3.0% within the first 10,000 sends, the circuit breaker trips, instantly pausing the campaign and paging the marketing operations team to investigate list hygiene.

Deep Dive 6: Dead-Letter Queue & Staged Exponential Jitter Backoff

When downstream delivery channels experience outages (e.g., Twilio experiencing carrier routing degradation), naive retry storms amplify the outage.

THE STAGED RETRY TOPOLOGY:

flowchart TB
  T["STAGED EXPONENTIAL JITTER RETRY PIPELINE<br/>Dispatch Worker encounters Transient HTTP 503 / Timeout:"]
  A1["Attempt 1 Fails -> Publish to Kafka: #96;notif.retry.30s#96;<br/>• Delayed consumer waits 30s + Full Jitter (rand(0, 15s))."]
  A2["Attempt 2 Fails -> Publish to Kafka: #96;notif.retry.5m#96;<br/>• Delayed consumer waits 5m + Full Jitter (rand(0, 60s))."]
  A3["Attempt 3 Fails -> Publish to Kafka: #96;notif.retry.30m#96;<br/>• Delayed consumer waits 30m."]
  A4["Attempt 4 Fails (Retry Budget Exhausted) -> Quarantine to #96;notif.dlq#96;<br/>• DLQ Record contains: full payload, error response, provider, and timestamps.<br/>• PagerDuty alert fires if #96;dlq.arrival.rate > 50 msg/min#96;.<br/>• Consuming team owns root-cause resolution within 4-hour SLA."]
  T --> A1 --> A2 --> A3 --> A4

Deep Dive 7: Multi-Device Notification Sync & Ephemeral Read Badges

Modern users own multiple devices: an iPhone, an iPad, a MacBook, and a browser session. If a user reads a notification on their laptop, the unread red badge icon on their iPhone must update immediately, rather than remaining stuck at "1" until tapped.

THE SYNC PROTOCOL:

1. The In-App Inbox Ledger: The database stores is_read = FALSE per notification.

2. The Read Action: When the user clicks the notification on their laptop:

   UPDATE user_notifications_inbox SET is_read = TRUE, read_at = NOW() WHERE notification_id = :id AND user_id = :uid;

3. Real-Time Badge Decrement: A Redis counter maintains the unread count: DECR notif:badge:<user_id>.

4. Silent Push Synchronization (Content-Available): The notification engine dispatches a Silent Push Notification (APNs header: apns-push-type: background, content-available: 1, badge: new_count) to all other registered devices for that user. The mobile app wakes in the background for 100ms, clears the local alert from the iOS Notification Center, updates the home screen app icon badge, and goes back to sleep without making a sound!

Deep Dive 8: Webhook Delivery Infrastructure & HMAC-SHA256 Signatures

Enterprise platforms (such as Stripe, GitHub, Shopify, or Twilio) must deliver reliable, authenticated HTTP callbacks (webhooks) to third-party customer endpoints upon critical business events (payment authorizations, code pushes, order fulfillment). Delivering webhooks at scale is notoriously fraught: external customer servers frequently experience catastrophic outages, introduce 10-second request latency, or return arbitrary HTTP status codes.

1. CRYPTOGRAPHIC PAYLOAD AUTHENTICATION & ANTI-REPLAY DEFENSE

Every outgoing HTTP POST request is signed using HMAC-SHA256 to allow recipient servers to verify payload authenticity and tamper-resistance:

The Wire Signature Header:

POST /api/webhooks/orders HTTP/1.1
Host: api.customer-merchant.com
User-Agent: Enterprise-Webhook-Engine/2.4
Content-Type: application/json
X-Webhook-ID: wh_evt_9a8b7c6d5e
X-Signature-SHA256: t=1718000450,v1=9b7c8a4192d...

Signature Computation: The signature is calculated across the concatenation of the epoch timestamp and the verbatim raw JSON request body using the customer's shared secret key (\(K_{\text{secret}}\)):

\[ \text{Signature} = \text{HMAC-SHA256}(K_{\text{secret}}, \, t \,\|\, "." \,\|\, \text{raw\_body}) \]

Verification & Anti-Replay Contract: The recipient calculates the HMAC over the received payload and compares it to v1 using constant-time byte comparison (crypto.timingSafeEqual) to eliminate timing attack vulnerabilities. The recipient validates that \(|\text{current\_time} - t| \le 300\text{ seconds}\); any webhook received outside this 5-minute clock drift window is rejected as a malicious replay attack.

2. HEAD-OF-LINE BLOCKING DEFENSE & PER-CUSTOMER CONCURRENCY ISOLATION

A single customer whose backend server is stalling on 10-second database locks must never degrade webhook delivery for thousands of other tenants:

• The Shared Fleet Vulnerability: If a shared worker fleet pulls all outgoing webhook tasks from a single queue, 5,000 requests destined for a stalled customer server will saturate all HTTP client worker threads, exhausting Linux ephemeral sockets and causing system-wide latency spikes.
• Per-Tenant Concurrency Caps: Outgoing webhook requests are partitioned by destination_host. The dispatch layer executes an atomic check against Redis:

-- Atomic check and reservation of per-destination in-flight limit
local active = redis.call('INCR', KEYS[1])
if active == 1 then redis.call('EXPIRE', KEYS[1], 30) end
if active > 5 then
    redis.call('DECR', KEYS[1])
    return 0 -- Defer dispatch: tenant concurrency saturated
end
return 1 -- Proceed with HTTP dispatch

Each customer endpoint is allocated a maximum of 5 concurrent in-flight HTTP requests. When an endpoint hits its concurrency ceiling, subsequent webhooks for that domain are held in partitioned delayed queues, ensuring 100% capacity remains available for healthy third-party destinations.

3. BACKPRESSURE TIMEOUTS, STAGED RETRY & CIRCUIT BREAKERS

Customer webhook servers are treated as inherently untrusted:

• Strict HTTP Client Timeouts: Egress HTTP clients configure a 1,500ms TCP connection timeout and a 3,000ms response read timeout. Long-running customer endpoints are severed immediately after 3.0 seconds to recycle TCP connections.
• Staged Exponential Jitter Retry: Transient failures (HTTP 408, 429, 500, 502, 503, 504, or socket timeouts) enter an exponential retry schedule: 15 seconds, 1 minute, 15 minutes, 1 hour, 6 hours, 24 hours (with full randomized jitter).
• Automated Degradation & Suspension Circuit Breaker: If an endpoint consecutively fails with HTTP 5xx or connection timeouts for 10 consecutive attempts, its state is marked DEGRADED, and dispatch polling rate is throttled. If failures persist continuously for 72 hours, the endpoint status is automatically transitioned to SUSPENDED. The system emits an administrative email alert to the developer, disabling future automated retries until the customer manually re-verifies their endpoint via the developer dashboard.