This lesson opens Chapter 10 (Part II, Production System Blueprints) and covers sections 1 to 3. It continues in Notification Fan-Out: Suppression, Rate Shaping, and Deep Dives.

Chapter 10: High-Throughput Notification & Fan-Out Engine

Production System Blueprint 7: Multi-Channel Routing, Intelligent Fatigue Suppression, and Priority-Shaped Egress

At a glance

1. The Scenario & Problem Framing

PROMPT:

"Design an enterprise-scale multi-channel notification and fan-out engine (like those powering Uber, Airbnb, or LinkedIn) serving 100,000,000 daily active users (DAU).

The system must ingest and deliver up to 1,000,000,000 notifications per day across Apple Push Notification service (APNs), Firebase Cloud Messaging (FCM), Email (Amazon SES), SMS (Twilio), and In-App WebSockets. Peak ingestion reaches 50,000 events/second.

The platform must guarantee sub-5-second delivery for critical transactional alerts (e.g. 2FA OTPs, fraud warnings, ride arrival notices), enforce dynamic quiet hours across global time zones, throttle user fatigue via sliding-window frequency caps, and adaptively shape outbound traffic to external third-party provider rate limits without dropping messages."

Sending an individual push notification or transactional email is trivial: any junior developer can write five lines of code to invoke Twilio's REST API or send an HTTP/2 payload to Apple's push gateway.

The monumental engineering challenge at 1,000,000,000 notifications per day is not message transmission; it is managing competing priorities across organizational boundaries while protecting users from cognitive exhaustion:

1. The Single-Queue Head-of-Line Starvation Disaster: A marketing team launches an unannounced Black Friday promotional campaign, blasting 40,000,000 push notifications into the system. In a naive, single-queue architecture, this massive bulk wave saturates worker pools. Simultaneously, a user attempts to log into their banking account and requests a Two-Factor Authentication (2FA) SMS OTP. The OTP lands behind 20,000,000 marketing messages. By the time the SMS arrives 35 minutes later, the login session has expired, and the user is locked out.
2. The "Notification Fatigue" Uninstallation Spiral: Product managers instinctively believe their specific feature deserves immediate user attention. Left unchecked, a single user receives 18 push notifications, 6 marketing emails, and 4 SMS alerts in a single day. The user does not adjust their notification settings; they permanently disable app notifications at the operating system level or uninstall the app entirely. Once push permissions are revoked, the user's lifetime value (LTV) collapses by over 60%.
3. The Downstream Rate-Limit Freeze: External delivery providers enforce unforgiving egress rate limits. Amazon SES caps outbound email at 1,500/second; Twilio shortcodes throttle SMS at 100/second; APNs drops connections if flooded with bursts exceeding certificate concurrency limits. Without egress token-bucket rate shapers, downstream providers return HTTP 429 (Too Many Requests), triggering uncoordinated client retry storms that crash internal dispatch worker fleets.
4. The Astronomical SMS Cost Explosion: SMS is the most reliable channel, but it is also an economic landmine. At $0.015 per message, blasting a single promotional campaign to 10,000,000 users via SMS costs $150,000 in minutes. Naive multi-channel fallback architectures that fall back to SMS whenever a push notification fails can bankrupt an early-stage company within a single quarter.

flowchart TB
  P["THE CORE PARADOX<br/>The system must ingest massive, spiky bulk marketing events at 50,000/sec while guaranteeing sub-second delivery for life-critical transactional alerts, yet the hardest architectural task is deciding what NOT to send."]
  G["Staff Goal: Decouple priority tiers into isolated Kafka topics, enforce multi-stage suppression (opt-outs, timezone quiet hours, sliding-window fatigue caps) in under 2ms in Redis, and shape external egress traffic via leaky-bucket rate shapers tuned to downstream provider contracts."]
  P --> G

The L5 vs L6 Contrast — What the Evaluator Actually Hears

The Senior Candidate (L5): Immediately draws an API Gateway, a single RabbitMQ or Kafka queue, a notification worker, and boxes for APNs, FCM, and Twilio. When asked how they handle a surge in marketing traffic, they say: "We will scale up the worker pods using Kubernetes HPA." When asked about user preferences, they describe a SQL table with a boolean opt_in column. They treat the system as a simple pipe that blindly forwards whatever upstream services produce.

The Staff Candidate (L6): Reframes the problem around delivery reliability contracts, multi-tier priority isolation, and user fatigue governance within the first 90 seconds: "A notification platform is not a transport pipe; it is an intelligent policy and rate-shaping gateway. At 1 Billion notifications/day, the architecture must enforce strict physical queue isolation: P0 Critical (OTPs, fraud) runs on dedicated unthrottled worker pools with zero-shared-infrastructure with P2 Marketing. Furthermore, suppression is a first-class subsystem: before any message touches a network dispatcher, it must clear a 5-stage filter evaluating legal opt-outs, timezone-aware Quiet Hours, sliding-window Redis fatigue buckets, and smart digest bundling. Finally, egress traffic is governed by leaky-bucket shapers matching external provider contracts, backed by staged retry budgets and a named DLQ ownership contract."

2. Step 1: Commit to Intent & Scale Math

Before designing database schemas or configuring channel dispatchers, an L6 Staff engineer establishes the core operational intent. Notification traffic divides into three strictly incompatible operational profiles:

The Three Incompatible Intents

1. Critical Transactional Delivery (P0 - 2FA OTP, Fraud, Ride Arrival):
   • The Constraint: Extreme temporal urgency. A 2FA code is worthless after 60 seconds.
   • The Architectural Commit: Dedicated queue topics, bypasses all quiet hours and fatigue rate limits, multi-channel synchronous fallback (Push -> SMS), zero batching.
2. Standard Transactional & Engagement (P1 - Order Receipts, Mentions, Chat):
   • The Constraint: Must deliver reliably, but user does not require sub-second latency.
   • The Architectural Commit: Respects user channel preferences, respects hard opt-outs, 60-second delivery SLA, smart digest aggregation during high-frequency bursts.
3. Scale Marketing & Promotions (P2 - Flash Sales, Re-engagement, Newsletters):
   • The Constraint: Massive volume (tens of millions of recipients), cost sensitivity, high risk of user fatigue.
   • The Architectural Commit: Strict quiet hours (10 PM to 8 AM local time), sliding-window frequency caps (max 1/day), trickle-drained over hours to prevent provider rate limits, zero SMS dispatch.

STAFF MOVE

Senior: "If a push notification fails to deliver, we will immediately retry by sending an SMS to ensure the user receives the message."

Staff: "Automatic SMS fallback for general notifications is a catastrophic anti-pattern. SMS costs $0.015 per message—over 150x more expensive than push. If an APNs service disruption causes 2,000,000 push notifications to fail, a naive SMS fallback will incur a $30,000 cloud bill in 10 minutes and wake thousands of users at 3 AM. SMS fallback must be strictly gated to P0 Critical security and payment events. For P1 engagement alerts, fallback routes silently to the user's In-App Notification Center."

Why: Demonstrating ruthless cost awareness and blast-radius containment distinguishes Staff engineers from theoretical architects.

Capacity Planning & Scale Math

At 100,000,000 Daily Active Users (DAU) and 1,000,000,000 notifications per day:

1. INGESTION THROUGHPUT & PEAK FAN-OUT SIZING

• Daily Ingestion Volume: 1,000,000,000 notifications / 86,400 seconds ≈ 11,574 events/second average.
• Peak Hour Traffic Factor: Marketing blasts, morning digest releases, and evening rush-hour activity generate a 4.5x peak burst multiplier:

Peak Ingestion Throughput = 11,574 * 4.5 ≈ 52,000 events/second

• Payload Size: Notification envelope (User ID: 16 bytes, Event Type: 16 bytes, Channel: 8 bytes, Locale: 8 bytes, Template Parameters: ~400 bytes, Metadata: ~64 bytes) ≈ 512 bytes average.

Inbound Bandwidth:

Peak Ingress Bandwidth = 52,000 events/sec * 512 bytes ≈ 26.6 Megabytes/second (213 Mbps)

2. CHANNEL DISTRIBUTION & EGRESS BREAKDOWN

Across 1 Billion daily notifications, historical enterprise delivery distribution splits across channels:

• Mobile Push (APNs / FCM): 70% of total volume -> 700,000,000 pushes/day (Peak: ~36,400 push/sec).
• In-App Notification Center: 100% of total volume (shadowed) -> 1,000,000,000 inbox rows/day.
• Transactional & Marketing Email: 15% of total volume -> 150,000,000 emails/day (Peak: ~7,800 emails/sec).
• Critical SMS (P0 Only): 2% of total volume -> 20,000,000 SMS/day (Peak: ~1,040 SMS/sec).
• Partner Webhooks: 5% of total volume -> 50,000,000 webhooks/day (Peak: ~2,600 calls/sec).

3. SUPPRESSION & RATE LIMITING STORAGE FOOTPRINT (REDIS)

To prevent fatigue, the system enforces a Sliding-Window Hourly Rate Limit per user per channel (e.g., max 3 push notifications per rolling 60 minutes):

• Active Daily Users: 100,000,000 users.
• Redis Key Layout: Sliding window counter stored as an in-memory integer with a 3600-second TTL:
  • Key: rl:notif:<user_id>:<channel>:<hour> (24 bytes).
  • Value: 8-byte integer counter.
  • Overhead per Redis key-value entry ≈ ~64 bytes.

Total Working Set RAM:

Active Working Set RAM = 100,000,000 users * 64 bytes ≈ 6.4 Gigabytes of RAM!

Even with multi-channel tracking (Push, Email, SMS), the entire global fatigue suppression working set requires under 20 Gigabytes of RAM, fitting comfortably on a modest Redis cluster!

4. APNS HTTP/2 CONNECTION SIZING

Apple Push Notification service (APNs) requires HTTP/2 with TLS persistent multiplexed connections. Apple guidelines mandate keeping connections open and multiplexing requests across concurrent streams (up to 500 streams per TCP connection).

At peak push throughput (36,400 pushes/sec):

Concurrent Connections Needed = 36,400 pushes/sec / 200 concurrent streams ≈ 182 persistent TCP sockets

With safety pooling across 10 dispatcher worker nodes, each worker maintains ~25 persistent HTTP/2 connections to api.push.apple.com, achieving line-rate push delivery with near-zero TCP handshake latency!

Key System Sizing Numbers Worth Memorizing

3. End-to-End System Architecture

The architecture decouples event ingestion, suppression policy evaluation, template personalization, and third-party channel dispatching:

Figure 10.1: End-to-End High-Throughput Notification & Fan-Out Architecture

The Architecture Components in Depth

flowchart TB
  subgraph NP["END-TO-END NOTIFICATION PLATFORM RUNTIME ARCHITECTURE"]
    UP["Upstream Services: Auth, Billing, Social, Marketing"]
    C1["COMPONENT 1: Priority Ingestion Gateway & Physical Topic Segregation (Kafka Broker Fleet)<br/>├─ notif.p0.critical (64 Partitions: Auth OTPs, Fraud Alerts, Security Password Resets)<br/>├─ notif.p1.transactional (128 Partitions: Order Confirmations, Delivery, Billing Receipts)<br/>└─ notif.p2.marketing (256 Partitions: Growth Blasts, Digests, Engagement Campaigns)"]
    C2["COMPONENT 2: Policy & Suppression Intelligence Engine (Go / Rust Stateless Workers)<br/>├─ Stage 1: Legal Opt-Out & Unsubscribe Registry (DynamoDB / Redis Bloom Filter)<br/>├─ Stage 2: Timezone-Aware Quiet Hours Evaluation (IANA String -> Redis ZSET Delay)<br/>├─ Stage 3: Sliding-Window Fatigue Throttling (Atomic Redis Lua Rate Shapers)<br/>├─ Stage 4: Smart Digest Bundling Buffer (5-Minute Redis Aggregation Lists)<br/>└─ Stage 5: Cryptographic P0 Priority Fast-Path Bypass"]
    C3["COMPONENT 3: Template Compilation, Localization & Personalization Engine<br/>├─ JIT Bytecode Template Renderer (Handlebars / Mustache with Locale Fallback Chains)<br/>├─ Dynamic Localization Dictionary (40+ Locales Cached in Process RAM, 60s TTL)<br/>└─ Deep-Link Cryptographic Signer & Dynamic A/B Variant Assignment Engine"]
    C4["COMPONENT 4: Multi-Channel Dispatch Gateways & Connection Multiplexers<br/>├─ Push Gateway: Long-lived HTTP/2 TCP multiplexed pools to APNs & FCM (JWT Auth)<br/>├─ Email Gateway: Distributed Leaky-Bucket Rate Shapers to Amazon SES / SendGrid<br/>├─ SMS Gateway: Pooled Shortcode REST Dispatchers to Twilio / Sinch (1,000 SMS/sec)<br/>└─ In-App Gateway: Distributed WebSocket Session Nodes & PostgreSQL Inbox Writers"]
    C5["COMPONENT 5: Tracking & Feedback Ledger<br/>• ClickHouse Analytics<br/>• APNs 410 Deactivation<br/>• Email Bounce Denylist"]
    C6["COMPONENT 6: Staged Retry & Dead-Letter Queue (DLQ)<br/>• notif.retry.30s (Jitter)<br/>• notif.retry.5m<br/>• notif.dlq (4h SRE SLA)"]
    C7["COMPONENT 7: In-App Notification Center<br/>• PostgreSQL Shards<br/>• Redis Badge Counters<br/>• Silent Push Sync"]
    UP -->|"gRPC / REST via Envoy Sidecars"| C1
    C1 -->|"Independent Consumer Worker Fleets"| C2
    C2 -->|"Suppression Passed"| C3
    C3 -->|"Compiled Payload Dispatch"| C4
    C4 -->|"Delivery Telemetry"| C5
    C4 -->|"HTTP 429 / 503 Failures"| C6
    C4 -->|"Inbox Feeds"| C7
  end

Component 1: Priority-Partitioned Event Ingestion Gateway & Ingress Cluster (Kafka / Pulsar)

• Responsibilities & Wire Protocol: Exposes high-throughput gRPC and HTTP/2 ingress endpoints fronted by Envoy sidecars. Ingests notification requests from upstream services (Authentication, Payment, Social Graph, Marketing Automation). Performs schema validation using Protocol Buffers and assigns each message a globally unique notification_id (Snowflake ID) and idempotency token (idempotency_key = sha256(user_id + event_type + timestamp_window)).
• Topic Partitioning & Physical Isolation: Routes incoming events into three physically isolated Kafka topics hosted on dedicated broker pools to guarantee zero resource contention:
  1. notif.p0.critical (64 partitions): Reserved exclusively for two-factor authentication (2FA) OTPs, password resets, and fraudulent transaction alerts. SLA: p99 < 1,500ms end-to-end delivery.
  2. notif.p1.transactional (128 partitions): Order confirmations, shipment tracking updates, flight change notices, and direct user mentions. SLA: p99 < 30 seconds.
  3. notif.p2.marketing (256 partitions): Promotional campaigns, weekly digests, engagement reminders, and bulk newsletters. SLA: p99 < 4 hours.
• Concurrency & Failure Semantics: Ingestion worker pods maintain non-blocking producer pools with in-memory ring buffers (librdkafka). In the event of a Kafka broker leader rebalance or network partition, P0 events failover to local disk spooling with immediate alerting, preventing upstream authentication services from experiencing backpressure.

Component 2: Policy & Suppression Intelligence Engine (The Core Brain)

• Responsibilities & Filtering Pipeline: A horizontally scalable fleet of stateless Go/Rust worker nodes that consumes raw events from the priority queues and acts as the gatekeeper before message formatting. Enforces a 5-stage evaluation pipeline:
  1. Legal Opt-Out & Regulatory Compliance: Checks global unsubscribe flags and channel-specific marketing consent (CAN-SPAM, GDPR, CCPA) against local Bloom filters backed by Redis and DynamoDB. Opted-out messages are immediately dropped.
  2. Timezone-Aware Quiet Hours: Maps the recipient's user ID to their stored IANA timezone string (America/New_York, Asia/Tokyo). If local time falls within 22:00–08:00 and priority is not P0, defers delivery by routing the message into a Redis ZSET delayed queue scheduled for 08:05 local time.
  3. Frequency Capping & Fatigue Throttling: Executes atomic Redis Lua scripts tracking sliding-window notification counts (e.g., maximum 3 push notifications per rolling 60 minutes). Excess messages are suppressed from mobile push and diverted to the silent in-app notification center.
  4. Smart Digest Bundling: Consolidates viral social events (e.g., 50 likes on a photo) into aggregated digests over a 5-minute sliding window.
  5. Cryptographic Priority Bypass: P0 Critical security and billing alerts contain verified internal cryptographic signatures that bypass stages 1–4 unconditionally.
• Concurrency & Memory Model: Stateless workers maintain local in-process LRU caches (60-second TTL) for tenant configuration rules and user preference snapshots, achieving suppression evaluation latencies of < 1.2ms p99.

Component 3: Template Compilation, Localization & Personalization Engine

• Responsibilities & Compilation Architecture: Compiles dynamic notification payloads by binding runtime event parameters to pre-compiled templates. Supports multi-channel output variations (plain text SMS, rich HTML email, structured JSON push payloads with action buttons, and in-app cards).
• Localization & Fallback Resolution: Dynamically resolves content into over 40 languages based on the recipient's preferred_locale. Implements deterministic fallback chains (e.g., pt_BR -> pt_PT -> en_US). Localization dictionaries are compiled into binary bytecode and cached in worker memory.
• Personalization & Cryptographic Link Wrapping: Injects dynamic user attributes (first name, cart items, account balances) and wraps all outgoing hyperlinks with tamper-proof HMAC-SHA256 tracking redirects to measure click-through rates (CTR) while defeating phishing attempts. Dynamically assigns A/B test variations using deterministic hash bucketing (hash(user_id + experiment_id) % 100).

Component 4: Multi-Channel Dispatch Gateways & Connection Multiplexers

• Push Gateway (APNs / FCM): Maintains persistent, long-lived HTTP/2 multiplexed TCP socket pools to Apple Push Notification service (api.push.apple.com) and Firebase Cloud Messaging (fcm.googleapis.com). Authenticates using JSON Web Tokens (JWT) signed with ES256 private keys (.p8), rotated every 50 minutes to eliminate TLS handshake overhead. Handles binary wire serialization, priority flags (apns-priority: 10), and background sync headers (content-available: 1).
• Email Gateway (Amazon SES / SendGrid): Dispatches transactional and bulk marketing emails through distributed connection pools. Enforces account-level egress rate shaping using a distributed Redis Leaky Bucket (e.g., 1,500 emails/second maximum). Cryptographically signs all outbound MIME messages with 2048-bit DKIM private keys and verifies SPF and DMARC alignment before socket transmission.
• SMS & WhatsApp Gateway (Twilio / Sinch / Infobip): Connects to cloud telephony REST APIs. Manages a dynamic pool of registered shortcodes and alphanumeric sender IDs. Enforces strict carrier-mandated throughput limits (100 SMS/sec per shortcode) and monitors carrier billing burn rates in real time.
• In-App Gateway: Pushes live alerts to currently active client sessions over persistent WebSocket connection clusters (backed by Redis Pub/Sub) and appends unread records to the persistent In-App Notification Center.

Component 5: Delivery Tracking, Ingestion Receipt & Feedback Ledger (ClickHouse + PostgreSQL)

• Storage Topology & Ledger Semantics: Maintains a dual-tier storage model:
  • PostgreSQL (OLTP): Manages mutable operational entities including registered device tokens, subscription preferences, unread notification counts, and customer webhook configurations. Partitioned by hash(user_id).
  • ClickHouse (OLAP): Ingests 100M+ immutable delivery lifecycle telemetry events per day (QUEUED -> DISPATCHED -> DELIVERED -> OPENED -> CLICKED -> BOUNCED -> FAILED). Provides sub-second analytical aggregations for campaign performance dashboards and automated deliverability monitoring.
• APNs 410 & Hard Bounce Feedback Loop: Asynchronously ingests delivery receipts from push and email providers. When APNs returns HTTP 410 DeviceTokenNotForTopic (app uninstalled), or SES reports a Permanent Hard Bounce, the feedback worker immediately publishes a revocation event that marks the device token or email address as inactive in PostgreSQL, preventing future wasted API calls and protecting sender reputation.

Component 6: Staged Retry & Dead-Letter Queue (DLQ) Pipeline

• Classification & Failure Isolation: Evaluates outbound dispatch failures and categorizes them into two distinct classes:
  • Transient Failures (Retryable): Downstream HTTP 429 (Rate Limited), HTTP 503 (Service Unavailable), DNS timeouts, and connection resets.
  • Fatal Failures (Non-Retryable): Malformed payloads, invalid phone numbers (E.164 syntax errors), revoked authentication tokens, and unsubscribed recipients. Fatal errors bypass retry queues and write directly to ClickHouse failure logs.
• Staged Exponential Jitter Backoff: Retryable events are pushed into staged Kafka retry topics with progressive backoff delays: notif.retry.30s (attempt 1), notif.retry.5m (attempt 2), and notif.retry.30m (attempt 3). Delayed consumers apply full randomized jitter (sleep = rand(0, base_backoff)) to prevent thundering-herd spikes on downstream carrier APIs.
• DLQ Quarantine & SRE Runbook: Messages exhausting all retry attempts enter notif.dlq. DLQ records preserve full error diagnostics, original payload, provider response codes, and hop history. If DLQ arrival rate exceeds 50 messages/minute, a PagerDuty alert pages the platform team with a mandatory 4-hour remediation SLA.

Component 7: In-App Notification Center & Read-State Synchronization Store

• Persistence & Cursor Pagination: A sharded PostgreSQL database cluster hosting the historical notification inbox for mobile and web clients. Partitioned by hash(user_id) with composite B-Tree indexes on (user_id, created_at DESC). Exposes cursor-based pagination APIs (WHERE user_id = :uid AND created_at < :cursor ORDER BY created_at DESC LIMIT 20) guaranteeing < 10ms p95 inbox retrieval under heavy concurrent access.
• Ephemeral Read Badges & Cross-Device Silent Push Sync: Maintains an in-memory Redis key per user: notif:badge:<user_id>. When a user reads a notification on the web application, a background worker decrements the Redis badge counter and emits a Silent Background Push (apns-push-type: background, content-available: 1, badge: new_count) via APNs to the user's registered iPhone and iPad. The mobile operating system wakes silently in the background, updates the app icon badge count, and dismisses the lock-screen banner without alerting the user.

Step-by-Step Priority Event Ingestion & Multi-Channel Dispatch Flow

The following sequence details the complete lifecycle of a notification from upstream trigger invocation to cross-channel edge delivery and cross-device read synchronization:

sequenceDiagram
  participant UA as Upstream App
  participant KI as Kafka Ingress
  participant PE as Policy Engine
  participant TE as Template Eng
  participant DG as Dispatch Gateway
  participant AS as APNs / SES
  participant CH as ClickHouse
  participant CD as Client Device
  UA->>KI: (1) Trigger Event (P0/P1/P2)
  KI->>PE: (2) Consume Event Batch
  PE->>TE: (3) Screen Opt-out, DND, Fatigue Limit
  TE->>DG: (4) JIT Compile & Personalize
  DG->>AS: (5) HTTP/2 Multiplex Send
  AS->>CD: (6) Push Delivery
  AS-->>DG: (7) Provider Receipt (200)
  DG->>CH: (8) Log State: DISPATCHED
  CD->>CH: (9) User Tap / Open Event
  Note over CH,CD: Log State: OPENED & CLICKED

STEP 1: UPSTREAM EVENT EMISSION & IDEMPOTENT INGESTION

An upstream microservice (e.g., Identity Service issuing a 2FA login verification code) issues a gRPC request to the Notification Ingress Gateway. The gateway validates the request schema, stamps an idempotency_key, and appends the event into the physically segregated Kafka topic notif.p0.critical. If the call is a duplicate submission within an active 5-minute sliding deduplication window, Redis returns the existing notification_id and drops the duplicate write.

STEP 2: CONSUMER FETCH & PRIORITY PARTITION PROCESSING

Dedicated P0 worker fleets consuming exclusively from notif.p0.critical pull message batches using non-blocking I/O. Because P0 workers share zero threads, memory pools, or network interfaces with the bulk marketing consumer fleet, processing begins within 5 milliseconds of topic append, completely insulated from any active multi-million-message promotional campaigns running on notif.p2.marketing.

STEP 3: POLICY SCREENING & MULTI-STAGE SUPPRESSION CHECK

The message passes through the 5-stage policy engine:

1. Legal Registry Check: Verifies the recipient has not opted out of critical operational notices (security notifications are legally mandatory and bypass commercial opt-outs).
2. Quiet Hours & Timezone Evaluation: Checks the user's localized IANA timezone. Because the event is tagged priority: P0, the Quiet Hours deferral check is bypassed; the message proceeds immediately without waiting for morning hours.
3. Fatigue Throttling Check: Evaluates the user's rolling push frequency in Redis. Security alerts bypass frequency caps unconditionally.

STEP 4: JUST-IN-TIME TEMPLATE COMPILATION & PERSONALIZATION

The template engine fetches the compiled Handlebars bytecode for auth.otp.verification from local process memory. It binds runtime variables (verification_code: 849201, expires_in: 5m), resolves the recipient's locale (es_MX), and formats the message into Spanish: "Su código de verificación es 849201. Expira en 5 minutos." Dynamic deep links are signed with HMAC-SHA256 tracking signatures.

STEP 5: MULTIPLEXED SOCKET DISPATCH OVER HTTP/2

The Push Dispatcher routes the compiled payload to the active Apple Push Notification service (APNs) connection pool. Rather than establishing a new TCP/TLS connection, the worker multiplexes the request as an HTTP/2 frame over an existing persistent TLS socket to api.push.apple.com. It sets the APNs headers: :method = POST, :path = /3/device/<device_token>, apns-priority = 10 (immediate wake), and apns-push-type = alert.

STEP 6: DOWNSTREAM CARRIER DELIVERY & EDGE TRANSMISSION

APNs routes the encrypted payload to Apple's push gateway infrastructure, which transmits the message over the persistent APNs connection maintained by the user's iOS device. The mobile operating system wakes the application notification daemon, displays the high-priority lock-screen banner, and triggers a physical haptic vibration.

STEP 7: ASYNCHRONOUS PROVIDER RECEIPT & STATUS EVALUATION

APNs returns an HTTP/2 response frame containing HTTP 200 OK with an apns-id tracking UUID. If APNs had instead returned HTTP 410 Unregistered (indicating the user deleted the application), the gateway would publish an immediate token invalidation event to device-token.invalidated to permanently deactivate the token in PostgreSQL.

STEP 8: TELEMETRY LEDGER INGESTION & STATE TRANSITION

The dispatch worker emits an immutable delivery record to the telemetry ingestion pipeline. ClickHouse appends the event to the notification_events table with state DISPATCHED, recording latency metrics, provider response times, and channel metadata. Simultaneously, if the notification requires in-app presence, the record is written to the sharded PostgreSQL user_notifications_inbox table.

STEP 9: USER ENGAGEMENT & CROSS-DEVICE SILENT BADGE SYNC

When the user taps the lock-screen banner, the mobile application fires an engagement webhook to the tracking gateway, which records OPENED and CLICKED in ClickHouse. When the user marks the notification as read on their desktop browser, a backend worker updates the database record to is_read = TRUE, decrements the Redis key notif:badge:<user_id>, and emits a Silent Background Push (content-available: 1) to the user's other registered devices, automatically clearing the unread red badge icon across their entire device ecosystem.