Design a Notification System

A notification system delivers messages to users through channels such as mobile push, SMS, and email, triggered by events like a shipped order, a new follower, or a security alert. The system must deliver reliably, respect user preferences, avoid duplicates, and absorb bursts (for example a marketing campaign to millions of users) without delaying time-critical messages.

Requirements

Functional

• Channels: mobile push (iOS via APNs, Android via FCM), SMS, email, and optionally in-app.
• Triggered by internal services (transactional) and by scheduled campaigns (bulk).
• Users can opt in and out per channel and per notification category.
• Templates with localization and variable substitution.
• Scheduling and delayed delivery.
• Delivery tracking: sent, delivered, failed, opened (where the channel supports it).

Non-functional

• Soft real-time: transactional messages (such as one-time passcodes) should arrive within seconds.
• Reliable: no lost notifications; at-least-once delivery with deduplication so users do not receive repeats.
• Scalable: tens of millions of notifications per day, with bursts.
• Isolation: a bulk campaign must not delay a password-reset message.
• Extensible: adding a channel or third-party provider should not change producers.

Estimation

Assume 10 million users and an average of 5 notifications per user per day across all channels.

• Volume: 10e6 x 5 = 50 million notifications/day.
• Average rate: 50e6 / 86,400 ≈ 580/s.
• Peak: assume 10x average during campaigns, about 5,800/s.
• Storage of notification log: 50e6/day x 500 bytes = 25 GB/day; retain 90 days = 2.25 TB.
• Device tokens: 10e6 users x 2 devices x 200 bytes = 4 GB. Fits in a single sharded key-value store and caches well.
• Worker concurrency: if a provider call takes about 200 ms (assumed), the peak of 5,800/s needs about 5,800 x 0.2 ≈ 1,160 concurrent sends, which asynchronous workers on a few machines handle.
• Provider limits matter more than internal capacity. Email and SMS providers throttle per account, so per-provider rate limiting is a design element.
• Bandwidth: outbound payloads are small (push about 1 to 4 KB, email 20 to 100 KB with content). Assume a mix of 50 percent push, 40 percent email, 10 percent SMS; peak 5,800/s x (0.5 x 2 KB + 0.4 x 50 KB) is about 120 MB/s outbound at worst, dominated by email, so keep large assets as links or CDN images rather than attachments.
• Queue volume: at peak, 5,800 messages/s x 1 KB = about 6 MB/s into queues, and a 10 million user campaign is 10 GB of queue data if enqueued at once. This is trivial for a log-based broker, which is why enqueueing is never the bottleneck.
• Campaign drain time: 10 million emails at a provider limit of 500/s (an assumed account limit) takes 20,000 s, about 5.5 hours; at 5,000/s it takes 33 minutes.

What the numbers imply: the internal system is small (a few dozen worker instances), so the design effort goes to isolation and provider throughput. A bulk send can occupy the provider for hours, so it must not share a queue or provider quota with transactional traffic; reserve a slice of the quota (for example 20 percent) for high priority, or use a separate provider account or IP pool.

API design

Producers call an internal API; user-facing endpoints manage preferences.

POST /v1/notifications
{
  "user_id": "u123",
  "type": "order_shipped",
  "channels": ["push", "email"],      // optional; defaults from type policy
  "template_id": "order_shipped_v3",
  "params": {"order_id": "o9", "eta": "2026-10-02"},
  "priority": "high",                  // high | normal | bulk
  "idempotency_key": "order-o9-shipped",
  "send_at": null
}
-> 202 Accepted { "notification_id": "n456" }

GET  /v1/users/{id}/preferences
PUT  /v1/users/{id}/preferences   {"marketing": {"email": false, "push": true}}
POST /v1/devices                  {"user_id": "u123", "platform": "ios", "token": "..."}
GET  /v1/notifications/{id}       -> status per channel

The 202 Accepted response reflects asynchronous processing. The idempotency_key lets a producer retry safely (see delivery semantics and idempotency).

Data model

Relational storage suits low-volume configuration data. The log is append-heavy and read by user and time, matching a wide-column model (see wide-column stores).

High-level design

flowchart LR
    Svc["Producer services"] --> API["Notification API"]
    Camp["Campaign scheduler"] --> API
    API --> Val["Validate, dedup, preference check"]
    Val --> Tmpl["Template renderer"]
    Tmpl --> QP["Push queues: high, normal, bulk"]
    Tmpl --> QE["Email queues: high, normal, bulk"]
    Tmpl --> QS["SMS queues: high, normal, bulk"]
    QP --> PushW["Push workers"]
    QE --> EmailW["Email workers"]
    QS --> SmsW["SMS workers"]
    PushW --> APNs["APNs / FCM"]
    EmailW --> EP["Email provider"]
    SmsW --> SP["SMS provider"]
    PushW --> Log[("Notification log")]
    EmailW --> Log
    SmsW --> Log
    APNs -.->|error response: invalid token| PushW
    EP -.->|webhooks: bounce, delivered| Hook["Status webhook receiver"]
    Hook --> Log

Component responsibilities:

• Notification API: authenticates producers, validates input, returns 202.
• Validation stage: deduplicates by idempotency key, loads the user's preferences, drops notifications for opted-out channels, applies quiet hours by delaying rather than dropping.
• Template renderer: merges template and parameters into channel-specific payloads, choosing the locale.
• Queues: decouple ingestion from delivery (message queues). One set of queues per channel, and separate queues per priority, so bulk traffic cannot starve transactional traffic.
• Channel workers: pull messages, call the third-party provider, record the result, and retry on failure.
• Status webhook receiver: providers report delivery, bounces, and unsubscribes asynchronously.

Message flow:

sequenceDiagram
    participant P as Producer
    participant A as Notification API
    participant Q as Channel queue
    participant W as Worker
    participant X as Provider
    participant L as Log
    P->>A: POST /notifications (idempotency_key)
    A->>A: dedup, check preferences, render
    A->>Q: enqueue
    A-->>P: 202 Accepted
    Q->>W: deliver message
    W->>X: send
    alt success
        X-->>W: accepted
        W->>L: status = sent
        W->>Q: ack
    else transient failure
        X-->>W: error or timeout
        W->>Q: nack, retry later with backoff
    end

Deep dive

Reliability and deduplication

Queues give at-least-once delivery: a worker that crashes after calling the provider but before acknowledging the message causes a redelivery and a possible duplicate send. Exactly-once delivery to an external provider is not achievable in general, so the design reduces duplicates:

• Before sending, the worker checks or atomically sets a (notification_id, channel) record in the log; a message already marked sent is acknowledged and skipped.
• The record is written before the provider call as in_progress, so a redelivery after a crash can see it. An in_progress record left by a crashed worker is ambiguous, because the provider call may or may not have happened. For critical categories the worker sends again (a possible duplicate is preferable to a lost alert). For non-critical categories it skips the send. Where a provider supports idempotency keys or client-supplied message IDs, the notification ID is passed through.
• Remaining duplicates are accepted for non-critical categories.

Alternative to the send-time record: a unique-key insert in the log store (INSERT ... IF NOT EXISTS) as the lock, which is why the dedup record is in a store with conditional writes. The dedup key TTL should exceed the longest producer retry window and queue redelivery horizon: 24 hours is typical for transactional events (the key store is then about 50e6 x 100 B = 5 GB). Queue semantics: at-least-once with a visibility timeout longer than the p99 provider call (for example 30 to 60 seconds against a 200 ms typical call), so a slow send is not redelivered while still running.

Retries use exponential backoff with jitter and a maximum attempt count. Concrete values: delays of about 1 s, 5 s, 30 s, 2 min, 10 min, with 5 attempts for push and email and 3 for SMS (cost); OTPs expire in minutes, so they get 2 quick retries and then failover to another provider or channel, not a long backoff. Delivering a stale OTP is worse than failing. Messages that exhaust retries move to a dead letter queue for inspection and replay (see backpressure, retries and DLQs). Permanent errors (an invalid token, a hard email bounce) are not retried; the address or token is marked invalid instead.

Priority isolation and rate limiting

A campaign to 10 million users can fill a shared queue for hours. Separate queues and worker pools per priority prevent that: one-time passcodes go to a high-priority queue with reserved worker capacity, and campaigns go to the bulk queue whose consumers are throttled.

Numbers for isolation: size the high-priority pool for peak transactional load with headroom (for example 1,000 msg/s at 200 ms needs about 200 concurrent sends, so provision 2x), and never let bulk consumers borrow it. Alternative: a single queue with priority ordering; it is simpler but a strict priority queue starves bulk under load and offers no isolation from a poison batch, so separate queues are chosen. Weighted sharing (for example 70/20/10 for high/normal/bulk) suits systems where bulk must still progress. Shard key for queues and the log is user_id, which keeps a user's messages ordered and lets per-user rate limits be enforced locally; a single very large campaign is handled by splitting the audience by user id range across partitions, not by key skew.

Rate limiting applies at three points:

• Per provider: stay under the provider's account limits (rate limiting algorithms); excess messages wait in the queue.
• Per user: cap notifications per user per time window, to avoid flooding (for example collapsing ten "new like" events into one digest).
• Per campaign: spread bulk sends over a window.

Provider failure and failover

Third-party providers fail or degrade. Keep at least two providers per channel where cost allows. A circuit breaker on each provider trips on error-rate thresholds, and workers route to the secondary provider while it is open. Different providers have different formats, so a provider adapter interface isolates them from the rest of the system.

Typical thresholds: open the breaker when the error rate exceeds about 50 percent over a 30 to 60 second window (with a minimum request count), stay open 30 seconds, then send a few probe requests (half-open). Failover trade-off: the secondary has different sender reputation and rate limits, and may duplicate a message the primary accepted but timed out on, so failover is safe only with dedup keys and for categories where a duplicate is acceptable. Keep the secondary warmed with a small share of traffic (for email, IP reputation needs warming).

Push token lifecycle

APNs and FCM return errors when a token is invalid (app uninstalled, token rotated). Workers process these responses and delete the token so future sends skip it. Clients re-register tokens on each app launch to refresh last_seen.

Scheduling

Delayed and recurring notifications are stored with send_at in a scheduler store indexed by time. A poller (or a delay queue) selects due entries and enqueues them. Quiet hours use the user's timezone from preferences to compute the next allowed time.

Failure modes

At 10x scale (500 million per day, 58,000/s peak): the log store needs about 250 GB/day so retention shrinks (for example 30 days hot, then archive to object storage); provider quotas require multiple accounts or providers and a quota-aware router; per-user preference reads of 58,000/s need a replicated cache with hit rate above 99 percent; and a broadcast to all users must use provider fan-out primitives (topics or batch APIs) and staged, region-by-region sends to avoid a self-inflicted load spike on your own backend, since millions of users open the app within seconds of a push.

Bottlenecks and scaling

• Ingestion: the API is stateless and scales horizontally behind a load balancer.
• Queues: partition by user or notification ID; add partitions and consumers to scale throughput (see Kafka internals if a log-based broker is used).
• Preference lookups: every notification needs one; cache in memory or Redis with short TTL and invalidate on update.
• Log writes: shard by user_id in the wide-column store.
• Worker scaling: autoscale on queue depth and message age, not CPU.
• Hot spikes: breaking-news pushes to all users send millions of messages at once. Fan out by splitting the audience into batches processed by many workers, and use provider topic or batch APIs where available.

Trade-offs and follow-up questions

• Push a notification now versus batching into a digest: fewer interruptions against added latency.
• Store rendered content versus rendering at send time: storing simplifies audit and retry, rendering late reflects the latest template.
• How to guarantee a critical alert (fraud) is not lost? Persist the notification durably before returning 202, use a high-priority path, and fall back to a second channel if the first is not confirmed delivered.
• How to support unsubscribe compliance? Honor provider unsubscribe events and store them in preferences; check preferences at send time, not only at enqueue time.
• What if a user gets 50 notifications in one minute? Per-user rate limit plus collapsing by a key (for example type + object), sending a digest after a short window (for example 1 to 5 minutes); exempt security categories.
• What if the same event is published twice by two services? Dedup on a business key (order-o9-shipped) rather than a random request id, so both callers produce one notification.
• What if the preference is changed while a message is queued? Re-check preferences at send time (a cached lookup with TTL of about 60 seconds bounds the staleness).
• What if a push is sent to a user with three devices? Fan out per device token; collapse keys (APNs collapse id, FCM collapse key) let the platform replace older undelivered notices.
• What if a campaign targets users across time zones? Schedule per-timezone send times from user preference and spread the sends to stay within provider limits.
• What if delivery must be exactly once? Not achievable to an external provider; at-least-once plus dedup makes duplicates rare (well below one percent in normal operation is a reasonable goal, not a guarantee).
• How to measure delivery? Combine provider callbacks with client-side open or receipt events.