# Notification System Design
A notification system turns product events into user-visible messages across email, push, SMS, in-app inboxes, webhooks, and realtime badges. Sending one message is easy: it is a provider SDK call. The hard part is **preference-aware, deduplicated, rate-limited, observable delivery across unreliable providers**, at a fanout that must never block the product write path.
## What we are building
Transactional notifications: login alerts, mentions, payment receipts, reminders, workflow updates. Five channels: in-app inbox, mobile push (APNs and FCM), email, SMS, and outbound webhooks for tenant integrations. The system must:
- respect user preferences, quiet hours, tenant policy, locale, and legal unsubscribe rules (CAN-SPAM and GDPR for email, TCPA for SMS),
- guarantee that retries and event replays never produce a second visible message,
- record delivery state and provider responses so support can answer "why did I not get this",
- absorb a tenant-wide announcement to 2 million users without delaying a password reset,
- keep product services unaware of channels and providers.
Out of scope: marketing campaign authoring, audience segmentation, subject-line experiments. Those share the delivery tier below but have their own creation path with different rules (a campaign is scheduled and cancellable; a login alert is neither).
## Scale and numbers
Assume a consumer product with 50 million monthly and 10 million daily active users. The numbers that matter are intents per second, deliveries per day, the worst single fanout, and the SMS bill.
| Quantity | Assumption | Result |
|---|---|---|
| Notifiable events | 20 per DAU per day | 200 million events per day |
| Recipients per event | 1.5 on average | 300 million intents per day |
| Average intent rate | 300 million over 86,400 s | about 3,500 intents per s |
| Peak factor | 5x (morning wave plus one hot broadcast) | about 17,500 intents per s |
| Suppression | 60 percent removed by preferences, dedupe, digests | 120 million deliveries per day |
| Channel mix | 55 push, 30 in-app, 14 email, 1 SMS (percent) | 1.2 million SMS per day |
| SMS cost | 0.0075 dollars per message | about 9,000 dollars per day |
| Intent row | 300 bytes | 90 GB per day |
| Delivery row | 500 bytes | 60 GB per day |
| Raw state per year | 150 GB per day times 365 | about 55 TB, or 13.5 TB at 90 day hot retention |
Two derived facts shape the design. First, the average is not the problem: 3,500 intents per second is one modest queue. The problem is the single event with 2 million recipients, which at 17,500 per second would occupy the whole system for two minutes if it shared a lane with everything else. Second, one percent of the volume produces most of the provider cost, so SMS gets its own policy gate and its own budget counter.
## High-level design
```mermaid
flowchart LR
App[Product service] --> Outbox[Outbox relay]
Outbox --> Bus[Event bus]
Bus --> Router[Notification router]
Router --> Prefs[Preference store]
Router --> Intents[Intent store]
Intents --> Lanes[Per channel lanes]
Lanes --> Push[Push worker]
Lanes --> Email[Email worker]
Lanes --> Inbox[Inbox writer]
Push --> Providers[Providers]
Email --> Providers
Providers --> Receipts[Receipt consumer]
Receipts --> Intents
```
Product services never call a provider. They commit their own transaction plus an outbox row, and a relay publishes the fact to the event bus (see [[wiki/event-bus-for-product-events]]). The router is the only component that knows about users, preferences, and channels. It turns one event into zero or more intents, and each intent is a durable row before it is a queue message. Per-channel lanes feed workers that talk to providers, and a receipt consumer folds provider callbacks back into intent state. Creation (the router) and delivery (the workers) are separate services with separate on-call, because they fail differently.
## Data model
| Entity | Key | Partition key | Notes |
|---|---|---|---|
| notification_event | event_id | event_id hash | Immutable fact from the bus, kept 30 days for replay and audit. |
| notification_intent | intent_id, a hash of event_id, user_id, channel | user_id | The decision to notify. Unique on event_id, user_id, channel. |
| notification_delivery | delivery_id | user_id | One attempt through one provider: provider message id, error class, timestamps. |
| notification_preference | user_id, tenant_id | user_id | Channel opt-ins, quiet hours, digest cadence, unsubscribe tokens. |
| device_token | user_id, token | user_id | Push tokens with platform and last-seen, invalidated by provider feedback. |
| notification_template | template_key, version | replicated everywhere | Localized content, immutable per version. |
| inbox_item | user_id, created_at desc, event_id | user_id | The in-app list. Same uniqueness as intent. |
Everything hot is partitioned by user_id. The product reads per user (inbox, badge, preference screen, support lookups), and the uniqueness constraint that enforces at-most-once visible delivery lives on (event_id, user_id, channel), so both the read and the guard sit inside one partition. Events partition by their own id because nobody reads them per user; they are the replay source.
## Write path
```mermaid
sequenceDiagram
participant P as Product service
participant DB as Product database
participant B as Event bus
participant R as Router
participant Pr as Preference store
participant I as Intent store
participant Q as Channel lane
P->>DB: commit change plus outbox row
DB-->>B: relay publishes event
B->>R: event with event id
R->>R: skip if event id already processed
R->>Pr: load preferences and tenant policy
R->>I: insert intents with deterministic id
I-->>R: inserted or duplicate
R->>Q: enqueue intent id per channel
```
1. The product service commits its change and an outbox row in one transaction. If the notification were published before the commit, a rollback would leave a message about something that never happened; the outbox removes that class of ghost.
2. The relay publishes the event with event_id, occurred_at, actor, subject ids, and a trace id. The router's consumer group reads it and first checks a short-lived processed-event set; a replayed event stops here.
3. The router loads preferences and tenant policy for each candidate recipient, evaluates quiet hours in the recipient's timezone, and applies the per-user rate limit (a mention storm on one thread should collapse into one push, not 40).
4. It inserts one intent per surviving recipient and channel with the deterministic intent_id. A duplicate key error is not an error; it means a previous run got here first, and the router moves on.
5. Only after the intent commits does the router enqueue a message per channel carrying the intent_id. If the process dies between steps 4 and 5, a sweeper enqueues any intent older than 30 seconds still in the created state. The intent row is the truth; the queue is a hint.
The template version is chosen at step 4 and stored on the intent. Workers render with that version, so a template edit does not change a message already decided.
## Read path and sync
Three reads exist, and none of them scan.
The inbox is a keyset-paginated read of inbox_item by user_id, newest first. The badge count is a separate projection in Redis, a sorted set per user of unseen event ids, cleared when the user opens the inbox (see [[wiki/newly-unread-indicator]] for why this is an acknowledgement, not read state). Delivery status for support tooling reads intent plus deliveries by user_id and event_id.
Sync is about two things that change underneath a queued intent. Preferences: the worker re-reads preferences at send time and marks the intent suppressed if the user opted out between creation and send; 30 seconds of queue lag must not defeat an unsubscribe. Push tokens: APNs and FCM report unregistered tokens on send, and the worker marks the device_token invalid immediately so the next intent does not retry a dead token.
```mermaid
stateDiagram-v2
[*] --> Created
Created --> Queued: enqueued
Created --> Suppressed: preference or limit
Queued --> Sending: worker claims
Sending --> Sent: provider accepted
Sending --> Retrying: retryable error
Sending --> Unknown: timeout
Retrying --> Sending: backoff elapsed
Retrying --> Failed: attempts exhausted
Unknown --> Sent: reconciler confirms
Unknown --> Failed: cannot confirm
Sent --> Delivered: receipt
```
## Deep dive: at-least-once processing, at-most-once visible delivery
The contract in one sentence: every event is processed at least once internally, and each (event, user, channel) becomes visible at most once. The system needs two layers of idempotency because it has two boundaries where duplicates enter.
The internal boundary is the queue. Consumers see replays after crashes, rebalances, and deliberate backfills. The deterministic intent_id and its unique constraint absorb all of them: a second insert fails, and a second queue message finds an intent already past the created state and drops itself. The sketch this walkthrough grew from included template_version in that key; that is a mistake, because a template republish during a retry would mint a new key and send the message twice. The key is the decision, and the template version is an attribute of it.
The external boundary is the provider, and here the ambiguity from [[wiki/retries-timeouts-idempotency]] bites: after a timeout the send may or may not have happened. The pattern is claim-then-send.
```mermaid
sequenceDiagram
participant W as Channel worker
participant D as Delivery store
participant Pv as Provider
participant Rc as Reconciler
W->>D: insert delivery in sending state with client message id
W->>Pv: send with client message id
alt provider responds
Pv-->>W: accepted or error class
W->>D: record result and provider message id
else timeout
W->>D: mark unknown
Rc->>Pv: look up by client message id
Pv-->>Rc: found or not found
Rc->>D: sent, or retry per channel policy
end
```
The worker writes a delivery row in the sending state with a client-generated message id, then calls the provider with that id where the provider supports one (collapse ids on APNs and FCM, client references on most SMS gateways). If the call times out, the worker does not retry blindly. A reconciler queries the provider by message id where the API allows it, and where it does not, most email APIs among them, the channel's policy decides: push and in-app tolerate a rare duplicate, so retry; SMS costs money and annoys, so mark unknown and stop. **The dedupe guarantee is only as strong as the provider's idempotency support, and for channels without it you are choosing between a duplicate and a miss, per channel, explicitly.**
## Deep dive: fanout, priority lanes, and the 2 million recipient announcement
One tenant announcement to 2 million users is a fanout-on-write problem (see [[wiki/fanout-patterns]]): one event becomes 2 million intents and several million deliveries. Three mechanisms keep it from starving the password reset that arrives midway.
Lanes. Queues are per channel and per priority: transactional, engagement, bulk. A bulk lane has its own worker pool and its own provider concurrency cap, so 2 million pushes queue behind each other and nothing else. This is the task-queue side of [[wiki/task-queue-vs-event-stream]]: the event stream carries the fact once for many consumers; the lanes carry work items with priority and retry state that must complete once.
Chunked fanout. The router does not enumerate 2 million recipients in one process. It writes a fanout job with a recipient cursor, and fanout workers pull 5,000 recipients per step, insert intents, enqueue, and advance the cursor. A crash mid-step replays the step, and the intent key makes that harmless.
Batching to the provider (see [[wiki/batching]]). FCM accepts 500 tokens per multicast call; SES bulk templated email accepts 50 destinations. Workers flush a batch at 500 items or 200 ms, whichever comes first, so the bulk lane makes 10 provider calls per second per worker instead of 500. Per-item provider responses go on per-item delivery rows, because a batch that half-fails must retry only its failed half.
Per-user rate limits sit in the router, keyed by user_id plus channel with a per-channel weight (see [[wiki/rate-limiter-placement-and-keys]]). Ten mentions in one minute become ten inbox items, one push, and one line in the hourly email digest. The limiter here is a product rule about attention rather than an infrastructure guard, and it lives where preferences and identity live. Each provider client also carries a circuit breaker (see [[wiki/circuit-breakers-and-timeouts]]) so a throttling email provider parks the email lane instead of tying up every worker in timeouts.
## Scaling and failure modes
| Failure | Symptom | Mitigation |
|---|---|---|
| Event published before commit | Notification about a rolled-back action | Transactional outbox; never publish from the request handler |
| Queue replay after consumer crash | Same event processed twice | Processed-event set plus deterministic intent key with unique constraint |
| Provider timeout on send | Unknown outcome, duplicate or missed message | Claim-then-send with client message id; per-channel reconcile policy |
| Preference change while intent queued | Message sent after unsubscribe | Worker re-reads preferences at send time |
| Push tokens expire silently | Rising failure rate on one platform | Mark token invalid on provider feedback; prune tokens unseen for 60 days |
| Provider throttles one channel | Email lane backs up | Per-channel lanes; circuit breaker per provider; secondary provider for email and SMS |
| Bulk fanout floods shared lanes | Password reset delayed by minutes | Bulk lane with its own workers and cap; chunked fanout with cursor |
| Hot user partition, such as a shared support account receiving millions of intents | One shard saturated | Cap intents per user per hour; collapse the rest into a digest |
| Template rendering bottleneck | Bulk sends CPU-bound | Pre-render once per locale for bulk; render per user only when content varies |
## Senior tradeoffs
| Decision | Default | When to change |
|---|---|---|
| Event bus vs direct send | Event bus with outbox | Direct send only for tiny systems or admin tools where a ghost notification is harmless |
| Per-channel and per-priority lanes | Yes, from day one | A single queue is acceptable until one slow channel or one bulk send delays anything transactional |
| Template rendering time | Worker side, at send | Pre-render at intent time when legal or audit review needs the exact text that was sent |
| In-app inbox | Durable store, partitioned by user | Skip only if notifications are purely transient badges |
| Provider abstraction | Thin wrapper exposing provider error classes | A deep abstraction hides the codes you need to decide retry versus invalidate token |
| Duplicate vs miss after timeout | Retry for push and in-app; stop for SMS | Flip for SMS one-time codes, where a miss locks the user out and a duplicate is cheap |
| Dedupe key scope | Per channel | Cross-channel collapse (skip the email if the push was opened within 5 minutes) is router policy, not part of the key |
## Operational checks
- Queue age by lane, not depth: the oldest transactional message should be under 30 seconds; bulk may be minutes.
- Send success and failure rate by provider and error class, with a page when a provider's failure rate crosses 5 percent for 5 minutes.
- Duplicate-suppression count: a sudden rise means a replay or a bug upstream.
- Preference-filtered and rate-limited counts, so a config change that silences everyone is visible within minutes.
- Provider latency p99 and throttle responses per lane.
- Intents stuck in created or sending for over 2 minutes: the sweeper and reconciler backlog.
- Invalid-token rate per platform and app version.
- Unsubscribe and complaint rate per template; a template above 0.1 percent complaints gets pulled.
## Fundamentals used
- [[wiki/task-queue-vs-event-stream]]: the bus carries the fact once for many consumers; lanes carry work items that must complete once with retries and priority.
- [[wiki/fanout-patterns]]: one event to 2 million intents is fanout-on-write, and the worst fanout, not the average, sizes the lanes.
- [[wiki/retries-timeouts-idempotency]]: the unknown outcome after a provider timeout is the whole reason for claim-then-send and per-channel reconcile policy.
- [[wiki/rate-limiter-placement-and-keys]]: per-user, per-channel, weighted limits live in the router because that is where preferences and identity are.
- [[wiki/batching]]: provider multicast APIs turn 500 sends into one call, with per-item results so partial failure retries only the failed part.
- [[wiki/event-bus-for-product-events]]: the outbox and the event contract keep product services from knowing about channels.
- [[wiki/newly-unread-indicator]]: the badge is an acknowledgement projection, separate from inbox read state.
- [[wiki/circuit-breakers-and-timeouts]]: a throttling provider parks its lane instead of consuming every worker in timeouts.
## Pro tip
Keep notification creation separate from delivery attempts. Creation is the product decision and must be idempotent, auditable, and fast. Delivery is an unreliable integration workflow with retries, provider quirks, and money attached. Mixing them makes audits, retries, and user support painful, and it is the first thing to untangle in an existing system.