Notification Service

Assume 300M daily active users, an average of 20 logical notifications per active user per day, and a peak multiplier of 20x during campaigns, incidents, and regional spikes. That gives 6B logical notifications per day before retries.

Assume channel mix is 65 percent push, 20 percent email, 5 percent SMS, and 10 percent in-app. Average logical notification metadata and rendered payload references are 1 KB, average delivery status event is 400 bytes, and each logical notification produces 2.5 status events on average including accepted, sent, delivered or failed, and engagement events. Retries add 15 percent more provider attempts on average, but more during incidents.

• Logical notifications: 300M daily active users times 20 notifications per day is 6B per day.
• Average ingestion QPS: 6B divided by 86,400 seconds is about 69,444 per second, rounded to 69,500.
• Peak ingestion QPS: a 20x burst multiplier gives about 1.39M notifications per second.
• Channel split: 65 percent push is 3.9B per day, 20 percent email is 1.2B per day, 5 percent SMS is 300M per day, and 10 percent in-app is 600M per day.
• Retry overhead: if transient failures add 15 percent more attempts, 6B logical notifications become 6.9B provider attempts per day.
• Queue ingress: 6B jobs times 1 KB is 6 TB raw per day. With replication, headers, and retention, provision several times that amount.
• Status events: 6B logical notifications times 2.5 events each is 15B status events per day.
• Tracking storage: 15B events times 400 bytes is 6 TB raw per day. Compression helps, but indexes and replication add overhead.
• Preferences: 1B users times 2 KB is about 2 TB raw. Hot preference cache size depends on active users and campaign fanout patterns.

• notification_requests.tenant_id, idempotency_key must be unique for the idempotency window.
• notification_requests.created_at and tenant_id, created_at support audit and operational queries.
• delivery_attempts.notification_id supports status lookup for one notification.
• delivery_attempts.provider_message_id supports provider receipt correlation.
• delivery_attempts.status, scheduled_at supports retry workers finding due attempts if the queue needs repair.
• user_preferences.user_id is the primary read key; updated_at drives cache invalidation.
• templates.template_id, tenant_id, locale, channel, version supports deterministic template lookup.

One notification request can create multiple delivery attempts, one per selected channel and retry. User preferences influence whether attempts are created or suppressed. Templates are immutable by version so in-flight notifications can be audited against the exact content that was rendered.

At high scale, keep recent notification status in a wide-column or key-value store keyed by notification_id and partitioned by tenant. Store delivery attempts in an append-friendly table keyed by notification_id plus attempt_id, with a secondary lookup by provider_message_id for receipts.

Queues hold the operational source of work. Analytics events belong in Kafka, Kinesis, Pub/Sub, or another log, then flow to OLAP storage such as ClickHouse, Druid, BigQuery, or Snowflake. User preferences can live in DynamoDB, Cassandra, Spanner, or a sharded relational store with Redis in front for hot reads during fanout.

A clean design accepts a logical notification intent first: who should receive something, why, how urgent it is, which template to use, and which channels are allowed. This lets product services stay independent of APNs payload rules, email headers, SMS character limits, and WebSocket session state.

Dispatch is where channel-specific behavior belongs. Push payloads need device tokens, platform-specific fields, collapse keys, and token cleanup. Email needs subject, HTML and text bodies, bounce handling, complaint handling, and unsubscribe headers. SMS needs strict length control, country rules, sender IDs, and cost controls. In-app needs mailbox persistence or online WebSocket delivery.

The boundary is important in interviews. If callers send provider-specific payloads directly, the platform cannot centralize preference enforcement, localization, fallback, analytics, or provider failover. If the core API hides every channel detail, it still needs extensible channel options for legitimate provider features.

Campaigns are dangerous because one API call can create tens or hundreds of millions of recipient jobs. Do not enqueue all recipients in one transaction or scan the audience from a single worker. Store campaign metadata, split the audience into shards, and let fanout workers checkpoint progress per shard.

Fanout should be paced by campaign budget, tenant quota, queue lag, provider quota, and time windows. A campaign for 100M users may need to run over hours to avoid provider throttling and user fatigue. Transactional queues must have reserved worker capacity so password reset or order updates are not stuck behind marketing sends.

Deduplication is also a fanout concern. The same user may appear in multiple segments or devices. Use campaign_id plus user_id plus notification_type as a dedupe key when product semantics require one message per user.

Preference enforcement must happen close to dispatch because user choices can change after ingestion but before delivery. The worker should read the current preference version and evaluate channel opt-in, topic subscription, quiet hours, timezone, age restrictions, country rules, and GDPR deletion flags.

Quiet hours are not always terminal suppression. A marketing push can be rescheduled until the next allowed local window, while a security alert or delivery driver arrival may bypass quiet hours based on policy. The request should include priority and notification type so the policy engine can make that distinction.

Unsubscribe and consent rules need special care. Email marketing must honor unsubscribe headers and suppression lists. SMS often needs explicit opt-in and STOP handling. GDPR deletion should remove or anonymize personal data from preferences, device tokens, and historical analytics according to retention policy.

Exactly-once delivery to users is not realistic. Clients retry API calls, queues redeliver messages, workers crash after provider calls, provider responses time out, and devices may show or drop notifications unpredictably. The practical goal is at-least-once internal processing with idempotent side effects and strong duplicate suppression.

On ingestion, use tenant_id plus idempotency_key to return the same notification_id for retried requests. On dispatch, use notification_id plus channel plus recipient plus attempt semantics to avoid sending the same user-visible message twice. If a provider call times out after the provider accepted it, the adapter should correlate provider_message_id when available and apply cautious retry policy.

Dedupe windows differ by use case. Password reset may allow a short resend window, order shipped should dedupe per order state transition, and daily digest should dedupe per day. The design should make dedupe keys explicit rather than guessing from rendered content.

There are multiple rate limits: per-user to prevent spam, per-tenant to enforce fairness, per-campaign to smooth fanout, and per-provider to stay within APNs, FCM, email, or SMS quotas. Token buckets in Redis work well for fast decisions, backed by configuration stored durably.

Retries should be based on error class. Retry timeouts, 429, and 5xx with exponential backoff and jitter. Do not retry invalid tokens, unsubscribed recipients, malformed payloads, or permanent bounces. Put attempts that exceed retry limits into a dead-letter queue with enough context for replay after fixes.

Failover is not always safe. Email can often fail over from SES to SendGrid if sender reputation, DKIM, and unsubscribe state are aligned. SMS failover may change sender identity or cost. Push failover is limited because APNs and FCM are platform-specific. A good answer describes channel-specific failover constraints.

A notification has multiple states. Accepted means the platform durably received the request. Sent means a provider accepted an API call. Delivered usually means the provider or device reported delivery, which is not available for every channel. Opened and clicked are engagement events and may be sampled or delayed.

Track events append-only, then compute latest status and aggregates asynchronously. This supports audit and analytics without turning the dispatch path into a transactional analytics workflow. Correlate events using notification_id, attempt_id, provider_message_id, user_id, campaign_id, channel, provider, and template version.

Analytics must tolerate duplicates and late arrivals because providers may retry webhooks. Use receipt IDs or provider message IDs for dedupe, watermark late events, and surface freshness in dashboards. For privacy, avoid storing raw IPs or full message bodies in analytics unless necessary.

Alternative one is direct provider integration from each product service. It is simple initially but leads to inconsistent templates, retries, compliance, and analytics.

Alternative two is a managed vendor such as Braze, Iterable, Customer.io, or Firebase for most messaging needs. This reduces engineering effort but may limit customization, data control, cost optimization, and deep product integration.

Alternative three is a pure event-bus model where product events are consumed by notification rules. This is powerful for automation, but it still needs the same dispatch, preference, dedupe, and provider layers underneath.

Do not build a full notification platform for a small product with one low-volume channel and no compliance complexity. A direct provider integration or managed notification tool is usually better until the organization needs shared templates, preference enforcement, fanout, multi-channel delivery, and operational analytics.