Three Notification Types
Modern notification systems must support three channels:
iOS Push Notification: the app server sends a notification payload (JSON containing a device token and the message) to Apple Push Notification Service (APNS). APNS delivers it to the iOS device. Each device has a unique device token registered when the user installs the app.
Android Push Notification: uses Firebase Cloud Messaging (FCM) instead of APNS. Same concept — server sends payload to FCM, FCM delivers to the Android device. FCM is Google's notification infrastructure and is available globally except China, which requires alternative providers like Jpush.
SMS: send via third-party SMS providers (Twilio, Nexmo). The notification server calls the SMS provider's API with the recipient's phone number and message body. Costs money per message — rate limiting and batching are important cost controls.
Email: use a commercial email service (SendGrid, Mailchimp) rather than hosting your own SMTP server. Commercial providers handle deliverability, spam filtering, and analytics.
Contact Info Gathering
To send notifications, the system needs device tokens (for push), phone numbers (for SMS), and email addresses (for email). These are collected at registration time.
When a user installs the app and creates an account, the client sends the device token (generated by the OS) along with the account creation request. The API server stores this in the user profile. A user can have multiple devices (phone + tablet + laptop), each with its own device token. On login from a new device, the new device token is registered.
Database design: a users table stores email and phone; a devices table stores device tokens with a foreign key to the user, allowing one user to have multiple device entries.
Initial Architecture and Its Problems
A naive implementation: a single notification server that receives notification requests, looks up user contact info from the database, and calls APNS/FCM/Twilio/SendGrid directly in the request handler.
Problems: (1) Single point of failure — if the notification server crashes, no notifications are sent. (2) Cannot scale horizontally — all logic in one server. (3) Performance bottleneck — making synchronous HTTP calls to external services (APNS, Twilio) blocks the thread. During peak hours (product launch, breaking news), the server is overwhelmed.
Improved Architecture with Message Queues
Solution: decouple notification creation from notification delivery using message queues.
Architecture: multiple notification servers (horizontally scaled, stateless) receive notification requests from upstream services and put notification events into type-specific message queues — one queue for iOS push, one for Android push, one for SMS, one for email. Separate worker pools consume from each queue and call the respective third-party services.
Benefits: notification servers and workers scale independently — if iOS push volume spikes, add more iOS workers without touching the email workers. If APNS is slow, the iOS queue backs up but does not affect the SMS pipeline. Notification servers can return immediately after enqueuing the job, without waiting for delivery confirmation. If a worker crashes, the message is requeued and another worker picks it up.
Reliability: Preventing Data Loss
Notifications must not be lost. If a third-party service (APNS, Twilio) is temporarily unavailable, the notification must survive and be retried later.
Persistence: before putting a notification into the queue, the notification server persists it to a notification log database with status "pending." Workers update the status to "sent" on success. A monitoring process scans for notifications stuck in "pending" status and retries them.
Retry mechanism: workers implement retry with exponential backoff and jitter. If APNS returns a failure, the worker waits 1 second, retries; if it fails again, waits 2 seconds, then 4, then 8. After a configurable maximum number of retries, the notification is moved to a dead-letter queue and an alert is raised for manual investigation.
Exactly-once delivery: message queues guarantee at-least-once delivery (a message may be delivered more than once in rare failure scenarios). Workers must be idempotent. Use a deduplication ID — when a notification is enqueued, assign it a unique ID. Workers check if this ID has been processed before (using a Redis set or a delivered table) and skip it if already processed.
User Settings and Rate Limiting
Users receive far too many notifications and quickly disable them entirely if the volume is overwhelming. Give users control via notification settings: per-channel opt-in/opt-out (push enabled, SMS disabled) and per-category control (marketing notifications off, transactional notifications on).
Before sending any notification, the notification server checks the user's preferences. If the user has opted out of push notifications, skip the iOS/Android queues entirely.
Rate limiting: cap the number of notifications a user can receive per time window — e.g., at most 10 marketing notifications per day. This requires a counter per (user, notification category, time window) stored in Redis. Rate limiting protects user experience and prevents marketing campaigns from flooding users.
Monitoring and Analytics
Key metrics to monitor: queue depth (notifications waiting to be processed — a growing queue means workers are falling behind), delivery success rate per channel, third-party service latency and error rates, and end-to-end notification latency (from trigger to device receipt).
Event tracking for analytics: capture open events (user opened the notification), click events (user tapped a link in the notification), and conversion events (user completed a purchase after clicking). This data drives A/B testing of notification copy, send-time optimization (what time of day does this user open notifications?), and notification personalization.