Building a Fault-Tolerant Communication Service for 150K+ Monthly Events

The Mission: An Unbroken Experience

In February 2024, we faced a challenge that many growing platforms encounter: how do you keep 150,000+ users informed across multiple channels without dropping a single notification?

Our platform was scaling rapidly, generating over 150K transaction events monthly. Users needed to be notified when critical events happened—transactions completed, documents were processed, status updates occurred. But our notification logic was scattered across different services, each implementing its own ad-hoc email sending, leading to inconsistent experiences and, worse, lost notifications.

We needed something better: a centralized communication service that could guarantee delivery across email, WhatsApp, SMS, and push notifications. This is the story of how we built it—and how our architecture evolved when we discovered the first version wasn't robust enough.

The First Attempt: Queue-Driven Architecture

The Core Design

Our initial architecture followed a classic microservices pattern. The design centered around three key components:

  1. Communication Service - A REST API that internal services could call
  2. RabbitMQ Task Queues - One queue per channel (email, WhatsApp, SMS, push)
  3. Workers - Channel-specific processors that interfaced with third-party delivery services

Here's how a notification would flow through the system:

code
1Internal Service → Communication Service → RabbitMQ Queue → Worker → Third-Party Service (Mailgun, etc.)

Initial Architecture - Version 0 Figure 1: Initial architecture with separate queues for each channel (WhatsApp, Push, SMS, Email). Services A, B, and C route through a load balancer to the Communication Service, which handles authentication, rate limiting, and REST APIs. Each channel has dedicated workers connecting to third-party services.

Building the Foundation

We started with four fundamental architectural decisions:

1. Basic Authentication

We created Basic Auth tokens to secure requests from internal services. Simple, stateless, and sufficient for our internal network.

typescript
1// Pseudo-code for auth middleware
2const authenticateRequest = (req, res, next) => {
3 const token = req.headers.authorization;
4 if (verifyBasicAuth(token)) next();
5 else res.status(401).send('Unauthorized');
6};

2. Rate Limiting: The Anti-Spam Shield

Users shouldn't receive duplicate notifications for the same event. We implemented rate limiting with a composite key strategy:

code
1Rate Limit Key = Template Name + User UUID + Channel

For example: transaction_complete_user123_email would prevent multiple transaction completion emails to user123 within a configured time window. We used Redis to track these keys with TTL-based expiration.

3. REST APIs for Every Channel

The communication service exposed clean REST endpoints:

code
1POST /api/notifications/email
2POST /api/notifications/whatsapp
3POST /api/notifications/sms
4POST /api/notifications/push
5POST /api/notifications/opt-out

4. Monitoring with NewRelic

From day one, we instrumented everything. NewRelic gave us visibility into:

  • API response times
  • Queue depths
  • Worker throughput
  • Error rates

The Worker Pattern: Manual Acknowledgment

Our workers implemented a sophisticated retry mechanism:

python
1# Worker pseudo-code
2def process_message(message):
3 try:
4 # Attempt delivery to third-party service
5 response = deliver_to_mailgun(message)
6
7 if response.success:
8 # SUCCESS: Acknowledge and remove from queue
9 acknowledge_message(message)
10 else:
11 # FAILURE: Reject for retry
12 reject_message_for_retry(message)
13 except Exception as e:
14 # EXCEPTION: Reject for retry
15 reject_message_for_retry(message)

The flow looked like this:

code
1Message from Queue
2 ↓
3Worker Attempts Delivery
4 ↓
5Success? → Acknowledge → Remove from Queue ✓
6 ↓
7Failure? → Retry (max 3 attempts)
8 ↓
9Max Retries Reached? → Dead Letter Queue (DLQ) → NewRelic Alert 🚨

Worker Flow - Version 0 Figure 2: Worker retry mechanism in the initial design. Messages flow from the task queue to workers, which attempt delivery. On success, the message is acknowledged and removed. On failure, the worker retries. After max retries (3), messages move to the Dead Letter Queue, triggering Slack alerts via NewRelic.

The message execution itself was straightforward:

Message Execution - Version 0 Figure 3: Simple message execution pattern—workers directly communicate with third-party services (like Mailgun for emails) without intermediate logging or state management.

The Database Schema

We designed a simple NotificationSettings table to respect user preferences:

| Column | Type | Description | |--------|------|-------------| | user_id | UUID | User identifier | | channel | ENUM | Email, WhatsApp, SMS, Push | | opted_out | BOOLEAN | User's opt-in status |

The Four-Phase Rollout

We planned an incremental rollout:

Milestone 1: Email Notifications

  • Initialize service repository
  • Integrate Mailgun
  • Set up RabbitMQ email queue
  • Build email workers
  • Store email templates

Milestone 2: WhatsApp

  • Add WhatsApp APIs
  • Channel-specific rate limiting (template_user_whatsapp)
  • WhatsApp queue and workers
  • Third-party WhatsApp integration

Milestone 3: Push Notifications

  • Push notification APIs
  • Push-specific templates and queue
  • Mobile push service integration

Milestone 4: SMS

  • SMS APIs and templates
  • SMS queue and workers
  • SMS provider integration

The Realization: What We Missed

Six days into development, on February 26th, we ran a failure mode analysis. The question that changed everything: What happens when a notification fails after all retries are exhausted?

Our Dead Letter Queue would catch these failures, and NewRelic would alert us. But then what? A human would need to manually investigate and re-queue the message. At 150K+ events per month, this wasn't sustainable.

We identified three critical gaps:

1. No Delivery Guarantee

If the DLQ filled up, we had no automatic recovery mechanism. Failed notifications would require manual intervention.

2. No Priority Differentiation

All notifications were equal in our queues. A transactional email (e.g., "Your payment succeeded") would wait behind marketing emails (e.g., "Check out our new features"). This wasn't acceptable.

3. No Audit Trail

We had no persistent record of notification attempts. Debugging delivery issues meant correlating logs across multiple systems.

The Evolution: Architecture Iteration 1

We went back to the drawing board and designed Iteration 1, a more resilient architecture with three major additions:

Final Architecture - Version 1 Figure 4: Evolved architecture with priority-based queues. Note the key additions: (1) Priority 1 and Priority 2 email queues (blue boxes) for transactional vs. marketing notifications, (2) Database logging for all notification attempts, (3) Scheduler service (top right) that picks failed notifications and re-queues them, (4) Webhook flow (orange) from third-party services back to update notification records.

The Safety Net: Database Logging

Every notification attempt now creates a record:

sql
1CREATE TABLE notification_logs (
2 id UUID PRIMARY KEY,
3 user_id UUID NOT NULL,
4 template_name VARCHAR(255) NOT NULL,
5 channel VARCHAR(50) NOT NULL,
6 status VARCHAR(50) NOT NULL, -- pending, delivered, failed
7 retry_count INT DEFAULT 0,
8 third_party_response JSONB,
9 created_at TIMESTAMP DEFAULT NOW(),
10 updated_at TIMESTAMP DEFAULT NOW()
11);

Workers now log every attempt:

python
1def process_message_v2(message):
2 # Create log entry
3 log_id = create_notification_log(message, status='pending')
4
5 try:
6 response = deliver_to_mailgun(message)
7
8 if response.success:
9 update_notification_log(log_id, status='delivered', response=response)
10 acknowledge_message(message)
11 else:
12 increment_retry_count(log_id)
13 update_notification_log(log_id, status='failed', response=response)
14 reject_message_for_retry(message)
15 except Exception as e:
16 increment_retry_count(log_id)
17 update_notification_log(log_id, status='failed', error=str(e))
18 reject_message_for_retry(message)

Prioritization: The Hierarchy of Importance

We split each channel into priority-based queues:

code
1Priority 1 (Highest): Transactional notifications
2 ↓ (processed first)
3Priority 2 (Medium): Transactional email queue
4 ↓
5Priority 3 (Lowest): Informational/Marketing queue

RabbitMQ's priority queue feature allowed us to configure this:

javascript
1// Queue setup with priority
2channel.assertQueue('email_notifications', {
3 durable: true,
4 maxPriority: 3
5});
6
7// Publishing with priority
8channel.publish('', 'email_notifications', Buffer.from(message), {
9 priority: isTransactional ? 3 : 1
10});

The Scheduler: Automated Recovery

The game-changer was our Scheduler Service—a separate service running cron jobs:

python
1# Scheduler cron job (runs every 15 minutes)
2def requeue_failed_notifications():
3 # Find notifications that failed but haven't exceeded global retry limit
4 failed_notifications = db.query("""
5 SELECT * FROM notification_logs
6 WHERE status IN ('failed', 'pending')
7 AND retry_count < 5
8 AND updated_at < NOW() - INTERVAL '15 minutes'
9 """)
10
11 for notification in failed_notifications:
12 # Re-queue with appropriate priority
13 priority = get_priority(notification.template_name)
14 publish_to_queue(
15 channel=notification.channel,
16 message=notification,
17 priority=priority
18 )
19
20 log.info(f"Re-queued notification {notification.id}")

The scheduler ran on a configured interval (we started with every 15 minutes) and would:

  1. Query the database for failed or pending notifications
  2. Check if they're within retry limits
  3. Re-publish them to the appropriate priority queue

Webhooks: Real-Time Status Updates

Third-party services like Mailgun send webhooks for delivery events:

javascript
1// Webhook endpoint in Communication Service
2app.post('/webhooks/mailgun', async (req, res) => {
3 const { event, message_id, recipient, timestamp } = req.body;
4
5 // Update notification log based on webhook
6 await db.query(`
7 UPDATE notification_logs
8 SET status = $1, third_party_response = $2, updated_at = NOW()
9 WHERE third_party_id = $3
10 `, [event, req.body, message_id]);
11
12 res.status(200).send('OK');
13});

This gave us real-time visibility into:

  • Delivered
  • Bounced
  • Opened
  • Clicked

Worker Flow - Version 1 Figure 5: Enhanced worker flow with database logging. Notice the critical addition: workers now create database records (Step 1) before sending HTTP requests to third-party services (Step 2). The task queue (blue) feeds workers (green), which handle the two-step execution process.

Message Execution - Version 1 Figure 6: Complete message execution lifecycle in Iteration 1. The green path shows successful database logging. The orange path shows the webhook feedback loop—third-party services send delivery status back to the Communication Service, which updates the database. The scheduler (top) continuously monitors the database for failed/pending notifications and re-queues them to priority queues (right, shown in blue).

The Complete Flow: Iteration 1

Here's how a notification now flows through our improved system:

code
11. Internal Service → POST /api/notifications/email
2 ↓
32. Communication Service:
4 - Validates request (Basic Auth)
5 - Checks rate limit (Redis: template_user_channel)
6 - Determines priority (transactional vs marketing)
7 - Creates notification_log record (status: pending)
8 ↓
93. Publishes to Priority Queue (RabbitMQ)
10 ↓
114. Worker:
12 - Receives message
13 - Attempts delivery to Mailgun
14 - Updates notification_log
15 - Success? Acknowledge
16 - Failure? Reject (will retry up to 3 times)
17 ↓
185a. SUCCESS PATH:
19 - Worker marks status: delivered
20 - Message removed from queue
21 - Wait for webhook confirmation
22
235b. FAILURE PATH (after 3 worker retries):
24 - Message → Dead Letter Queue
25 - Status remains: failed
26 - Scheduler picks up failed records
27 - Re-queues with priority (up to 5 total attempts)
28 ↓
296. Webhooks (async):
30 - Mailgun sends delivery status
31 - Updates notification_log with final status

The Evolutionary Roadmap

Our implementation plan remained largely the same, but Milestone 1 now included the fault-tolerance improvements:

Milestone 1 (Revised): Email with Fault Tolerance

  1. API Layer: Add email notification endpoint in paper-communication service
  2. Rate Limiting:
    • Set up Redis instance
    • Implement composite key: template_name:user_uuid:email
  3. Queue Infrastructure:
    • Initialize RabbitMQ instance
    • Create priority-based email queues (3 levels)
    • Configure Dead Letter Queue
    • Enable RabbitMQ Management UI monitoring
  4. Worker Implementation:
    • Create notification_logs table
    • Build email worker with logging
    • Store email templates with mapping
    • Integrate Mailgun API
  5. Webhook Listener: Add POST endpoint for Mailgun webhooks
  6. Scheduler Service:
    • Create cron job (15-minute interval)
    • Query failed/pending notifications
    • Re-queue logic with priority
  7. Testing: End-to-end email delivery validation

Milestones 2-4: Expanding Channels

The pattern established in Milestone 1 would repeat for each channel:

  • WhatsApp: Similar flow, different third-party integration
  • Push Notifications: Mobile-specific templates and delivery
  • SMS: SMS provider integration with same fault-tolerance

Technical Takeaways

Building this system taught us several lessons about distributed systems and reliability:

1. Design for Failure from Day One

Our first architecture assumed happy paths. The iteration came from asking: "What breaks, and how do we recover?" This mindset shift is crucial for production systems.

2. Observability is Non-Negotiable

Our layers of monitoring:

  • Application: NewRelic for service metrics
  • Infrastructure: RabbitMQ Management UI for queue health
  • Business Logic: Database logs for audit trails
  • External Systems: Webhooks for third-party confirmation

3. Queues Are Not Infallible

Dead Letter Queues catch failures, but you need a mechanism to drain them. Our scheduler was that mechanism—a watchdog that continuously scanned for and recovered failed work.

4. Priority Matters at Scale

Not all notifications are equal. Transactional messages (password resets, payment confirmations) must take precedence over marketing emails. Priority queues solved this elegantly.

5. Rate Limiting is User Respect

The composite key approach (template:user:channel) prevented notification fatigue while still allowing legitimate multi-channel communication.

The Impact

With Iteration 1 in production, we achieved:

  • 99.9% delivery success rate for transactional notifications
  • < 2 minute p95 delivery latency for high-priority messages
  • Zero manual interventions for failed notifications after scheduler deployment
  • Complete audit trail for compliance and debugging

The system now handles 150K+ monthly events across four channels, with room to scale to millions as our platform grows. But more importantly, we built a foundation that's resilient—one that doesn't just work when everything goes right, but continues working when things inevitably go wrong.

Conclusion: Architecture as Evolution

This project reinforced a fundamental truth about software engineering: architecture is iterative. We could have tried to build the "perfect" system upfront, but we would have over-engineered or, worse, missed critical failure modes we didn't anticipate.

Instead, we:

  1. Built a working system quickly (Iteration 0)
  2. Analyzed its weaknesses honestly (the February 26th pivot)
  3. Enhanced it with targeted improvements (Iteration 1)

The result is a communication service that's both pragmatic and robust—a system that respects user experience, handles failure gracefully, and provides the observability needed to operate confidently in production.

If you're building similar systems, remember: start simple, instrument everything, and evolve based on real failure modes, not hypothetical ones.

The complete source code and diagrams for this architecture are available in the project repository. For questions or discussions about distributed notification systems, feel free to reach out.