Scaling Notification Systems

Scaling notifications is not just "add more servers." More workers can overload providers. More queues can complicate ordering. More database writes can create hot partitions. Scaling is controlled pressure management.

Baseline Capacity Math

Incoming notifications: 1,000,000/hour
Average channel jobs: 2 per notification
Total jobs: 2,000,000/hour
Jobs per second: 555

If one worker safely handles 25 jobs/second, you need at least 23 workers before headroom.

555 / 25 = 22.2 workers
Add 30 percent headroom -> 30 workers

Horizontal Worker Scaling

Queue
  |
  +--> worker-1
  +--> worker-2
  +--> worker-3
  +--> worker-N

Workers should be stateless. State belongs in the database, queue, cache, or provider.

Priority Isolation

security-critical queue -> small fast worker pool
transactional queue -> normal worker pool
marketing queue -> bulk worker pool with throttling

This prevents campaigns from delaying OTP and fraud alerts.

Database Scaling

Hot tables:

notifications
delivery_attempts
device_tokens
preferences

Use indexes for common queries, partition large append-heavy tables, and archive old delivery attempts.

CREATE INDEX notifications_tenant_status_created_idx
ON notifications (tenant_id, status, created_at);

For multi-tenant SaaS, tenant-aware indexes are often essential.

Caching

Cache stable reads:

active templates
user preferences
provider routing rules
rate-limit counters
device token lists

Redis is common, but cache invalidation matters. When a user unsubscribes, preference cache must expire or be invalidated.

Provider Rate Limits

If SendGrid allows 10,000 requests/minute, scaling workers to 100,000 sends/minute creates failures. Rate limit before the provider.

await rateLimiter.consume(`provider:sendgrid`, 1);
await sendgrid.send(email);

Provider Failover

Primary email provider fails
  |
  v
Route critical transactional emails to backup provider
  |
  v
Keep marketing paused

Failover should be selective. Sending every marketing email through backup can be expensive and may hurt reputation.

Regional Deployment

For global products, region matters:

India users -> India SMS route
EU users -> EU data retention rules
US users -> US provider account

Regional routing can reduce latency, cost, and compliance risk.

Common Mistakes

1. Scaling workers without provider rate limits.
2. Mixing marketing and security traffic.
3. No database archiving plan.
4. Cache invalidation missing for preferences.
5. Assuming one provider works equally well in every country.

Interview Questions

1. How would you estimate worker count?
2. Why separate queues by priority?
3. How do provider rate limits affect scaling?
4. When would you use regional routing?

Exercise

Calculate workers needed for 5 million notifications/day with 1.8 channel jobs each and each worker processing 20 jobs/second.