Scaling Notification Systems

Scaling notifications is not just "add more servers." More workers can overload providers. More queues can complicate ordering. More database writes can create hot partitions. Scaling is controlled pressure management.

Baseline Capacity Math

If one worker safely handles 25 jobs/second, you need at least 23 workers before headroom.

Horizontal Worker Scaling

1. 01
   Queue

Parallel branches

Fan-out

worker-1worker-2worker-3worker-N

Workers should be stateless. State belongs in the database, queue, cache, or provider.

Priority Isolation

This prevents campaigns from delaying OTP and fraud alerts.

Database Scaling

Hot tables:

Use indexes for common queries, partition large append-heavy tables, and archive old delivery attempts.

CREATE INDEX notifications_tenant_status_created_idx
ON notifications (tenant_id, status, created_at);

For multi-tenant SaaS, tenant-aware indexes are often essential.

Caching

Cache stable reads:

Redis is common, but cache invalidation matters. When a user unsubscribes, preference cache must expire or be invalidated.

Provider Rate Limits

If SendGrid allows 10,000 requests/minute, scaling workers to 100,000 sends/minute creates failures. Rate limit before the provider.

await rateLimiter.consume(`provider:sendgrid`, 1);
await sendgrid.send(email);

Provider Failover

1. 01
   Primary email provider fails
2. 02
   Route critical transactional emails to backup provider
3. 03
   Keep marketing paused

Failover should be selective. Sending every marketing email through backup can be expensive and may hurt reputation.

Regional Deployment

For global products, region matters:

Regional routing can reduce latency, cost, and compliance risk.

Common Mistakes

1. Scaling workers without provider rate limits.
2. Mixing marketing and security traffic.
3. No database archiving plan.
4. Cache invalidation missing for preferences.
5. Assuming one provider works equally well in every country.

Interview Questions

1. How would you estimate worker count?
2. Why separate queues by priority?
3. How do provider rate limits affect scaling?
4. When would you use regional routing?

Exercise

Calculate workers needed for 5 million notifications/day with 1.8 channel jobs each and each worker processing 20 jobs/second.