Lesson 3 of 15High Level Architecture

High Level Architecture

The simplest notification design is also the easiest one to outgrow.

Frontend
  |
  v
Backend
  |
  v
Send Email

This can work for a prototype. It fails when traffic grows, providers slow down, retries are needed, or multiple channels must be sent. The product backend becomes responsible for checkout, payment, order state, email templates, provider rate limits, and retry behavior. That is too much responsibility in one request path.

Why Bad Design Fails

Imagine 100,000 orders per minute. If each email provider call takes 3 seconds, synchronous sending creates 300,000 seconds of external waiting every minute.

100,000 orders/minute * 3 seconds/provider call = 300,000 blocked seconds/minute

Even if calls run concurrently, your backend now needs connection pools, timeouts, retry policy, provider throttling, and duplicate prevention. A provider outage becomes a checkout outage.

Better Architecture

Frontend
  |
  v
Product Backend
  |
  v
Notification Service
  |
  v
Message Queue
  |
  v
Worker Pool
  |
  +--> Email Provider
  +--> SMS Provider
  +--> Push Provider

The product backend finishes its work quickly and emits notification intent. The Notification Service validates and persists that intent. The queue buffers work. Workers process jobs independently. Providers remain external dependencies, not checkout dependencies.

Component Responsibilities

Component             Owns                         Should Not Own
Product Backend       business event               provider retries
Notification Service  templates, preferences       payment logic
Queue                 buffering for workers        source of truth
Worker                provider call and status     product transaction
Database              durable notification state   long-running execution

Request Flow

1. User completes checkout.
2. Product backend stores order.
3. Backend publishes OrderPlaced.
4. Notification Service creates notification rows.
5. Service enqueues channel jobs.
6. Worker sends email, push, or SMS.
7. Worker stores provider response.
8. Dashboard or API shows final status.

API Example

await notificationClient.create({
  eventId: order.eventId,
  eventType: "OrderPlaced",
  tenantId: order.tenantId,
  userId: order.customerId,
  templateKey: "order_confirmation",
  variables: {
    orderId: order.id,
    deliveryDate: order.estimatedDeliveryDate
  }
});

The backend sends facts, not rendered messages. Template rendering belongs inside the notification domain because templates evolve independently from order logic.

Why Queue Exists

A queue gives three powers:

  1. Burst absorption: traffic spikes do not overload providers instantly.
  2. Retry control: failed jobs can be retried with delay.
  3. Worker scaling: add more workers without changing product code.

A queue turns user-facing latency into background throughput work.

Production Variants

Small SaaS:

Next.js API -> PostgreSQL -> BullMQ/Redis -> Node worker -> SendGrid

Large marketplace:

Backend services -> Kafka -> Notification service -> Kafka topics -> worker fleet -> multi-provider routing

The pattern is the same. The technology changes with scale, team skill, compliance, and operational maturity.

Common Mistakes

  1. Treating the queue as the only source of truth.
  2. Rendering templates in product services.
  3. Having one worker process all channels forever.
  4. Not separating transactional and promotional workloads.
  5. Ignoring provider rate limits until production.

Interview Questions

  1. Why should checkout not wait for email delivery?
  2. What is the role of the Notification Service?
  3. How does a queue improve reliability?
  4. When would you split workers by channel?

Exercise

Draw high-level architecture for a food delivery app. Include customer, restaurant, delivery partner, and support notifications.

What you will learn

Why direct provider calls from the backend fail at scale.

How queues protect product latency.

How workers isolate slow provider integrations.

How to reason mathematically about blocking calls.

Production checklist

  • Direct-send design is avoided
  • Notification service owns delivery workflow
  • Queue absorbs bursts
  • Workers are horizontally scalable
  • Provider calls are isolated
  • Status updates are stored
High Level Architecture - Production Notification System Design | Niraj Kumar