Lesson 3 of 15High Level Architecture

High Level Architecture

The simplest notification design is also the easiest one to outgrow.

Process flow3 steps
  1. 01
    Frontend
  2. 02
    Backend
  3. 03
    Send Email

This can work for a prototype. It fails when traffic grows, providers slow down, retries are needed, or multiple channels must be sent. The product backend becomes responsible for checkout, payment, order state, email templates, provider rate limits, and retry behavior. That is too much responsibility in one request path.

Why Bad Design Fails

Imagine 100,000 orders per minute. If each email provider call takes 3 seconds, synchronous sending creates 300,000 seconds of external waiting every minute.

Blocking cost
100,000 orders/minute * 3 seconds/provider call = 300,000 blocked seconds/minute

Even if calls run concurrently, your backend now needs connection pools, timeouts, retry policy, provider throttling, and duplicate prevention. A provider outage becomes a checkout outage.

Better Architecture

Process flow5 steps
  1. 01
    Frontend
  2. 02
    Product Backend
  3. 03
    Notification Service
  4. 04
    Message Queue
  5. 05
    Worker Pool

Parallel branches

Email ProviderSMS ProviderPush Provider

The product backend finishes its work quickly and emits notification intent. The Notification Service validates and persists that intent. The queue buffers work. Workers process jobs independently. Providers remain external dependencies, not checkout dependencies.

Component Responsibilities

ComponentOwnsShould Not Own
Product BackendBusiness eventProvider retries
Notification ServiceTemplates, preferencesPayment logic
QueueBuffering for workersSource of truth
WorkerProvider call and statusProduct transaction
DatabaseDurable notification stateLong-running execution

Request Flow

Process flow8 steps
  1. 01
    User completes checkout
  2. 02
    Product backend stores order
  3. 03
    Backend publishes OrderPlaced
  4. 04
    Notification Service creates notification rows
  5. 05
    Service enqueues channel jobs
  6. 06
    Worker sends email, push, or SMS
  7. 07
    Worker stores provider response
  8. 08
    Dashboard or API shows final status

API Example

await notificationClient.create({
  eventId: order.eventId,
  eventType: "OrderPlaced",
  tenantId: order.tenantId,
  userId: order.customerId,
  templateKey: "order_confirmation",
  variables: {
    orderId: order.id,
    deliveryDate: order.estimatedDeliveryDate
  }
});

The backend sends facts, not rendered messages. Template rendering belongs inside the notification domain because templates evolve independently from order logic.

Why Queue Exists

A queue gives three powers:

  1. Burst absorption: traffic spikes do not overload providers instantly.
  2. Retry control: failed jobs can be retried with delay.
  3. Worker scaling: add more workers without changing product code.

A queue turns user-facing latency into background throughput work.

Production Variants

Small SaaS:

Process flow5 steps
  1. 01
    Next.js API
  2. 02
    PostgreSQL
  3. 03
    BullMQ/Redis
  4. 04
    Node worker
  5. 05
    SendGrid

Large marketplace:

Process flow6 steps
  1. 01
    Backend services
  2. 02
    Kafka
  3. 03
    Notification service
  4. 04
    Kafka topics
  5. 05
    Worker fleet
  6. 06
    Multi-provider routing

The pattern is the same. The technology changes with scale, team skill, compliance, and operational maturity.

Common Mistakes

  1. Treating the queue as the only source of truth.
  2. Rendering templates in product services.
  3. Having one worker process all channels forever.
  4. Not separating transactional and promotional workloads.
  5. Ignoring provider rate limits until production.

Interview Questions

  1. Why should checkout not wait for email delivery?
  2. What is the role of the Notification Service?
  3. How does a queue improve reliability?
  4. When would you split workers by channel?

Exercise

Draw high-level architecture for a food delivery app. Include customer, restaurant, delivery partner, and support notifications.

What you will learn

Why direct provider calls from the backend fail at scale.

How queues protect product latency.

How workers isolate slow provider integrations.

How to reason mathematically about blocking calls.

Production checklist

  • Direct-send design is avoided
  • Notification service owns delivery workflow
  • Queue absorbs bursts
  • Workers are horizontally scalable
  • Provider calls are isolated
  • Status updates are stored