Message Queue Deep Dive

A queue is not just a list of jobs. In a notification platform, it is the shock absorber between product traffic and provider capacity.

Core Terms

Technology Comparison

For a personal SaaS or startup, BullMQ or SQS can be enough. For an event platform at marketplace scale, Kafka may be justified.

Queue Design for Notifications

Do not put OTP, password-change alerts, and marketing campaigns in one queue forever. If marketing floods the queue, security alerts should not wait behind it.

Retry Strategy

await queue.add("send-email", payload, {
  attempts: 5,
  backoff: {
    type: "exponential",
    delay: 10_000
  },
  removeOnComplete: true,
  removeOnFail: false
});

Retry only failures that can recover. A provider timeout can recover. An invalid email address probably cannot. Classify errors.

Dead Letter Queue

A DLQ is where messages go when normal processing cannot finish.

1. 01
   Worker fails job 5 times
2. 02
   Move to notification-dlq
3. 03
   Alert engineering or operations
4. 04
   Inspect, fix, replay, or discard

Ordering

Some notifications need order. "Order shipped" should not arrive before "Order confirmed." Kafka partitions can preserve ordering per key, such as orderId. BullMQ queues process jobs in order only under specific concurrency conditions. SQS FIFO can preserve group ordering with throughput limits.

If strict ordering is required, define the ordering key.

Back Pressure

Back pressure means incoming jobs exceed processing capacity.

You can respond by scaling workers, reducing marketing sends, routing to another provider, or temporarily delaying low-priority queues.

Common Mistakes

1. Choosing Kafka because it sounds senior.
2. Not separating high-priority and bulk queues.
3. Retrying permanent failures.
4. Having no DLQ replay tool.
5. Forgetting provider rate limits while scaling consumers.

Interview Questions

1. When would you choose Kafka over SQS?
2. What is a dead-letter queue?
3. How do consumer groups improve throughput?
4. How do you handle back pressure?

Exercise

Design queue names and retry policies for OTP, invoice email, and promotional campaign notifications.