Notification Workers

Workers are where notification intent becomes provider calls. They are also where many production bugs live: duplicate sends, missing variables, infinite retries, unbounded timeouts, and confusing status updates.

Worker Lifecycle

Take Job
  |
  v
Validate
  |
  v
Load Template
  |
  v
Replace Variables
  |
  v
Call Provider
  |
  v
Update Status
  |
  v
Success

Every box can fail. A strong worker treats failure as expected, not exceptional.

Job Payload

type SendNotificationJob = {
  notificationId: string;
  channel: "email" | "sms" | "push" | "in_app";
  attempt: number;
};

Keep job payloads small. Store durable state in the database. If the queue message contains the full rendered email, stale data and replay behavior become harder to reason about.

Worker Implementation

async function processNotificationJob(job: SendNotificationJob) {
  const notification = await db.notification.findUniqueOrThrow({
    where: { id: job.notificationId }
  });

  if (notification.status === "sent") return;

  const template = await loadActiveTemplate(
    notification.templateKey,
    job.channel
  );

  const rendered = renderTemplate(template.body, notification.variables);
  validateRenderedMessage(rendered);

  const attempt = await createDeliveryAttempt(notification.id, job.channel);

  try {
    const response = await providerFor(job.channel).send(rendered, {
      timeoutMs: 10_000
    });

    await markAttemptSuccess(attempt.id, response.messageId);
    await maybeMarkNotificationSent(notification.id);
  } catch (error) {
    await markAttemptFailed(attempt.id, error);
    throw error;
  }
}

Idempotency matters. If the worker crashes after the provider accepts a message but before the database update, the queue may retry. Provider idempotency keys can reduce duplicate sends.

Provider Idempotency

const idempotencyKey = `${notification.id}:${job.channel}`;

await emailProvider.send({
  to: recipient.email,
  subject,
  html,
  headers: {
    "Idempotency-Key": idempotencyKey
  }
});

Not all providers support this. If they do not, store provider message IDs and make your duplicate detection as strong as possible.

Timeouts

Never let provider calls hang forever.

const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 10_000);

try {
  await fetch(providerUrl, {
    method: "POST",
    body: JSON.stringify(payload),
    signal: controller.signal
  });
} finally {
  clearTimeout(timer);
}

Timeouts protect worker capacity. Without them, a provider issue can freeze the whole pool.

Poison Messages

A poison message fails every time because the data is bad. Example: template requires firstName, but the notification variables do not contain it.

Missing variable -> retry -> missing variable -> retry -> DLQ

The fix is not more retries. The fix is validation before enqueue or before provider call.

Common Mistakes

1. Putting full business data in queue messages.
2. Not setting provider timeouts.
3. Updating final status before provider acceptance.
4. Retrying validation failures.
5. Running one giant worker for every channel and priority.

Interview Questions

1. Why should worker jobs be small?
2. How can a duplicate send happen after a worker crash?
3. What is a poison message?
4. Why should provider calls have timeouts?

Exercise

Write worker pseudo-code for a push notification job. Include token validation, provider call, retryable failure, and final status update.