Monitoring and Observability

You cannot operate a notification system by checking whether the worker process is alive. The worker can be alive while the queue grows, provider calls fail, or users never receive security alerts.

The same principle applies to synchronous APIs. The Node.js API performance guide connects p50, p95, and p99 latency with PostgreSQL pool waits, Redis hit rates, event-loop delay, and capacity limits.

Core Metrics

Queue age is often more important than queue length. A queue of 10,000 marketing jobs may be fine. A queue where OTP jobs are 5 minutes old is not.

Dashboard Layout

Prometheus Metrics

notification_jobs_processed_total{channel="email",status="success"}
notification_jobs_failed_total{channel="sms",provider="twilio"}
notification_delivery_latency_seconds_bucket{channel="push"}
notification_queue_depth{queue="security-critical"}
notification_dlq_total{queue="transactional"}

Labels are powerful but can explode cardinality. Do not label metrics by user ID or notification ID.

Structured Logs

{
  "level": "info",
  "message": "email provider accepted",
  "notificationId": "noti_123",
  "eventId": "evt_456",
  "channel": "email",
  "provider": "sendgrid",
  "providerMessageId": "msg_789",
  "traceId": "trace_abc"
}

Logs should let support or engineering follow one notification across services.

Tracing

1. 01
   Product API request
2. 02
   Outbox publish
3. 03
   Notification consumer
4. 04
   Queue job
5. 05
   Worker provider call

OpenTelemetry can carry trace context through events and jobs if you include trace IDs in metadata.

Alerting

Alert on user impact, not just noise.

Good alerts:

Noisy alerts:

Common Mistakes

1. Monitoring only worker uptime.
2. Alerting on every individual job failure.
3. Not separating provider failures from validation failures.
4. Using high-cardinality metric labels.
5. Having no dashboard for queue age.

Interview Questions

1. Which metric best shows notification delay?
2. Why is queue age important?
3. How would you trace one notification from event to provider?
4. What alerts would you set for OTP delivery?

Exercise

Design a Grafana dashboard for a notification system. Include at least eight panels and two alerts.