Monitoring and Observability

You cannot operate a notification system by checking whether the worker process is alive. The worker can be alive while the queue grows, provider calls fail, or users never receive security alerts.

Core Metrics

Metric                  Why It Matters
Queue length            shows backlog
Queue age               shows user-facing delay
Delivery latency        event to provider acceptance
Failure rate            provider or system health
Retry count             instability signal
DLQ count               unhandled failures
Provider latency        external dependency health
Send volume by channel  cost and capacity

Queue age is often more important than queue length. A queue of 10,000 marketing jobs may be fine. A queue where OTP jobs are 5 minutes old is not.

Dashboard Layout

Top row:
  - notifications created/minute
  - jobs processed/minute
  - p95 delivery latency
  - failure rate

Middle row:
  - queue length by priority
  - queue age by priority
  - provider latency
  - retry count

Bottom row:
  - DLQ messages
  - bounce/complaint rate
  - SMS spend estimate
  - worker CPU/memory

Prometheus Metrics

notification_jobs_processed_total{channel="email",status="success"}
notification_jobs_failed_total{channel="sms",provider="twilio"}
notification_delivery_latency_seconds_bucket{channel="push"}
notification_queue_depth{queue="security-critical"}
notification_dlq_total{queue="transactional"}

Labels are powerful but can explode cardinality. Do not label metrics by user ID or notification ID.

Structured Logs

{
  "level": "info",
  "message": "email provider accepted",
  "notificationId": "noti_123",
  "eventId": "evt_456",
  "channel": "email",
  "provider": "sendgrid",
  "providerMessageId": "msg_789",
  "traceId": "trace_abc"
}

Logs should let support or engineering follow one notification across services.

Tracing

Product API request
  |
  v
Outbox publish
  |
  v
Notification consumer
  |
  v
Queue job
  |
  v
Worker provider call

OpenTelemetry can carry trace context through events and jobs if you include trace IDs in metadata.

Alerting

Alert on user impact, not just noise.

Good alerts:

security queue age p95 > 30 seconds for 5 minutes
DLQ count for transactional queue > 0
SMS provider failure rate > 10 percent for 10 minutes
email bounce rate > normal baseline

Noisy alerts:

one job failed once
worker CPU above 60 percent for 30 seconds
marketing queue has backlog during campaign

Common Mistakes

1. Monitoring only worker uptime.
2. Alerting on every individual job failure.
3. Not separating provider failures from validation failures.
4. Using high-cardinality metric labels.
5. Having no dashboard for queue age.

Interview Questions

1. Which metric best shows notification delay?
2. Why is queue age important?
3. How would you trace one notification from event to provider?
4. What alerts would you set for OTP delivery?

Exercise

Design a Grafana dashboard for a notification system. Include at least eight panels and two alerts.