Architecture Review: Notification Delivery Platform

Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.

Affects 2 nodes. (Event Streaming, Slow Consumer)

When a downstream service begins rate limiting requests from an upstream service, the upstream's retry logic with insufficient backoff amplifies the request rate : exceeding the rate limit further and potentially pushing the rejection downstream to other upstream callers, producing a cascade of rate-limited retries across the call graph.

Affects 0 nodes

Seed 'Queue Consumer Backlog' identifies 4 metrics relevant to queue_backlog_accumulation.

Metrics to instrument: queue_depth, consumer_lag_seconds, consumer_throughput

Trigger: Notification provider latency (SendGrid, FCM) adding 200–500ms to business transaction p99 latency; a provider outage causing business transaction failures (e.g., order placement failing because email send fails); no ability to retry failed notifications without replaying the entire business transaction. Migrate from 'Direct synchronous notification sends in application code (inline with business transaction)' to 'Kafka-decoupled async notification pipeline with RabbitMQ per-channel fan-out'. Deploy the Kafka event publication before removing the synchronous send code. Run both in parallel for 1 week: publish to Kafka AND send synchronously, validating that the Kafka path delivers the same notifications. Remove the synchronous path only after the Kafka path is validated and the inbox deduplication layer is live.

The application code change from synchronous send to event publication requires careful handling of cases where the business logic previously used the notification send result (e.g., "if email send fails, block the operation"): these cases must be explicitly identified and decoupled; Until the inbox pattern is in place, the async pipeline has at-least-once delivery with no deduplication; early deployments may produce duplicate notifications if upstream events are retried

Trigger: Product requirement to add mobile push notifications and SMS for critical transactional alerts; need to route different notification types to different channels based on user preference; single delivery code path unable to handle channel-specific retry semantics and rate limits. Migrate from 'Single-channel notification delivery (email only)' to 'Multi-channel notification delivery (email + push + SMS + in-app) with per-channel RabbitMQ queues'. Add channels one at a time. Start with in-app notifications (no external provider dependency) to validate the RabbitMQ channel routing architecture, then add push, then SMS. Each channel requires its own dead-letter queue configuration and delivery monitoring dashboard before it is considered production-ready.

Each new channel introduces a new provider dependency with its own authentication, rate limiting, and failure mode; adding push before implementing per-channel circuit breakers risks a FCM outage cascading to the entire notification pipeline; Mobile push token management (FCM/APNs token refresh, invalid token handling) is operationally complex: invalid tokens must be removed from delivery attempts and their invalid status must be recorded to prevent repeated failed delivery attempts

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: RabbitMQ queue depth for email channel growing > 100k messages; SendGrid 429 responses visible in delivery worker logs; email delivery p95 latency > 2 minutes; delivery worker retry thread pool saturated; dead-letter queue receiving messages from retry exhaustion despite provider being available

Bottleneck: Delivery worker retry policy not aligned with provider rate limit reset window; workers retrying before the rate limit has reset, accumulating failed attempts. Evolution: Implement provider-aware retry backoff: on 429 response, parse the Retry-After header and schedule the next retry attempt at exactly that time, not on a fixed exponential backoff schedule; add per-provider circuit breakers that open after 5 consecutive 429s and attempt a probe request at the retry-after interval; scale email consumer replicas to process the backlog faster when the rate limit window resets

Signal: Kafka consumer lag for notification event consumers growing; notification delivery volume much higher than upstream business event volume (ratio > 5:1); RabbitMQ aggregate message rate across all channel queues elevated; one upstream event type (e.g., new_comment) accounting for disproportionate share of notification volume

Bottleneck: Upstream event fan-out generating more notification sends per event than expected; or a notification rule misconfiguration triggering notifications for every event regardless of user preference. Evolution: Audit notification fan-out ratio per upstream event type; add fan-out cost metrics (notifications_generated per upstream event) as a monitored SLA; implement notification preference filtering before fan-out: only generate delivery attempts for users with the corresponding notification type enabled; implement a notification aggregation layer that batches multiple low-priority events into digest notifications rather than individual sends