Architecture Review: Notification Delivery Platform

Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.

Affects 2 nodes. (Event Streaming, Slow Consumer)

When a downstream service begins rate limiting requests from an upstream service, the upstream's retry logic with insufficient backoff amplifies the request rate : exceeding the rate limit further and potentially pushing the rejection downstream to other upstream callers, producing a cascade of rate-limited retries across the call graph.

Affects 0 nodes

Seed 'Queue Consumer Backlog' identifies 4 metrics relevant to queue_backlog_accumulation.

Metrics to instrument: queue_depth, consumer_lag_seconds, consumer_throughput

Trigger: Notification provider latency (SendGrid, FCM) adding 200–500ms to business transaction p99 latency; a provider outage causing business transaction failures (e.g., order placement failing because email send fails); no ability to retry failed notifications without replaying the entire business transaction. Migrate from 'Direct synchronous notification sends in application code (inline with business transaction)' to 'Kafka-decoupled async notification pipeline with RabbitMQ per-channel fan-out'. Deploy the Kafka event publication before removing the synchronous send code. Run both in parallel for 1 week: publish to Kafka AND send synchronously, validating that the Kafka path delivers the same notifications. Remove the synchronous path only after the Kafka path is validated and the inbox deduplication layer is live.

The application code change from synchronous send to event publication requires careful handling of cases where the business logic previously used the notification send result (e.g., "if email send fails, block the operation"): these cases must be explicitly identified and decoupled; Until the inbox pattern is in place, the async pipeline has at-least-once delivery with no deduplication; early deployments may produce duplicate notifications if upstream events are retried

Trigger: Product requirement to add mobile push notifications and SMS for critical transactional alerts; need to route different notification types to different channels based on user preference; single delivery code path unable to handle channel-specific retry semantics and rate limits. Migrate from 'Single-channel notification delivery (email only)' to 'Multi-channel notification delivery (email + push + SMS + in-app) with per-channel RabbitMQ queues'. Add channels one at a time. Start with in-app notifications (no external provider dependency) to validate the RabbitMQ channel routing architecture, then add push, then SMS. Each channel requires its own dead-letter queue configuration and delivery monitoring dashboard before it is considered production-ready.

Each new channel introduces a new provider dependency with its own authentication, rate limiting, and failure mode; adding push before implementing per-channel circuit breakers risks a FCM outage cascading to the entire notification pipeline; Mobile push token management (FCM/APNs token refresh, invalid token handling) is operationally complex: invalid tokens must be removed from delivery attempts and their invalid status must be recorded to prevent repeated failed delivery attempts

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

'Notification Delivery Platform' includes a cache in its topology. If the cache becomes unavailable, the primary database receives the cache's full request load until the cache recovers.

Capacity-plan the primary database for this fallback load, not only for the steady-state cached load.

Cache invalidation for Notification Delivery Platform is event-driven: kafka refreshes or invalidates redis. This couples cache freshness to consumer lag on that event stream, not to the primary write path directly.

If the event-stream consumer falls behind, the cache serves stale data until it catches up -- monitor consumer lag as a cache-freshness signal, not only a backlog signal.

Notification Delivery Platform carries retry_with_backoff for its event publication with no referenced dead_letter_queue. Exhausted retries need an explicit destination, or a permanently failing message either blocks the queue or is silently discarded.

Add a dead-letter path for exhausted retries, and monitor its depth as an active-incident signal, not just a static count.