Use Notification Delivery Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Notification Delivery Platform. Traceable to YAML knowledge entities.
Context
Notification platforms sit at the intersection of two competing failure modes: under-delivery (critical transactional notifications: password reset, payment confirmation: must never be silently dropped) and over-delivery (marketing and engagement notifications must be rate-limited to prevent user fatigue and spam classification). The delivery path traverses external provider APIs (SendGrid, FCM, Twilio) that each have their own rate limits, availability SLAs, and failure modes. A provider outage triggers retry storms that can exhaust retry budgets for legitimate delivery attempts. Deduplication is mandatory : at-least-once Kafka delivery combined with upstream event retries means the same notification may be triggered multiple times; sending a duplicate password reset SMS is a security and trust incident. Primary operational risks include: Provider rate limit cascade: a SendGrid API rate limit response (429) causes email delivery workers to back off and retry; if retry backoff is not calibrated to the provider's rate limit reset window, workers continuously retry against the limit, consuming all retry budget without delivering messages; the email queue grows unboundedly while retries consume worker threads; Deduplication token expiry causing duplicate sends: per-notification deduplication tokens in Redis are set with a TTL (e.g., 24 hours) to bound memory growth; if an upstream system replays a notification event after the deduplication TTL has expired (e.g., a batch retry job 30 hours after the original event), the duplicate check misses and the notification is re-sent; for transactional notifications (password reset, payment confirmation), this is a user-visible error; Notification suppression misconfiguration: user-level rate limiting in Redis (e.g., max 3 push notifications per hour per user) is implemented as a sliding window counter; if the rate limit key is accidentally over-broad (keyed by user_id without channel type), a high-volume email campaign suppresses push notifications for the same users, silently dropping critical transactional notifications behind a marketing suppression.
Decision
We will adopt the **Notification Delivery Platform** architecture pattern. This is a moderate-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A multi-channel notification delivery architecture that accepts upstream business events (order placed, payment received, comment posted, threshold alert triggered) and routes them to per-channel delivery workers (push via FCM/APNs, email via SendGrid, SMS via Twilio, in-app via WebSocket). Kafka carries raw business events from upstream producers. RabbitMQ handles per-channel fan-out with separate exchanges and queues per delivery channel, isolating email queue backlog from push notification delivery. PostgreSQL provides durable notification state tracking (sent, failed, bounced, suppressed). Redis enforces per-user rate limiting (notification frequency caps to prevent fatigue) and stores deduplication tokens to prevent duplicate sends across retry attempts. The inbox pattern on the consumer side ensures idempotent delivery even when Kafka produces duplicate events. Core technology stack: kafka, rabbitmq, postgresql, redis.
Accepted Tradeoffs
- ⚠RabbitMQ per-channel queues (email, push, SMS, in-app) provide channel isolation : a provider outage causing email queue backlog does not delay push notifications : but multiply operational surface: each channel has its own queue depth metric, consumer SLA, and dead-letter queue to monitor
- ⚠Inbox pattern on the consumer side ensures idempotent delivery regardless of Kafka duplicate events, but requires a per-notification inbox table in PostgreSQL; at high notification volume (>10M/day), the inbox table requires aggressive TTL-based cleanup to prevent unbounded growth and index bloat
- ⚠Circuit breaker per provider prevents a degraded provider from exhausting worker thread pools with blocked I/O, but introduces a window where notifications are intentionally not attempted; the circuit breaker open period (typically 30–60s) must be tuned to balance provider recovery detection time against delivery latency
- ⚠Per-user rate limiting prevents notification fatigue and reduces spam classification risk, but requires distinguishing notification priority tiers: transactional notifications (password reset, payment failed) must bypass rate limiting entirely; this requires a notification type registry with explicit priority classification maintained as application configuration
- ⚠At-least-once delivery semantics from Kafka require idempotent delivery workers; the simplest model (check deduplication token → deliver → record delivery state) has a TOCTOU race: two concurrent workers can both pass the dedup check simultaneously and both deliver; the inbox pattern closes this race by using PostgreSQL upsert on the notification_id as the idempotency gate
Risks
Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.
When a downstream service begins rate limiting requests from an upstream service, the upstream's retry logic with insufficient backoff amplifies the request rate : exceeding the rate limit further and potentially pushing the rejection downstream to other upstream callers, producing a cascade of rate-limited retries across the call graph.
A single inbound request triggers N downstream calls, amplifying load on downstream services by the fanout factor. At sustained inbound rates, the amplified load overwhelms downstream services that appear correctly sized for direct traffic.
A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.
A subset of service instances fails while others remain healthy, producing a low aggregate error rate that masks significant per-instance failures and causes consistent errors for specific request patterns or user segments.
Alternatives Considered
AI Retrieval-Augmented Generation Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Notification Delivery Platform's moderate complexity.
Analytics Data Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Notification Delivery Platform's moderate complexity.
API Gateway Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Notification Delivery Platform's moderate complexity.
Audit and Compliance Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Notification Delivery Platform's moderate complexity.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Provider Rate Limit Backlog
Signal: RabbitMQ queue depth for email channel growing > 100k messages; SendGrid 429 responses visible in delivery worker logs; email delivery p95 latency > 2 minutes; delivery worker retry thread pool saturated; dead-letter queue receiving messages from retry exhaustion despite provider being available
Evolution: Implement provider-aware retry backoff: on 429 response, parse the Retry-After header and schedule the next retry attempt at exactly that time, not on a fixed exponential backoff schedule; add per-provider circuit breakers that open after 5 consecutive 429s and attempt a probe request at the retry-after interval; scale email consumer replicas to process the backlog faster when the rate limit window resets
Tier 2: Notification Fan-Out Amplification
Signal: Kafka consumer lag for notification event consumers growing; notification delivery volume much higher than upstream business event volume (ratio > 5:1); RabbitMQ aggregate message rate across all channel queues elevated; one upstream event type (e.g., new_comment) accounting for disproportionate share of notification volume
Evolution: Audit notification fan-out ratio per upstream event type; add fan-out cost metrics (notifications_generated per upstream event) as a monitored SLA; implement notification preference filtering before fan-out: only generate delivery attempts for users with the corresponding notification type enabled; implement a notification aggregation layer that batches multiple low-priority events into digest notifications rather than individual sends
Tier 3: Inbox Table Growth and Query Pressure
Signal: PostgreSQL inbox table row count > 500M; inbox deduplication query latency > 10ms (above the acceptable delivery path overhead); inbox table VACUUM running continuously; index bloat on notification_id index visible in pg_stat_user_indexes; dead tuple count in pg_stat_user_tables for inbox table growing faster than autovacuum can clear
Evolution: Partition the inbox table by created_at date range; implement automated partition drop for partitions older than the deduplication TTL (e.g., drop partitions > 7 days old); this replaces row-level DELETE with partition DROP, which is orders of magnitude faster; separate the delivery_state tracking table from the deduplication inbox table to reduce update churn on the primary dedup index
Tier 4: Multi-Provider Delivery Architecture
Signal: Single provider (SendGrid or Twilio) outage causing sustained delivery failure across all email or SMS notifications; no automatic fallback to secondary provider; delivery SLA breach for transactional notifications (password reset, payment confirmation) during provider maintenance window; provider cost becoming material and single-vendor lock-in a business risk
Evolution: Implement provider abstraction layer with per-channel provider routing config; add a secondary provider (e.g., Postmark as email fallback behind SendGrid) with circuit-breaker-driven automatic failover; route transactional notifications (priority=critical) through the primary provider with automatic failover to secondary on circuit open; route marketing notifications through the cheaper secondary provider with no failover (acceptable to drop marketing notifications during outages)
Migration Path
Direct synchronous notification sends in application code (inline with business transaction) → Kafka-decoupled async notification pipeline with RabbitMQ per-channel fan-out
Notification provider latency (SendGrid, FCM) adding 200–500ms to business transaction p99 latency; a provider outage causing business transaction failures (e.g., order placement failing because email send fails); no ability to retry failed notifications without replaying the entire business transaction
Single-channel notification delivery (email only) → Multi-channel notification delivery (email + push + SMS + in-app) with per-channel RabbitMQ queues
Product requirement to add mobile push notifications and SMS for critical transactional alerts; need to route different notification types to different channels based on user preference; single delivery code path unable to handle channel-specific retry semantics and rate limits
Fixed notification preference (all users receive all notification types) → Per-user notification preference management with suppression and rate limiting
User complaints about notification volume increasing; spam classification rate rising for marketing notifications; regulatory requirement (GDPR, CAN-SPAM) to honor notification opt-out within 10 business days; need to suppress notifications for churned users to avoid wasting provider quota
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has moderate operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 2 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.