Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'Notification provider latency (SendGrid, FCM) adding 200–500'. Financial Ledger Platform: triggered by 'Audit requirement to reconstruct account state at any histor'.

Migration Step 2

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'Product requirement to add mobile push notifications and SMS'. Financial Ledger Platform: triggered by 'Kafka publish failures causing financial transaction rollbac'.

Migration Step 3

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'User complaints about notification volume increasing; spam c'. Financial Ledger Platform: triggered by 'Financial dashboard query p99 > 500ms causing dashboard-driv'.

RabbitMQ queue depth for email channel growing > 100k messages; SendGrid 429 responses visible in delivery worker logs; email delivery p95 latency > 2 minutes; delivery worker retry thread pool saturated; dead-letter queue receiving messages from retry exhaustion despite provider being available

Tier 1: Provider Rate Limit Backlog: Delivery worker retry policy not aligned with provider rate limit reset window; workers retrying before the rate limit has reset, accumulating failed attempts. Recommended evolution: Implement provider-aware retry backoff: on 429 response, parse the Retry-After header and schedule the next retry attempt at exactly that time, not on a fixed exponential backoff schedule; add per-provider circuit breakers that open after 5 consecutive 429s and attempt a probe request at the retry-after interval; scale email consumer replicas to process the backlog faster when the rate limit window resets .

Kafka consumer lag for notification event consumers growing; notification delivery volume much higher than upstream business event volume (ratio > 5:1); RabbitMQ aggregate message rate across all channel queues elevated; one upstream event type (e.g., new_comment) accounting for disproportionate share of notification volume

Tier 2: Notification Fan-Out Amplification: Upstream event fan-out generating more notification sends per event than expected; or a notification rule misconfiguration triggering notifications for every event regardless of user preference. Recommended evolution: Audit notification fan-out ratio per upstream event type; add fan-out cost metrics (notifications_generated per upstream event) as a monitored SLA; implement notification preference filtering before fan-out: only generate delivery attempts for users with the corresponding notification type enabled; implement a notification aggregation layer that batches multiple low-priority events into digest notifications rather than individual sends .

pg_locks shows contended rows on accounts table; write p99 > 50ms; deadlock errors in application logs; pg_stat_activity showing many transactions waiting for RowExclusiveLock on the same account rows

Tier 1: Hot Account Lock Contention: Concurrent debit/credit transactions competing for the same account row versions. Recommended evolution: Implement optimistic locking with version column and retry; or queue concurrent updates for the same account entity through an account-scoped serialization queue at the application layer; or partition the accounts table by account range .

Write p99 > 100ms with synchronous_commit = remote_apply; replica WAL apply lag visible in pg_stat_replication; network jitter between primary and replica causing write latency spikes correlating with replication ACK delays

Tier 2: Synchronous Replication Write Latency: Synchronous replication write-ahead wait amplifying network latency for every committed transaction. Recommended evolution: Co-locate primary and replica in the same availability zone for lowest replication RTT; tune wal_sender_timeout and recovery_min_apply_delay; evaluate whether synchronous_commit = on (durable to primary WAL only) is acceptable for your regulatory risk model .