Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'Notification provider latency (SendGrid, FCM) adding 200–500'. Distributed Job Queue Platform: triggered by 'Background jobs competing with user-facing API requests for '.

Migration Step 2

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'Product requirement to add mobile push notifications and SMS'. Distributed Job Queue Platform: triggered by 'High-priority transactional jobs (e.g., payment processing, '.

Migration Step 3

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'User complaints about notification volume increasing; spam c'. Distributed Job Queue Platform: triggered by 'Multi-step jobs (e.g., ingest file → validate → transform → '.

RabbitMQ queue depth for email channel growing > 100k messages; SendGrid 429 responses visible in delivery worker logs; email delivery p95 latency > 2 minutes; delivery worker retry thread pool saturated; dead-letter queue receiving messages from retry exhaustion despite provider being available

Tier 1: Provider Rate Limit Backlog: Delivery worker retry policy not aligned with provider rate limit reset window; workers retrying before the rate limit has reset, accumulating failed attempts. Recommended evolution: Implement provider-aware retry backoff: on 429 response, parse the Retry-After header and schedule the next retry attempt at exactly that time, not on a fixed exponential backoff schedule; add per-provider circuit breakers that open after 5 consecutive 429s and attempt a probe request at the retry-after interval; scale email consumer replicas to process the backlog faster when the rate limit window resets .

Kafka consumer lag for notification event consumers growing; notification delivery volume much higher than upstream business event volume (ratio > 5:1); RabbitMQ aggregate message rate across all channel queues elevated; one upstream event type (e.g., new_comment) accounting for disproportionate share of notification volume

Tier 2: Notification Fan-Out Amplification: Upstream event fan-out generating more notification sends per event than expected; or a notification rule misconfiguration triggering notifications for every event regardless of user preference. Recommended evolution: Audit notification fan-out ratio per upstream event type; add fan-out cost metrics (notifications_generated per upstream event) as a monitored SLA; implement notification preference filtering before fan-out: only generate delivery attempts for users with the corresponding notification type enabled; implement a notification aggregation layer that batches multiple low-priority events into digest notifications rather than individual sends .

Worker idle rate > 20% despite queue depth > 10k pending jobs; PostgreSQL pg_locks showing wait events on job table index; worker job claim p99 latency > 50ms (claim should be sub-10ms with correct indexing); CPU on PostgreSQL elevated from index scan overhead on job claim queries

Tier 1: Job Claim Lock Contention: Missing or misconfigured partial index on the job claim query; high worker concurrency driving SELECT FOR UPDATE SKIP LOCKED contention on a narrow hot page. Recommended evolution: Add a partial index on (priority DESC, created_at ASC) WHERE status = 'pending' AND run_at <= NOW(): the WHERE clause reduces the index to only claimable jobs, dramatically reducing index scan range; if contention persists, implement a job dispatch service (single dispatcher process) that batches claim queries and distributes job IDs to workers via an in-memory channel, removing per-worker database claims; tune FILLFACTOR on the job table to 70% to reduce hot page contention on SKIP LOCKED .

High-priority job queue depth growing despite workers available; low-priority batch jobs showing high throughput while transactional job latency (time from enqueue to execution start) p95 > 30s; worker pool metrics showing workers claiming jobs uniformly across priority levels rather than draining the high- priority queue first

Tier 2: Priority Inversion Under Load: Single worker pool consuming from all priority queues with equal weight; no priority-weighted polling implementation. Recommended evolution: Separate worker pools per priority tier (e.g., dedicated transactional workers for high-priority jobs, shared workers for low-priority batch); or implement priority-weighted polling in a unified worker pool (poll high-priority queue N times before polling low-priority queue once, where N is the priority weight ratio); add high-priority job execution latency as a first-class SLA metric with alerting threshold separate from batch job latency .