Architecture Review: Distributed Job Queue Platform

Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.

Affects 2 nodes. (Event Streaming, Slow Consumer)

Two or more transactions each hold a lock the other needs, forming a cycle in the lock wait-for graph that no participant can escape on its own. The database breaks the cycle by aborting one transaction, surfacing a serialization-class error the application must catch and retry. Under sustained contention, naive immediate retries re-enter the same cycle and amplify it into a retry storm.

Affects 1 node. (PostgreSQL)

Seed 'Queue Consumer Backlog' identifies 4 metrics relevant to queue_backlog_accumulation.

Metrics to instrument: queue_depth, consumer_lag_seconds, consumer_throughput

Trigger: Background jobs competing with user-facing API requests for application server resources (CPU, memory, thread pool); a long-running job blocking the application process for minutes; need for job retry on failure without application restart; need to scale job execution independently from the API request handling capacity. Migrate from 'In-process job execution (synchronous, within the same application process)' to 'PostgreSQL-backed distributed job queue with Redis visibility leasing'. Identify and categorize all in-process jobs by idempotency before migration begins. Migrate idempotent jobs first (image resizing, report generation, email send). Build the dead-letter queue, retry policy, and monitoring before migrating non-idempotent jobs. Never migrate financial or state-mutation jobs to the async path until idempotency is verified and load-tested.

The transition from synchronous to async execution means the caller no longer has a direct result from the job operation; all callers that block on synchronous job results must be refactored to poll a job status endpoint or receive a callback; The first version of the distributed job queue must implement visibility leasing and idempotent job operations before any non-idempotent jobs are migrated to the async path: migrating a non-idempotent job before leasing is implemented guarantees duplicate execution during worker failures

Trigger: High-priority transactional jobs (e.g., payment processing, user account operations) delayed by low-priority batch jobs (e.g., report generation, data export) consuming all worker capacity; job execution latency SLA differentiated by job type but not enforceable with a single worker pool; need to independently autoscale high-priority workers without scaling batch workers. Migrate from 'Single-worker-pool job queue (all jobs processed by one pool)' to 'Priority-separated worker pools with dedicated transactional and batch pools'. Start with two pools: one for high-priority (all transactional jobs), one for low-priority (all batch jobs). Add more priority tiers only if the two-tier model proves insufficient. Each pool has its own job table query (filtered by priority tier), its own autoscale policy, and its own SLA dashboard. The job type-to-pool mapping is maintained as application configuration, not database metadata.

Separate worker pools require separate monitoring and autoscaling configuration; doubling the operational surface requires commensurate monitoring investment before the separation is made; If a job is mis-classified (transactional job enqueued as batch), it enters the wrong queue and violates its own SLA silently; job priority classification must be enforced at enqueue time with validation, not as a convention

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Worker idle rate > 20% despite queue depth > 10k pending jobs; PostgreSQL pg_locks showing wait events on job table index; worker job claim p99 latency > 50ms (claim should be sub-10ms with correct indexing); CPU on PostgreSQL elevated from index scan overhead on job claim queries

Bottleneck: Missing or misconfigured partial index on the job claim query; high worker concurrency driving SELECT FOR UPDATE SKIP LOCKED contention on a narrow hot page. Evolution: Add a partial index on (priority DESC, created_at ASC) WHERE status = 'pending' AND run_at <= NOW(): the WHERE clause reduces the index to only claimable jobs, dramatically reducing index scan range; if contention persists, implement a job dispatch service (single dispatcher process) that batches claim queries and distributes job IDs to workers via an in-memory channel, removing per-worker database claims; tune FILLFACTOR on the job table to 70% to reduce hot page contention on SKIP LOCKED

Signal: High-priority job queue depth growing despite workers available; low-priority batch jobs showing high throughput while transactional job latency (time from enqueue to execution start) p95 > 30s; worker pool metrics showing workers claiming jobs uniformly across priority levels rather than draining the high- priority queue first

Bottleneck: Single worker pool consuming from all priority queues with equal weight; no priority-weighted polling implementation. Evolution: Separate worker pools per priority tier (e.g., dedicated transactional workers for high-priority jobs, shared workers for low-priority batch); or implement priority-weighted polling in a unified worker pool (poll high-priority queue N times before polling low-priority queue once, where N is the priority weight ratio); add high-priority job execution latency as a first-class SLA metric with alerting threshold separate from batch job latency