Use Distributed Job Queue Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Distributed Job Queue Platform. Traceable to YAML knowledge entities.
Context
Background job queues have a deceptively simple interface (enqueue job, execute job) that conceals significant distributed systems complexity. At-least-once execution semantics (the standard for durable queues) means a job may be executed more than once: when a worker crashes mid-execution, the job's visibility timeout expires and it is re-queued, potentially running again on a different worker. If the job is not idempotent, the duplicate execution causes incorrect state (double-charged customer, double-sent email, duplicate database record). Job visibility timeout is the central operational knob: too short, and long-running jobs self-interrupt before completion; too long, and a crashed worker leaves a job invisible to other workers for the full timeout period. Multi-step jobs that involve external API calls, file processing, or sequential database operations require workflow orchestration with explicit step checkpointing: otherwise a failure in step 4 of 5 restarts from step 1. Primary operational risks include: Visibility timeout miscalibration causing duplicate execution: a job processing a large file takes 90 seconds but the visibility timeout is set to 60 seconds; the job is re-queued and claimed by a second worker while the first worker is still executing; both workers execute the same job to completion, producing duplicate output; this is silent: no error is raised unless the job operation explicitly checks for prior execution state; PostgreSQL job table lock contention under high enqueue rate: the job table uses an advisory lock or SELECT FOR UPDATE SKIP LOCKED for job claiming; at high worker concurrency (50+ workers querying simultaneously), SKIP LOCKED contention on the job table index produces elevated p99 claim latency and worker idle time that appears as low throughput despite a full queue and available workers; Temporal workflow history bloat: long-running workflows that accumulate thousands of activity completions produce large workflow history objects; Temporal history is stored in its backing PostgreSQL database and replayed on worker restart; workflows with >10k events are slow to replay and can cause workflow worker memory pressure; workflows that never complete (stuck in a waiting state) accumulate history indefinitely.
Decision
We will adopt the **Distributed Job Queue Platform** architecture pattern. This is a moderate-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded. Core technology stack: postgresql, redis, temporal, kafka.
Accepted Tradeoffs
- ⚠PostgreSQL as the durable job store provides ACID guarantees and SQL query access to job state (job history, retry counts, error logs queryable via SQL), but requires careful index management: the job claim query (SELECT ... WHERE status = 'pending' AND run_at <= NOW() ORDER BY priority DESC, created_at ASC FOR UPDATE SKIP LOCKED) must be supported by a partial index, not a full-table scan
- ⚠Redis visibility timeout leases give fast in-flight state checks (O(1) lease expiry check vs. PostgreSQL query), but Redis is not the source of truth for job state: Redis and PostgreSQL can diverge if a Redis failure occurs mid-execution; the reconciliation process (PostgreSQL is authoritative, Redis is rebuilt from PostgreSQL job state on startup) must be implemented before production use
- ⚠Temporal provides durable, resumable multi-step workflow execution that eliminates the need to design step checkpointing manually, but adds a significant operational dependency: Temporal requires its own PostgreSQL database, worker pool, and monitoring stack; it is over-engineered for single-step jobs and should only be introduced when multi-step workflow complexity justifies the overhead
- ⚠Competing consumer worker pools scale horizontally, but work stealing across priority queues requires careful design: a single worker pool consuming from both high-priority and low-priority queues risks priority inversion (low-priority batch jobs consuming all workers, starving high-priority transactional jobs) if the worker pool does not implement priority-weighted polling
- ⚠Backpressure at the enqueue API layer (rejecting new jobs when the queue depth exceeds a threshold) prevents unbounded queue growth during worker degradation, but requires the caller to handle the rejection gracefully: callers that enqueue from synchronous user-facing flows must not propagate the backpressure rejection as a user error
Risks
Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.
Two or more transactions each hold a lock the other needs, forming a cycle in the lock wait-for graph that no participant can escape on its own. The database breaks the cycle by aborting one transaction, surfacing a serialization-class error the application must catch and retry. Under sustained contention, naive immediate retries re-enter the same cycle and amplify it into a retry storm.
Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.
A subset of service instances fails while others remain healthy, producing a low aggregate error rate that masks significant per-instance failures and causes consistent errors for specific request patterns or user segments.
A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.
Alternatives Considered
AI Retrieval-Augmented Generation Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Distributed Job Queue Platform's moderate complexity.
Analytics Data Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Distributed Job Queue Platform's moderate complexity.
API Gateway Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Distributed Job Queue Platform's moderate complexity.
Audit and Compliance Platform was not selected because its high operational complexity exceeds the current team's readiness relative to Distributed Job Queue Platform's moderate complexity.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Job Claim Lock Contention
Signal: Worker idle rate > 20% despite queue depth > 10k pending jobs; PostgreSQL pg_locks showing wait events on job table index; worker job claim p99 latency > 50ms (claim should be sub-10ms with correct indexing); CPU on PostgreSQL elevated from index scan overhead on job claim queries
Evolution: Add a partial index on (priority DESC, created_at ASC) WHERE status = 'pending' AND run_at <= NOW(): the WHERE clause reduces the index to only claimable jobs, dramatically reducing index scan range; if contention persists, implement a job dispatch service (single dispatcher process) that batches claim queries and distributes job IDs to workers via an in-memory channel, removing per-worker database claims; tune FILLFACTOR on the job table to 70% to reduce hot page contention on SKIP LOCKED
Tier 2: Priority Inversion Under Load
Signal: High-priority job queue depth growing despite workers available; low-priority batch jobs showing high throughput while transactional job latency (time from enqueue to execution start) p95 > 30s; worker pool metrics showing workers claiming jobs uniformly across priority levels rather than draining the high- priority queue first
Evolution: Separate worker pools per priority tier (e.g., dedicated transactional workers for high-priority jobs, shared workers for low-priority batch); or implement priority-weighted polling in a unified worker pool (poll high-priority queue N times before polling low-priority queue once, where N is the priority weight ratio); add high-priority job execution latency as a first-class SLA metric with alerting threshold separate from batch job latency
Tier 3: Temporal Workflow History Size
Signal: Temporal workflow worker memory usage growing with age of oldest active workflow; workflow replay time (on worker restart or task routing) > 5s for specific workflow types; Temporal UI showing workflow history event count > 10k for specific workflow instances; Temporal backing PostgreSQL storage growing disproportionately to active workflow count
Evolution: Implement Continue-As-New in long-running Temporal workflows to reset workflow history at safe checkpoints (typically every 1000–2000 events); use workflow signals sparingly in loops: each signal creates a history event; for workflows waiting on external events for > 24 hours, implement a timer-based wakeup with Continue-As-New rather than an open-ended wait; add workflow history size monitoring as an operational metric
Tier 4: PostgreSQL Job Table Storage and Query Pressure
Signal: PostgreSQL job table row count > 500M (including completed jobs not yet archived); autovacuum running continuously on job table; completed job retention queries (SELECT ... WHERE completed_at < NOW() - INTERVAL '7 days' DELETE) taking > 60s; job history query latency (for admin/audit queries on completed jobs) > 5s; PostgreSQL storage cost for job table exceeding budget
Evolution: Partition the job table by created_at date range (weekly or monthly partitions); implement automated partition archival: move completed partitions to cold storage (S3 as Parquet, queryable via Trino/Athena) after a retention window; keep only current + previous partition hot in PostgreSQL; this replaces row-level DELETE with partition-level DETACH + COPY, which completes in seconds vs. minutes; add FILLFACTOR 70 to the job table to leave room for in-place updates of status transitions without creating dead tuples on every row
Migration Path
In-process job execution (synchronous, within the same application process) → PostgreSQL-backed distributed job queue with Redis visibility leasing
Background jobs competing with user-facing API requests for application server resources (CPU, memory, thread pool); a long-running job blocking the application process for minutes; need for job retry on failure without application restart; need to scale job execution independently from the API request handling capacity
Single-worker-pool job queue (all jobs processed by one pool) → Priority-separated worker pools with dedicated transactional and batch pools
High-priority transactional jobs (e.g., payment processing, user account operations) delayed by low-priority batch jobs (e.g., report generation, data export) consuming all worker capacity; job execution latency SLA differentiated by job type but not enforceable with a single worker pool; need to independently autoscale high-priority workers without scaling batch workers
Simple job queue with single-step job execution → Temporal-orchestrated multi-step workflows for complex job pipelines
Multi-step jobs (e.g., ingest file → validate → transform → load → notify → archive) failing at step 3 and restarting from step 1 on retry, causing duplicate work and incorrect intermediate state; step failure debugging requiring full job log analysis with no visibility into individual step state; need to pause and resume long-running workflows based on external events (user approval, payment confirmation)
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has moderate operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 3 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.