Distributed Job Queue Platform

Topology Notes

• ·The job enqueue API writes a job record to PostgreSQL with status='pending' and simultaneously writes an outbox event for downstream consumers (job completion subscribers). The outbox write is in the same transaction as the job INSERT, if the transaction rolls back, neither the job nor the outbox event is committed. This ensures that every successfully enqueued job generates exactly one downstream event on completion, with no dual-write consistency hazard.
• ·Workers claim jobs using SELECT FOR UPDATE SKIP LOCKED on the pending job table. SKIP LOCKED is critical: it allows multiple workers to query simultaneously without blocking each other on locked rows. Each claimed job is assigned a lease token stored in Redis (SET job_lease:{job_id} {worker_id} EX {visibility_timeout_seconds} NX). Workers must extend the lease heartbeat every visibility_timeout/3 seconds for long-running jobs. If a worker crashes, the Redis lease expires and the job becomes claimable again.
• ·The dead-letter mechanism is PostgreSQL-native: jobs that exhaust their retry budget have status set to 'dead_letter' with the final error recorded. A separate dead-letter worker provides manual replay capability (re-enqueue DLQ jobs after investigation). DLQ depth per job type is a monitored metric with alerting threshold at 100 messages, DLQ growth beyond this threshold triggers an incident. DLQ items are never auto-replayed without operator review.
• ·Temporal orchestrates multi-step workflows as a separate execution path alongside the core job queue. Temporal workflows call the same underlying job queue workers as activities, enabling reuse of existing worker logic within orchestrated pipelines. The Temporal server uses its own PostgreSQL database, isolated from the job queue PostgreSQL to prevent Temporal's internal state management from competing with job claim queries.
• ·Kafka receives job completion events from the outbox relay. Downstream consumers include: analytics (job throughput, latency, failure rate by type), billing triggers (job completion events that trigger downstream billing operations), and notification fan-out (notify users when their requested job completes). Kafka consumer groups for each downstream system are independent: a slow billing consumer does not delay notification fan-out for completed jobs.