Distributed Job Queue Platform

Summary

A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.

Problem Statement

Background job queues have a deceptively simple interface (enqueue job, execute job) that conceals significant distributed systems complexity. At-least-once execution semantics (the standard for durable queues) means a job may be executed more than once: when a worker crashes mid-execution, the job's visibility timeout expires and it is re-queued, potentially running again on a different worker. If the job is not idempotent, the duplicate execution causes incorrect state (double-charged customer, double-sent email, duplicate database record). Job visibility timeout is the central operational knob: too short, and long-running jobs self-interrupt before completion; too long, and a crashed worker leaves a job invisible to other workers for the full timeout period. Multi-step jobs that involve external API calls, file processing, or sequential database operations require workflow orchestration with explicit step checkpointing: otherwise a failure in step 4 of 5 restarts from step 1.