Transactional Outbox Pattern
matureSummary
Atomically persist a business state change and its outgoing event in one database transaction, then relay that event to the message broker asynchronously. The event and the state it describes commit or roll back together, which closes the dual-write gap between database and broker.
Problem
A service that writes state to a database and must publish events to a broker faces a dual-write consistency gap: either the event is lost after a crash, or a spurious event is published when the write fails after the publish. Neither is acceptable for financial, inventory, or event-driven architectures.
Description
The dual-write problem. Writing to a database and then to a message broker in sequence (write DB, then publish to Kafka) is not atomic. If the process crashes after the DB write but before the publish, the event is lost. If the publish happens first and the DB write then fails, an event exists for a state change that never happened. There is no ordering of the two writes that makes them atomic, because they are two separate systems with no shared transaction.
The outbox pattern moves the second write into the first system. Within the same database transaction that commits the business state change, the application inserts a row into an outbox table describing the event. Because both writes are in one transaction, they share its atomicity: commit makes both durable, rollback leaves neither. The broker is no longer in the critical path of the state change, so a broker outage cannot fail or corrupt the write.
A separate relay process then reads committed outbox rows and publishes them to the broker. This is the key point that makes the pattern correct: the relay only ever sees rows from transactions that already committed, so it never publishes an event for a rolled-back change. There are two relay forms. A polling relay runs SELECT from the outbox where published is false, ordered by id, and marks or deletes rows once the broker acknowledges them; it is simple but adds query load and latency equal to its poll interval. A change-data-capture relay (Debezium is the common choice) reads committed changes from the PostgreSQL write-ahead log through logical decoding, which requires wal_level = logical and a logical replication slot; it adds sub-second latency and no polling load.
The relay guarantees at-least-once delivery: if it crashes after publishing but before recording success, it republishes on restart. Consumers must therefore be idempotent, for example with the inbox pattern, which together give effectively-once processing. Exactly-once delivery is not what the outbox provides, and claiming it would be an overclaim.
The logical replication slot is the main operational hazard of the CDC relay, and it is easy to miss. A slot pins the WAL: PostgreSQL cannot recycle any WAL segment past the slot's restart_lsn until the consumer advances it. If the Debezium consumer stops or falls far behind, WAL accumulates on the primary and can fill the disk and take the database down, an outage caused by a dead consumer rather than the database itself. pg_replication_slots must be monitored and abandoned slots dropped.
Ordering is set by the relay, not the outbox table. The broker sees events in the order the relay publishes them. A single ordered relay preserves per-key order; parallel relays or multiple partitions can reorder events, so when order matters the relay must key events onto broker partitions deliberately (for example by aggregate id) rather than assume the outbox insert order survives.
Outbox cleanup is an operational responsibility. Published rows must be deleted or archived on a schedule; an unbounded outbox table degrades the relay's own queries and consumes disk.
Tradeoffs
Guaranteed at-least-once delivery; event loss is impossible after commit
Broker unavailability does not fail primary writes; the outbox accumulates and drains later
The extra outbox INSERT in the transaction adds negligible latency (<1ms)
Relay process, outbox cleanup, and (for CDC) a replication slot add operational surface
Relay lag means events are not instant: CDC relay under 100ms, polling relay seconds
At-least-once delivery requires idempotent consumers
When to use
Service writes state to a database and must publish events to a broker
The outbox pattern is designed for exactly this topology; it closes the consistency gap with a single database transaction.
Event loss is unacceptable (financial events, order state changes, CDC feeds)
The transactional guarantee ensures every committed state change produces exactly one outbox row, and the relay guarantees eventual publication.
Application already uses PostgreSQL or another ACID-capable database
The pattern requires a transaction covering both the business write and the outbox insert; without ACID transactions that atomicity cannot be made.
Message consumers can be made idempotent
At-least-once relay delivery means consumers must handle duplicates; a non-idempotent consumer would need exactly-once delivery, which the outbox does not provide.
When not to use
Events must be published within single-digit milliseconds of the state change
Relay lag (polling: the poll interval; CDC: under 100ms but not zero) adds latency; direct broker writes are faster if the consistency gap can be tolerated.
Database does not support transactions (e.g., DynamoDB single-item)
Without a transaction covering both the business write and the outbox insert, the atomicity guarantee is lost.
Event volume is so high that the outbox table becomes a write bottleneck
Above ~100,000 events/second a single outbox table is a hotspot; partition it by aggregate type, or use native broker transactions instead.
Operational Requirements
Deploy the outbox relay with redundancy and monitor relay lag
If the relay stops, the outbox accumulates; alert when the outbox has unpublished rows older than 30 seconds, and ensure a restarted relay resumes from the last published row.
Monitor logical replication slot lag when using a CDC relay
A stopped or lagging CDC consumer leaves its slot behind, pinning WAL past restart_lsn until it advances. WAL then accumulates and can fill the disk. Alert on pg_replication_slots lag and drop abandoned slots.
Purge published outbox rows on a schedule
Accumulated published rows degrade the relay's queries; purge with a daily DELETE WHERE published_at < NOW() - INTERVAL '24 hours'.
Ensure all message consumers are idempotent
The relay guarantees at-least-once; a consumer that processes a duplicate must produce the same result as processing it once. See the inbox pattern.
Index the outbox table on (published, created_at) for efficient relay queries
A polling relay issues frequent WHERE published = false ORDER BY created_at queries; without this index they degrade as the table grows.
Characteristics
Technologies
Canonical
Alternatives
Relationships
Evolves to
Complements
Basis
Well-established solution to the dual-write problem; the at-least-once relay, idempotency requirement, and CDC replication-slot WAL-retention hazard are real and operationally significant.
Implementation Playbooks
Related Architecture Knowledge
Outbound: this entity affects
The outbox pattern eliminates split-brain between a database write and a message broker publish by writing both the domain record and the outbox event in a single ACID transaction, ensuring events are published if and only if the database write committed.
Tradeoffs
- ·Adds ~1ms write overhead per transaction for the outbox INSERT
- ·Relay is a single point of failure: relay high availability requires careful deployment
- ·At-least-once delivery means consumers must handle duplicate events: idempotency key required
Inbound: affects this entity
Fat events produced via the outbox pattern carry entity state that consumers can use to update projections without calling back to the source service.
Full relationship →The outbox pattern guarantees at-least-once delivery from producer to broker; the inbox pattern guarantees idempotent processing at the consumer. Together they provide end-to-end exactly-once processing.
Full relationship →Write-heavy transactional workloads that emit downstream events (order placed, payment captured) benefit from the outbox pattern to ensure events are published exactly when the database transaction commits: never before, never after.
Tradeoffs
- ·Adds one INSERT per transaction to the outbox table: minor but nonzero write amplification
- ·Relay is an additional component to maintain and monitor
- ·At-least-once delivery requires all downstream consumers to be idempotent
Used In Architecture Scenarios
Financial Ledger
An append-only audit log architecture for capturing system actions: user operations, data access events, configuration changes, and financial operations: with cryptographic integrity chaining, immutable storage, and dual-path query serving. PostgreSQL stores the authoritative append-only event log in time-partitioned tables; ClickHouse serves historical aggregate queries and retention analytics; Kafka streams audit events in real time to SIEM integrations and security dashboards; Redis caches hot audit query results for compliance-facing read paths. No record is ever updated or deleted: only inserts are permitted. Each record includes a hash of the previous record, forming a tamper-evident chain verifiable without external tooling.
Multi-Tenant SaaS
A multi-tenant developer tooling platform providing CI/CD pipeline execution, log aggregation, code analysis, and dependency scanning across isolated tenant organizations. Tenant isolation is the primary correctness constraint: a security boundary violation between tenants is a critical incident, not a performance event. PostgreSQL row-level security enforces data isolation; Redis manages job queues and distributed locks; Elasticsearch indexes pipeline log output for search; Kafka delivers webhook events to tenant-registered endpoints; MinIO stores pipeline artifacts. Resource quota enforcement prevents any single tenant's burst from affecting others.
Write-Heavy Application
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Financial Ledger
A strict consistency financial architecture for double-entry accounting, payment processing, and audit trail management where every debit must have a corresponding credit, every state transition must be ordered and durable, and the complete history must be reconstructable without gaps. Event sourcing provides an append-only audit log; the outbox pattern guarantees at-least-once downstream event delivery without distributed two-phase commit (idempotent consumers deduplicate on event ID for effectively-once processing); PostgreSQL provides ACID guarantees for ledger entry atomicity.
Realtime Collaboration
A real-time location tracking platform for moving entities: vehicles, delivery couriers, field workers, and assets: receiving position updates at 1–30 second intervals from thousands of simultaneously tracked objects. Redis geospatial indexes (GEOADD / GEOSEARCH) serve sub-10ms proximity queries against the live position surface; PostgreSQL stores durable entity metadata and historical location records; TimescaleDB stores high-frequency time-series location data with automatic hypertable chunking and retention policies; Kafka streams location events to downstream consumers (dispatch systems, customer tracking apps, analytics pipelines). Geofence evaluation runs at ingestion time: each incoming position update is tested against active geofences for the entity's region, and geofence entry/exit events are emitted as Kafka messages to downstream consumers.
Financial Ledger
An electronic health record (EHR) architecture built around strict auditability, HIPAA compliance, and append-only correctness. Clinical records are mutable by design (amendments, addenda) but corrections must be explicitly attributed, not silently overwritten. Event sourcing provides a reconstructable audit log; PostgreSQL row-level security enforces patient-level access control at the database layer; Kafka streams HL7 FHIR events to downstream clinical systems. The architecture must support breach detection, access auditing, and state reconstruction at any historical point: not just current state retrieval.
Event-Driven System
A multi-channel notification delivery architecture that accepts upstream business events (order placed, payment received, comment posted, threshold alert triggered) and routes them to per-channel delivery workers (push via FCM/APNs, email via SendGrid, SMS via Twilio, in-app via WebSocket). Kafka carries raw business events from upstream producers. RabbitMQ handles per-channel fan-out with separate exchanges and queues per delivery channel, isolating email queue backlog from push notification delivery. PostgreSQL provides durable notification state tracking (sent, failed, bounced, suppressed). Redis enforces per-user rate limiting (notification frequency caps to prevent fatigue) and stores deduplication tokens to prevent duplicate sends across retry attempts. The inbox pattern on the consumer side ensures idempotent delivery even when Kafka produces duplicate events.
Event-Driven System
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.
Write-Heavy Application
A high-volume transactional write architecture anchored on PostgreSQL, where write throughput, durability guarantees, and audit completeness must coexist. The outbox pattern ensures reliable event publishing to Kafka without two-phase commit, and WAL-based CDC provides a durable change log that can reconstruct system state. Connection pooling via PgBouncer bounds connection overhead at the database layer.