Lock Contention
criticalSummary
Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.
Description
PostgreSQL uses multi-version concurrency control, so readers never block writers and writers never block readers: a reader sees the last committed version while a writer builds a new one. Contention is therefore a writer-versus-writer problem. To update a row, a transaction takes an exclusive row lock and holds it until commit or rollback. A second transaction updating the same row blocks until the first finishes. The database is doing no more work, but the second transaction's latency now includes the first's entire remaining duration.
This serialization is the whole failure. Writers to one row form a queue, and the Nth writer waits behind the N-1 ahead of it, so its latency is the sum of their hold times, not the cost of its own statement. A hot counter row (inventory quantity, account balance, a global sequence) is the worst case: at a thousand concurrent updates to a single row, p99 latency moves from roughly 1ms for a lone writer to over a second, purely from queuing, while committed throughput stops rising because the writes execute one at a time no matter how many are offered.
What sits inside each transaction sets the hold time, and long transactions widen the whole queue. A transaction that makes an external call (a payment processor, an email send) between acquiring a row lock and committing holds that lock for the round trip, so every writer behind it waits for a network call it has nothing to do with. The contention is not the lock itself but how long it is held, which is why keeping external I/O and slow computation outside the transaction boundary is the highest-leverage fix.
Contention also appears at table scope through DDL. ALTER TABLE and VACUUM FULL take an ACCESS EXCLUSIVE lock that conflicts with every read and write. That request queues behind the current transactions on the table, and because it is exclusive, every statement that arrives after it queues behind it in turn. A thirty-minute rewrite can lock out all traffic to the table for its full duration, so a background migration becomes a foreground outage.
The secondary blast radius is the connection pool. A blocked writer is not idle from the pool's point of view: it holds its connection while it waits. Hundreds of writers stalled on one hot row hold hundreds of connections, and once the pool is exhausted, requests that would never have touched the contended row cannot get a connection at all, so contention on a single row degrades endpoints that have nothing to do with it. When contending writers also acquire more than one lock in inconsistent orders, the same queue can close into a cycle and surface as a deadlock; that is a distinct, co-occurring failure, covered under deadlock.
Characteristics
Triggers
- ·High-concurrency updates to the same row (counter increment, inventory decrement, balance update)
- ·Long transactions that hold row locks across external API calls or network operations
- ·DDL operations (ALTER TABLE, VACUUM FULL) taking an ACCESS EXCLUSIVE lock under live traffic
- ·Batch jobs updating rows concurrently with OLTP transactions on the same table
Detection Signals
Mitigation Strategies
Hold time is what turns a lock into a queue. Fetch external data before BEGIN, commit the database writes, then do external I/O afterward, so a lock is never held across a network round trip. This is the highest-leverage change because it shrinks every queue at once; the cost is refactoring code that currently spans a transaction around external work.
When many workers pull from a shared table and any available row will do, SKIP LOCKED lets each worker take a row no one else holds instead of queuing on a locked one, so the workers stop serializing entirely. It changes semantics rather than adding cost: a skipped row is claimed by another worker, which suits work-queue draining but not a case where one specific row must be handled here.
Split one hot counter into N rows and write to a random shard, spreading the writers across N locks so contention falls by roughly N. The cost moves to reads: the true value is now a SUM over N rows rather than a single lookup, and the schema and write path both change. N around 16 is a common balance between write relief and read cost.
When the thing being serialized is a concept (a user session, a job id) rather than a specific row, pg_try_advisory_lock takes an application-defined lock without holding a row lock, and its non-blocking form lets the caller do other work instead of queuing. The cost is that the lock's meaning now lives in application code, not in the data.
Recovery Steps
- 1.Find the waiters: SELECT pid, wait_event, query FROM pg_stat_activity WHERE wait_event_type = 'Lock' AND state = 'active'
- 2.Find the blocker at the head of the queue: SELECT pg_blocking_pids(pid) FROM pg_stat_activity WHERE wait_event_type = 'Lock'
- 3.If the blocker is stuck on an external call inside its transaction, terminate it: SELECT pg_terminate_backend(blocking_pid)
- 4.After immediate relief, audit the contended path for external calls held inside the transaction
- 5.Address the structural cause: shorten transactions, sharded counters for hot rows, or SKIP LOCKED for queue tables
Estimated recovery time: Immediate relief is seconds once the head-of-queue blocker is identified and terminated. Structural fixes (moving external calls out of transactions, sharding a hot counter, adopting SKIP LOCKED) are code and schema changes on a release cycle, hours to days to ship.
Affected Systems
Patterns
Technologies
Basis
PostgreSQL row-locking and MVCC writer-versus-writer behavior is precisely specified and observable through pg_locks and pg_stat_activity; hot-row queuing, the connection-pool secondary failure, and sharded-counter relief are well documented and reproducible.
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
Write-heavy transactional workloads amplify lock contention: many concurrent writers contend for row-level locks on the same records (e.g., shared account balances, inventory counts), causing transactions to queue, latency to spike, and throughput to plateau well below hardware limits.
Tradeoffs
- ·Optimistic locking (check-and-compare) reduces lock duration but increases retry rate under high contention
- ·Saga pattern eliminates distributed locks but introduces compensating transactions
- ·Short transactions reduce lock hold time but increase commit overhead at high throughput
Used In Architecture Scenarios
Financial Ledger
An append-only audit log architecture for capturing system actions: user operations, data access events, configuration changes, and financial operations: with cryptographic integrity chaining, immutable storage, and dual-path query serving. PostgreSQL stores the authoritative append-only event log in time-partitioned tables; ClickHouse serves historical aggregate queries and retention analytics; Kafka streams audit events in real time to SIEM integrations and security dashboards; Redis caches hot audit query results for compliance-facing read paths. No record is ever updated or deleted: only inserts are permitted. Each record includes a hash of the previous record, forming a tamper-evident chain verifiable without external tooling.
Write-Heavy Application
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Financial Ledger
A strict consistency financial architecture for double-entry accounting, payment processing, and audit trail management where every debit must have a corresponding credit, every state transition must be ordered and durable, and the complete history must be reconstructable without gaps. Event sourcing provides an append-only audit log; the outbox pattern guarantees at-least-once downstream event delivery without distributed two-phase commit (idempotent consumers deduplicate on event ID for effectively-once processing); PostgreSQL provides ACID guarantees for ledger entry atomicity.
Financial Ledger
An electronic health record (EHR) architecture built around strict auditability, HIPAA compliance, and append-only correctness. Clinical records are mutable by design (amendments, addenda) but corrections must be explicitly attributed, not silently overwritten. Event sourcing provides a reconstructable audit log; PostgreSQL row-level security enforces patient-level access control at the database layer; Kafka streams HL7 FHIR events to downstream clinical systems. The architecture must support breach detection, access auditing, and state reconstruction at any historical point: not just current state retrieval.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.
Write-Heavy Application
A high-volume transactional write architecture anchored on PostgreSQL, where write throughput, durability guarantees, and audit completeness must coexist. The outbox pattern ensures reliable event publishing to Kafka without two-phase commit, and WAL-based CDC provides a durable change log that can reconstruct system state. Connection pooling via PgBouncer bounds connection overhead at the database layer.