Lock Contention
SEV-2, Significant ImpactLinear propagation · capacity · Affects 7 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. This failure mode appears in 7 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Lock Contention begins at the source component. Trigger: High-concurrency updates to the same row (counter increment, inventory decrement, balance update).
Immediate (T+0) · Signal: Latency Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Contention shows up in query latency percentiles within one to five minutes of percentile monitoring, and more directly in pg_locks: alerting on the count of non-granted locks, or on pg_stat_activity rows with wait_event_type = 'Lock', gives near-real-time detection. pg_blocking_pids(pid) names the transaction at the head of the queue. Without lock monitoring, contention is visible only as unexplained write-latency growth that tracks concurrency. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Blast Radius
Direct impact is confined to the contended rows or table: writers there queue and their tail latency climbs. The radius widens through the connection pool. A hot row can hold hundreds of connections in its wait queue, and once the pool is exhausted those connections are unavailable to unrelated queries, so latency and errors spread to endpoints that never touched the contended row. Table-level DDL contention widens it further, blocking every user of the table for the lock's duration.
Contributing Factors
Write-heavy transactional workloads amplify lock contention: many concurrent writers contend for row-level locks on the same records (e.g., shared account balances, inventory counts), causing transactions to queue, latency to spike, and throughput to plateau well below hardware limits.
This operational trigger enables Lock Contention: High-concurrency updates to the same row (counter increment, inventory decrement, balance update)
This operational trigger enables Lock Contention: Long transactions that hold row locks across external API calls or network operations
This operational trigger enables Lock Contention: DDL operations (ALTER TABLE, VACUUM FULL) taking an ACCESS EXCLUSIVE lock under live traffic
Remediation Plan
Find the waiters: SELECT pid, wait_event, query FROM pg_stat_activity WHERE wait_event_type = 'Lock' AND state = 'active'
Effort: Minutes to hours (on-call response)
Find the blocker at the head of the queue: SELECT pg_blocking_pids(pid) FROM pg_stat_activity WHERE wait_event_type = 'Lock'
Effort: Minutes to hours (on-call response)
If the blocker is stuck on an external call inside its transaction, terminate it: SELECT pg_terminate_backend(blocking_pid)
Effort: Minutes to hours (on-call response)
Hold time is what turns a lock into a queue. Fetch external data before BEGIN, commit the database writes, then do external I/O afterward, so a lock is never held across a network round trip. This is the highest-leverage change because it shrinks every queue at once; the cost is refactoring code that currently spans a transaction around external work.
Effort: 1 day to 1 week
When many workers pull from a shared table and any available row will do, SKIP LOCKED lets each worker take a row no one else holds instead of queuing on a locked one, so the workers stop serializing entirely. It changes semantics rather than adding cost: a skipped row is claimed by another worker, which suits work-queue draining but not a case where one specific row must be handled here.
Effort: 1 day to 1 week
When the thing being serialized is a concept (a user session, a job id) rather than a specific row, pg_try_advisory_lock takes an application-defined lock without holding a row lock, and its non-blocking form lets the caller do other work instead of queuing. The cost is that the lock's meaning now lives in application code, not in the data.
Effort: 1 day to 1 week
Configure alerts for: latency spike, queue depth, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Split one hot counter into N rows and write to a random shard, spreading the writers across N locks so contention falls by roughly N. The cost moves to reads: the true value is now a SUM over N rows rather than a single lookup, and the schema and write path both change. N around 16 is a common balance between write relief and read cost.
Effort: 1–4 sprints
Lock Contention affects 7 architecture scenarios (Audit and Compliance Platform, Distributed Job Queue Platform, E-Commerce Order Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.