Lock Contention
criticalContainedcapacity failure · linear propagation
Blast Radius
0%
0/19 nodes
Time to Detect
Contention shows up in query latency percentiles within one to five minutes of percentile monitoring, and more directly in pg_locks: alerting on the count of non-granted locks, or on pg_stat_activity rows with wait_event_type = 'Lock', gives near-real-time detection. pg_blocking_pids(pid) names the transaction at the head of the queue. Without lock monitoring, contention is visible only as unexplained write-latency growth that tracks concurrency.
Preventive Mitigations
3
Confidence
StrongFailure Cascade
(no topology nodes mapped for this failure mode)
This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.
Severity at this step: critical
Detection Signals
Recovery time estimate: Immediate relief is seconds once the head-of-queue blocker is identified and terminated. Structural fixes (moving external calls out of transactions, sharding a hot counter, adopting SKIP LOCKED) are code and schema changes on a release cycle, hours to days to ship.
Mitigation Checklist(3 preventive, 1 reactive)
Split one hot counter into N rows and write to a random shard, spreading the writers across N locks so contention falls by roughly N. The cost moves to reads: the true value is now a SUM over N rows rather than a single lookup, and the schema and write path both change. N around 16 is a common balance between write relief and read cost.
Hold time is what turns a lock into a queue. Fetch external data before BEGIN, commit the database writes, then do external I/O afterward, so a lock is never held across a network round trip. This is the highest-leverage change because it shrinks every queue at once; the cost is refactoring code that currently spans a transaction around external work.
When many workers pull from a shared table and any available row will do, SKIP LOCKED lets each worker take a row no one else holds instead of queuing on a locked one, so the workers stop serializing entirely. It changes semantics rather than adding cost: a skipped row is claimed by another worker, which suits work-queue draining but not a case where one specific row must be handled here.
When the thing being serialized is a concept (a user session, a job id) rather than a specific row, pg_try_advisory_lock takes an application-defined lock without holding a row lock, and its non-blocking form lets the caller do other work instead of queuing. The cost is that the lock's meaning now lives in application code, not in the data.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.