Deadlock
criticalContainedconcurrency failure · isolated propagation
Blast Radius
5%
1/19 nodes
Time to Detect
The database logs a deadlock the moment it resolves one (PostgreSQL: ERROR: deadlock detected, with the conflicting statements listed when log_lock_waits is on; MySQL: the LATEST DETECTED DEADLOCK section of SHOW ENGINE INNODB STATUS). Application monitoring catches the spike within 30 to 60 seconds only if deadlock errors are tracked as a distinct category. Many deadlock spikes go unnoticed because the application retries successfully and only the database log records that anything happened.
Preventive Mitigations
2
Confidence
StrongImpacted Components
PostgreSQL
primary datastore · Directly connected to failure mode 'Deadlock' via risk propagation path in the topology.
Failure Cascade
PostgreSQL
Deadlock (isolated propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: Each individual deadlock resolves automatically: PostgreSQL within one deadlock_timeout, InnoDB effectively immediately. Application retry adds roughly 100 to 500ms per attempt. Bringing a sustained deadlock rate down is a code change on a release cycle, and the rate typically drops as soon as consistent lock ordering ships.
Mitigation Checklist(2 preventive, 2 reactive)
For work-queue access where any available row will do, SKIP LOCKED skips rows already locked by other workers instead of waiting, so competing workers never form a wait cycle. It changes semantics rather than adding cost: a skipped row is simply handled by someone else, so it fits "claim any pending row" but not "this specific row must be processed here", and it does nothing for multi-row transactional updates.
Establish a canonical order for acquiring locks on multiple rows, for example sorting by primary key before locking. If every path acquires locks in the same order, a cycle cannot form. The cost is discipline that lives in application code and code review, and the limit is coverage: it prevents lock-order-inversion deadlocks but not gap-lock or foreign-key-cascade cycles, and it is unworkable when the lock set is discovered mid-transaction rather than known up front.
Shorter transactions hold locks for less time, shrinking the window a cycle can form in. Move external calls and computation outside the transaction, and acquire locks as late as possible. The cost is refactoring, and the limit is that a business operation genuinely needing several locks held together cannot shed them.
On a deadlock error, roll back and retry after a short randomized delay (roughly 100 to 500ms with jitter), bounded to 3 to 5 attempts before surfacing the error. Backoff spaces the contenders out so they stop reforming the same cycle. This does not reduce the deadlock rate, it absorbs it: cost moves into added tail latency, and the transaction must be safe to replay, since a retried write runs again from the start.
Affected Systems
Workloads
Technologies
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.