Deadlock
SEV-2, Significant ImpactIsolated propagation · concurrency · Affects 3 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. This failure mode appears in 3 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Deadlock begins at the source component. Trigger: Concurrent transactions acquiring locks on the same rows in different orders.
Immediate (T+0) · Signal: Error Rate Spike
Blast Radius
A single deadlock touches only the transactions in the cycle: one is aborted and must retry. The blast radius grows through the application's response, not the database's. If retries fire immediately without backoff, the same cycle reforms and deadlock frequency can climb into the hundreds per second on a contested resource, turning a rare abort into a sustained error rate and elevated latency for the affected feature. If deadlock errors are not handled at all and surface to users as raw failures, the radius reaches the user directly.
Contributing Factors
Financial transaction workloads are vulnerable to deadlocks when concurrent transactions acquire locks on the same account or balance rows in different orders.
CockroachDB is known to be susceptible to Deadlock under high operational burden. Using monotonically increasing primary keys (SERIAL, SEQUENCE) in high-write tables: all inserts target the same 'hot end' range. Use UUID v4 or hash-prefixed keys to distribute write load
MySQL is known to be susceptible to Deadlock under medium operational burden. Not deploying ProxySQL: direct application connections to MySQL primary cause connection storms during failover and cannot route reads to replicas
This operational trigger enables Deadlock: Concurrent transactions acquiring locks on the same rows in different orders
This operational trigger enables Deadlock: InnoDB gap or next-key locking under range DELETE or UPDATE with concurrent INSERT into the range
This operational trigger enables Deadlock: Foreign key cascade operations concurrent with direct updates to the same parent or child rows
Remediation Plan
Read the database deadlock log to identify the exact statements and the rows or index ranges in the cycle
Effort: Minutes to hours (on-call response)
Determine which lock-ordering, gap-lock, or foreign-key-cascade pattern produced the cycle
Effort: Minutes to hours (on-call response)
Enforce consistent lock ordering on every code path that touches the contested resource
Effort: Minutes to hours (on-call response)
Establish a canonical order for acquiring locks on multiple rows, for example sorting by primary key before locking. If every path acquires locks in the same order, a cycle cannot form. The cost is discipline that lives in application code and code review, and the limit is coverage: it prevents lock-order-inversion deadlocks but not gap-lock or foreign-key-cascade cycles, and it is unworkable when the lock set is discovered mid-transaction rather than known up front.
Effort: 1 day to 1 week
On a deadlock error, roll back and retry after a short randomized delay (roughly 100 to 500ms with jitter), bounded to 3 to 5 attempts before surfacing the error. Backoff spaces the contenders out so they stop reforming the same cycle. This does not reduce the deadlock rate, it absorbs it: cost moves into added tail latency, and the transaction must be safe to replay, since a retried write runs again from the start.
Effort: 1 day to 1 week
Shorter transactions hold locks for less time, shrinking the window a cycle can form in. Move external calls and computation outside the transaction, and acquire locks as late as possible. The cost is refactoring, and the limit is that a business operation genuinely needing several locks held together cannot shed them.
Effort: 1 day to 1 week
Configure alerts for: error rate spike, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Deadlock affects 3 architecture scenarios (Distributed Job Queue Platform, E-Commerce Order Platform, Healthcare Records Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.