Connection Pool Exhaustion
SEV-2, Significant ImpactFan-Out propagation · capacity · Affects 5 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 5 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Connection Pool Exhaustion begins at the source component. Trigger: Slow query or lock contention holds connections longer than expected.
Immediate (T+0) · Signal: Queue Depth
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Seconds if pool saturation alerting is configured. Minutes if only detected through latency monitors or user reports. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
All application requests that require a database connection are blocked. Typically affects the entire application. May cascade to upstream services if they retry exhausted requests.
Contributing Factors
This operational trigger enables Connection Pool Exhaustion: Slow query or lock contention holds connections longer than expected
This operational trigger enables Connection Pool Exhaustion: Traffic spike without corresponding pool expansion
This operational trigger enables Connection Pool Exhaustion: App replica count scaled up without adjusting total pool ceiling
Mitigation Gaps
Add 'connection pooling' to the runbook. A connection pool bounds the total database connections an application can open, preventing connection storms during traffic spikes and protecting the database server from exceeding its connection limit.
Add 'pgbouncer' to the runbook. PgBouncer multiplexes many client connections onto a small pool of PostgreSQL server connections, directly preventing connection exhaustion by bounding the number of server connections regardless of client count.
Remediation Plan
Identify query holding connections: check pg_stat_activity for long-running queries
Effort: Minutes to hours (on-call response)
Terminate blocking queries if safe: SELECT pg_terminate_backend(pid)
Effort: Minutes to hours (on-call response)
Reduce app replica count temporarily to shed connection pressure
Effort: Minutes to hours (on-call response)
Configure both the connection pool timeout (how long to wait for a connection) and the query timeout (how long a query can run). This prevents connections from being held indefinitely.
Effort: 1 day to 1 week
Total connections across all app replicas must not exceed database max_connections minus administrative headroom (typically 10–20%). pool_size_per_replica × replica_count < max_connections × 0.8
Effort: 1 day to 1 week
Open the circuit when pool wait time exceeds threshold, failing fast rather than queuing requests that will time out anyway.
Effort: 1 day to 1 week
Configure alerts for: queue depth, latency spike, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Connection Pool Exhaustion affects 5 architecture scenarios (API Gateway Platform, Gaming Backend Platform, Multi-Tenant SaaS Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.