Connection Timeout Storm
SEV-2, Significant ImpactFeedback Loop propagation · cascading · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The feedback loop propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Connection Timeout Storm begins at the source component. Trigger: Downstream database enters GC pause, lock contention, or I/O saturation.
Immediate (T+0) · Signal: Latency Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Connection pool saturation signals (pool wait queue depth) appear within seconds of the downstream slowdown. Application-level error rate spikes within seconds to a minute. Without pool saturation monitoring, detected through user-visible errors after pool exhaustion completes. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Self-amplification loop
Retry behavior and back-pressure cause the failure to amplify. Resource exhaustion accelerates, the system cannot self-recover without intervention.
Minutes after fan-out · Signal: Cascading alert storms; all downstream indicators deteriorating
Blast Radius
Connection pool exhaustion affects all requests to the upstream service, not just those related to the slow downstream operation. If the service uses one pool for all database operations, a slow query for one feature exhausts the pool for all features. The blast radius expands from the specific slow call path to total service unavailability.
Contributing Factors
This operational trigger enables Connection Timeout Storm: Downstream database enters GC pause, lock contention, or I/O saturation
This operational trigger enables Connection Timeout Storm: External API dependency becomes slow (rate limiting, upstream overload)
This operational trigger enables Connection Timeout Storm: Network latency to downstream increases significantly
Remediation Plan
Confirm it is a timeout storm vs traffic spike: check downstream query duration (pg_stat_activity), not just pool utilization
Effort: Minutes to hours (on-call response)
Terminate long-running downstream queries: SELECT pg_terminate_backend(pid) WHERE query_duration > '10s'
Effort: Minutes to hours (on-call response)
Temporarily reduce inbound request rate to reduce downstream load (feature flag, load shedding)
Effort: Minutes to hours (on-call response)
A circuit breaker stops forwarding calls to the downstream when it is detected as slow. Instead of accumulating N connections waiting 30 seconds each, the circuit opens and all callers fail fast in microseconds. The downstream receives no new load during the open window, allowing recovery. This is the primary prevention for connection timeout storms.
Effort: 1 day to 1 week
If the service SLO is 500ms p99, a 30-second query timeout provides no protection: the user already failed at 500ms. Set timeouts to 2–3× the expected normal latency (e.g., 200ms for a 20ms query). Shorter timeouts reduce the connection hold duration, limiting pool exhaustion.
Effort: 1 day to 1 week
Pool utilization at 90% is acceptable under normal load. Pool queue depth > 0 means requests are waiting for a connection: an early warning that the pool cannot serve the current request rate at the current downstream latency. Alert on queue depth, not just utilization.
Effort: 1 day to 1 week
Configure alerts for: latency spike, connection exhaustion, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.