Post-Mortem Framework · Capacity: Connection Leak

Connection Leak

SEV-2, Significant Impact

Linear propagation · capacity · Affects 0 scenario(s)

Severity Classification

Classified as CRITICAL based on failure mode severity.

Propagation Chain

Origin component

Connection Leak begins at the source component. Trigger: Exception path that bypasses connection.close() or try-with-resources.

Immediate (T+0) · Signal: Connection Exhaustion

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

With active connection count monitoring, detectable within minutes as a monotonically increasing trend. Without monitoring, detected when pool exhaustion causes visible errors: typically hours to days after the leak begins. The gradual onset makes it easy to miss as normal growth. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Blast Radius

As leaked connections accumulate, the pool drains and new requests begin queuing at the pool boundary. After full exhaustion, the failure is equivalent to connection exhaustion: all database-dependent requests fail or time out. Unlike spike-driven exhaustion, the pool does not self-recover when traffic drops: leaked connections do not return to the pool on their own. The service must be restarted or connections forcibly terminated.

Contributing Factors

Trigger Condition: Exception path that bypasses connection.close() or try-with-operational

This operational trigger enables Connection Leak: Exception path that bypasses connection.close() or try-with-resources

Trigger Condition: ORM session not closed on error exit pathsoperational

This operational trigger enables Connection Leak: ORM session not closed on error exit paths

Trigger Condition: Connection borrowed in a worker thread that terminates abnoroperational

This operational trigger enables Connection Leak: Connection borrowed in a worker thread that terminates abnormally

Remediation Plan

ImmediateConfirm leak via pg_stat_activity: look for idle connections accumulating over h

Confirm leak via pg_stat_activity: look for idle connections accumulating over hours

Effort: Minutes to hours (on-call response)

ImmediateRestart the application instance to return all connections to the pool (fast rec

Restart the application instance to return all connections to the pool (fast recovery)

Effort: Minutes to hours (on-call response)

ImmediateEnable HikariCP leakDetectionThreshold to identify the leak callsite in logs

Enable HikariCP leakDetectionThreshold to identify the leak callsite in logs

Effort: Minutes to hours (on-call response)

Short-TermEnable HikariCP leakDetectionThreshold

Set leakDetectionThreshold (milliseconds) to slightly above the expected maximum connection hold time. HikariCP logs a stack trace identifying the connection acquisition callsite when a connection is held longer than the threshold. This identifies the leak location without requiring production reproduction.

Effort: 1 day to 1 week

Short-TermSet connectionMaxLifetime to force periodic connection recycling

HikariCP connectionMaxLifetime (default 30 minutes) forces connections to be retired and replaced after the specified lifetime, even if not returned to the pool. This provides a safety valve: leaked connections are recycled at worst after the max lifetime, limiting leak accumulation. Does not fix the leak but bounds its impact.

Effort: 1 day to 1 week

Short-TermAudit all connection usage with try-with-resources

Replace manual connection.close() calls with Java try-with-resources or equivalent (Python context managers, Go defer). This guarantees connection return on all exit paths including exceptions.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: connection exhaustion, log errors. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermArchitecture review for Connection Leak resilience

Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.

Effort: 1–2 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.