Connection Leak

critical

Summary

Database or network connections are acquired from the pool but never returned, causing the pool to drain slowly over hours or days until exhausted, manifesting as a gradual degradation rather than a sudden spike.

Description

Unlike connection exhaustion from a traffic spike, which is sudden and correlates with load, connection leaks accumulate silently over time. The pool active connection count grows monotonically. Each leaked connection represents a connection acquired but never returned: typically because the code path that called connection.close() or used try-with-resources was not reached.

Common causes: - Exception path bypass: the connection is closed in the normal path but the

exception handler does not close it. Without try-with-resources or finally

block, any unchecked exception skips the close.

- ORM session not closed: JPA EntityManager or SQLAlchemy Session acquired in a

request and added to request context but not explicitly closed on error paths.

- Connection passed across thread boundaries: connection borrowed in one thread

and never returned when the thread dies or times out.

- Framework misconfiguration: Spring @Transactional on a method that does not

complete the transaction on all exit paths.

Diagnostic signals distinguishing leak from spike: - Traffic spike exhaustion: pool fills suddenly, correlates with request rate spike - Connection leak exhaustion: pool fills gradually over hours/days independent of load;

active count grows even during low-traffic periods

PostgreSQL diagnostic: SELECT pid, usename, application_name, state, query_start, state_change FROM pg_stat_activity WHERE state != 'active' ORDER BY state_change;connections idle for hours with no activity are likely leaked.

HikariCP leak detection: set leakDetectionThreshold to a value slightly above normal query duration (e.g., 2000ms). HikariCP logs a stack trace when a connection is held longer than the threshold, identifying the allocation site.

Characteristics

Propagationlinear

Time to detectWith active connection count monitoring, detectable within minutes as a monotonically increasing trend. Without monitoring, detected when pool exhaustion causes visible errors: typically hours to days after the leak begins. The gradual onset makes it easy to miss as normal growth.

Blast radiusAs leaked connections accumulate, the pool drains and new requests begin queuing at the pool boundary. After full exhaustion, the failure is equivalent to connection exhaustion: all database-dependent requests fail or time out. Unlike spike-driven exhaustion, the pool does not self-recover when traffic drops: leaked connections do not return to the pool on their own. The service must be restarted or connections forcibly terminated.

Triggers

·Exception path that bypasses connection.close() or try-with-resources
·ORM session not closed on error exit paths
·Connection borrowed in a worker thread that terminates abnormally
·Missing finally block around JDBC connection usage
·Framework-managed transaction that does not complete on all code paths

Detection Signals

connection exhaustionlog errors

Mitigation Strategies

Enable HikariCP leakDetectionThresholdcomplexity: low

Set leakDetectionThreshold (milliseconds) to slightly above the expected maximum connection hold time. HikariCP logs a stack trace identifying the connection acquisition callsite when a connection is held longer than the threshold. This identifies the leak location without requiring production reproduction.

Set connectionMaxLifetime to force periodic connection recyclingcomplexity: low

HikariCP connectionMaxLifetime (default 30 minutes) forces connections to be retired and replaced after the specified lifetime, even if not returned to the pool. This provides a safety valve: leaked connections are recycled at worst after the max lifetime, limiting leak accumulation. Does not fix the leak but bounds its impact.

Audit all connection usage with try-with-resourcespreventscomplexity: medium

Replace manual connection.close() calls with Java try-with-resources or equivalent (Python context managers, Go defer). This guarantees connection return on all exit paths including exceptions.

Monitor active connection count trend, not just current valuecomplexity: low

Alert not on absolute pool utilization but on a monotonically increasing trend over a 30-minute window during low-traffic periods. Spike-driven exhaustion follows traffic; leak-driven exhaustion grows during quiet periods.

Recovery Steps

1.Confirm leak via pg_stat_activity: look for idle connections accumulating over hours
2.Restart the application instance to return all connections to the pool (fast recovery)
3.Enable HikariCP leakDetectionThreshold to identify the leak callsite in logs
4.Search code for manual connection management without try-with-resources or finally blocks
5.After fix, verify pool active count stabilizes at baseline under normal traffic

Estimated recovery time: Immediate on service restart (seconds). The root cause (code leak) requires a code fix and deployment: typically hours to days depending on investigation time. connectionMaxLifetime provides a temporary bound on re-accumulation.

Affected Systems

Patterns

connection pooling

Technologies

postgresqlmysql

Basis

Common production failure with well-understood diagnostic path; HikariCP documentation covers leak detection explicitly