Connection Pool Exhaustion
criticalSummary
All database connections in the pool are in use; new requests queue and then time out, causing cascading latency and errors across all dependent services.
Description
When every connection in the pool is held: whether by slow queries, lock waits, or a traffic burst: new requests cannot acquire a connection and queue at the pool boundary. Queue depth grows. Request latency rises sharply. Timeouts begin firing. Upstream services begin failing or retrying, amplifying the traffic. The database itself may appear healthy while the application is completely unavailable.
This failure is common when: - A single slow query holds connections for many seconds - A traffic spike overwhelms the pool without back-pressure - A downstream dependency (external API) causes queries to slow - Autoscaling of app replicas outpaces pool size expectations
Characteristics
Triggers
- ·Slow query or lock contention holds connections longer than expected
- ·Traffic spike without corresponding pool expansion
- ·App replica count scaled up without adjusting total pool ceiling
- ·Downstream dependency slowdown causes queries to hold connections
Detection Signals
Mitigation Strategies
Configure both the connection pool timeout (how long to wait for a connection) and the query timeout (how long a query can run). This prevents connections from being held indefinitely.
Total connections across all app replicas must not exceed database max_connections minus administrative headroom (typically 10–20%). pool_size_per_replica × replica_count < max_connections × 0.8
Open the circuit when pool wait time exceeds threshold, failing fast rather than queuing requests that will time out anyway.
PgBouncer multiplexes thousands of client connections to a small server-side pool. App replicas connect to PgBouncer (cheap), which routes to PostgreSQL (expensive).
Recovery Steps
- 1.Identify query holding connections: check pg_stat_activity for long-running queries
- 2.Terminate blocking queries if safe: SELECT pg_terminate_backend(pid)
- 3.Reduce app replica count temporarily to shed connection pressure
- 4.Check for lock waits: SELECT * FROM pg_locks WHERE NOT granted
- 5.After recovery, review pool configuration and add PgBouncer if not present
Estimated recovery time: Seconds once blocking queries are terminated. Minutes if waiting for natural query completion.
Affected Systems
Patterns
Technologies
Basis
One of the most common production database failure modes; well documented
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
A connection pool bounds the total database connections an application can open, preventing connection storms during traffic spikes and protecting the database server from exceeding its connection limit.
Tradeoffs
- ·Pooler becomes a new single point of failure if not replicated
- ·Transaction-mode pooling incompatible with LISTEN/NOTIFY or explicit transactions across requests
- ·Pool saturation under extreme load shifts the bottleneck to the pool queue
PgBouncer multiplexes many client connections onto a small pool of PostgreSQL server connections, directly preventing connection exhaustion by bounding the number of server connections regardless of client count.
Full relationship →Rate limiting bounds the inbound request rate per identity, preventing any single caller from consuming the entire connection pool and exhausting capacity for other callers.
Full relationship →Redis clients hold persistent TCP connections per thread or goroutine. Under connection pool misconfiguration or sudden traffic spikes, the Redis server can exhaust its maxclients limit, causing cascading cache misses that amplify load on the primary database.
Tradeoffs
- ·Mitigation (client-side pooling) adds configuration complexity
- ·Proxy layer adds latency and another failure point
Used In Architecture Scenarios
Multi-Tenant SaaS
A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart.
Realtime Collaboration
An online multiplayer game backend built around authoritative server game state synchronization. Game rooms run as distributed state machines where the server is the single source of truth for game state: client predictions are reconciled against the server state at every tick. WebSocket connections provide low-latency bidirectional communication for state deltas. Redis stores active game room state (in-flight, with sub-millisecond access), session affinity tokens (routing players to the same server instance as their game room), and matchmaking queues. PostgreSQL is the durable store for player profiles, persistent inventory, leaderboards, and achievement records. Kafka carries post-game event streams for analytics, anti-cheat processing, and achievement evaluation. NATS provides low-latency pub/sub for intra-cluster game state broadcasting when multiple game server instances must coordinate on shared state. Event sourcing records every game action as an immutable event for replay, dispute resolution, and anti-cheat audit.
Multi-Tenant SaaS
A multi-tenant SaaS architecture where multiple customers are served from a shared deployment, with PostgreSQL row-level security providing logical tenant isolation, Redis delivering per-tenant caching, and connection pooling managing the aggregate connection demand across tenant workloads. Tenant isolation, resource fairness, and operational simplicity are the three competing forces this architecture must balance.
Read-Heavy Application
A standard SaaS API architecture optimized for read-dominant workloads. PostgreSQL serves as the primary data store, Redis provides a caching layer for hot data, connection pooling bounds database concurrency, and read replicas scale read throughput without scaling write capacity.
Realtime Collaboration
An architecture for multi-user document editing where users see each other's changes in near real-time. PostgreSQL provides durable state persistence, Redis coordinates ephemeral session state and pub/sub for live change propagation, and connection pooling protects the database from WebSocket-induced connection churn.