Thundering Herd (Cache Stampede)
criticalSummary
When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.
Description
Under normal operation, a hot cache key absorbs hundreds or thousands of requests per second with sub-millisecond Redis reads. When that key expires : or when the entire cache is cold after a restart or flush: every concurrent request misses the cache. Each one independently fetches from PostgreSQL, generating N simultaneous identical queries. For a key with 500 requests/second and a 200ms database query time, a single key expiry can produce 100+ concurrent identical database queries in the first 200ms window.
PostgreSQL handles this poorly: each query acquires shared locks, consumes connection pool capacity, and competes for shared_buffers. If the query is expensive (full table scan, complex join), CPU and I/O saturate quickly. Connection pools exhaust. New requests queue at the pool. Queue depth grows faster than it drains. Application-level timeouts begin firing. Upstream services see error spikes. The failure looks like a sudden database outage even though the underlying query is correct.
The stampede is self-amplifying: during the database overload, query latency increases, which means the first wave of requests holds connections longer, which means subsequent requests wait longer for a connection, which means they arrive at the database even more simultaneously when they finally execute.
A variation occurs when a service recovers from downtime (restart, network partition recovery): all pending requests that accumulated during the outage arrive simultaneously, producing an identical spike pattern without any cache involvement.
Characteristics
Triggers
- ·High-traffic cache key TTL expiry under sustained concurrent load (>100 req/s per key)
- ·Full Redis cache flush (restart, eviction policy switch, maintenance)
- ·Application restart with empty local caches after deployment
- ·Service recovery after downtime with accumulated request queue
- ·Deliberate cache invalidation on a hot key (config change, manual flush)
Detection Signals
Mitigation Strategies
On a cache miss, acquire a distributed lock (Redis SETNX with short TTL, e.g., 5 seconds) before executing the database query. Only the lock holder executes the query and populates the cache; all other concurrent misses wait and then re-check the cache after the lock is released. Reduces N simultaneous database queries to 1 per cache miss event.
Instead of expiring a key exactly at TTL, each read recomputes expiry with a probability that increases as TTL approaches zero: should_refresh = (random() < exp((current_ttl - TTL) / beta)). A small fraction of reads trigger a background cache refresh before the key expires, eliminating simultaneous expiry. Beta = 1.0 is a standard starting point; tune based on query cost and acceptable staleness.
Add random jitter (±10–20% of TTL) to cache key expiry times to prevent multiple hot keys from expiring simultaneously. Reduces the probability of a simultaneous multi-key stampede; does not prevent single-key stampede under sustained load.
Before switching traffic to a new application instance, run a cache warm-up pass that pre-populates all known hot keys. Eliminates the cold-start stampede on deployment. Requires maintaining a list of hot keys or using a traffic-replay mechanism to identify them.
On cache miss, immediately return the last known (potentially expired) value while triggering an asynchronous background refresh. Caller receives stale data for at most one request cycle. Requires storing the previous value alongside the TTL metadata.
Recovery Steps
- 1.Identify the hot key(s) that expired: check Redis keyspace_misses and correlate with timestamp of incident
- 2.Immediately repopulate the cache keys manually (warm-up query executed once by operator)
- 3.If database is saturated, terminate the duplicate queries: SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE query = '<stampede query>' AND state = 'active'
- 4.Temporarily increase connection pool size if requests are queuing, to absorb the backlog
- 5.After immediate recovery, implement mutex or probabilistic early expiry to prevent recurrence
- 6.Review all cache key TTLs for hot keys and add TTL jitter across the set
Estimated recovery time: 30–120 seconds once hot keys are manually repopulated and database load subsides. Without manual intervention, recovery depends on whether the database can absorb the sustained stampede load or whether it collapses into connection exhaustion requiring a longer restart sequence.
Affected Systems
Patterns
Technologies
Basis
Thoroughly documented failure mode with well-understood mechanics and multiple proven prevention strategies; occurs predictably in any cache- aside system with hot keys under sustained concurrent load
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
Redis distributed locks (via SET NX EX or Redlock) prevent thundering herd by ensuring only one caller repopulates a cache entry at a time, with other callers either waiting or returning a stale value until the cache is warm.
Tradeoffs
- ·Distributed locking adds one Redis round-trip to every cache miss that triggers population
- ·If the lock holder crashes mid-population, the lock TTL must expire before recovery: causing a cache gap
- ·Redlock (multi-node locking) adds complexity: for most cache stampede cases, single-node SET NX is sufficient
Redis is itself vulnerable to thundering herd when it restarts or flushes: all cache entries expire simultaneously, and many concurrent requests all miss and race to repopulate the same keys from the database, causing a stampede that can overwhelm the downstream database.
Tradeoffs
- ·Pre-warming requires either a warm replica or a cache loader process: adds operational complexity
- ·Jitter reduces synchronized expiry risk but requires all cache writers to implement TTL jitter
- ·Redis Cluster (with replicas) reduces but does not eliminate cold-start risk: slot resharding warms new nodes
Used In Architecture Scenarios
AI / RAG Application
A Retrieval-Augmented Generation (RAG) architecture that combines vector similarity search for semantic document retrieval with relational metadata filtering, using PostgreSQL with pgvector as the unified store for both embeddings and structured data. Redis provides a semantic cache to avoid redundant embedding model inference and reduce vector index query load for repeated or similar queries. Kafka manages the asynchronous embedding generation pipeline that keeps the vector index current as source documents are added or updated.
Multi-Tenant SaaS
A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart.
Read-Heavy Application
A CMS for publishing and serving structured content: articles, documentation, product pages, and localized variants: where read APIs serve 50–100x more traffic than editorial write APIs. PostgreSQL stores the content graph (articles, authors, categories, taxonomy) and workflow state (draft, in-review, scheduled, published). Redis caches published content objects for read APIs. Elasticsearch powers full-text content search with faceting and relevance ranking. Cache invalidation on publish must be fast and complete; N+1 query patterns on content relationship traversal are the dominant database performance risk during reads.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Search-Heavy Application
A content platform architecture centered on Elasticsearch for full-text search, faceted navigation, and ranked results, with PostgreSQL as the transactional source of truth and Redis for session management and hot content caching. WAL-based CDC maintains index freshness by streaming PostgreSQL changes into Elasticsearch asynchronously. The core tension is between search index freshness, query performance, and index maintenance cost under high write volume.
Event-Driven System
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth.
Event-Driven System
A video and audio streaming architecture where content ingestion triggers an async multi-variant transcoding pipeline, CDN delivery handles 95%+ of playback traffic, and Cassandra absorbs the write volume of per-user viewing history. Kafka decouples upload events from transcoding workers; MinIO stores raw and encoded assets; Redis maintains playback session state and view counters. The architecture must handle upload spikes without blocking delivery, and cache cold starts without cascading database load.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.