Thundering Herd (Cache Stampede)
SEV-2, Significant ImpactFan-Out propagation · capacity · Affects 8 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 8 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Thundering Herd (Cache Stampede) begins at the source component. Trigger: High-traffic cache key TTL expiry under sustained concurrent load (>100 req/s per key).
Immediate (T+0) · Signal: Alert
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
15–60 seconds with cache miss rate monitoring and database QPS alerting. 2–10 minutes if detected only through application error rate monitors. The failure signature is a near-simultaneous cache miss rate spike followed by a database QPS spike: these two metrics together are the primary signal. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
The origin database bears the full stampede load. If the database saturates, connection pool exhaustion affects all application requests requiring database access, not just those related to the hot key. This expands the blast radius from the specific feature to total application unavailability. If the database serves multiple application services, the stampede from one service's cache miss can degrade all other services sharing the same database.
Contributing Factors
Redis is itself vulnerable to thundering herd when it restarts or flushes: all cache entries expire simultaneously, and many concurrent requests all miss and race to repopulate the same keys from the database, causing a stampede that can overwhelm the downstream database.
This operational trigger enables Thundering Herd (Cache Stampede): High-traffic cache key TTL expiry under sustained concurrent load (>100 req/s per key)
This operational trigger enables Thundering Herd (Cache Stampede): Full Redis cache flush (restart, eviction policy switch, maintenance)
This operational trigger enables Thundering Herd (Cache Stampede): Application restart with empty local caches after deployment
Mitigation Gaps
Add 'redis' to the runbook. Redis distributed locks (via SET NX EX or Redlock) prevent thundering herd by ensuring only one caller repopulates a cache entry at a time, with other callers either waiting or returning a stale value until the cache is warm.
Remediation Plan
Identify the hot key(s) that expired: check Redis keyspace_misses and correlate with timestamp of incident
Effort: Minutes to hours (on-call response)
Immediately repopulate the cache keys manually (warm-up query executed once by operator)
Effort: Minutes to hours (on-call response)
If database is saturated, terminate the duplicate queries: SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE query = '<stampede query>' AND state = 'active'
Effort: Minutes to hours (on-call response)
On a cache miss, acquire a distributed lock (Redis SETNX with short TTL, e.g., 5 seconds) before executing the database query. Only the lock holder executes the query and populates the cache; all other concurrent misses wait and then re-check the cache after the lock is released. Reduces N simultaneous database queries to 1 per cache miss event.
Effort: 1 day to 1 week
Instead of expiring a key exactly at TTL, each read recomputes expiry with a probability that increases as TTL approaches zero: should_refresh = (random() < exp((current_ttl - TTL) / beta)). A small fraction of reads trigger a background cache refresh before the key expires, eliminating simultaneous expiry. Beta = 1.0 is a standard starting point; tune based on query cost and acceptable staleness.
Effort: 1 day to 1 week
Add random jitter (±10–20% of TTL) to cache key expiry times to prevent multiple hot keys from expiring simultaneously. Reduces the probability of a simultaneous multi-key stampede; does not prevent single-key stampede under sustained load.
Effort: 1 day to 1 week
Configure alerts for: alert, queue depth, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Before switching traffic to a new application instance, run a cache warm-up pass that pre-populates all known hot keys. Eliminates the cold-start stampede on deployment. Requires maintaining a list of hot keys or using a traffic-replay mechanism to identify them.
Effort: 1–4 sprints
Thundering Herd (Cache Stampede) affects 8 architecture scenarios (AI Retrieval-Augmented Generation Platform, API Gateway Platform, Content Management Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.