Cache Stampede (Dog-Pile)
criticalContainedcaching failure · fan out propagation
Blast Radius
5%
1/21 nodes
Time to Detect
15–30 seconds for cache miss rate monitoring and database QPS alerting. The signature is a simultaneous spike in cache misses and database QPS for the same time period. Without specific cache miss rate alerting, the event may appear as a database overload of unknown origin, taking 2–5 minutes to diagnose.
Preventive Mitigations
3
Confidence
StrongImpacted Components
Read-Heavy API Backend
workload · Directly connected to failure mode 'Cache Stampede (Dog-Pile)' via risk propagation path in the topology.
Failure Cascade
Read-Heavy API Backend
Cache Stampede (Dog-Pile) (fan out propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: 30–120 seconds after manual cache repopulation. The database load subsides as soon as a cached value is available to serve concurrent requests. Without manual intervention, the stampede persists for the duration of the database query latency under overload conditions: potentially minutes if the database degrades into connection exhaustion.
Mitigation Checklist(3 preventive, 1 reactive)
On a cache miss, acquire a distributed lock (Redis SET NX PX 5000) before querying the database. Only the lock holder executes the query. Other concurrent misses poll the cache at 50ms intervals until the lock holder populates it. Reduces N database queries to 1 per stampede event.
Before a key expires, a fraction of reads voluntarily trigger a background refresh. The probability increases as the key approaches expiry: P(refresh) ∝ exp(-remaining_ttl / beta). A background worker refreshes the key while the current value continues to be served. Key never reaches zero: stampede window is eliminated.
Serve the expired (stale) cache entry while triggering an asynchronous background refresh. The stale entry is served for at most one request cycle; the background refresh updates the cache before the next request arrives. Requires storing the previous value with an extended TTL alongside the logical expiry.
Add random jitter to all cache key TTLs (±10–20% of base TTL) to prevent co-expiry of keys set simultaneously. Eliminates the mass expiry scenario. Does not prevent single-key stampede under sustained load.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.