Cache Stampede (Dog-Pile)
SEV-2, Significant ImpactFan-Out propagation · caching · Affects 3 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 3 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Cache Stampede (Dog-Pile) begins at the source component. Trigger: Hot cache key reaches its TTL and expires under sustained concurrent access.
Immediate (T+0) · Signal: Alert
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
15–30 seconds for cache miss rate monitoring and database QPS alerting. The signature is a simultaneous spike in cache misses and database QPS for the same time period. Without specific cache miss rate alerting, the event may appear as a database overload of unknown origin, taking 2–5 minutes to diagnose. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
The origin database receives a sudden spike of concurrent identical queries. If the database connection pool is exhausted, all application features requiring database access degrade: not just the feature related to the hot key. The blast radius can expand from a single feature to full application unavailability if the stampede fully saturates the database. In systems with multiple caches and services, a cache flush can trigger simultaneous stampedes across all services sharing the origin.
Contributing Factors
Read-heavy API workloads are vulnerable to cache stampede when popular cache keys expire under sustained concurrent load, causing all requests to simultaneously miss and query the origin database.
This operational trigger enables Cache Stampede (Dog-Pile): Hot cache key reaches its TTL and expires under sustained concurrent access
This operational trigger enables Cache Stampede (Dog-Pile): Full cache flush (Redis FLUSHDB, Memcached restart) causes all requests to miss simultaneously
This operational trigger enables Cache Stampede (Dog-Pile): Coordinated cache invalidation across multiple keys at the same wall-clock time
Remediation Plan
Identify the expired/invalidated hot key(s) from cache miss metrics and database slow query log
Effort: Minutes to hours (on-call response)
Manually repopulate hot keys (execute the population query once and SET the result)
Effort: Minutes to hours (on-call response)
Terminate or limit duplicate in-flight database queries for the stampede pattern
Effort: Minutes to hours (on-call response)
On a cache miss, acquire a distributed lock (Redis SET NX PX 5000) before querying the database. Only the lock holder executes the query. Other concurrent misses poll the cache at 50ms intervals until the lock holder populates it. Reduces N database queries to 1 per stampede event.
Effort: 1 day to 1 week
Before a key expires, a fraction of reads voluntarily trigger a background refresh. The probability increases as the key approaches expiry: P(refresh) ∝ exp(-remaining_ttl / beta). A background worker refreshes the key while the current value continues to be served. Key never reaches zero: stampede window is eliminated.
Effort: 1 day to 1 week
Add random jitter to all cache key TTLs (±10–20% of base TTL) to prevent co-expiry of keys set simultaneously. Eliminates the mass expiry scenario. Does not prevent single-key stampede under sustained load.
Effort: 1 day to 1 week
Configure alerts for: alert, error rate spike, queue depth. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Cache Stampede (Dog-Pile) affects 3 architecture scenarios (API Gateway Platform, Content Management Platform, ML Feature Serving Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.