Hot Key Cache Eviction
SEV-3, Limited ImpactFeedback Loop propagation · caching · Affects 0 scenario(s)
Severity Classification
Classified as PARTIAL based on failure mode severity. The feedback loop propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Hot Key Cache Eviction begins at the source component. Trigger: Redis memory utilization exceeds maxmemory threshold, triggering eviction.
Immediate (T+0) · Signal: Error Rate Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
15–60 seconds via Redis keyspace_misses spike + database QPS spike correlation. Without combined Redis and database monitoring, the failure appears as a sudden database error rate increase with no obvious cause, taking 2–5 minutes to diagnose correctly. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Self-amplification loop
Retry behavior and back-pressure cause the failure to amplify. Resource exhaustion accelerates, the system cannot self-recover without intervention.
Minutes after fan-out · Signal: Cascading alert storms; all downstream indicators deteriorating
Blast Radius
The database bears the full simultaneous load of all requests that were hitting the evicted key. If the key was absorbing 500 req/s and the database query takes 200ms, the initial stampede delivers up to 100 concurrent queries. If the connection pool exhausts, all requests requiring database access fail, not just those for the evicted key. In systems where Redis also serves as a session store or rate limiter, memory pressure that causes evictions can also corrupt session data or reset rate limit counters.
Contributing Factors
This operational trigger enables Hot Key Cache Eviction: Redis memory utilization exceeds maxmemory threshold, triggering eviction
This operational trigger enables Hot Key Cache Eviction: Sudden traffic spike increases total key footprint (new users, new feature launch) until memory saturates
This operational trigger enables Hot Key Cache Eviction: Large value keys inserted that consume a disproportionate share of memory, forcing aggressive eviction of small hot keys
Remediation Plan
Check Redis keyspace_misses in INFO stats: a sudden increase confirms a cache miss event
Effort: Minutes to hours (on-call response)
Identify the evicted key by correlating database slow query log with the cache miss timestamp
Effort: Minutes to hours (on-call response)
Manually re-populate the evicted key by running the backing query once and setting the result in Redis
Effort: Minutes to hours (on-call response)
Switch Redis eviction policy to volatile-lru and ensure that hot keys are NOT given a TTL. Keys without a TTL are never evicted under volatile-lru (only keys with TTL are candidates for eviction). Hot keys that must not be evicted are set with SET key value (no EX/PX). This requires discipline in application code to distinguish hot durable keys from warm expendable keys, but completely prevents hot key eviction.
Effort: 1 day to 1 week
On a cache miss for any key serving >50 req/s, acquire a short-lived Redis lock (SETNX hot_key_lock 1 PX 5000) before querying the database. Only the lock holder executes the database query and repopulates the cache. All other concurrent misses for the same key poll the lock and wait. This limits N simultaneous database queries to 1, regardless of whether the miss was caused by eviction or TTL expiry. Reduces stampede impact by 99% for hot keys.
Effort: 1 day to 1 week
Increase maxmemory-samples from 5 to 10 or 20. Higher sample counts make the LRU approximation more accurate, significantly reducing the probability that a recently-accessed hot key is incorrectly selected for eviction. At maxmemory-samples=10, eviction quality approaches true LRU. The CPU cost increase is less than 10% on typical workloads. This does not prevent eviction under extreme memory pressure but reduces the probability of hot key eviction substantially.
Effort: 1 day to 1 week
Configure alerts for: error rate spike, latency spike, memory pressure. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.