Thundering Herd
criticalContainedcapacity failure · fan out propagation
Blast Radius
0%
0/21 nodes
Time to Detect
15–60 seconds when reconnect/retry rate is monitored directly as a distinct signal from steady-state traffic. Longer, and easy to misdiagnose as "the recovery didn't work," when the only visibility is downstream error rate: the recovering component can appear to be failing again when it is actually saturated by its own returning clients.
Preventive Mitigations
3
Confidence
StrongFailure Cascade
(no topology nodes mapped for this failure mode)
This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.
Severity at this step: critical
Detection Signals
Recovery time estimate: 1–10 minutes depending on whether admission control or traffic ramping is available to apply reactively; without either, recovery depends on the herd naturally desynchronizing as individual clients time out and retry independently, which can take longer than the original outage.
Mitigation Checklist(3 preventive, 1 reactive)
Before switching traffic to a new application instance, run a cache warm-up pass that pre-populates known hot keys, so a deployment-triggered herd is not compounded by every instance also being cold on its local cache.
Route traffic to a recovering component gradually (a canary or ramped percentage) rather than switching all traffic back at once when a health check passes, giving connection pools and local caches time to warm before receiving full load.
The recovering component itself limits the rate of new connection acceptance or authentication attempts immediately after recovery, queueing or rejecting excess reconnects rather than accepting the full burst and collapsing under it. Trades immediate availability for controlled, survivable recovery.
Clients reconnecting or retrying after a shared triggering event add randomized jitter to their backoff/retry delay, spreading the reconnect burst over a window instead of concentrating it at the instant the trigger fires. This is the general-case analogue of TTL jitter for cache keys, applied to connection and retry logic instead of cache expiry.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.