Thundering Herd
SEV-2, Significant ImpactFan-Out propagation · capacity · Affects 7 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 7 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Thundering Herd begins at the source component. Trigger: Service recovery after downtime, releasing a backlog of queued or retrying clients simultaneously.
Immediate (T+0) · Signal: Alert
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
15–60 seconds when reconnect/retry rate is monitored directly as a distinct signal from steady-state traffic. Longer, and easy to misdiagnose as "the recovery didn't work," when the only visibility is downstream error rate: the recovering component can appear to be failing again when it is actually saturated by its own returning clients. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
The recovering component receives a burst far exceeding its steady-state capacity at the exact moment it is least able to absorb one: connection acceptance, authentication, and cold-cache paths are all more expensive per request than normal. If the component saturates, the burst can push it back into the same failure state it was just recovering from, extending the outage rather than ending it. When many services share the same downstream dependency (a database, an auth service), a recovery-triggered herd from one service can degrade that shared dependency for every other service depending on it, even ones that were never part of the original outage.
Contributing Factors
This operational trigger enables Thundering Herd: Service recovery after downtime, releasing a backlog of queued or retrying clients simultaneously
This operational trigger enables Thundering Herd: Network partition healing, causing connection pools to re-establish every dropped connection at once
This operational trigger enables Thundering Herd: Coordinated DNS or configuration propagation causing many clients to re-resolve or reconnect at the same moment
Remediation Plan
Confirm the recovering component is not actually unhealthy, but saturated by its own returning clients: check reconnect/retry rate, not just error rate
Effort: Minutes to hours (on-call response)
Apply or increase server-side admission control to shed excess reconnect load while the component stabilizes
Effort: Minutes to hours (on-call response)
If a load balancer or orchestrator supports it, ramp traffic back in gradually rather than restoring it all at once
Effort: Minutes to hours (on-call response)
Clients reconnecting or retrying after a shared triggering event add randomized jitter to their backoff/retry delay, spreading the reconnect burst over a window instead of concentrating it at the instant the trigger fires. This is the general-case analogue of TTL jitter for cache keys, applied to connection and retry logic instead of cache expiry.
Effort: 1 day to 1 week
Route traffic to a recovering component gradually (a canary or ramped percentage) rather than switching all traffic back at once when a health check passes, giving connection pools and local caches time to warm before receiving full load.
Effort: 1 day to 1 week
The recovering component itself limits the rate of new connection acceptance or authentication attempts immediately after recovery, queueing or rejecting excess reconnects rather than accepting the full burst and collapsing under it. Trades immediate availability for controlled, survivable recovery.
Effort: 1 day to 1 week
Configure alerts for: alert, queue depth, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Before switching traffic to a new application instance, run a cache warm-up pass that pre-populates known hot keys, so a deployment-triggered herd is not compounded by every instance also being cold on its local cache.
Effort: 1–4 sprints
Thundering Herd affects 7 architecture scenarios (AI Retrieval-Augmented Generation Platform, API Gateway Platform, Content Management Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.