Thundering Herd
criticalSummary
When a shared outage, network partition, or coordinated recovery event ends, every client, connection, or worker that was waiting or blocked resumes activity at nearly the same instant, producing a synchronized burst of retries, reconnects, or requests that can overwhelm the system just as it is recovering.
Description
Thundering herd is the general class of failure where a large number of independent actors, previously idle, blocked, or waiting, are released by the same triggering event and act at nearly the same moment. The trigger is not traffic growth: it is synchronization. A system that could comfortably absorb the same total request volume spread over seconds collapses when that volume arrives in a single burst, because connection acceptance, authentication, and initial query paths are typically far more expensive per request than steady-state traffic.
Common triggers, beyond cache expiry: - Service recovery after downtime: every client that queued or retried during an outage reconnects and replays its backlog the instant the service becomes reachable again, rather than trickling back in. - Network partition healing: connection pools on one side of a healed partition attempt to re-establish every dropped connection simultaneously. - Coordinated configuration or DNS propagation: many clients independently poll or are notified of a change (a failover, a new endpoint) and re-resolve or reconnect at the same moment. - Load balancer or health-check state flips: a fleet of instances flipping from unhealthy to healthy at once causes the balancer to route a full burst of traffic to instances that have not yet warmed their own connection pools or local state.
Self-amplification mechanism, common to all of these triggers: the recovering system is momentarily slower than steady state (cold connection pools, cold local caches, authentication or handshake overhead paid by every reconnecting client at once). Slower handling means more retrying clients accumulate before any of them succeed, which means the next retry wave is even larger, which means the system takes longer to recover than it would have if the same clients had reconnected independently. The failure can look like the recovery itself failed, when the underlying service is actually healthy and simply saturated by the synchronized reconnect burst.
Cache-key expiry under sustained concurrent read load is one specific, very common trigger for this same synchronization mechanism, but it has its own name, its own detection signature, and its own purpose-built prevention techniques (distributed mutex on miss, probabilistic early expiry, TTL jitter): see `cache_stampede` for that case in full. This entry covers the broader class: synchronized reconnect, retry, and recovery amplification that has nothing to do with a cache key expiring, alongside the general principle both share.
Characteristics
Triggers
- ·Service recovery after downtime, releasing a backlog of queued or retrying clients simultaneously
- ·Network partition healing, causing connection pools to re-establish every dropped connection at once
- ·Coordinated DNS or configuration propagation causing many clients to re-resolve or reconnect at the same moment
- ·Load balancer or health-check state flipping a fleet of instances from unhealthy to healthy simultaneously, routing a full traffic burst to cold instances
- ·Cache-key TTL expiry under sustained concurrent load is a related but distinct trigger; see `cache_stampede`
Detection Signals
Mitigation Strategies
Clients reconnecting or retrying after a shared triggering event add randomized jitter to their backoff/retry delay, spreading the reconnect burst over a window instead of concentrating it at the instant the trigger fires. This is the general-case analogue of TTL jitter for cache keys, applied to connection and retry logic instead of cache expiry.
Route traffic to a recovering component gradually (a canary or ramped percentage) rather than switching all traffic back at once when a health check passes, giving connection pools and local caches time to warm before receiving full load.
Before switching traffic to a new application instance, run a cache warm-up pass that pre-populates known hot keys, so a deployment-triggered herd is not compounded by every instance also being cold on its local cache.
The recovering component itself limits the rate of new connection acceptance or authentication attempts immediately after recovery, queueing or rejecting excess reconnects rather than accepting the full burst and collapsing under it. Trades immediate availability for controlled, survivable recovery.
Recovery Steps
- 1.Confirm the recovering component is not actually unhealthy, but saturated by its own returning clients: check reconnect/retry rate, not just error rate
- 2.Apply or increase server-side admission control to shed excess reconnect load while the component stabilizes
- 3.If a load balancer or orchestrator supports it, ramp traffic back in gradually rather than restoring it all at once
- 4.After recovery, add client-side backoff jitter and server-side admission control to prevent recurrence on the next shared-trigger event
Estimated recovery time: 1–10 minutes depending on whether admission control or traffic ramping is available to apply reactively; without either, recovery depends on the herd naturally desynchronizing as individual clients time out and retry independently, which can take longer than the original outage.
Affected Systems
Patterns
Technologies
Basis
The general reconnect/recovery-amplification mechanism and its mitigations (jittered backoff, gradual traffic ramp, admission control) are well-documented distributed-systems practices; confidence is held slightly below `cache_stampede`'s because the broadened framing here has not yet had a dedicated sourcing pass distinguishing it from the cache-specific case the way the cache literature does.
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Used In Architecture Scenarios
AI / RAG Application
A Retrieval-Augmented Generation (RAG) architecture that combines vector similarity search for semantic document retrieval with relational metadata filtering, using PostgreSQL with pgvector as the unified store for both embeddings and structured data. Redis provides a semantic cache to avoid redundant embedding model inference and reduce vector index query load for repeated or similar queries. Kafka manages the asynchronous embedding generation pipeline that keeps the vector index current as source documents are added or updated.
Multi-Tenant SaaS
A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart.
Read-Heavy Application
A CMS for publishing and serving structured content: articles, documentation, product pages, and localized variants: where read APIs serve 50–100x more traffic than editorial write APIs. PostgreSQL stores the content graph (articles, authors, categories, taxonomy) and workflow state (draft, in-review, scheduled, published). Redis caches published content objects for read APIs. Elasticsearch powers full-text content search with faceting and relevance ranking. Cache invalidation on publish must be fast and complete; N+1 query patterns on content relationship traversal are the dominant database performance risk during reads.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Event-Driven System
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth.
Event-Driven System
A video and audio streaming architecture where content ingestion triggers an async multi-variant transcoding pipeline, CDN delivery handles 95%+ of playback traffic, and Cassandra absorbs the write volume of per-user viewing history. Kafka decouples upload events from transcoding workers; MinIO stores raw and encoded assets; Redis maintains playback session state and view counters. The architecture must handle upload spikes without blocking delivery, and cache cold starts without cascading database load.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.