Cascading Failure
criticalSummary
A failure or degradation in one service causes increased load, held resources, or error propagation in its callers, which in turn degrade their callers, until the failure front propagates through the entire dependency graph and brings down services with no direct dependency on the original failure point.
Description
Cascading failure is the failure mode that transforms a single-service incident into a system-wide outage. The chain begins with a degraded downstream service: slow response times, elevated error rates, or high latency. Callers of that service hold threads, connections, or coroutines open waiting for responses that take longer than usual. This resource accumulation depletes the callers' own thread pools or connection pools. The callers themselves become slow to respond, appearing degraded to their own callers. The degradation propagates up the call chain.
A canonical production scenario: the payment service experiences database slowness (p99 = 5s instead of 50ms). The checkout service calls the payment service synchronously. Each checkout request now holds a thread for 5 seconds waiting for payment. With 200 checkout threads, the checkout service saturates within 200/5 = 40 seconds. The API gateway calls the checkout service. Checkout now times out to the API gateway. API gateway accumulates queued requests. Within 2–3 minutes of the original payment service slowdown, the entire checkout and API gateway tier appears to be down, even though only the payment database is the root cause.
Cascading failures are amplified by retry storms: callers that receive errors or timeouts retry the request, adding more load to the already-degraded downstream service. Without exponential backoff and jitter, retries hit the failing service in synchronised waves, preventing recovery.
The failure is difficult to diagnose because the most visible symptoms are in the top-level services (API gateway timeouts) while the root cause is deep in the dependency graph. Service dependency maps and distributed tracing are essential for identifying the origin.
Characteristics
Triggers
- ·Database slowdown causing synchronous callers to hold threads waiting for query results
- ·Downstream service experiencing GC pressure or I/O saturation with elevated response times
- ·External API dependency (payment gateway, fraud detection) experiencing latency spike
- ·Memory pressure on one service causing GC pauses that cascade to all synchronous callers
- ·Retry storms amplifying load on a recovering service, preventing it from returning to baseline
Detection Signals
Mitigation Strategies
Trip a circuit breaker when the downstream service error rate or latency exceeds a threshold. Once open, fail fast rather than holding threads. Prevents thread pool exhaustion in the caller when the callee is slow. Recovery is automatic via HALF-OPEN probe.
Assign a dedicated, fixed-size thread pool to each downstream dependency. Payment service threads cannot consume checkout service's allocation for database calls. A slow payment API exhausts only its own 10-thread pool, not the shared 100-thread pool used for everything else.
Set aggressive timeouts on all downstream calls: e.g., payment API timeout = 500ms, total checkout request timeout = 1000ms. Fast failure bounds the maximum thread hold time regardless of downstream behaviour.
Retries must use exponential backoff (wait = base × 2^attempt) with random jitter (wait = wait × (0.5 + random())), and a maximum retry count. Prevents retry storms from re-saturating a recovering service.
When queue depth or response latency exceeds a threshold, actively reject new requests with HTTP 503 rather than accepting them and timing out. Controlled shedding prevents queue depth from growing unboundedly and allows faster recovery by keeping the service within its operational range.
Recovery Steps
- 1.Use distributed tracing to identify the root cause service (deepest span with high latency)
- 2.Open circuit breakers manually or reduce retry load on the root cause service to stop amplification
- 3.If root cause is database: terminate blocking queries, reduce connection count, verify index usage
- 4.Allow thread pools in upstream services to drain as fast-fail replaces held threads
- 5.Monitor error rates at each service layer: they should drop from bottom to top as the cascade unwinds
- 6.After recovery, implement circuit breakers and bulkheads to prevent recurrence
Estimated recovery time: 5–30 minutes once the root cause is identified and addressed. Without circuit breakers, the cascade may require manual intervention at each service layer to release held resources. With circuit breakers already in place, the system self-heals within 1–5 minutes of the root cause recovering.
Affected Systems
Patterns
Technologies
Basis
One of the most common and well-documented distributed systems failure modes; clear propagation mechanics with multiple proven prevention patterns
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
Bulkhead isolation partitions resources (thread pools, connection pools, queues) per downstream dependency, preventing a slow or failing dependency from consuming all shared resources and causing cascading failure across unrelated services.
Tradeoffs
- ·Bulkhead isolation requires separate resource allocation per dependency: higher total resource consumption
- ·Fine-grained bulkheads increase operational complexity: each bulkhead needs sizing, monitoring, and alerting
- ·Bulkheads do not prevent failure: they prevent failure spread; the failing dependency still fails
Circuit breakers prevent cascading failure by stopping the propagation of downstream errors to upstream callers, converting unbounded connection wait into fast failure with a predictable error response and giving the downstream dependency time to recover without continued load.
Tradeoffs
- ·False positives: transient blips can trip the circuit, causing callers to see errors the dependency could have served
- ·Half-open recovery probe rate must be conservative to avoid re-triggering overload on a recovering dependency
- ·Circuit breakers treat the symptom (connection accumulation), not the cause: the downstream issue requires separate remediation
- ·Requires explicit fallback logic at every call site, adding implementation surface area
Rate limiting at service ingress caps the load each upstream can place on a downstream, preventing the overload cascade triggered by burst traffic from multiple callers.
Full relationship →Used In Architecture Scenarios
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Event-Driven System
A video and audio streaming architecture where content ingestion triggers an async multi-variant transcoding pipeline, CDN delivery handles 95%+ of playback traffic, and Cassandra absorbs the write volume of per-user viewing history. Kafka decouples upload events from transcoding workers; MinIO stores raw and encoded assets; Redis maintains playback session state and view counters. The architecture must handle upload spikes without blocking delivery, and cache cold starts without cascading database load.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.