Cascading Failure
SEV-2, Significant ImpactFan-Out propagation · cascading · Affects 3 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 3 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Cascading Failure begins at the source component. Trigger: Database slowdown causing synchronous callers to hold threads waiting for query results.
Immediate (T+0) · Signal: Latency Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
60–180 seconds for automated alerting to fire at the user-facing layer. Identifying the root cause service typically takes 5–15 minutes without distributed tracing, or 1–3 minutes with distributed tracing and clear dependency span data. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
In the worst case, the entire application is unavailable within 2–5 minutes of a single downstream service degradation. The blast radius expands with each level of the call graph that saturates. Typically the top-level user-facing service (API gateway, frontend BFF) is the last to saturate and the most visible to users and monitoring. The blast radius includes every service in the transitive call graph of the failed dependency.
Contributing Factors
This operational trigger enables Cascading Failure: Database slowdown causing synchronous callers to hold threads waiting for query results
This operational trigger enables Cascading Failure: Downstream service experiencing GC pressure or I/O saturation with elevated response times
This operational trigger enables Cascading Failure: External API dependency (payment gateway, fraud detection) experiencing latency spike
Mitigation Gaps
Add 'bulkhead isolation' to the runbook. Bulkhead isolation partitions resources (thread pools, connection pools, queues) per downstream dependency, preventing a slow or failing dependency from consuming all shared resources and causing cascading failure across unrelated services.
Add 'circuit breaker' to the runbook. Circuit breakers prevent cascading failure by stopping the propagation of downstream errors to upstream callers, converting unbounded connection wait into fast failure with a predictable error response and giving the downstream dependency time to recover without continued load.
Remediation Plan
Use distributed tracing to identify the root cause service (deepest span with high latency)
Effort: Minutes to hours (on-call response)
Open circuit breakers manually or reduce retry load on the root cause service to stop amplification
Effort: Minutes to hours (on-call response)
If root cause is database: terminate blocking queries, reduce connection count, verify index usage
Effort: Minutes to hours (on-call response)
Trip a circuit breaker when the downstream service error rate or latency exceeds a threshold. Once open, fail fast rather than holding threads. Prevents thread pool exhaustion in the caller when the callee is slow. Recovery is automatic via HALF-OPEN probe.
Effort: 1 day to 1 week
Assign a dedicated, fixed-size thread pool to each downstream dependency. Payment service threads cannot consume checkout service's allocation for database calls. A slow payment API exhausts only its own 10-thread pool, not the shared 100-thread pool used for everything else.
Effort: 1 day to 1 week
Set aggressive timeouts on all downstream calls: e.g., payment API timeout = 500ms, total checkout request timeout = 1000ms. Fast failure bounds the maximum thread hold time regardless of downstream behaviour.
Effort: 1 day to 1 week
Configure alerts for: latency spike, disk saturation, queue depth. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Cascading Failure affects 3 architecture scenarios (E-Commerce Order Platform, Two-Sided Marketplace Platform, Streaming Media Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.