Rate Limit Cascade
criticalContainedresilience failure · feedback loop propagation
Blast Radius
0%
0/17 nodes
Time to Detect
Immediately visible in 429 response rate metrics if monitored. Without explicit 429 monitoring, the failure appears as elevated error rates and latency across services connected to the downstream: potentially attributed to other causes and taking minutes to diagnose.
Preventive Mitigations
2
Confidence
StrongFailure Cascade
(no topology nodes mapped for this failure mode)
This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.
Severity at this step: critical
Detection Signals
Recovery time estimate: Stopping the retry amplification provides immediate relief (seconds). Root cause fix (implementing proper 429 handling) requires a code deployment (hours to days). Downstream capacity recovery (if the downstream was overloaded) takes minutes after upstream retry rate drops.
Mitigation Checklist(2 preventive, 2 reactive)
Implement a client-side rate limiter for calls to each downstream dependency. Set the limit to the downstream's documented quota. Requests above the limit are queued or rejected client-side before being sent: preventing rate limit events entirely by never exceeding the downstream's quota.
When receiving a 429, extract the Retry-After header and wait the specified duration before retrying. If 429 responses persist, open a circuit breaker for that dependency rather than continuing to retry. The circuit breaker prevents the retry amplification loop.
Rate limiting enforced at the API boundary prevents the retry amplification loop that causes rate limit cascades by ensuring callers never exceed the downstream quota in the first place.
Configure retry logic with exponential backoff (delay doubles on each attempt) and random jitter (prevent synchronized retries from multiple instances). A starting backoff of 100ms × 2^attempt + rand(0, 100ms) prevents immediate re-requests. Maximum backoff should be 10–30 seconds.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.