Rate Limit Cascade
SEV-2, Significant ImpactFeedback Loop propagation · resilience · Affects 2 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The feedback loop propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 2 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Rate Limit Cascade begins at the source component. Trigger: Downstream service begins rate limiting without the upstream implementing proper 429 handling.
Immediate (T+0) · Signal: Error Rate Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Immediately visible in 429 response rate metrics if monitored. Without explicit 429 monitoring, the failure appears as elevated error rates and latency across services connected to the downstream: potentially attributed to other causes and taking minutes to diagnose. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Self-amplification loop
Retry behavior and back-pressure cause the failure to amplify. Resource exhaustion accelerates, the system cannot self-recover without intervention.
Minutes after fan-out · Signal: Cascading alert storms; all downstream indicators deteriorating
Blast Radius
The rate limit cascade propagates upstream through the dependency graph: from B to A to C. In a deep call chain (C → A → B → D), a rate limit at D can propagate to C within seconds if each hop retries aggressively. At scale, the combined retry traffic from all upstreams can push a service far beyond its capacity, converting a rate limit event into a sustained overload.
Contributing Factors
This operational trigger enables Rate Limit Cascade: Downstream service begins rate limiting without the upstream implementing proper 429 handling
This operational trigger enables Rate Limit Cascade: Traffic spike causes the upstream to exceed its downstream quota
This operational trigger enables Rate Limit Cascade: Downstream reduces its rate limit quota without notifying upstreams
Mitigation Gaps
Add 'rate limiting' to the runbook. Rate limiting enforced at the API boundary prevents the retry amplification loop that causes rate limit cascades by ensuring callers never exceed the downstream quota in the first place.
Remediation Plan
Identify the origin rate limit event in downstream metrics (first 429 responses)
Effort: Minutes to hours (on-call response)
Reduce upstream retry rate or apply circuit breaker to the 429-returning dependency
Effort: Minutes to hours (on-call response)
Check Retry-After headers in 429 responses and configure callers to honor them
Effort: Minutes to hours (on-call response)
When receiving a 429, extract the Retry-After header and wait the specified duration before retrying. If 429 responses persist, open a circuit breaker for that dependency rather than continuing to retry. The circuit breaker prevents the retry amplification loop.
Effort: 1 day to 1 week
Configure retry logic with exponential backoff (delay doubles on each attempt) and random jitter (prevent synchronized retries from multiple instances). A starting backoff of 100ms × 2^attempt + rand(0, 100ms) prevents immediate re-requests. Maximum backoff should be 10–30 seconds.
Effort: 1 day to 1 week
Implement a client-side rate limiter for calls to each downstream dependency. Set the limit to the downstream's documented quota. Requests above the limit are queued or rejected client-side before being sent: preventing rate limit events entirely by never exceeding the downstream's quota.
Effort: 1 day to 1 week
Configure alerts for: error rate spike, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Rate Limit Cascade affects 2 architecture scenarios (API Gateway Platform, Notification Delivery Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.