Retry with Exponential Backoff

Retry transient failures with exponentially increasing wait times and added jitter to prevent synchronized retry storms while ensuring eventual operation success without saturating a recovering dependency.

Transient failures in distributed systems cause request loss when not retried, but naive immediate retries amplify load on a recovering dependency and can cause retry storms that prevent recovery.

Transient failures: network blips, brief database overload, momentary downstream unavailability: are ubiquitous in distributed systems. A fixed immediate retry amplifies the problem: if 1000 clients fail simultaneously and all retry immediately, the recovering dependency receives 1000 requests in the same instant, extending or deepening the failure.

Exponential backoff: after the first failure, wait 100ms. After the second, 200ms. After the third, 400ms. The wait doubles each attempt up to a configured ceiling (typically 30–60 seconds). This gives the dependency breathing room between retry waves.

Jitter: add a random factor to the wait time. Without jitter, all clients that failed at the same moment back off for the same duration and retry simultaneously, recreating the original spike. With full jitter (uniform random from 0 to the computed backoff), retries are spread across the backoff window, dramatically reducing the peak retry load.

AWS recommends "full jitter" (random between 0 and computed_sleep) or "decorrelated jitter" (random between base and previous_sleep × 3) based on empirical benchmarks showing either outperforms equal jitter at scale.

Max attempts: always set a finite limit (e.g., 3–5 attempts). Infinite retry loops consume resources indefinitely and mask permanent failures. Operations that exhaust retries should fail to a dead-letter queue for human inspection or fall back to a degraded response.

Idempotency requirement: retries assume the operation can be safely repeated. A charge $100 call cannot be retried blindly: the second attempt may also succeed, charging the customer twice. Non-idempotent operations require idempotency keys (a client-generated unique token that the server uses to detect and deduplicate repeated requests).