Cascading Failure

Trip a circuit breaker when the downstream service error rate or latency exceeds a threshold. Once open, fail fast rather than holding threads. Prevents thread pool exhaustion in the caller when the callee is slow. Recovery is automatic via HALF-OPEN probe.

Assign a dedicated, fixed-size thread pool to each downstream dependency. Payment service threads cannot consume checkout service's allocation for database calls. A slow payment API exhausts only its own 10-thread pool, not the shared 100-thread pool used for everything else.

When queue depth or response latency exceeds a threshold, actively reject new requests with HTTP 503 rather than accepting them and timing out. Controlled shedding prevents queue depth from growing unboundedly and allows faster recovery by keeping the service within its operational range.

Bulkhead isolation partitions resources (thread pools, connection pools, queues) per downstream dependency, preventing a slow or failing dependency from consuming all shared resources and causing cascading failure across unrelated services.

Circuit breakers prevent cascading failure by stopping the propagation of downstream errors to upstream callers, converting unbounded connection wait into fast failure with a predictable error response and giving the downstream dependency time to recover without continued load.

Rate limiting at service ingress caps the load each upstream can place on a downstream, preventing the overload cascade triggered by burst traffic from multiple callers.

Set aggressive timeouts on all downstream calls: e.g., payment API timeout = 500ms, total checkout request timeout = 1000ms. Fast failure bounds the maximum thread hold time regardless of downstream behaviour.

Retries must use exponential backoff (wait = base × 2^attempt) with random jitter (wait = wait × (0.5 + random())), and a maximum retry count. Prevents retry storms from re-saturating a recovering service.