Circuit Breaker
matureSummary
Detect repeated failures to a downstream dependency and stop attempting calls to it for a recovery window, preventing cascading failure and allowing the dependency time to recover.
Problem
When a downstream service becomes slow or unavailable, callers accumulate blocked threads and held connections waiting for timeouts that may take 30 seconds to fire. This resource exhaustion cascades to the caller's callers, propagating failure across the system.
Description
A circuit breaker wraps outbound calls to a downstream service (database, HTTP API, cache, message broker) and tracks the success/failure rate of those calls within a sliding time window. The circuit operates in three states.
CLOSED (normal operation): calls are forwarded to the downstream service. Failures are counted. If the failure rate exceeds a threshold (e.g., 50% of calls in the last 10 seconds fail, or 5 consecutive failures), the circuit trips to OPEN.
OPEN (failing fast): calls are rejected immediately without contacting the downstream service. The caller receives a predictable error (or a fallback response) in microseconds instead of waiting for a connection timeout. This prevents the calling service from accumulating blocked threads or connections waiting for an unavailable dependency, which would otherwise cascade into resource exhaustion. The circuit stays OPEN for a configurable recovery timeout (e.g., 30–60 seconds).
HALF-OPEN (probing recovery): after the recovery timeout expires, the circuit allows a single probe request through. If the probe succeeds, the circuit closes. If it fails, the recovery timeout resets and the circuit returns to OPEN.
Circuit breaker state is typically stored in-process (per-instance) for low latency, but can be externalised to Redis for shared state across replicas when coordinated circuit opening is required. Per-instance state is simpler and sufficient for most cases.
Tradeoffs
Prevents resource exhaustion cascade; fails fast with predictable cost
HALF-OPEN probe detects recovery within one recovery-timeout period
Circuit state transitions are highly observable and alertable events
Open circuit silently drops calls; must ensure callers handle this
Requires calibration per dependency; wrong thresholds cause false trips
Per-instance state leads to inconsistent tripping across replicas
When to use
Service calls a downstream dependency that can fail independently
Any outbound network call is a failure surface; HTTP APIs, databases, caches, and brokers all benefit from circuit breaker protection
Downstream dependency recovery time is predictable (seconds to minutes)
Circuit breakers are most effective when the dependency recovers within the OPEN window; if recovery takes hours, the strategy changes
A degraded or fallback response is acceptable when the circuit is open
The circuit breaker must have somewhere to go when it trips; either a fallback value, a cached response, or a graceful error to the caller
Service is under sustained load with multiple concurrent callers
The benefit of failing fast scales with caller concurrency; a single- threaded caller with short timeouts gets less value from a circuit breaker
When not to use
Downstream call must succeed and no fallback is acceptable
An open circuit still needs to return something; if the operation is non-optional, a retry-with-backoff or queue-based approach is needed
Downstream service does not have transient failure patterns
Circuit breakers solve temporary outages; if failures are permanent, the circuit will never close and the problem is architectural
Operational Requirements
Instrument circuit state transitions as observable events (OPEN/HALF-OPEN/CLOSED)
Circuit trips are the primary signal that a downstream dependency is failing; alert on every OPEN transition in production
Tune failure threshold and recovery timeout per dependency
A database circuit needs different thresholds (lower tolerance) than a non-critical third-party API; use separate instances per dependency
Define and test fallback behaviour for every protected call site
The circuit breaker must return something when open; undefined fallback causes NullPointerExceptions or unexpected behaviour downstream
Test circuit trip and recovery in staging under simulated dependency failure
Chaos engineering or fault injection should verify the circuit trips correctly and that the application degrades gracefully when open
Characteristics
Technologies
Canonical
Alternatives
Relationships
Evolves to
Complements
Basis
Extensively documented pattern with well-understood states and failure modes; threshold tuning challenge is the primary operational risk
Related Architecture Knowledge
Outbound: this entity affects
Circuit breakers prevent cascading failure by stopping the propagation of downstream errors to upstream callers, converting unbounded connection wait into fast failure with a predictable error response and giving the downstream dependency time to recover without continued load.
Tradeoffs
- ·False positives: transient blips can trip the circuit, causing callers to see errors the dependency could have served
- ·Half-open recovery probe rate must be conservative to avoid re-triggering overload on a recovering dependency
- ·Circuit breakers treat the symptom (connection accumulation), not the cause: the downstream issue requires separate remediation
- ·Requires explicit fallback logic at every call site, adding implementation surface area
Circuit breakers fast-fail requests when downstream latency is elevated, releasing connections back to the pool rather than holding them open for slow responses.
Full relationship →Inbound: affects this entity
API gateways are the natural enforcement point for circuit breakers: the gateway intercepts all inbound requests, tracks per-service error rates, and can open circuits to specific backend services while returning cached responses or 503s to callers: without any changes to individual service code.
Tradeoffs
- ·Gateway-level circuit breaking is coarser-grained than per-call circuit breaking in application code
- ·A centralized gateway is itself a potential single point of failure: requires high-availability deployment
- ·Gateway circuit breakers may not capture partial failures within a service (e.g., only one endpoint degraded)
Backpressure controls the rate at which producers emit to consumers; circuit breakers fast-fail calls to overloaded downstream services. Together they provide complete flow control for both producer-consumer and request-response topologies.
Full relationship →Used In Architecture Scenarios
Multi-Tenant SaaS
A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart.
Write-Heavy Application
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Realtime Collaboration
An online multiplayer game backend built around authoritative server game state synchronization. Game rooms run as distributed state machines where the server is the single source of truth for game state: client predictions are reconciled against the server state at every tick. WebSocket connections provide low-latency bidirectional communication for state deltas. Redis stores active game room state (in-flight, with sub-millisecond access), session affinity tokens (routing players to the same server instance as their game room), and matchmaking queues. PostgreSQL is the durable store for player profiles, persistent inventory, leaderboards, and achievement records. Kafka carries post-game event streams for analytics, anti-cheat processing, and achievement evaluation. NATS provides low-latency pub/sub for intra-cluster game state broadcasting when multiple game server instances must coordinate on shared state. Event sourcing records every game action as an immutable event for replay, dispute resolution, and anti-cheat audit.
Event-Driven System
A multi-channel notification delivery architecture that accepts upstream business events (order placed, payment received, comment posted, threshold alert triggered) and routes them to per-channel delivery workers (push via FCM/APNs, email via SendGrid, SMS via Twilio, in-app via WebSocket). Kafka carries raw business events from upstream producers. RabbitMQ handles per-channel fan-out with separate exchanges and queues per delivery channel, isolating email queue backlog from push notification delivery. PostgreSQL provides durable notification state tracking (sent, failed, bounced, suppressed). Redis enforces per-user rate limiting (notification frequency caps to prevent fatigue) and stores deduplication tokens to prevent duplicate sends across retry attempts. The inbox pattern on the consumer side ensures idempotent delivery even when Kafka produces duplicate events.