Partial Service Failure
partialSummary
A subset of service instances fails while others remain healthy, producing a low aggregate error rate that masks significant per-instance failures and causes consistent errors for specific request patterns or user segments.
Description
Partial failure is the hardest variant of service failure to detect and diagnose. The aggregate service appears mostly healthy: overall error rate is 2%, latency is elevated but not catastrophically so. But some users experience consistent failures on every request. The discrepancy between aggregate health and per-instance health is the defining characteristic.
Causes and patterns:
Partial bad deploy: a rolling deployment has pushed a new version to 30% of pods. The new version has a bug that triggers on certain input patterns. Requests to the old pods (70%) succeed; requests to new pods (30%) fail. Load balancer routes randomly, so affected users hit the bad pod 30% of the time. Their error rate is 30%, not the aggregate 2%.
AZ-specific issue: one availability zone has a network issue affecting its pods. Requests to pods in that AZ fail; pods in other AZs succeed. Users whose requests are sticky to the affected AZ (session affinity, geographic routing) experience 100% failure. Users in other AZs experience no failure. Aggregate error rate is proportional to the AZ's traffic share.
Shard-specific degradation: a specific database shard is degraded (hot partition, index corruption, disk I/O issue). Only requests whose keys hash to that shard fail. Users whose data is on that shard experience complete failure; other users are unaffected.
Data-dependent bug: a bug that triggers only on records with certain properties (large records, specific character encoding, records from before a migration). Not an infrastructure issue at all: looks like one because the failures appear randomly distributed.
Load balancer masking: Kubernetes and load balancers remove unhealthy instances quickly (based on health checks or error rate circuit). This is correct behavior: but it means partial failure shrinks the service's effective capacity. As bad pods are removed, the remaining healthy pods handle more load. A partial failure that removes 30% of capacity can trigger capacity pressure on the remaining 70%.
Detection requirement: per-instance and per-shard metrics. Aggregate metrics (cluster-level error rate, cluster-level latency) can be healthy even when individual instances are failing. Only per-instance error rate metrics and structured logs with instance ID reveal the pattern.
Characteristics
Triggers
- ·Rolling deployment with a bug that triggers on specific input patterns
- ·Single availability zone network degradation affecting its pods
- ·Database shard degradation affecting only keys in that shard's range
- ·Memory leak or GC pressure on a subset of pods
- ·Kubernetes node issue affecting pods scheduled on that node
Detection Signals
Mitigation Strategies
Include pod name, node name, and availability zone as labels in all error rate metrics. Alert on per-instance error rate, not just aggregate. A single pod at 50% error rate should alert even if the aggregate is 2%.
Every log line must include the pod name, host, or instance ID as a structured field. This is the minimum requirement for per-instance error correlation during investigation. Without it, distinguishing per-instance from per-user issues requires log sampling.
Deploy new versions to 1–5% of pods first. Monitor per-instance error rate and latency for 5–15 minutes. Promote only if canary metrics match baseline. This limits the blast radius of a bad deploy to the canary fraction.
Monitor error rate and latency per availability zone separately. Alert on AZ-level divergence (e.g., one AZ error rate 10× others). Configure load balancer to deprioritize a degraded AZ when detected.
Recovery Steps
- 1.Check per-pod error rate metrics: identify which specific pods are failing
- 2.Check per-AZ metrics: is the failure correlated with a specific AZ?
- 3.For partial deploy: identify the bad pods (new version), roll back to previous version
- 4.For AZ issue: mark the AZ as degraded in load balancer; escalate to infrastructure team
- 5.For shard degradation: identify the affected shard, check database metrics, escalate to DBA
- 6.Verify aggregate error rate returns to baseline after isolating failing instances
Estimated recovery time: Rollback of a partial bad deploy: minutes (Kubernetes rollout undo). AZ issue resolution: depends on infrastructure root cause: minutes to hours. Shard degradation: depends on database issue: minutes to hours.
Affected Systems
Patterns
Technologies
Basis
Common production failure pattern; per-instance observability requirement is well-documented in SRE literature
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
The inbox pattern ensures idempotent message processing by deduplicating based on message ID, so partial failures that cause redelivery do not result in duplicate side effects.
Full relationship →Used In Architecture Scenarios
Write-Heavy Application
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Realtime Collaboration
An online multiplayer game backend built around authoritative server game state synchronization. Game rooms run as distributed state machines where the server is the single source of truth for game state: client predictions are reconciled against the server state at every tick. WebSocket connections provide low-latency bidirectional communication for state deltas. Redis stores active game room state (in-flight, with sub-millisecond access), session affinity tokens (routing players to the same server instance as their game room), and matchmaking queues. PostgreSQL is the durable store for player profiles, persistent inventory, leaderboards, and achievement records. Kafka carries post-game event streams for analytics, anti-cheat processing, and achievement evaluation. NATS provides low-latency pub/sub for intra-cluster game state broadcasting when multiple game server instances must coordinate on shared state. Event sourcing records every game action as an immutable event for replay, dispute resolution, and anti-cheat audit.
Financial Ledger
An electronic health record (EHR) architecture built around strict auditability, HIPAA compliance, and append-only correctness. Clinical records are mutable by design (amendments, addenda) but corrections must be explicitly attributed, not silently overwritten. Event sourcing provides a reconstructable audit log; PostgreSQL row-level security enforces patient-level access control at the database layer; Kafka streams HL7 FHIR events to downstream clinical systems. The architecture must support breach detection, access auditing, and state reconstruction at any historical point: not just current state retrieval.
Event-Driven System
A multi-channel notification delivery architecture that accepts upstream business events (order placed, payment received, comment posted, threshold alert triggered) and routes them to per-channel delivery workers (push via FCM/APNs, email via SendGrid, SMS via Twilio, in-app via WebSocket). Kafka carries raw business events from upstream producers. RabbitMQ handles per-channel fan-out with separate exchanges and queues per delivery channel, isolating email queue backlog from push notification delivery. PostgreSQL provides durable notification state tracking (sent, failed, bounced, suppressed). Redis enforces per-user rate limiting (notification frequency caps to prevent fatigue) and stores deduplication tokens to prevent duplicate sends across retry attempts. The inbox pattern on the consumer side ensures idempotent delivery even when Kafka produces duplicate events.