Partial Service Failure
partialContainedcascading failure · fan out propagation
Blast Radius
0%
0/19 nodes
Time to Detect
Aggregate error rate monitoring detects partial failures proportionally to the failing fraction: a 2% aggregate error rate from 30% bad pods takes minutes to breach alert thresholds. Per-instance error rate monitoring (at pod granularity) detects within seconds. Kubernetes removes failing pods from load balancing within 10–30 seconds of readiness probe failure.
Preventive Mitigations
1
Confidence
StrongFailure Cascade
(no topology nodes mapped for this failure mode)
This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.
Severity at this step: partial
Detection Signals
Recovery time estimate: Rollback of a partial bad deploy: minutes (Kubernetes rollout undo). AZ issue resolution: depends on infrastructure root cause: minutes to hours. Shard degradation: depends on database issue: minutes to hours.
Mitigation Checklist(1 preventive, 4 reactive)
Deploy new versions to 1–5% of pods first. Monitor per-instance error rate and latency for 5–15 minutes. Promote only if canary metrics match baseline. This limits the blast radius of a bad deploy to the canary fraction.
Monitor error rate and latency per availability zone separately. Alert on AZ-level divergence (e.g., one AZ error rate 10× others). Configure load balancer to deprioritize a degraded AZ when detected.
The inbox pattern ensures idempotent message processing by deduplicating based on message ID, so partial failures that cause redelivery do not result in duplicate side effects.
Include pod name, node name, and availability zone as labels in all error rate metrics. Alert on per-instance error rate, not just aggregate. A single pod at 50% error rate should alert even if the aggregate is 2%.
Every log line must include the pod name, host, or instance ID as a structured field. This is the minimum requirement for per-instance error correlation during investigation. Without it, distinguishing per-instance from per-user issues requires log sampling.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.