Post-Mortem Framework · Cascading: Partial Service Failure

Partial Service Failure

SEV-3, Limited Impact

Fan-Out propagation · cascading · Affects 5 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 5 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Partial Service Failure begins at the source component. Trigger: Rolling deployment with a bug that triggers on specific input patterns.

Immediate (T+0) · Signal: Error Rate Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

Aggregate error rate monitoring detects partial failures proportionally to the failing fraction: a 2% aggregate error rate from 30% bad pods takes minutes to breach alert thresholds. Per-instance error rate monitoring (at pod granularity) detects within seconds. Kubernetes removes failing pods from load balancing within 10–30 seconds of readiness probe failure. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Downstream of dependents (fan-out)

Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.

Within minutes of initial failure · Signal: Multiple services reporting elevated error rates

Blast Radius

Directly affects users or request patterns routed to failing instances. Indirectly affects all users if health check removal of failing instances reduces total service capacity below demand, causing overload on healthy instances.

Contributing Factors

Trigger Condition: Rolling deployment with a bug that triggers on specific inpuoperational

This operational trigger enables Partial Service Failure: Rolling deployment with a bug that triggers on specific input patterns

Trigger Condition: Single availability zone network degradation affecting its poperational

This operational trigger enables Partial Service Failure: Single availability zone network degradation affecting its pods

Trigger Condition: Database shard degradation affecting only keys in that shardoperational

This operational trigger enables Partial Service Failure: Database shard degradation affecting only keys in that shard's range

Mitigation Gaps

MEDIUMKnown mitigator 'inbox_pattern' not in runbook

Add 'inbox pattern' to the runbook. The inbox pattern ensures idempotent message processing by deduplicating based on message ID, so partial failures that cause redelivery do not result in duplicate side effects.

Remediation Plan

ImmediateCheck per-pod error rate metrics: identify which specific pods are failing

Check per-pod error rate metrics: identify which specific pods are failing

Effort: Minutes to hours (on-call response)

ImmediateCheck per-AZ metrics: is the failure correlated with a specific AZ?

Check per-AZ metrics: is the failure correlated with a specific AZ?

Effort: Minutes to hours (on-call response)

ImmediateFor partial deploy: identify the bad pods (new version), roll back to previous v

For partial deploy: identify the bad pods (new version), roll back to previous version

Effort: Minutes to hours (on-call response)

Short-TermEmit per-instance error rate metrics and alert at instance granularity

Include pod name, node name, and availability zone as labels in all error rate metrics. Alert on per-instance error rate, not just aggregate. A single pod at 50% error rate should alert even if the aggregate is 2%.

Effort: 1 day to 1 week

Short-TermInclude instance ID in all structured log lines

Every log line must include the pod name, host, or instance ID as a structured field. This is the minimum requirement for per-instance error correlation during investigation. Without it, distinguishing per-instance from per-user issues requires log sampling.

Effort: 1 day to 1 week

Short-TermUse canary deployments to validate new versions before full rollout

Deploy new versions to 1–5% of pods first. Monitor per-instance error rate and latency for 5–15 minutes. Promote only if canary metrics match baseline. This limits the blast radius of a bad deploy to the canary fraction.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: error rate spike, latency spike, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermEliminate cross-scenario Partial Service Failure exposure

Partial Service Failure affects 5 architecture scenarios (Distributed Job Queue Platform, E-Commerce Order Platform, Gaming Backend Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.