Post-Mortem Framework · Network: Network Partition

Network Partition

SEV-2, Significant Impact

Fan-Out propagation · network · Affects 1 scenario(s)

Severity Classification

Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.

Propagation Chain

Origin component

Network Partition begins at the source component. Trigger: AWS availability zone network degradation or inter-AZ routing failure.

Immediate (T+0) · Signal: Error Rate Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

Connection refused or timeout errors surface immediately, within seconds. Cluster coordination systems detect node unreachability within their configured failure detection window (typically 5 to 30 seconds). Application-level health checks typically catch it within 10 to 60 seconds. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Downstream of dependents (fan-out)

Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.

Within minutes of initial failure · Signal: Multiple services reporting elevated error rates

Blast Radius

Every cross-partition service call fails for the duration. CP systems stop accepting writes (and current reads) from the minority side; AP systems keep serving but accumulate divergence that must be resolved later. Coordination services (etcd, ZooKeeper) losing quorum during a partition cascades to every service that depends on them for configuration or leader election, turning a network event into a wide application outage even for services with no direct network problem of their own.

Contributing Factors

Trigger Condition: AWS availability zone network degradation or inter-AZ routinoperational

This operational trigger enables Network Partition: AWS availability zone network degradation or inter-AZ routing failure

Trigger Condition: Firewall rule change blocking inter-node communicationoperational

This operational trigger enables Network Partition: Firewall rule change blocking inter-node communication

Trigger Condition: Network equipment failure on a cross-datacenter linkoperational

This operational trigger enables Network Partition: Network equipment failure on a cross-datacenter link

Remediation Plan

ImmediateIdentify partition scope: which nodes cannot reach which? Check network connecti

Identify partition scope: which nodes cannot reach which? Check network connectivity between AZs

Effort: Minutes to hours (on-call response)

ImmediateCheck cluster membership (etcd member list, Kafka broker metadata): do members s

Check cluster membership (etcd member list, Kafka broker metadata): do members see each other?

Effort: Minutes to hours (on-call response)

ImmediateFor CP systems: confirm the majority side is functional; the minority will stay

For CP systems: confirm the majority side is functional; the minority will stay unavailable until the partition heals

Effort: Minutes to hours (on-call response)

Short-TermRoute writes to a single AZ and reads to any AZ

Send writes only to the primary AZ; serve reads from any AZ. On partition, the primary AZ keeps accepting writes at the cost of reads from the isolated AZ going stale or unavailable. AZ-aware load balancing (Kubernetes topology spread constraints) supports this routing.

Effort: 1 day to 1 week

Short-TermUse quorum-based consensus (Raft or Paxos) for anything requiring a single leader

A quorum-based consensus system can only elect a leader with majority agreement, so a partition minority cannot independently elect one and stops accepting writes. etcd, CockroachDB, and Kafka in KRaft mode implement this correctly; the specific failure this prevents, and what happens without it, is detailed in split_brain.

Effort: 1 day to 1 week

Short-TermTest partition behavior with chaos engineering

Inject network partitions in staging with tc netem or a chaos engineering tool (Chaos Monkey, Chaos Mesh) and verify the system behaves as designed: CP systems stop writes on the minority side, AP systems converge correctly once the partition heals.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: error rate spike, replication lag, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermChoose CP or AP deliberately per data store, not by inherited default

Coordination state and financial data need CP; user activity feeds and analytics can tolerate AP. The cost of CP is minority-side unavailability during a partition; the cost of AP is divergence that must be resolved after. Pick per data store based on which cost is acceptable for that data, rather than accepting whatever a chosen technology defaults to.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.