Leader Election Storm
SEV-2, Significant ImpactFan-Out propagation · consistency · Affects 1 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Leader Election Storm begins at the source component. Trigger: Network flap causing intermittent heartbeat loss without actual node failure.
Immediate (T+0) · Signal: Error Rate Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Election events are logged immediately by all distributed coordination systems. Write failure errors to callers appear within seconds of election start. Monitoring election frequency (etcd election counter, Kafka controller epoch changes) detects storms within the first election. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
All write operations to the cluster fail during election. Services depending on the cluster for coordination (etcd: Kubernetes; ZooKeeper: Kafka, HBase; Kafka controller: all Kafka producers and consumers needing metadata) are blocked. For etcd election storms, the entire Kubernetes control plane is unavailable. For Kafka controller election, all topic metadata operations are blocked. Duration of impact equals the sum of election durations across the storm.
Contributing Factors
This operational trigger enables Leader Election Storm: Network flap causing intermittent heartbeat loss without actual node failure
This operational trigger enables Leader Election Storm: Leader node overloaded (CPU, GC, I/O) causing heartbeat latency to exceed follower timeout
This operational trigger enables Leader Election Storm: Election timeout configured too aggressively (less than 3× observed heartbeat latency)
Remediation Plan
Identify the trigger: check if the leader node is overloaded (CPU, I/O, GC logs)
Effort: Minutes to hours (on-call response)
Check network: is there a flapping link? Check network interface error counters
Effort: Minutes to hours (on-call response)
If leader is overloaded: reduce load on the leader node (shed traffic, restart problematic processes)
Effort: Minutes to hours (on-call response)
Measure the 99th percentile heartbeat round-trip time under production load. Set election timeout to 3–5× this value. For etcd with 1ms p99 heartbeat, a 5–10ms election timeout is aggressive; 50–100ms is safer. Kafka session.timeout.ms for brokers should be well above broker heartbeat delivery time.
Effort: 1 day to 1 week
Network I/O saturation from data replication can delay heartbeat delivery. Binding coordination (Raft heartbeats, etcd peer traffic) to a dedicated network interface isolates it from data traffic contention.
Effort: 1 day to 1 week
Monitor leader node CPU, I/O, and GC metrics. When the leader is approaching saturation, proactively step down (force a leader transfer to a healthy follower) before heartbeat timeouts cause uncontrolled election. etcdctl move-leader and Kafka preferred replica election support this.
Effort: 1 day to 1 week
Configure alerts for: error rate spike, log errors, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.