Leader Election Storm
criticalContainedconsistency failure · fan out propagation
Blast Radius
0%
0/18 nodes
Time to Detect
Election events are logged immediately by all distributed coordination systems. Write failure errors to callers appear within seconds of election start. Monitoring election frequency (etcd election counter, Kafka controller epoch changes) detects storms within the first election.
Preventive Mitigations
2
Confidence
StrongFailure Cascade
(no topology nodes mapped for this failure mode)
This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.
Severity at this step: critical
Detection Signals
Recovery time estimate: A single election completes in 1–10 seconds in a healthy cluster. An election storm requires addressing the underlying trigger first: resolving network flap or leader overload. Once the trigger is resolved, storm subsides within one election cycle (seconds to minutes).
Mitigation Checklist(2 preventive, 2 reactive)
Network I/O saturation from data replication can delay heartbeat delivery. Binding coordination (Raft heartbeats, etcd peer traffic) to a dedicated network interface isolates it from data traffic contention.
Measure the 99th percentile heartbeat round-trip time under production load. Set election timeout to 3–5× this value. For etcd with 1ms p99 heartbeat, a 5–10ms election timeout is aggressive; 50–100ms is safer. Kafka session.timeout.ms for brokers should be well above broker heartbeat delivery time.
Monitor leader node CPU, I/O, and GC metrics. When the leader is approaching saturation, proactively step down (force a leader transfer to a healthy follower) before heartbeat timeouts cause uncontrolled election. etcdctl move-leader and Kafka preferred replica election support this.
Track the leader election counter metric for etcd, ZooKeeper, and Kafka. Alert when elections occur more than once per hour. This provides early warning before a storm develops.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.