Leader Election Storm
criticalSummary
A distributed cluster repeatedly cycles through leader election: each new leader is deposed shortly after taking over: causing the cluster to be unavailable for write operations for most of the storm duration.
Description
Distributed systems that require a single leader (Raft-based etcd, Kafka controller, ZooKeeper) rely on a stable leader to process writes. During leader election, the cluster cannot commit new writes: callers receive "not leader" or timeout errors. A brief election (< 1 second in a healthy cluster) is acceptable. An election storm: where elections occur repeatedly in rapid succession: can cause minutes of write unavailability.
Mechanisms that cause election storms:
Network flap: a network interface or switch port flaps (link comes up and down repeatedly). The leader's heartbeat to followers alternates between reaching and not reaching them. Followers time out and start elections. The network recovers, a leader is elected, but the flap recurs, starting another election.
Overloaded leader: the leader node is under CPU or I/O pressure (GC pause, disk I/O saturation). Heartbeat send latency exceeds the follower timeout. Followers initiate election. The new leader is elected from a healthy node, but if all candidates are overloaded, the new leader immediately begins missing heartbeats. The election storm continues until load is reduced.
Clock skew: election timeout logic that uses wall clocks can be affected by clock skew. If the leader's clock jumps forward due to NTP correction, it may believe its lease has expired and step down unnecessarily.
Split-brain variant: aggressive election timeouts in a partitioned network cause both partitions to elect leaders simultaneously. With Raft, this should not result in split-brain (the minority cannot achieve quorum) but it does result in elevated election frequency and reduced availability for the minority partition.
Kafka controller election: the Kafka controller manages partition leadership for all topics. Controller election is triggered when the controller node fails or is overloaded. During controller election, broker metadata is unavailable: producers and consumers may not be able to get metadata to discover partition leaders. Kafka with KRaft mode (replacing ZooKeeper) has made controller election faster and more stable.
etcd election impact: services that depend on etcd for distributed locks, leader election, or configuration (Kubernetes itself) lose access to these primitives during election. Kubernetes controller manager and scheduler cannot process new work during an etcd election storm.
Election frequency is a critical health metric: more than one election per hour in a healthy cluster is a warning sign. More than one per 10 minutes indicates an active problem.
Characteristics
Triggers
- ·Network flap causing intermittent heartbeat loss without actual node failure
- ·Leader node overloaded (CPU, GC, I/O) causing heartbeat latency to exceed follower timeout
- ·Election timeout configured too aggressively (less than 3× observed heartbeat latency)
- ·Split-brain from network partition where minority partition repeatedly attempts elections
- ·etcd or ZooKeeper session expiry under heavy coordination traffic
Detection Signals
Mitigation Strategies
Measure the 99th percentile heartbeat round-trip time under production load. Set election timeout to 3–5× this value. For etcd with 1ms p99 heartbeat, a 5–10ms election timeout is aggressive; 50–100ms is safer. Kafka session.timeout.ms for brokers should be well above broker heartbeat delivery time.
Network I/O saturation from data replication can delay heartbeat delivery. Binding coordination (Raft heartbeats, etcd peer traffic) to a dedicated network interface isolates it from data traffic contention.
Monitor leader node CPU, I/O, and GC metrics. When the leader is approaching saturation, proactively step down (force a leader transfer to a healthy follower) before heartbeat timeouts cause uncontrolled election. etcdctl move-leader and Kafka preferred replica election support this.
Track the leader election counter metric for etcd, ZooKeeper, and Kafka. Alert when elections occur more than once per hour. This provides early warning before a storm develops.
Recovery Steps
- 1.Identify the trigger: check if the leader node is overloaded (CPU, I/O, GC logs)
- 2.Check network: is there a flapping link? Check network interface error counters
- 3.If leader is overloaded: reduce load on the leader node (shed traffic, restart problematic processes)
- 4.Force a leader transfer to a healthy node if the current leader is consistently problematic
- 5.Verify election frequency returns to zero after addressing the trigger
- 6.Review election timeout configuration against measured heartbeat latency
Estimated recovery time: A single election completes in 1–10 seconds in a healthy cluster. An election storm requires addressing the underlying trigger first: resolving network flap or leader overload. Once the trigger is resolved, storm subsides within one election cycle (seconds to minutes).
Affected Systems
Patterns
Technologies
Basis
Well-documented failure mode for etcd, Kafka, and ZooKeeper; timeout tuning recommendations are in official documentation
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.