Leader Election

Elect a single active node among a group of distributed replicas to perform writes, coordinate state, or execute singleton tasks, preventing conflicting operations from multiple concurrent actors. The guarantee this actually provides depends entirely on the election mechanism: consensus-based election can be made linearizable, lease-based election cannot, no matter how carefully it is tuned.

Distributed systems with multiple replicas cannot safely perform singleton operations (writes, scheduled tasks, cluster coordination) without a mechanism to designate exactly one active actor; without it, concurrent actors produce conflicting state or duplicate work.

In any distributed system where multiple nodes can act concurrently, some operations must run on exactly one node at a time: primary database writes, scheduled job execution, distributed lock management, cluster state coordination. Running these on more than one node at once causes conflicts, duplicate work, or split-brain. Leader election is the mechanism nodes use to agree on which one of them holds that role right now.

Election quorum versus read/write quorum. A consensus system's leader election requires a majority (N/2 + 1) of nodes to agree, and because any two majorities of the same cluster must overlap by at least one node, at most one leader can be elected per term; two disjoint groups cannot both produce a valid leader at the same time. This is a different concept from the read/write quorum (W + R > N) used by leaderless replication systems like Dynamo, where any sufficiently large set of nodes can satisfy a read or write independently, with no election at all. Both are called "quorum," and conflating them is a common source of confusion: one elects a single coordinator, the other coordinates without electing anyone.

Consensus-based election (Raft, used by etcd, CockroachDB, Consul) can give linearizable writes, but the guarantee depends on more than the election, and more than majority overlap alone. A write is linearizable because it is not acknowledged until a majority of replicas has it. Surviving a leader change needs a second ingredient beyond that overlap: a candidate can only win a vote from a node whose log is at least as up to date as its own (compared by last log term, then index), so Raft's RequestVote handling refuses a candidate that is missing entries the voter already has. Majority overlap on its own only guarantees that some voter in the next election also saw the old commit; the up-to-date check is what forces that voter to withhold its vote from any candidate lacking that entry, and only their combination guarantees the new leader has every write the old one committed. Citing the overlap alone as the reason is a common shorthand, but it understates the mechanism: a majority-quorum system without Raft's vote-eligibility rule does not get this durability property for free. Reads do not inherit either guarantee automatically; a read served by whichever node happens to answer, including a follower or a leader that has been silently partitioned away, can return stale data. A linearizable read requires either routing to the current leader with a read-index or lease check confirming it is still leader, or a follower proving it has applied the log up to a known committed index before answering. Leader election gives you a leader; it does not by itself give you fresh reads.

Lease-based election (Redis SETNX with a TTL) is a fundamentally weaker mechanism, and the weakness is not a tuning problem, it is structural. A node acquires leadership by setting a key with SETNX and a TTL, must renew it before expiry, and loses it to another node if it does not. This depends on clock and timing assumptions that distributed systems cannot actually guarantee: a leader that stalls (a GC pause, disk I/O, a scheduler preemption) for longer than the TTL has its lease silently expire while it still believes it holds leadership, and a second node can acquire the lease and start acting as leader before the first one notices. No amount of TTL tuning fixes this, because it is impossible to bound how long a real process can be paused without its own knowledge; shortening the TTL only makes false expiry under normal load more likely, and lengthening it only makes the unsafe window longer. This is acceptable for low-stakes coordination like deduplicating a cron job, where a brief double-run is a minor cost, and it is not acceptable for coordinating writes to shared state, where the same failure is a correctness violation.

Fencing tokens are what makes any leader-exclusive operation safe even when the election mechanism itself cannot guarantee uniqueness. Every operation the leader performs against shared storage carries a monotonically increasing token tied to its term or lease generation; the storage layer remembers the highest token it has accepted and rejects anything lower. A stale leader that wakes from a long pause and tries to write with token 33 is rejected once the storage layer has already seen token 34 from its successor, even though the stale leader still believes itself to be in charge. Fencing tokens do not prevent a second leader from existing briefly; they prevent that second leader's actions from taking effect. This is the general-purpose version of what split_brain calls STONITH fencing for a primary-replica pair: fencing at the storage layer instead of fencing the node itself.

PostgreSQL advisory locks provide a simpler, single-instance form of election: a session holds a session-level advisory lock while acting as leader, and the lock releases immediately if the session disconnects, letting another node acquire it. This is reliable within one PostgreSQL instance but has no meaning across multiple database nodes, since there is only one lock table to hold it.