Split-Brain

A failover mechanism promotes a new leader without confirming the old one has stopped, so two nodes simultaneously believe they hold the primary role and both accept writes. The two histories diverge, and when the partition that triggered the failover heals, one set of committed transactions must be discarded.

Split-brain is the specific failure that happens when a network_partition triggers a failover, and the failover has no way to guarantee the old primary has actually stopped accepting writes before the new one starts. It is not partition tolerance gone wrong in the abstract; it is a concrete gap in the promotion procedure.

In a PostgreSQL primary-standby setup without fencing, the primary and standby sit behind a virtual IP, each with its own connectivity to application servers. A partition isolates the primary from the standby but not from either one's own application servers. Patroni or repmgr on the standby detects the primary is unreachable and, after the failover timeout (typically 30 seconds), promotes the standby. Both nodes now accept writes: the original primary from its own application servers, which never learned it should stop, and the new primary from its own. When the network heals, the two write-ahead logs have diverged with conflicting committed transactions, and one side's must be discarded.

In Redis Sentinel with three sentinels, a partition isolating the primary from two of the three sentinels triggers promotion of a replica. If the original primary keeps connectivity to some application instances (its own availability zone, say), those instances keep writing to it. When the partition heals, the original primary holds writes the new primary does not, and because Redis replication is one-directional, those writes are simply lost during forced resynchronization rather than merged.

Fencing is the defense, and there are two different mechanisms sold under that name. STONITH-style fencing (Shoot The Other Node In The Head) acts on the old primary from the outside: cut its network access, power it off via IPMI, revoke its cloud network interface. It works, but it requires infrastructure access and takes time to execute. Fencing-token-style fencing acts at the storage layer instead: every write carries a monotonically increasing token from the current leadership term, and the storage layer rejects any write bearing a token older than the highest it has already accepted. A demoted primary that never got the message keeps sending writes with its old token, and those are rejected on arrival rather than accepted and later discovered to conflict; no external action against the old primary is required. The leader_election pattern covers fencing tokens as the general-purpose mechanism; STONITH is the infrastructure-level equivalent used when the leadership role itself controls a physical or network resource rather than a storage write path.

Consensus-based election prevents split-brain by construction, but the precise claim is narrower than "impossible to have two leaders": Raft guarantees at most one leader per term, because election requires a majority vote and any two majorities of the same cluster must overlap by at least one node, so two disjoint groups cannot each elect a leader in the same term. A new term with a new leader can still follow an old one; what cannot happen is two simultaneously valid leaders in one term, which is exactly the split-brain condition.

The data damage from split-brain is permanent once it occurs: committed transactions on the demoted node are lost. Depending on what those transactions were (financial writes, order state changes, account modifications), the damage may require manual reconciliation or may be unrecoverable.