Split-Brain
criticalContainedconsistency failure · isolated propagation
Blast Radius
0%
0/18 nodes
Time to Detect
Split-brain is often undetected until the network heals and the conflict surfaces. Real-time detection requires monitoring that asserts at most one primary exists at a time; checking topology every 10 to 30 seconds catches the condition within seconds of it forming. Without topology monitoring, it is discovered during reconciliation after the network heals, minutes to hours after the divergence began.
Preventive Mitigations
4
Confidence
StrongFailure Cascade
(no topology nodes mapped for this failure mode)
This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.
Severity at this step: critical
Detection Signals
Recovery time estimate: 1 to 6 hours for technical recovery. Business data reconciliation may take days if the lost transactions involve financial, inventory, or user-visible state. Damage scope is proportional to the split-brain window's duration and the write rate during it.
Mitigation Checklist(4 preventive, 2 reactive)
Before a new primary starts accepting writes, the failover manager fences the old one: cutting its network access, powering it off via IPMI, or revoking its cloud network interface, and only then releasing the new primary to accept writes. Patroni with etcd implements this via a leader key; cloud environments can use security-group changes or instance-stop APIs. The cost is the time the fencing action itself takes, during which the system is unavailable rather than split.
Replace ad-hoc failover scripts with a Raft-based consensus system (etcd, Consul, Patroni with a DCS). Electing a leader requires a majority, and a node that cannot reach one cannot be elected, so two disjoint groups cannot each produce a valid leader in the same term. This is the mechanism, not just a policy: it holds as long as the storage layer's writes are actually gated by the elected leader's term.
Configure synchronous_standby_names = 'ANY 1 (replica1, replica2)'. The primary will not confirm a write until at least one replica acknowledges, so a primary that loses all replica connectivity stalls on new writes rather than completing ones no replica can confirm. This bounds what the original primary can do during a partition; it does not by itself fence a promoted replica, so pair it with an explicit fencing mechanism. Adds one round trip of write latency.
Require a human operator to confirm the primary is unreachable before promoting a replica. Eliminates automated split-brain at the cost of a longer mean time to recovery during genuine primary failures.
Leader election ensures only one node is authoritative at any time, preventing split-brain by using a consensus protocol that requires a quorum of nodes to agree before a leader is promoted: making it impossible for two nodes to simultaneously believe they are the leader.
The outbox pattern eliminates split-brain between a database write and a message broker publish by writing both the domain record and the outbox event in a single ACID transaction, ensuring events are published if and only if the database write committed.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.