Split-Brain During Network Partition
A network partition between PostgreSQL primary and standby causes the standby to promote itself to primary after the failover timeout expires. Both nodes accept writes simultaneously: the classic split-brain condition. When the partition heals, the original primary reconnects to find a diverged WAL timeline. Conflicting writes from both nodes cannot be automatically reconciled. One node's writes must be discarded, and the decision about which writes to keep is a manual, irreversible data operation.
PostgreSQL primary + synchronous standby + Patroni/pgautofailover HA manager with 30-second failover timeout
Degradation Replay
Nominal: Primary-Standby Fully Synchronized
Primary and standby communicating normally; replication lag near zero
Critical: 2 exceeds critical threshold of 0 count
- ·Replication lag < 100ms: WAL streaming to standby in near real-time
- ·HA manager reporting both nodes healthy
- ·Both application and HA manager can reach primary on expected endpoint
- ·No timeline divergence: both nodes on same WAL timeline
- ·Standby in hot-standby mode: available for read queries
- ·Primary WAL sender streaming continuously to standby
- !Normal operation: HA cluster healthy and synchronized
Operational simulation model only, not a production forecast. Degradation stages are derived from structured operational knowledge, not measured telemetry. Do not use for capacity planning or incident response.
Run With Your Parameters
Adjust the parameters below to see how metric values shift across degradation stages. Formulas are deterministic: same inputs always produce the same output.
Simulation Parameters
Monthly probability of a node failure event
Computed Degradation Stages
Primary and standby communicating normally; replication lag near zero
Network partition between primary and standby begins; HA manager loses visibility to primary
Failover timeout expires; HA manager promotes standby to primary; both nodes accepting writes
Network partition heals; original primary reconnects; PostgreSQL detects conflicting WAL timelines
Original primary fenced; diverged WAL documented; cluster rebuilt with new primary and fresh standby
Threshold Events
System availability at 99.75% with 1 replica(s). Failover time 30s exposes 150,000 requests per event.
System availability at 84.79% with 1 replica(s). Failover time 30s exposes 150,000 requests per event.
Interpretation
With 1 replica(s) and 5% monthly failure rate, availability at 69.82%. Failover window: 30s.
Single point of failure: 1 replica(s) insufficient for 5% failure rate
Add 1 more replicas to achieve 99.9% availability target. Reduce failover time from 30s to <10s with pre-warmed standby. Test failover monthly in production: MTTR is only reliable when practiced.
Parameterized simulation: not a production forecast. Values derived from deterministic formulas applied to your parameters. Do not use for capacity planning or operational decisions without validation.
Propagation Model
HA manager cannot reach primary for failover_timeout seconds; promotes standby to primary; both nodes now accept writes: original primary from existing application connections, new primary from new connections
Stabilizes: Does not stabilize: divergence grows with every write accepted by both nodes during the partition
Every second the split-brain persists, both primaries accumulate independent WAL; at 100 writes/second on each node, divergence grows at 200 writes/second; after 5 minutes: 60,000 conflicting writes
Stabilizes: Divergence freezes when partition heals and one node is fenced (demoted)
When partition heals, PostgreSQL cannot automatically reconcile diverged WAL timelines; one node must be demoted and its WAL since the divergence point is discarded: all writes on that node since promotion are lost
Stabilizes: Resolved only by manual operator decision on which node's writes to keep
Recovery Patterns
Fence original primary + rebuild standby from new primary
Cluster rebuild: 30-60 minutes via pg_basebackup; reconciliation: hours to days depending on divergence- ·Rebuilding standby from pg_basebackup requires transferring full database: hours of replication lag before standby is caught up
- ·Lost writes from original primary timeline are permanent: no merge path exists
- !If fencing fails and original primary remains running, split-brain will recur on partition heal
Implement synchronous replication with synchronous_standby_names
Hours to implement; prevents all future split-brain at cost of write latency- ·synchronous_commit=on requires standby acknowledgment before commit returns: adds 1-5ms round-trip latency per write
- ·If standby is unreachable, writes block indefinitely: availability tradeoff
- !Synchronous replication makes primary unavailable if standby is down: requires timeout tuning
Operational Summary
Split-brain is the CAP theorem applied in the worst way: a system claiming CP properties (consistency + partition tolerance) but actually providing AP behavior (availability + partition tolerance, at the cost of consistency) because fencing was not configured. The result is two primaries accepting writes simultaneously on diverged timelines that PostgreSQL cannot merge.
The data loss from split-brain is always exactly the writes accepted by the original primary after the standby's promotion. If the partition lasts 5 minutes at 100 writes/second, that is 30,000 potentially lost writes. PostgreSQL has no mechanism to reconcile diverged WAL timelines: one timeline must be discarded entirely, and all writes on it are gone.
Prevention requires three things: proper fencing (STONITH), an appropriate failover timeout, and a quorum-aware HA manager. Any one of these alone is insufficient. Fencing without a long enough timeout causes false positives. A long timeout without fencing still allows split-brain on a genuine partition. Quorum without fencing still risks split-brain if the quorum decision cannot be enforced.