Post-Mortem Framework · Data Consistency: Split-Brain

Split-Brain

SEV-2, Significant Impact

Isolated propagation · consistency · Affects 2 scenario(s)

Severity Classification

Classified as CRITICAL based on failure mode severity. This failure mode appears in 2 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Split-Brain begins at the source component. Trigger: Network partition between primary and replica with no fencing mechanism.

Immediate (T+0) · Signal: Alert

Blast Radius

All writes accepted by both primaries during the split-brain window are at risk; one side's writes are lost when the conflict resolves. The window's duration sets the blast radius directly: a 30-second window at 1,000 writes per second puts up to 30,000 transactions at risk. If the lost transactions are financial ledger entries, inventory decrements, or user-facing state changes, the business impact can require manual reconciliation of every affected record.

Contributing Factors

Trigger Condition: Network partition between primary and replica with no fencinoperational

This operational trigger enables Split-Brain: Network partition between primary and replica with no fencing mechanism

Trigger Condition: Failover automation that promotes a replica before confirminoperational

This operational trigger enables Split-Brain: Failover automation that promotes a replica before confirming the primary has stopped

Trigger Condition: A partition pattern where primary and replica each retain apoperational

This operational trigger enables Split-Brain: A partition pattern where primary and replica each retain application connectivity in their own segment

Mitigation Gaps

MEDIUMKnown mitigator 'leader_election' not in runbook

Add 'leader election' to the runbook. Leader election ensures only one node is authoritative at any time, preventing split-brain by using a consensus protocol that requires a quorum of nodes to agree before a leader is promoted: making it impossible for two nodes to simultaneously believe they are the leader.

MEDIUMKnown mitigator 'outbox_pattern' not in runbook

Add 'outbox pattern' to the runbook. The outbox pattern eliminates split-brain between a database write and a message broker publish by writing both the domain record and the outbox event in a single ACID transaction, ensuring events are published if and only if the database write committed.

Remediation Plan

ImmediateImmediately isolate both nodes from all application traffic to stop further dive

Immediately isolate both nodes from all application traffic to stop further divergent writes

Effort: Minutes to hours (on-call response)

ImmediateDetermine which node has the most recent data (compare pg_current_wal_lsn or the

Determine which node has the most recent data (compare pg_current_wal_lsn or the transaction timestamp range on each)

Effort: Minutes to hours (on-call response)

ImmediateDemote the node with less data: stop PostgreSQL and remove its primary designati

Demote the node with less data: stop PostgreSQL and remove its primary designation

Effort: Minutes to hours (on-call response)

Short-TermUse synchronous replication with quorum commit

Configure synchronous_standby_names = 'ANY 1 (replica1, replica2)'. The primary will not confirm a write until at least one replica acknowledges, so a primary that loses all replica connectivity stalls on new writes rather than completing ones no replica can confirm. This bounds what the original primary can do during a partition; it does not by itself fence a promoted replica, so pair it with an explicit fencing mechanism. Adds one round trip of write latency.

Effort: 1 day to 1 week

Short-TermDisable automatic failover and require manual promotion

Require a human operator to confirm the primary is unreachable before promoting a replica. Eliminates automated split-brain at the cost of a longer mean time to recovery during genuine primary failures.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: alert, replication lag, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermImplement STONITH fencing

Before a new primary starts accepting writes, the failover manager fences the old one: cutting its network access, powering it off via IPMI, or revoking its cloud network interface, and only then releasing the new primary to accept writes. Patroni with etcd implements this via a leader key; cloud environments can use security-group changes or instance-stop APIs. The cost is the time the fencing action itself takes, during which the system is unavailable rather than split.

Effort: 1–4 sprints

Long-TermUse consensus-based leader election (Raft)

Replace ad-hoc failover scripts with a Raft-based consensus system (etcd, Consul, Patroni with a DCS). Electing a leader requires a majority, and a node that cannot reach one cannot be elected, so two disjoint groups cannot each produce a valid leader in the same term. This is the mechanism, not just a policy: it holds as long as the storage layer's writes are actually gated by the elected leader's term.

Effort: 1–4 sprints

Long-TermEliminate cross-scenario Split-Brain exposure

Split-Brain affects 2 architecture scenarios (Financial Ledger Platform, Gaming Backend Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.