Replica Divergence

A replica in a single-leader replication topology applies changes differently than the primary did, producing a permanent state difference rather than a temporary lag. Reads from the replica return data the primary never contained, not stale data that will eventually catch up, and nothing about the replica's own error state signals that anything is wrong.

This entity is scoped to single-leader (primary/replica) replication, PostgreSQL and MySQL being the canonical cases, where there is exactly one authoritative source and a replica's job is to reproduce it exactly. Divergence there is always a bug: the replica should be byte-for-byte reconstructable from the primary's change stream, and any difference is a defect to fix. Leaderless and multi-leader systems (Cassandra, DynamoDB, Riak) do not have a single authoritative source to diverge from; temporary disagreement between replicas is an expected, designed-for state there, resolved by read repair, hinted handoff, and anti-entropy (typically Merkle-tree) repair rather than by the row-based-replication and checksum tooling this entity covers. None of this entity's mitigations (binlog row format, a read-only flag, pt-table-checksum) apply to that model, which is why it is out of scope here rather than folded in.

Divergence is distinct from replication lag. Lag is temporal: the replica is correct but behind, and will catch up. Divergence is permanent: the replica holds data that does not match the primary and never will without manual intervention, because nothing in ordinary replication will overwrite it back to the correct value.

Divergence causes, in the single-leader model:

1. Non-deterministic functions in statement-based replication: RAND(), NOW(), UUID()

produce different values when the same SQL statement re-executes on the replica.

"INSERT INTO t VALUES (NOW())" inserts the primary's execution time on the

primary, but a different time on the replica if the statement itself, rather than

its resulting row values, is what gets shipped. Row-based replication eliminates

this entirely by shipping the actual row values instead of the statement.

2. Direct writes to a replica: a misrouted connection or an operator error writes to

a replica directly. That write is invisible to the primary; a later primary

update that touches the same row overwrites it, or, if no later update touches

it, the divergent value persists silently forever.

3. Non-replicated DDL or manual data manipulation: an operator runs ALTER TABLE or a

manual UPDATE directly against a replica for an "emergency fix." The replica now

has a different schema or different data with no record of why.

4. Clock skew combined with timestamp-based conflict resolution: this is specifically

a multi-leader-adjacent case (DynamoDB, Cassandra LWW) resolving concurrent writes

by comparing timestamps rather than a single-leader case, included here because

the mechanism is the same divergence-through-ambiguous-ordering that clock_skew

covers directly. If node clocks drift, "last write wins" can pick the wrong write,

since the timestamps being compared are not actually measuring true order, only

each node's own clock at write time.

5. Partial transaction application failure: in MySQL GTID replication, a transaction

that errors on the replica and gets silently skipped (sql_slave_skip_counter, or

equivalent) leaves the replica permanently missing those changes, with no error

surfaced afterward to indicate the gap.