Replica Divergence
SEV-2, Significant ImpactIsolated propagation · replication · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Replica Divergence begins at the source component. Trigger: Statement-based replication with non-deterministic SQL functions.
Immediate (T+0) · Signal: Alert
Blast Radius
Reads from the diverged replica return incorrect data, not stale data. Depending on what diverged, the impact ranges from incorrect analytics aggregations (non-critical) to incorrect financial balances or access-control state (critical). If the primary fails and the diverged replica is promoted, the divergent data becomes the new authoritative state, and the corruption propagates to every subsequent replica of it.
Contributing Factors
MySQL with statement-based binlog replication is vulnerable to replica divergence when SQL contains non-deterministic functions; row-based replication eliminates this vulnerability.
Read-heavy APIs that serve reads from replicas are vulnerable to replica divergence, where the replica contains data that never existed on the primary due to non-deterministic replication.
This operational trigger enables Replica Divergence: Statement-based replication with non-deterministic SQL functions
This operational trigger enables Replica Divergence: Application writes routed to a read replica (a misconfigured connection string)
This operational trigger enables Replica Divergence: Manual DML executed directly on a replica for a "quick fix"
Remediation Plan
Run pt-table-checksum or equivalent to identify which tables and rows have diverged
Effort: Minutes to hours (on-call response)
Use pt-table-sync (Percona Toolkit) to repair diverged rows by syncing from the primary to the replica
Effort: Minutes to hours (on-call response)
Identify the root cause before repairing: repairing without fixing the cause just re-diverges
Effort: Minutes to hours (on-call response)
Configure MySQL binlog_format=ROW (PostgreSQL's WAL-based replication is row-based by default and unaffected). Row-based replication ships actual before/after row values instead of the SQL statement, eliminating non-determinism from re-executing functions on the replica entirely.
Effort: 1 day to 1 week
Set read_only=ON (MySQL) or default_transaction_read_only=on (PostgreSQL) on every replica instance. A direct write attempt then fails at the database layer itself, not just at an application-level guard that a misrouted connection or an operator with direct access can bypass.
Effort: 1 day to 1 week
Run pt-table-checksum (Percona Toolkit for MySQL) or pgcompare on a schedule (daily or weekly for critical tables). It computes and compares checksums per table chunk across the primary and its replicas, surfacing diverged rows as an alert rather than waiting for a user to notice.
Effort: 1 day to 1 week
Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.