Asymmetric Replication Topology Failure
SEV-2, Significant ImpactLinear propagation · replication · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Asymmetric Replication Topology Failure begins at the source component. Trigger: Manual data modification executed directly on the replica (not through the primary replication path).
Immediate (T+0) · Signal: Replication Lag
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Days to weeks without proactive consistency checking. Immediately detectable when replication stops (MySQL with strict mode, PostgreSQL logical replication). pt-table-checksum can detect divergence within 1–4 hours of a scheduled run. Real-time detection requires application-level data reconciliation queries comparing primary and replica values for sampled records. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Blast Radius
All reads served from the diverged replica return potentially incorrect data. The divergence affects every table that had replication events applied incorrectly after the divergence point. In a read-heavy system where 90% of reads go to replicas, the majority of user-facing data is sourced from incorrect state. Downstream systems populated from replica data (search indexes, analytics pipelines, caches) also contain the incorrect values, extending the blast radius beyond the immediate read path.
Contributing Factors
This operational trigger enables Asymmetric Replication Topology Failure: Manual data modification executed directly on the replica (not through the primary replication path)
This operational trigger enables Asymmetric Replication Topology Failure: MySQL slave_exec_mode=IDEMPOTENT configured, silently skipping rows that fail to apply
This operational trigger enables Asymmetric Replication Topology Failure: Software bug in the MySQL or PostgreSQL replication implementation misapplying a specific event type
Remediation Plan
Stop routing reads to the suspected diverged replica immediately
Effort: Minutes to hours (on-call response)
Run pt-table-checksum (MySQL) or a manual checksum query (SELECT md5(array_agg(id ORDER BY id)::text) from critical tables) comparing primary vs replica
Effort: Minutes to hours (on-call response)
Identify which tables and approximately how many rows are diverged
Effort: Minutes to hours (on-call response)
Run Percona Toolkit's pt-table-checksum weekly (or daily for critical tables) against the primary, which checksums all tables and compares against each replica. Diverged rows are reported by table and approximate row count. Use pt-table-sync to repair diverged rows on the replica. Configure an alert to fire if pt-table-checksum reports diverged chunks on any table. This detects silent divergence within the check interval.
Effort: 1 day to 1 week
Create a read-only database user for the replica and configure the application to use this user for replica reads. Explicitly deny WRITE and UPDATE permissions on the replica. For operator access, add a policy requiring that all data fixes be applied to the primary and replicated, never directly on the replica. This eliminates the most common cause of divergence (manual modifications) through access control rather than policy.
Effort: 1 day to 1 week
Configure alerts for: replication lag, alert, log errors. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Replace MySQL row-based replication or PostgreSQL logical replication with PostgreSQL physical streaming replication (WAL shipping). Physical replication cannot diverge because it replays WAL records byte-for-byte. Silent divergence is structurally impossible. The trade-off: physical replicas cannot be used for databases other than the primary's exact version and schema; logical replication allows partial table subscriptions and cross-version replication. For most standard read-scaling use cases, physical replication is the safer choice.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.