Post-Mortem Framework · Data Replication: Cross-Region Replication Drift

Cross-Region Replication Drift

SEV-3, Limited Impact

Linear propagation · replication · Affects 0 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity.

Propagation Chain

Origin component

Cross-Region Replication Drift begins at the source component. Trigger: Write throughput spike exceeding replication apply thread capacity (>5,000 writes/second sustained for >30 seconds).

Immediate (T+0) · Signal: Replication Lag

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

5–30 minutes via replication lag monitoring if lag SLO alerts are configured with a 10-second threshold. Without dedicated lag monitoring, detection occurs only when users report stale data, which may be 30–120 minutes after the drift event begins. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Blast Radius

All reads served from the secondary region return stale data for the duration of the lag accumulation. This affects all users in that region, not just the feature that triggered the write spike. If the secondary feeds downstream caches or materialized stores, those caches may be populated with stale values that persist beyond the replication recovery window, extending the staleness duration further.

Contributing Factors

Trigger Condition: Write throughput spike exceeding replication apply thread caoperational

This operational trigger enables Cross-Region Replication Drift: Write throughput spike exceeding replication apply thread capacity (>5,000 writes/second sustained for >30 seconds)

Trigger Condition: Cross-region network congestion causing apply thread to staloperational

This operational trigger enables Cross-Region Replication Drift: Cross-region network congestion causing apply thread to stall

Trigger Condition: Large transaction (bulk import, schema change) on primary geoperational

This operational trigger enables Cross-Region Replication Drift: Large transaction (bulk import, schema change) on primary generating a single large WAL event

Remediation Plan

ImmediateCheck current replication lag on all secondaries via monitoring dashboard or SEL

Check current replication lag on all secondaries via monitoring dashboard or SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) AS lag_seconds on PostgreSQL

Effort: Minutes to hours (on-call response)

ImmediateIf lag exceeds SLO, redirect all reads for affected region to the primary immedi

If lag exceeds SLO, redirect all reads for affected region to the primary immediately

Effort: Minutes to hours (on-call response)

ImmediateMonitor lag metric to confirm it is decreasing; lag should recover at the apply

Monitor lag metric to confirm it is decreasing; lag should recover at the apply thread throughput rate

Effort: Minutes to hours (on-call response)

Short-TermReplication lag SLO alerting with automatic read routing fallback

Monitor secondary replication lag in real time (PostgreSQL: pg_stat_replication.write_lag; MySQL: Seconds_Behind_Master). Configure a PagerDuty alert at >5 seconds lag. Implement a lag-aware read router in the application that automatically falls back to primary reads when secondary lag exceeds a configurable threshold (e.g., 500ms for financial data, 5 seconds for content data). The router checks lag once per second and caches the result to avoid per-request overhead.

Effort: 1 day to 1 week

Short-TermParallel replication apply threads

Enable multi-threaded replication on the secondary to match primary write concurrency. MySQL: set slave_parallel_workers=8 and slave_parallel_type=LOGICAL_CLOCK. PostgreSQL logical replication: use multiple subscription workers or pglogical with parallel apply. This reduces the apply bottleneck for workloads with many concurrent independent transactions, though it does not help for single large-transaction lag.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: replication lag, alert, log errors. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermSynchronous replication for critical write paths

For data that must be strongly consistent across regions (financial balances, inventory counts), configure synchronous commit on the primary. PostgreSQL: synchronous_commit = remote_write for at least one secondary. MySQL: semi-sync replication with rpl_semi_sync_master_wait_for_slave_count=1. Accepts higher write latency (+40–150ms per write for cross-region ACK) in exchange for read-your-writes consistency guarantee.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.