Cross-Region Replication Drift

When multi-region replication falls behind during write spikes, reads directed to secondary regions silently return stale data beyond the application SLA. Unlike an outage, the system appears healthy: queries succeed, latency is normal: but users in secondary regions observe data that may be minutes or hours old, with no automatic alerting unless explicit replication lag SLO monitoring is configured.

Multi-region architectures commonly route writes to a primary region and serve reads from regional secondaries for latency reduction. Under normal load, replication lag stays below 100–500ms, which is acceptable for most use cases. During write spikes: bulk imports, viral traffic events, end-of-period batch processing : the primary generates WAL or binlog at a rate that exceeds the secondary region's ability to apply it. Lag accumulates from 500ms to 5 seconds, then 30 seconds, then minutes. Reads from the secondary return increasingly stale data.

The silent nature of this failure is its defining characteristic. The database reports replication as "running" because the secondary is connected and receiving data: it simply hasn't applied it yet. Application-level health checks query the secondary and get successful responses. The staleness is invisible unless the application explicitly checks the secondary's replication_delay metric or the data has a timestamp field that reveals its age. Users notice before monitoring does: they write data, reload the page routed to a secondary, and see the old value.

Network jitter between regions amplifies the problem. Cross-region links have higher latency (40–150ms typical US-EU) and bandwidth limits compared to intra-region links. The apply thread on the secondary is single-threaded in many configurations (MySQL binlog replication is single-threaded by default without parallel replication enabled). A burst of 50,000 writes in 10 seconds at the primary may take 30–90 seconds to fully apply at the secondary, depending on network bandwidth and apply thread capacity.

The failure compounds when the secondary is used for reads that feed into write decisions. An inventory check reads stale stock levels from the secondary and returns "in stock"; the order is placed and written to the primary; the true stock level (visible only at the primary) was already zero. This produces inventory oversell: a correctness violation that manifests as a business error rather than a technical alert.