DBRaven
Blast Radius Analysis · Healthcare Records Platform

Replication Lag Cascade

partialContained

replication failure · linear propagation

Blast Radius

5%

1/19 nodes

Time to Detect

Seconds if replication lag (pg_stat_replication replay_lsn distance, or seconds- behind) is monitored with an alert threshold. Otherwise minutes to hours, surfacing indirectly as "my save disappeared" support tickets rather than as a system alarm.

Preventive Mitigations

3

Confidence

Strong

Impacted Components

Immediate

Read Replica

architecture pattern · Directly connected to failure mode 'Replication Lag Cascade' via risk propagation path in the topology.

Failure Cascade

1

Read Replica

Replication Lag Cascade (linear propagation) directly affects these components.

Severity at this step: partial

Detection Signals

Replication LagLog ErrorsAlert

Recovery time estimate: Minutes to hours, set by how far behind the replica fell and its serial apply rate. Recovery is automatic once WAL production drops below the replica's apply throughput; it is not automatic if the trigger (a standby query holding back replay) is still present.

Mitigation Checklist(3 preventive, 2 reactive)

Quorum synchronous replication, read from the synchronous standby (bounds staleness on that path)preventshigh

synchronous_standby_names with ANY 1 (s1, s2, s3) and synchronous_commit = on makes a commit wait until any one of several standbys has the WAL, so a single slow or dead standby does not block writes the way a single named synchronous standby would. Be precise about what this buys you: it bounds staleness only for reads routed to a synchronous, caught-up standby, not for the whole read tier. Other asynchronous replicas can still lag freely. To turn it into a freshness guarantee you must direct the sensitive reads to an eligible synchronous standby, not just any replica. Reserve it for the write paths that truly need it; applying it everywhere pays the latency tax on writes that never had a freshness requirement.

Fence reads on a log position (restores read-after-write)preventsmedium

After a write, capture a WAL position to fence on and carry it with the user (session, cookie, or token). pg_current_wal_lsn() read just after commit is a conservative fence: it returns a position at or after the transaction's commit LSN, so it never serves stale data but may occasionally route a read to the primary that a caught-up replica could have served. Capture the transaction's exact commit LSN if the driver exposes it and you want a tighter fence. Then serve that user's later reads from a replica only if pg_last_wal_replay_lsn() has reached the fenced position; otherwise read the primary. This gives read-your-writes for the one user who needs it without pinning all traffic to the primary, which is the difference between a targeted fence and throwing away your read scaling.

Pin each user to one replica (restores monotonic reads)preventslow

Route a given user's reads to the same replica by hashing a stable key such as user id, so their reads never jump to a more-lagged replica within a session. Cheaper than an LSN fence and sufficient when the problem is time going backward rather than not seeing one's own write.

Shed or smooth write bursts with backpressuremedium

Rate-limit or queue burst write operations so the primary does not produce WAL faster than replicas can sustain. Treats the cause (burst) rather than the symptom (stale reads), at the cost of added write latency during spikes.

Alert and auto-eject lagging replicas from the read poollow

Alert on a lag budget derived from the application's tolerance (for example warn at 500ms, page and demote at 5s). A replica past the budget is removed from the read pool until it catches up, so stale reads stop even before the root cause is found.

Affected Systems

Workloads

Event StreamingMixed OLTP (SaaS Core)Write-Heavy Transactional

Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.