Replication Lag Cascade
SEV-3, Limited ImpactLinear propagation · replication · Affects 7 scenario(s)
Severity Classification
Classified as PARTIAL based on failure mode severity. This failure mode appears in 7 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Replication Lag Cascade begins at the source component. Trigger: Write burst that exceeds the replica's serial WAL apply rate.
Immediate (T+0) · Signal: Replication Lag
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Seconds if replication lag (pg_stat_replication replay_lsn distance, or seconds- behind) is monitored with an alert threshold. Otherwise minutes to hours, surfacing indirectly as "my save disappeared" support tickets rather than as a system alarm. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Blast Radius
During a write burst, every asynchronous replica can lag at once, so the blast radius is the entire read tier, not one node. The specific reads that break are the ones that depend on a guarantee lag has removed: writer-then-reader flows lose read-after-write, users load-balanced across replicas lose monotonic reads, and causally linked writes across partitions lose consistent prefix. Reads with no freshness requirement (analytics, browse, cacheable content) are unaffected, which is why the impact is easy to miss in aggregate dashboards.
Contributing Factors
The read replica pattern is structurally vulnerable to replication lag cascade because its value proposition: serving reads from replicas: depends on replica data being sufficiently current. Any condition that delays WAL replay degrades or invalidates the replica's usefulness.
This operational trigger enables Replication Lag Cascade: Write burst that exceeds the replica's serial WAL apply rate
This operational trigger enables Replication Lag Cascade: Long-running analytical or reporting query on a hot standby, delaying WAL replay up to max_standby_streaming_delay
This operational trigger enables Replication Lag Cascade: Replica I/O saturation: disk cannot apply WAL as fast as the primary produces it
Remediation Plan
Identify and throttle the source of the write burst if it is still running
Effort: Minutes to hours (on-call response)
Read current lag on every replica (replay_lsn distance from the primary's write_lsn)
Effort: Minutes to hours (on-call response)
Fence or route freshness-sensitive reads to the primary until replicas catch up
Effort: Minutes to hours (on-call response)
After a write, capture a WAL position to fence on and carry it with the user (session, cookie, or token). pg_current_wal_lsn() read just after commit is a conservative fence: it returns a position at or after the transaction's commit LSN, so it never serves stale data but may occasionally route a read to the primary that a caught-up replica could have served. Capture the transaction's exact commit LSN if the driver exposes it and you want a tighter fence. Then serve that user's later reads from a replica only if pg_last_wal_replay_lsn() has reached the fenced position; otherwise read the primary. This gives read-your-writes for the one user who needs it without pinning all traffic to the primary, which is the difference between a targeted fence and throwing away your read scaling.
Effort: 1 day to 1 week
Route a given user's reads to the same replica by hashing a stable key such as user id, so their reads never jump to a more-lagged replica within a session. Cheaper than an LSN fence and sufficient when the problem is time going backward rather than not seeing one's own write.
Effort: 1 day to 1 week
Alert on a lag budget derived from the application's tolerance (for example warn at 500ms, page and demote at 5s). A replica past the budget is removed from the read pool until it catches up, so stale reads stop even before the root cause is found.
Effort: 1 day to 1 week
Configure alerts for: replication lag, log errors, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
synchronous_standby_names with ANY 1 (s1, s2, s3) and synchronous_commit = on makes a commit wait until any one of several standbys has the WAL, so a single slow or dead standby does not block writes the way a single named synchronous standby would. Be precise about what this buys you: it bounds staleness only for reads routed to a synchronous, caught-up standby, not for the whole read tier. Other asynchronous replicas can still lag freely. To turn it into a freshness guarantee you must direct the sensitive reads to an eligible synchronous standby, not just any replica. Reserve it for the write paths that truly need it; applying it everywhere pays the latency tax on writes that never had a freshness requirement.
Effort: 1–4 sprints
Replication Lag Cascade affects 7 architecture scenarios (Audit and Compliance Platform, Content Management Platform, Event-Driven Analytics Pipeline). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.