Replication Lag Cascade
partialSummary
Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.
Description
Replication lag is not a bug. It is the defining tradeoff of asynchronous replication. The primary acknowledges a commit as soon as the write is durable locally, before any replica has applied it. That is what keeps writes fast and keeps the primary writable when a replica is down or slow. The price is a window during which every replica holds a strictly older version of the data than the primary. A write burst widens that window: the primary appends WAL faster than a replica can replay it, and on a PostgreSQL standby replay is largely a single serial process, so it cannot simply add cores to keep up. Lag grows until the burst subsides.
The reason this failure is dangerous is that reads keep returning HTTP 200 with the wrong version of the row. Error rate and latency stay flat, so the only direct signal is the lag metric itself. What actually breaks is one of three consistency guarantees that lag silently removes, and naming which one is broken is what tells you how to fix it:
Read-after-write (read-your-writes). A user who just wrote does not see their own write, because their next read was routed to a replica that has not yet applied it. This is the anomaly people hit first: save a comment, the page reloads from a replica, the comment is gone. It looks like lost data and generates support tickets.
Monotonic reads. Successive reads move backward in time. A read hits an up-to-date replica and returns a fresh value, then a later read from the same user lands on a more-lagged replica and returns an older value. Data appears to vanish and return. It is caused by spreading one user's reads across replicas with different lag.
Consistent prefix reads. Causally ordered writes are observed out of order: an observer sees the answer before the question. This one is specific to partitioned or multi-source replication, where two writes with a happens-before relationship travel different paths and arrive with different lag. A single-leader, single- partition setup does not exhibit it; a sharded or multi-primary one can.
A long analytical query on the replica is a second, less obvious trigger. On a hot standby the query conflicts with WAL replay. PostgreSQL resolves the conflict either by delaying replay up to max_standby_streaming_delay (lag grows) or by canceling the query with "canceling statement due to conflict with recovery." Setting hot_standby_feedback = on stops the cancellations but moves the cost to the primary as table bloat, because the primary must now retain the old row versions the standby is still reading. There is no free option here; you are choosing where the cost lands.
Consistency guarantees removed
The specific guarantees this failure takes away. The right fix depends on which one you actually need.
Characteristics
Triggers
- ·Write burst that exceeds the replica's serial WAL apply rate
- ·Long-running analytical or reporting query on a hot standby, delaying WAL replay up to max_standby_streaming_delay
- ·Replica I/O saturation: disk cannot apply WAL as fast as the primary produces it
- ·Network congestion or bandwidth limit on the primary-to-standby stream
- ·Replica CPU starved by concurrent read load, slowing the single apply process
Detection Signals
Mitigation Strategies
After a write, capture a WAL position to fence on and carry it with the user (session, cookie, or token). pg_current_wal_lsn() read just after commit is a conservative fence: it returns a position at or after the transaction's commit LSN, so it never serves stale data but may occasionally route a read to the primary that a caught-up replica could have served. Capture the transaction's exact commit LSN if the driver exposes it and you want a tighter fence. Then serve that user's later reads from a replica only if pg_last_wal_replay_lsn() has reached the fenced position; otherwise read the primary. This gives read-your-writes for the one user who needs it without pinning all traffic to the primary, which is the difference between a targeted fence and throwing away your read scaling.
Route a given user's reads to the same replica by hashing a stable key such as user id, so their reads never jump to a more-lagged replica within a session. Cheaper than an LSN fence and sufficient when the problem is time going backward rather than not seeing one's own write.
synchronous_standby_names with ANY 1 (s1, s2, s3) and synchronous_commit = on makes a commit wait until any one of several standbys has the WAL, so a single slow or dead standby does not block writes the way a single named synchronous standby would. Be precise about what this buys you: it bounds staleness only for reads routed to a synchronous, caught-up standby, not for the whole read tier. Other asynchronous replicas can still lag freely. To turn it into a freshness guarantee you must direct the sensitive reads to an eligible synchronous standby, not just any replica. Reserve it for the write paths that truly need it; applying it everywhere pays the latency tax on writes that never had a freshness requirement.
Alert on a lag budget derived from the application's tolerance (for example warn at 500ms, page and demote at 5s). A replica past the budget is removed from the read pool until it catches up, so stale reads stop even before the root cause is found.
Rate-limit or queue burst write operations so the primary does not produce WAL faster than replicas can sustain. Treats the cause (burst) rather than the symptom (stale reads), at the cost of added write latency during spikes.
Recovery Steps
- 1.Identify and throttle the source of the write burst if it is still running
- 2.Read current lag on every replica (replay_lsn distance from the primary's write_lsn)
- 3.Fence or route freshness-sensitive reads to the primary until replicas catch up
- 4.Confirm lag is draining; a healthy replica catches up once WAL production falls below its apply rate
- 5.If a long-running query on a standby is the cause, cancel it or move reporting to a dedicated replica
- 6.Restore replica reads once lag is back inside the application's lag budget
Estimated recovery time: Minutes to hours, set by how far behind the replica fell and its serial apply rate. Recovery is automatic once WAL production drops below the replica's apply throughput; it is not automatic if the trigger (a standby query holding back replay) is still present.
Affected Systems
Patterns
Technologies
Basis
Mechanism and consistency-guarantee taxonomy are standard and well documented (single-leader asynchronous replication; read-after-write, monotonic reads, and consistent prefix as the three lag anomalies). PostgreSQL specifics (serial WAL replay, max_standby_streaming_delay, hot_standby_feedback, LSN fencing, quorum synchronous_standby_names) are current and verifiable in the official documentation.
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
The read replica pattern is structurally vulnerable to replication lag cascade because its value proposition: serving reads from replicas: depends on replica data being sufficiently current. Any condition that delays WAL replay degrades or invalidates the replica's usefulness.
Tradeoffs
- ·Synchronous commit eliminates lag but halves write throughput
- ·Lag-aware routing adds application complexity and requires replica health metadata
- ·Aggressive lag thresholds shift all traffic to primary under load, negating scale benefits
Used In Architecture Scenarios
Financial Ledger
An append-only audit log architecture for capturing system actions: user operations, data access events, configuration changes, and financial operations: with cryptographic integrity chaining, immutable storage, and dual-path query serving. PostgreSQL stores the authoritative append-only event log in time-partitioned tables; ClickHouse serves historical aggregate queries and retention analytics; Kafka streams audit events in real time to SIEM integrations and security dashboards; Redis caches hot audit query results for compliance-facing read paths. No record is ever updated or deleted: only inserts are permitted. Each record includes a hash of the previous record, forming a tamper-evident chain verifiable without external tooling.
Read-Heavy Application
A CMS for publishing and serving structured content: articles, documentation, product pages, and localized variants: where read APIs serve 50–100x more traffic than editorial write APIs. PostgreSQL stores the content graph (articles, authors, categories, taxonomy) and workflow state (draft, in-review, scheduled, published). Redis caches published content objects for read APIs. Elasticsearch powers full-text content search with faceting and relevance ranking. Cache invalidation on publish must be fast and complete; N+1 query patterns on content relationship traversal are the dominant database performance risk during reads.
Event-Driven System
A streaming architecture that captures database changes via WAL-based CDC, publishes them to an event stream (Kafka), and routes them to analytics consumers. Decouples the write path from the read path while maintaining a durable, replayable event log.
Financial Ledger
An electronic health record (EHR) architecture built around strict auditability, HIPAA compliance, and append-only correctness. Clinical records are mutable by design (amendments, addenda) but corrections must be explicitly attributed, not silently overwritten. Event sourcing provides a reconstructable audit log; PostgreSQL row-level security enforces patient-level access control at the database layer; Kafka streams HL7 FHIR events to downstream clinical systems. The architecture must support breach detection, access auditing, and state reconstruction at any historical point: not just current state retrieval.
Read-Heavy Application
A standard SaaS API architecture optimized for read-dominant workloads. PostgreSQL serves as the primary data store, Redis provides a caching layer for hot data, connection pooling bounds database concurrency, and read replicas scale read throughput without scaling write capacity.
Search-Heavy Application
A content platform architecture centered on Elasticsearch for full-text search, faceted navigation, and ranked results, with PostgreSQL as the transactional source of truth and Redis for session management and hot content caching. WAL-based CDC maintains index freshness by streaming PostgreSQL changes into Elasticsearch asynchronously. The core tension is between search index freshness, query performance, and index maintenance cost under high write volume.
Event-Driven System
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth.