WAL Saturation
SEV-2, Significant ImpactFan-Out propagation · capacity · Affects 4 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 4 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
WAL Saturation begins at the source component. Trigger: Bulk data import or ETL generating WAL at storage bandwidth ceiling (>200MB/s).
Immediate (T+0) · Signal: Queue Depth
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Replication lag detection: seconds with pg_stat_replication monitoring. WAL disk space accumulation: minutes if disk monitoring is active. Write stall from WAL buffer exhaustion: immediate (visible as write latency spike). WAL disk full causing complete write halt: 0 seconds (instant detection from write errors in application). · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
WAL saturation stalls all write transactions. If WAL disk is full, PostgreSQL halts all write operations until WAL is cleared: complete write unavailability. Replication lag affects all read replicas simultaneously. If CDC replication slots are lagging and filling pg_wal, even read-only queries may be affected if the data disk fills completely. Replica lag means all reads-from-replica return stale data, potentially hours stale.
Contributing Factors
Write-heavy transactional workloads generate high WAL volume that can saturate WAL writer throughput, fill the WAL buffer, and: in the extreme: cause write transactions to block waiting for WAL to be flushed to disk or consumed by replicas.
MySQL is known to be susceptible to WAL Saturation under medium operational burden. Not deploying ProxySQL: direct application connections to MySQL primary cause connection storms during failover and cannot route reads to replicas
PostgreSQL is known to be susceptible to WAL Saturation under medium operational burden. Not running PgBouncer: PostgreSQL connections are expensive; >500 direct connections degrade performance
This operational trigger enables WAL Saturation: Bulk data import or ETL generating WAL at storage bandwidth ceiling (>200MB/s)
This operational trigger enables WAL Saturation: High UPDATE rate on wide tables with many indexes (high WAF producing large WAL records)
This operational trigger enables WAL Saturation: full_page_writes = on generating full 8KB page images to WAL after each checkpoint
Remediation Plan
Check pg_replication_slots for lagging slots: any slot with >5GB retained WAL is an immediate risk
Effort: Minutes to hours (on-call response)
Drop lagging slots that belong to offline or slow consumers
Effort: Minutes to hours (on-call response)
If pg_wal is filling disk: pg_switch_wal() to force WAL segment rotation, then check archive_status
Effort: Minutes to hours (on-call response)
An idle or lagging replication slot is the most common cause of unconstrained WAL accumulation. SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff( pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag FROM pg_replication_slots. Drop slots for consumers that have been inactive >24 hours.
Effort: 1 day to 1 week
In PostgreSQL 13+, max_slot_wal_keep_size limits how much WAL a lagging slot can retain. When the limit is reached, the slot is invalidated rather than filling the disk. Set to 10–50GB based on disk headroom. Trade-off: invalidated slots lose their position and must restart from scratch.
Effort: 1 day to 1 week
For bulk imports or ETL jobs, add rate limiting (INSERT N rows, sleep M ms). At 10,000 rows/batch with 10ms sleep: ~1M rows/minute. This bounds WAL generation to a rate that replicas and WAL archive can sustain.
Effort: 1 day to 1 week
Configure alerts for: queue depth, disk saturation, replication lag. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
WAL Saturation affects 4 architecture scenarios (Audit and Compliance Platform, IoT Telemetry Ingestion Platform, Observability Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.