WAL Saturation
criticalContainedcapacity failure · fan out propagation
Blast Radius
5%
1/19 nodes
Time to Detect
Replication lag detection: seconds with pg_stat_replication monitoring. WAL disk space accumulation: minutes if disk monitoring is active. Write stall from WAL buffer exhaustion: immediate (visible as write latency spike). WAL disk full causing complete write halt: 0 seconds (instant detection from write errors in application).
Preventive Mitigations
1
Confidence
StrongImpacted Components
Write-Heavy Transactional
workload · Directly connected to failure mode 'WAL Saturation' via risk propagation path in the topology.
Failure Cascade
Write-Heavy Transactional
WAL Saturation (fan out propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: Seconds to minutes once lagging replication slots are dropped and bulk writes are throttled. Replica lag recovery depends on how far behind replicas have fallen (1GB lag at 100MB/s replication bandwidth takes 10s; 1TB lag takes ~3 hours). Disk space recovery from dropped slots is near-immediate.
Mitigation Checklist(1 preventive, 3 reactive)
In PostgreSQL 13+, max_slot_wal_keep_size limits how much WAL a lagging slot can retain. When the limit is reached, the slot is invalidated rather than filling the disk. Set to 10–50GB based on disk headroom. Trade-off: invalidated slots lose their position and must restart from scratch.
Place pg_wal on a dedicated NVMe volume or high-IOPS EBS io2 volume, separate from data files. WAL I/O is sequential; NVMe achieves 3–7 GB/s sequential writes, far exceeding EBS limits. Data I/O and WAL I/O no longer compete.
An idle or lagging replication slot is the most common cause of unconstrained WAL accumulation. SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff( pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag FROM pg_replication_slots. Drop slots for consumers that have been inactive >24 hours.
For bulk imports or ETL jobs, add rate limiting (INSERT N rows, sleep M ms). At 10,000 rows/batch with 10ms sleep: ~1M rows/minute. This bounds WAL generation to a rate that replicas and WAL archive can sustain.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.