Disk I/O Saturation
criticalContainedcapacity failure · fan out propagation
Blast Radius
5%
1/19 nodes
Time to Detect
Seconds with I/O utilisation monitoring (iowait alerting, EBS IOPS monitoring). Application latency impact is nearly immediate when saturation begins. EBS burst credit exhaustion (GP2) is detectable 10–60 minutes before the drop to baseline IOPS if BurstBalance is monitored.
Preventive Mitigations
0
Confidence
StrongImpacted Components
Time-Series Metrics
workload · Directly connected to failure mode 'Disk I/O Saturation' via risk propagation path in the topology.
Failure Cascade
Time-Series Metrics
Disk I/O Saturation (fan out propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: Seconds to minutes once the saturating I/O source (checkpoint, autovacuum) is throttled or eliminated. EBS IOPS provisioning change takes effect within minutes. Full architectural fixes (separate WAL volume, NVMe migration) require maintenance windows of 30–60 minutes.
Mitigation Checklist(0 preventive, 4 reactive)
Configure PostgreSQL with PGDATA on one volume and pg_wal on a separate volume. WAL writes are sequential and high-frequency; separating WAL from data page writes eliminates I/O contention between them. Each volume gets its own IOPS budget.
Identify queries spilling to disk via pg_stat_statements (temp_blks_read and temp_blks_written > 0). Increase work_mem for sessions executing these queries. A query that sorts in memory generates zero temporary disk I/O. Balance against total memory budget (see memory_pressure_oom).
Move from GP2/GP3 baseline to io2 Block Express (up to 64,000 IOPS, 1,000 MB/s). For AWS: modify the EBS volume type and IOPS parameter. Immediate effect after modification completes. Most effective short-term fix. Ongoing cost increase proportional to provisioned IOPS.
Set checkpoint_completion_target = 0.9 (spread checkpoint I/O over 90% of the checkpoint interval). Set autovacuum_vacuum_cost_delay = 10ms and autovacuum_vacuum_cost_limit = 200 to throttle autovacuum I/O. These settings reduce I/O peaks at the cost of slightly more sustained background I/O.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.