Post-Mortem Framework · Capacity: Disk I/O Saturation

Disk I/O Saturation

SEV-2, Significant Impact

Fan-Out propagation · capacity · Affects 5 scenario(s)

Severity Classification

Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 5 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Disk I/O Saturation begins at the source component. Trigger: PostgreSQL checkpoint flushing large volumes of dirty shared_buffers to disk.

Immediate (T+0) · Signal: Disk Saturation

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

Seconds with I/O utilisation monitoring (iowait alerting, EBS IOPS monitoring). Application latency impact is nearly immediate when saturation begins. EBS burst credit exhaustion (GP2) is detectable 10–60 minutes before the drop to baseline IOPS if BurstBalance is monitored. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Downstream of dependents (fan-out)

Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.

Within minutes of initial failure · Signal: Multiple services reporting elevated error rates

Blast Radius

All database operations that require disk I/O are affected simultaneously. This includes every write (WAL stall), every uncached read (page fault stall), and background processes (autovacuum, checkpoint). The blast radius is the entire database: all tables, all queries, all connections. Replication is affected as WAL sends slow. All applications using the database experience simultaneous degradation.

Contributing Factors

Workload: Time Series Metricsoperational

Time-series metric workloads generate write throughput that can saturate disk I/O: 100,000-1,000,000 data points per second produce continuous sequential write load that exceeds spinning disk capacity and requires NVMe or storage-optimized instances to sustain.

Trigger Condition: PostgreSQL checkpoint flushing large volumes of dirty sharedoperational

This operational trigger enables Disk I/O Saturation: PostgreSQL checkpoint flushing large volumes of dirty shared_buffers to disk

Trigger Condition: Autovacuum performing full-table vacuum on a large table witoperational

This operational trigger enables Disk I/O Saturation: Autovacuum performing full-table vacuum on a large table with many dead tuples

Trigger Condition: Workload shift to larger queries that exceed work_mem and geoperational

This operational trigger enables Disk I/O Saturation: Workload shift to larger queries that exceed work_mem and generate temporary disk sort files

Mitigation Gaps

HIGHNo preventive mitigation documented

Identify and document at least one architectural change that prevents Disk I/O Saturation from occurring (not just recovering from it).

Remediation Plan

ImmediateConfirm I/O saturation: iostat -x 1 showing util > 90% or iowait > 30%

Confirm I/O saturation: iostat -x 1 showing util > 90% or iowait > 30%

Effort: Minutes to hours (on-call response)

ImmediateIdentify top I/O consumers: iotop or pg_stat_bgwriter to find checkpoint/autovac

Identify top I/O consumers: iotop or pg_stat_bgwriter to find checkpoint/autovacuum vs. query I/O

Effort: Minutes to hours (on-call response)

ImmediateIf checkpoint is the driver: immediately reduce checkpoint_completion_target = 0

If checkpoint is the driver: immediately reduce checkpoint_completion_target = 0.9 and set autovacuum_vacuum_cost_delay

Effort: Minutes to hours (on-call response)

Short-TermUpgrade to higher-IOPS storage tier

Move from GP2/GP3 baseline to io2 Block Express (up to 64,000 IOPS, 1,000 MB/s). For AWS: modify the EBS volume type and IOPS parameter. Immediate effect after modification completes. Most effective short-term fix. Ongoing cost increase proportional to provisioned IOPS.

Effort: 1 day to 1 week

Short-TermSeparate WAL and data to distinct storage volumes

Configure PostgreSQL with PGDATA on one volume and pg_wal on a separate volume. WAL writes are sequential and high-frequency; separating WAL from data page writes eliminates I/O contention between them. Each volume gets its own IOPS budget.

Effort: 1 day to 1 week

Short-TermTune checkpoint and autovacuum to smooth I/O

Set checkpoint_completion_target = 0.9 (spread checkpoint I/O over 90% of the checkpoint interval). Set autovacuum_vacuum_cost_delay = 10ms and autovacuum_vacuum_cost_limit = 200 to throttle autovacuum I/O. These settings reduce I/O peaks at the cost of slightly more sustained background I/O.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: disk saturation, queue depth, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermEliminate cross-scenario Disk I/O Saturation exposure

Disk I/O Saturation affects 5 architecture scenarios (Audit and Compliance Platform, Geospatial Tracking Platform, IoT Telemetry Ingestion Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.