Write Amplification Cascade
SEV-2, Significant ImpactLinear propagation · capacity · Affects 6 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. This failure mode appears in 6 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Write Amplification Cascade begins at the source component. Trigger: High UPDATE rate on tables with 4+ secondary indexes.
Immediate (T+0) · Signal: Disk Saturation
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
10–30 minutes with I/O utilisation and WAL generation monitoring. The failure degrades gradually rather than spiking instantly; by the time application latency is affected, I/O has been saturated for some time. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Blast Radius
When storage I/O saturates, all PostgreSQL operations that require disk I/O are affected: reads (uncached pages), writes (WAL, heap, indexes), autovacuum, and checkpoint. Replication lag grows as WAL sender waits for WAL flush. If replication falls behind, lag-sensitive read routing may redirect traffic to the primary, amplifying its load further. Total write throughput collapses from the sustained high rate to near zero while the I/O queue drains.
Contributing Factors
PostgreSQL is known to be susceptible to Write Amplification Cascade under medium operational burden. Not running PgBouncer: PostgreSQL connections are expensive; >500 direct connections degrade performance
This operational trigger enables Write Amplification Cascade: High UPDATE rate on tables with 4+ secondary indexes
This operational trigger enables Write Amplification Cascade: Bulk import or migration generating WAL at storage bandwidth ceiling
This operational trigger enables Write Amplification Cascade: Workload shift from INSERT-heavy to UPDATE-heavy without index audit
Remediation Plan
Identify I/O saturation: iostat -x 1 showing util > 90% on database disk device
Effort: Minutes to hours (on-call response)
Reduce write load: throttle batch jobs, bulk imports, or autovacuum (autovacuum_vacuum_cost_delay)
Effort: Minutes to hours (on-call response)
Check pg_stat_user_indexes for unused indexes on high-write tables and DROP them
Effort: Minutes to hours (on-call response)
Each secondary index maintained during UPDATE adds one index page write per UPDATE row. Identify indexes with zero or near-zero scans via pg_stat_user_indexes.idx_scan. Drop indexes unused in the last 30 days. Reducing from 8 to 4 indexes can halve write amplification.
Effort: 1 day to 1 week
Set checkpoint_completion_target = 0.9 and checkpoint_timeout = 15min (default 5min). checkpoint_completion_target defaults to 0.5 on PostgreSQL 13 and earlier but already defaults to 0.9 from PostgreSQL 14 onward, so confirm the running version and current value before assuming this is a needed change. Spreading checkpoint dirty page flushing over a longer period reduces instantaneous I/O peaks. Does not reduce total I/O; trades peaks for a more even sustained rate. See checkpoint_amplification for the full mechanism.
Effort: 1 day to 1 week
Move from GP3 (3,000–16,000 IOPS) to io2 (up to 64,000 IOPS) or local NVMe. Increases the I/O ceiling, buying headroom. Does not fix the write amplification factor; defers the problem to higher load levels.
Effort: 1 day to 1 week
Configure alerts for: disk saturation, queue depth, latency spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
PostgreSQL table partitioning by time range or hash distributes write I/O across partition-level pages, potentially benefiting from OS-level I/O parallelism. Each partition has its own set of index pages; hot-partition writes are distributed across the newer partition's pages.
Effort: 1–4 sprints
For high-volume append-only workloads (event logs, analytics), route to ClickHouse or Cassandra, which use LSM trees with sequential I/O and significantly lower write amplification for append patterns (WAF ~ 3–10x vs. PostgreSQL's 20–100x on update-heavy workloads).
Effort: 1–4 sprints
Write Amplification Cascade affects 6 architecture scenarios (Audit and Compliance Platform, Financial Ledger Platform, Geospatial Tracking Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.