Checkpoint Amplification
SEV-3, Limited ImpactLinear propagation · capacity · Affects 1 scenario(s)
Severity Classification
Classified as PARTIAL based on failure mode severity.
Propagation Chain
Origin component
Checkpoint Amplification begins at the source component. Trigger: large shared_buffers with high dirty-page fraction at checkpoint time.
Immediate (T+0) · Signal: Disk Saturation
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Detectable within minutes with pg_stat_bgwriter monitoring and I/O utilisation trending. The periodic pattern (spikes at exactly checkpoint_timeout intervals) is the distinguishing characteristic: it is not random latency, it is clock-driven. Without checkpoint-specific monitoring, the pattern may be attributed to generic I/O variability for weeks before the periodicity is noticed. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Blast Radius
Checkpoint amplification primarily affects write latency on the primary. All write transactions compete with checkpoint I/O for the same storage bandwidth. During a checkpoint burst, write transaction commit latency may increase from 1ms to 5–50ms. Replication WAL sender may experience brief stalls as WAL writes compete with checkpoint writes. Read latency is unaffected unless checkpoint I/O saturates the storage device and queues uncached read I/O.
Contributing Factors
Write-heavy transactional workloads trigger frequent PostgreSQL checkpoints that flush large numbers of dirty pages to disk simultaneously, causing I/O spikes that interrupt query execution and increase write amplification beyond the WAL baseline.
MySQL is known to be susceptible to Checkpoint Amplification under medium operational burden. Not deploying ProxySQL: direct application connections to MySQL primary cause connection storms during failover and cannot route reads to replicas
PostgreSQL is known to be susceptible to Checkpoint Amplification under medium operational burden. Not running PgBouncer: PostgreSQL connections are expensive; >500 direct connections degrade performance
This operational trigger enables Checkpoint Amplification: large shared_buffers with high dirty-page fraction at checkpoint time
This operational trigger enables Checkpoint Amplification: checkpoint_completion_target left at the pre-PostgreSQL-14 default of 0.5, or manually lowered from the 14+ default of 0.9, concentrating checkpoint I/O into a shorter window
This operational trigger enables Checkpoint Amplification: full_page_writes = on (required for crash safety) amplifying WAL after each checkpoint
Mitigation Gaps
Identify and document at least one architectural change that prevents Checkpoint Amplification from occurring (not just recovering from it).
Remediation Plan
Confirm checkpoint I/O is the driver: check pg_stat_bgwriter.checkpoint_write_time trend
Effort: Minutes to hours (on-call response)
Check the PostgreSQL major version and current checkpoint_completion_target via SHOW checkpoint_completion_target; on 14+ it should already read 0.9
Effort: Minutes to hours (on-call response)
If it is below 0.9 (pre-14 instance, or manually lowered), set checkpoint_completion_target = 0.9 via ALTER SYSTEM and SELECT pg_reload_conf()
Effort: Minutes to hours (on-call response)
ALTER SYSTEM SET checkpoint_completion_target = 0.9; SELECT pg_reload_conf(). On PostgreSQL 14 and later this is already the default, so check pg_settings before assuming action is needed; the setting only needs changing if it was manually lowered. On PostgreSQL 13 and earlier, this is a required change: the shipped default is 0.5. Either way, 0.9 spreads checkpoint writes over 90% of the checkpoint interval instead of 50%, reducing peak I/O spike at checkpoint time. Takes effect immediately without restart. Does not reduce total checkpoint I/O volume; spreads the same work more evenly.
Effort: 1 day to 1 week
Set checkpoint_timeout = '15min' or '30min'. Fewer checkpoints per hour means fewer I/O bursts. However: longer checkpoint intervals increase WAL retained for crash recovery (max_wal_size may also need increasing) and extend crash recovery time. Trade-off: more time between checkpoints increases recovery time after a crash.
Effort: 1 day to 1 week
Smaller shared_buffers means fewer dirty pages to flush per checkpoint. Counter-intuitive: reducing shared_buffers from 16GB to 8GB on a write- heavy workload can reduce checkpoint I/O amplitude at the cost of slightly reduced cache hit rate. Measure cache hit rate before and after.
Effort: 1 day to 1 week
Configure alerts for: disk saturation, alert, latency spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.