Blast Radius Analysis · Search-Heavy Content Platform

Table and Index Bloat

partialContained

capacity failure · linear propagation

Blast Radius

0/13 nodes

Time to Detect

Days to weeks for bloat to reach a level that degrades performance noticeably. With regular bloat queries (pg_bloat monitoring), detectable within days. Without proactive monitoring, typically detected through unexplained storage growth or gradual query performance degradation over weeks.

Preventive Mitigations

Confidence

Strong

Failure Cascade

(no topology nodes mapped for this failure mode)

This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.

Severity at this step: partial

Detection Signals

Disk SaturationAlertQueue Depth

Recovery time estimate: VACUUM on a bloated table runs at approximately 10–50GB/hour depending on disk I/O. A 100GB bloated table may take 2–10 hours to vacuum. pg_repack takes similar time. Tuning autovacuum settings takes effect immediately but prevents future bloat accumulation, not existing bloat.

Mitigation Checklist(1 preventive, 3 reactive)

Tune autovacuum thresholds for high-update tablespreventslow

Override per-table autovacuum settings for high-update tables: ALTER TABLE orders SET (autovacuum_vacuum_scale_factor = 0.01, autovacuum_vacuum_threshold = 1000). This triggers vacuum after 1% dead tuples instead of 20%, keeping bloat bounded for large tables.

Run pg_repack on bloated tablesmedium

pg_repack rebuilds tables and indexes online without ACCESS EXCLUSIVE lock, reclaiming dead space while the table remains available. Unlike VACUUM FULL, it does not block reads or writes. Run during off-peak hours for large tables.

Identify and terminate long-running transactions blocking vacuumlow

Query pg_stat_activity for transactions older than 1 hour: SELECT pid, now() - xact_start AS duration, query FROM pg_stat_activity WHERE xact_start < now() - interval '1 hour'. Terminate long-running analytics queries or move them to replicas.

Drop or set_inactive inactive replication slotslow

Lagging replication slots prevent vacuum from reclaiming dead tuples. SELECT slot_name, xmin, catalog_xmin FROM pg_replication_slots WHERE xmin IS NOT NULL AND age(xmin) > 1000000. Drop or pause consumer-inactive slots immediately.

Affected Systems

Workloads

Read-Heavy API BackendSearch Heavy

Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.