Read Amplification (LSM Tree)
criticalContainedstorage failure · isolated propagation
Blast Radius
5%
1/21 nodes
Time to Detect
Read amplification is detectable through storage engine metrics: Cassandra: SSTable count per partition, SSTable reads per query (nodetool tpstats) RocksDB: level_stats, estimated_num_keys, total_sst_files_size per level Alerts on high SSTable read counts per query detect the condition in under a minute.
Preventive Mitigations
0
Confidence
StrongImpacted Components
Apache Cassandra
primary datastore · Directly connected to failure mode 'Read Amplification (LSM Tree)' via risk propagation path in the topology.
Failure Cascade
Apache Cassandra
Read Amplification (LSM Tree) (isolated propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: Increasing compaction concurrency shows improvement within minutes to hours as SSTable count decreases. Full major compaction on a large node (several TB) takes hours. Architectural changes (compaction strategy switch, data model changes) require a maintenance window and take longer.
Mitigation Checklist(0 preventive, 5 reactive)
Increase compaction thread count and I/O bandwidth allocation. For Cassandra, switch from SizeTieredCompaction (good for write-heavy) to LeveledCompaction (good for read-heavy) if reads are the primary bottleneck. LCS maintains bounded SSTable count per level by compacting aggressively.
For Cassandra, excessive tombstones (from row deletions or TTL expiry) degrade reads because tombstones must be checked during row reconstruction. Use TimeWindowCompaction for TTL-heavy workloads; compact tombstone-heavy partitions proactively. Avoid partition widths exceeding 100MB.
Snapshots bound the number of events that must be replayed to reconstruct aggregate state, reducing the read I/O required to serve aggregate loads compared to full event log replay.
Reduce the Bloom filter false positive rate (increase bits per key from the default of 10 to 15–20). This increases memory usage but reduces disk I/O for point reads by eliminating more false-positive SSTable checks.
Increase the storage engine's block cache (Cassandra key_cache and row_cache, RocksDB block_cache_size) to keep more SSTable blocks in memory. Reduces disk I/O for repeated reads on the same data. Most effective when the working set fits in memory.
Affected Systems
Workloads
Technologies
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.