Read Amplification (LSM Tree)
SEV-2, Significant ImpactIsolated propagation · storage · Affects 1 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Read Amplification (LSM Tree) begins at the source component. Trigger: Insufficient compaction bandwidth relative to write rate (LSM tree depth increases).
Immediate (T+0) · Signal: Alert
Blast Radius
Elevated read latency for queries that hit many levels. P99 latency spikes are more pronounced than P50, because P50 reads frequently hit the cache or upper levels. If the read latency increase causes cascading timeouts in the application, the blast radius expands to the feature level. RocksDB-backed systems (TiKV, MyRocks) sharing the same instance may see read latency increase across all keyspaces if the overall SSTable depth grows.
Contributing Factors
Apache Cassandra is known to be susceptible to Read Amplification (LSM Tree) under very_high operational burden. Not running regular nodetool repair: inconsistencies accumulate silently and manifest during node replacement
This operational trigger enables Read Amplification (LSM Tree): Insufficient compaction bandwidth relative to write rate (LSM tree depth increases)
This operational trigger enables Read Amplification (LSM Tree): Bloom filter disabled or undersized (false positive rate too high)
This operational trigger enables Read Amplification (LSM Tree): Range queries on keys not co-located in SSTable sort order
Mitigation Gaps
Identify and document at least one architectural change that prevents Read Amplification (LSM Tree) from occurring (not just recovering from it).
Add 'snapshot pattern' to the runbook. Snapshots bound the number of events that must be replayed to reconstruct aggregate state, reducing the read I/O required to serve aggregate loads compared to full event log replay.
Remediation Plan
Measure current SSTable count per level using storage engine metrics
Effort: Minutes to hours (on-call response)
Identify whether compaction is falling behind: check pending compaction size
Effort: Minutes to hours (on-call response)
Increase compaction concurrency (Cassandra: concurrent_compactors; RocksDB: max_background_compactions)
Effort: Minutes to hours (on-call response)
Increase compaction thread count and I/O bandwidth allocation. For Cassandra, switch from SizeTieredCompaction (good for write-heavy) to LeveledCompaction (good for read-heavy) if reads are the primary bottleneck. LCS maintains bounded SSTable count per level by compacting aggressively.
Effort: 1 day to 1 week
Reduce the Bloom filter false positive rate (increase bits per key from the default of 10 to 15–20). This increases memory usage but reduces disk I/O for point reads by eliminating more false-positive SSTable checks.
Effort: 1 day to 1 week
Increase the storage engine's block cache (Cassandra key_cache and row_cache, RocksDB block_cache_size) to keep more SSTable blocks in memory. Reduces disk I/O for repeated reads on the same data. Most effective when the working set fits in memory.
Effort: 1 day to 1 week
Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.