LSM Compaction Debt
SEV-2, Significant ImpactFeedback Loop propagation · storage · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The feedback loop propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
LSM Compaction Debt begins at the source component. Trigger: Sustained write throughput exceeds the compaction strategy's steady-state merge throughput.
Immediate (T+0) · Signal: Queue Depth
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Minutes to hours with compaction-specific monitoring (Cassandra nodetool compactionstats pending count, RocksDB level-0 file count, or an equivalent metrics exporter tracking pending compaction bytes/tasks), since the backlog trend is visible well before writes actually stall. Without that monitoring, debt is typically first noticed as unexplained read-latency growth on an otherwise unremarkable write workload, which can take days to connect to compaction unless someone checks SSTable count per read directly. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Self-amplification loop
Retry behavior and back-pressure cause the failure to amplify. Resource exhaustion accelerates, the system cannot self-recover without intervention.
Minutes after fan-out · Signal: Cascading alert storms; all downstream indicators deteriorating
Blast Radius
Debt starts as a per-table (Cassandra/ScyllaDB) or per-column-family (RocksDB) condition, but its effects widen in two directions. First, read latency on the affected table degrades for every reader, not just the write-heavy path, because read amplification is a property of the table's SSTable count, not of who is reading. Second, once compaction debt triggers a write stall or throttle, every write to that table blocks or slows, and on a shared node, compaction I/O and write-stall backpressure can starve other tables sharing the same disk and compaction thread pool, so a single hot table's debt can degrade unrelated tables on the same node. In a distributed cluster, if debt is uneven across nodes (for example one node behind on repair-driven compaction), that node's elevated read latency skews the cluster's tail latency even though most nodes are healthy.
Contributing Factors
Apache Cassandra is known to be susceptible to LSM Compaction Debt under very_high operational burden. Not running regular nodetool repair: inconsistencies accumulate silently and manifest during node replacement
This operational trigger enables LSM Compaction Debt: Sustained write throughput exceeds the compaction strategy's steady-state merge throughput
This operational trigger enables LSM Compaction Debt: compaction_throughput_mb_per_sec (Cassandra) set too low for the actual write rate, deliberately capping compaction I/O below what is needed to keep up
This operational trigger enables LSM Compaction Debt: Large bulk load, backfill, or bulk import that spikes the memtable flush rate far above normal steady-state writes
Remediation Plan
Confirm compaction debt is the cause: check pending compaction tasks (nodetool compactionstats) or level-0 SSTable file count, and correlate with the read-latency trend
Effort: Minutes to hours (on-call response)
Temporarily raise the compaction throughput budget or thread count to work down the backlog faster, accepting the foreground I/O contention this adds
Effort: Minutes to hours (on-call response)
Identify whether a specific event (bulk load, repair, TTL expiry wave) triggered the spike, and throttle or reschedule that source if it is still running
Effort: Minutes to hours (on-call response)
Increase compaction_throughput_mb_per_sec (Cassandra) or the equivalent background I/O rate limit for a RocksDB-based engine, and/or increase the number of concurrent compaction threads. This lets compaction consume more of the disk's I/O capacity to work down the backlog faster. The cost moves directly to foreground I/O: compaction and live reads/writes share the same disk, so raising compaction's budget takes bandwidth away from application traffic while the backlog drains, and can itself cause a temporary latency spike on the traffic it is trying to protect.
Effort: 1 day to 1 week
Rather than disabling or fighting level0_slowdown_writes_trigger and level0_stop_writes_trigger (RocksDB) or the equivalent Cassandra backpressure, treat the resulting write latency or rejection as the intended signal: the engine is bounding worst-case read amplification on purpose. The mitigation here is application-side, queue or shed writes gracefully when the engine signals backpressure, rather than retrying aggressively into a stalled write path, which only adds queued load that discharges as a burst once the stall clears.
Effort: 1 day to 1 week
Every overwrite and every delete (tombstone) of the same key adds compaction work later. Using TTLs so expired data ages out via a time-window strategy instead of accumulating as tombstones, batching writes to the same partition to reduce per-write memtable churn, and avoiding read-modify-write patterns that repeatedly overwrite hot keys all reduce the volume compaction has to merge. This is a workload-shape change, not a configuration change, and it moves cost to application design and, for TTL-based expiry, to the read path needing to tolerate eventual rather than immediate space reclamation.
Effort: 1 day to 1 week
Configure alerts for: queue depth, latency spike, disk saturation. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Size-tiered compaction favors write throughput but tolerates more read amplification and space amplification; leveled compaction favors read latency and bounds space amplification but costs more write amplification per byte, since each byte tends to be rewritten across more levels; a time-window strategy fits time-series data with natural expiry (TTL-heavy workloads) and avoids compacting old, cold data at all. Picking the wrong strategy for the write and read pattern is a common root cause of a backlog that raising throughput alone cannot fix. Changing strategy requires re-compacting existing data under the new strategy, which is itself compaction work and temporarily adds to the debt before it reduces it.
Effort: 1–4 sprints
Add nodes and let the partitioner redistribute data, so each node's write rate, and therefore its memtable flush rate and compaction workload, drops proportionally. This addresses the root cause (per-node write rate exceeding per-node compaction throughput) rather than compaction's symptoms, at the cost of additional hardware and the operational work of a rebalance, which itself generates streaming and compaction I/O while it runs.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.