LSM Compaction Debt

In a log-structured merge-tree (LSM) engine, sustained write rate outpaces the background compaction process that merges on-disk SSTables. The number of SSTables a read must check grows (read amplification climbs), and once the backlog crosses an engine-defined threshold, the engine deliberately slows or stops accepting writes to keep worst-case read latency bounded, rather than let read amplification grow without limit.

An LSM engine never updates data in place. Writes land first in an in-memory memtable and are also appended to a write-ahead log for durability. When the memtable fills, it is frozen and flushed to disk as an immutable SSTable (sorted string table). Over time this produces many SSTables, each holding its own sorted run of keys, some overlapping in key range with others. A read for a single key cannot assume the key lives in exactly one place: in the worst case it must check every SSTable that could contain that key range, plus the active memtable, and merge what it finds by recency. This is read amplification, and it grows with the number of SSTables a read has to touch.

Compaction is the background process that bounds this. It merges multiple SSTables into fewer, larger ones, combining overlapping key ranges into a single sorted run, discarding obsolete versions of overwritten keys, and dropping keys whose tombstone (the marker a LSM engine writes instead of deleting in place) has aged past the point where it could still shadow a value in an older, not-yet-compacted SSTable. Compaction is what makes deletes and overwrites actually free space, and what keeps the number of files a read must check from growing without bound.

Compaction debt is what accumulates when the write rate, and therefore the memtable flush rate, produces new SSTables faster than compaction can merge them away. The backlog is visible directly: the count of pending compaction tasks grows, the number of SSTables per level (or, in a tiered strategy, per bucket) grows past what the compaction strategy was sized for, and read latency degrades because reads now check more files per key. This is a feedback loop, not a one-time cost: a larger backlog means the next compaction pass has more data to merge, which takes longer, which lets the backlog grow further while it runs. Left alone, debt does not self-correct once the write rate that created it continues.

Because unbounded read amplification is worse for correctness-adjacent reasons than slower writes, both Cassandra and RocksDB deliberately choose to protect reads by throttling or halting writes once debt crosses a threshold, rather than let SSTable count grow indefinitely. This is the mechanism, not an incidental side effect: a write stall under compaction debt is the engine enforcing a bound on worst-case read cost at the expense of write availability.

Cassandra exposes this operationally through nodetool compactionstats, which reports pending compaction tasks; a sustained, growing pending-compaction count under continuous write load is the direct signal of debt building. compaction_throughput_mb_per_sec (in cassandra.yaml) caps the I/O bandwidth compaction is allowed to consume in the background, so it competes with foreground read and write I/O for the same disk; raising it lets compaction catch up faster but takes I/O bandwidth away from live traffic while it does. SSTable count per read, visible per-table via nodetool tablestats/cfstats, is the read-amplification signal that debt drives up directly.

RocksDB names the mechanism as an explicit column-family option pair: level0_slowdown_writes_trigger and level0_stop_writes_trigger. When the number of level-0 SSTable files (files flushed directly from memtables, before they have been organized into non-overlapping runs by compaction) crosses the slowdown trigger, RocksDB deliberately inserts write delay to give compaction room to catch up; if it keeps growing and crosses the stop trigger, RocksDB stops accepting writes entirely until level-0 file count drops back down. This is the concrete stall/throttle mechanism this failure describes for RocksDB-based engines. The exact default numeric values for both triggers are not asserted here and should be confirmed against the RocksDB version in use; withholding the number is deliberate rather than an oversight, the mechanism is what matters operationally and the default has shifted across RocksDB releases.

This failure is a specific, LSM-only instance of the general write-amplification problem: compaction rewrites the same logical data multiple times as it moves through levels, which is a cost a B-Tree engine's in-place update model does not pay the same way (see write_amplification_cascade for the cross-engine comparison and why the two amplification models must not be collapsed into one bucket). It is also a distinct phenomenon from checkpoint_amplification: a PostgreSQL checkpoint is a periodic, bounded, time-boxed flush of dirty buffer-pool pages on a B-Tree engine, while compaction debt is an open-ended, backlog-driven condition on an LSM engine that has no fixed period and does not resolve itself on a timer, only by compaction throughput exceeding write rate for a sustained interval.