time_series_metrics vulnerable to disk_io

time series metricsVulnerable Todisk io saturation

Source: Workload·Target: Failure Mode

Summary

Time-series metric workloads generate write throughput that can saturate disk I/O: 100,000-1,000,000 data points per second produce continuous sequential write load that exceeds spinning disk capacity and requires NVMe or storage-optimized instances to sustain.

Evidence

·Prometheus remote write at 1M samples/second generates 500MB-2GB/s of WAL and index writes
·InfluxDB 2.0 documentation requires NVMe SSDs for ingestion above 100K points/second
·TimescaleDB chunk compression at the end of a chunk's lifecycle generates significant I/O spikes
·{'Spinning disk throughput ceiling': '~150-200MB/s sequential, 100-200 IOPS random: insufficient for 500K points/second'}
·AWS io1/io2 volumes are recommended for time-series databases exceeding 50K writes/second

Operational Context

·Monitor disk write latency (iostat await): sustained >10ms write latency indicates I/O saturation
·Use WAL and data on separate volumes to prevent WAL writes from competing with data reads
·Enable compression at the storage engine level: ClickHouse achieves 10:1 compression on metric data

Tradeoffs

·NVMe SSDs are 10-20x more expensive than spinning disks per GB but required for high-throughput time-series
·Write batching (accumulate 1000+ points before writing) amortizes I/O overhead : increases latency from ms to seconds
·Column-oriented storage (ClickHouse, TimescaleDB compression) reduces I/O dramatically but requires columnar query patterns

Evidence grounding

Grounded, 5 supporting items

Disk I/O saturation from time-series write throughput is documented in Prometheus, InfluxDB, and TimescaleDB operational guides and production post-mortems.