Architecture Review: Observability Platform

The storage device reaches its IOPS or throughput ceiling, causing all disk- dependent database operations to queue behind I/O requests, driving latency from sub-millisecond to hundreds of milliseconds and degrading all database operations simultaneously.

Affects 1 node. (Time-Series Metrics)

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 0 nodes

Seed 'Disk I/O Saturation Risk Probe' identifies 2 metrics relevant to disk_io_saturation.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: Prometheus storage limits reached at production metric volume; ClickHouse required for sub-second aggregate queries across multi-day windows that Prometheus cannot serve; need to retain metrics beyond Prometheus 15-day default retention without capacity-limited remote write targets; cross-service correlation queries not possible in PromQL. Migrate from 'Prometheus + Grafana stack with local time-series storage' to 'Kafka-buffered ClickHouse ingestion with Redis-backed alert evaluation'. Migrate dashboards domain by domain (infrastructure first, then application metrics) rather than all at once. Run Prometheus and ClickHouse in parallel for 30 days, validating that dashboard values match before switching alert evaluation to ClickHouse.

Prometheus metric naming conventions and label schemas must be preserved exactly in ClickHouse: a schema transformation at migration time breaks all existing Grafana dashboards; Alert rules expressed in PromQL must be rewritten for the new alert evaluation engine: the migration window requires running both systems in parallel and validating alert equivalence

Trigger: Elasticsearch indexing pressure causing log rejection (HTTP 429) back-pressuring into application services during incidents; log volume spikes during incidents causing Elasticsearch cluster instability; inability to replay historical logs when the indexing pipeline falls behind. Migrate from 'Log shipping directly to Elasticsearch without Kafka buffer' to 'Kafka-buffered log ingestion with backpressure and sampling controls'. Set Kafka log topic retention to 24 hours minimum. Deploy the Kafka-to-Elasticsearch consumer with auto-scaling enabled before switching log producers to write to Kafka. Validate consumer throughput at 10x normal volume with a load test before production cutover.

Introducing Kafka adds infrastructure complexity for the log shipping agents: every application must be updated to write to Kafka instead of directly to Elasticsearch; Kafka retention must be set to at least the Elasticsearch indexing catch-up time under worst-case incident volume: insufficient retention causes log loss if indexing falls behind more than the retention window

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql, kafka

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning

Bottleneck: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness. Evolution: Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance.

Signal: Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation

Bottleneck: Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods. Evolution: Size the log consumer for 10x normal throughput, not 1x: observability platform capacity must be planned for the incident scenario, not the steady state. Implement consumer autoscaling triggered by consumer lag metric: when Kafka consumer lag exceeds a threshold, add consumer instances automatically. Implement log sampling at the producer side for DEBUG and INFO level messages during identified spike periods : preserve all ERROR and WARN messages, sample INFO at 10%, sample DEBUG at 1%. This bounds the worst-case log volume without sacrificing diagnostic signal.