Observability Platform

Summary

A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume.

Problem Statement

Observability platforms face a pathological operational property: the highest telemetry volume occurs exactly when the platform is most needed: during incidents. A log pipeline that degrades under log spike conditions fails at the moment it is most critical. The ingestion path must accept burst write volume without back-pressuring into the production systems generating the telemetry, or those systems begin losing observability data during their own incidents. Separately, metric cardinality is a silent cost driver: a single metric with 20 high-cardinality label dimensions (e.g., user_id × request_path × datacenter × version × status_code) can generate millions of distinct time series from a single service, exhausting storage and query planner resources without a visible failure event.