Observability Platform

Topology Notes

• ·Kafka is the ingestion entry point for all telemetry: metrics, logs, and traces arrive on separate topics partitioned by source service or host. Topic retention is set to 24 hours for logs, 7 days for metrics (to support replay during Elasticsearch or ClickHouse write failures). Consumer groups are separated by data type : the metrics consumer, log consumer, and trace consumer operate independently with independent scaling and lag monitoring.
• ·ClickHouse receives metrics data from the Kafka metrics consumer via a Kafka table engine + materialized view pipeline. Raw metrics land in a raw_metrics table partitioned by toStartOfHour(timestamp). Continuous materialized views aggregate into 1-minute and 1-hour rollup tables. Dashboard queries run against the appropriate rollup table for the selected time range: raw table is queried only for time ranges within the last 6 hours.
• ·Elasticsearch receives structured log events from the Kafka log consumer. Index lifecycle management (ILM) controls rollover (new index when current index reaches 50GB), transition to warm tier (replica count reduced after 7 days), and deletion (after 30 days). Dynamic mapping is disabled: all indexed fields must be declared in the index template. Unstructured JSON logs with undeclared fields are logged to a separate unmapped_logs index for manual schema review.
• ·Redis stores active alert state (which alert rules are currently firing, at what severity, with what label set) and dashboard query result cache (pre-computed query results for the most-frequently-loaded dashboards, TTL 30 seconds). Alert state is the source of truth for the alert routing engine: Kafka alert events update Redis state before triggering notifications. Redis keyspace notifications are used to trigger alert routing on state transitions.
• ·The alert evaluation engine reads from TimescaleDB continuous aggregates for threshold rules and from ClickHouse for anomaly rules requiring historical context. Alert evaluation is scheduled independently of ingestion: a 30-second evaluation interval is the default. Alert rules are evaluated in parallel across rule groups; each rule group has a dedicated evaluation worker with a timeout to prevent slow rules blocking the evaluation scheduler.