Use Observability Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Observability Platform. Traceable to YAML knowledge entities.
Context
Observability platforms face a pathological operational property: the highest telemetry volume occurs exactly when the platform is most needed: during incidents. A log pipeline that degrades under log spike conditions fails at the moment it is most critical. The ingestion path must accept burst write volume without back-pressuring into the production systems generating the telemetry, or those systems begin losing observability data during their own incidents. Separately, metric cardinality is a silent cost driver: a single metric with 20 high-cardinality label dimensions (e.g., user_id × request_path × datacenter × version × status_code) can generate millions of distinct time series from a single service, exhausting storage and query planner resources without a visible failure event. Primary operational risks include: Cardinality explosion from unbounded label dimensions: a metric emitted with a user_id label from a high-traffic service generates a distinct time series per user. At 1 million active users, a single metric name creates 1 million distinct time series. ClickHouse handles high-cardinality columnar data better than Prometheus-based systems, but cardinality-blind metric instrumentation is the most common cause of observability platform storage exhaustion and query timeout in production.; Log volume spike during incident overwhelming Elasticsearch indexing throughput: when a production service enters a failure loop (rapid retry storms, panic-level logging), it can emit 100–1000x its normal log volume in seconds. If the Kafka log consumer is sized for normal operating throughput, the consumer falls behind. The Elasticsearch indexing pipeline runs out of buffer, and the logs most needed for incident diagnosis arrive hours after the incident closes.; Alert evaluation fanout amplification: a single alert rule evaluated against a high-cardinality metric (e.g., "error rate > 5% per service × region × version") evaluates independently for every distinct label combination. An alert rule that fires on 5,000 distinct series simultaneously produces 5,000 individual alert notifications, overwhelming the alert routing system and burying the 5 truly critical alerts in a notification flood that ops teams learn to ignore..
Decision
We will adopt the **Observability Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume. Core technology stack: clickhouse, kafka, elasticsearch, timescaledb, redis.
Accepted Tradeoffs
- ⚠Kafka ingestion buffer decouples ingest acceptance rate from ClickHouse write throughput, enabling the platform to absorb write bursts without back-pressuring production systems, but Kafka consumer lag is a silent consistency gap: dashboards and alerts run against data that may be 10–60 seconds behind real-time during sustained high-volume periods
- ⚠ClickHouse columnar storage achieves scan throughput that makes sub-second dashboard queries possible across billions of rows, but ClickHouse's merge tree architecture is not designed for low-latency single-row lookups: individual metric point queries are significantly slower than aggregate scans, so alert evaluation must be designed around aggregate queries, not row lookups
- ⚠TimescaleDB continuous aggregates enable sub-100ms alert evaluation against pre-computed rollup tables, but continuous aggregate refresh introduces a minimum alert evaluation latency equal to the aggregate refresh interval: real-time alerting at 1-second resolution requires the aggregate to refresh every second, which is impractical at high cardinality
- ⚠Elasticsearch full-text log search provides the query flexibility required for incident diagnosis, but Elasticsearch index mapping explosions (unbounded dynamic field mapping from unstructured JSON logs) degrade index performance and storage efficiency faster than structured columnar storage: log schema enforcement at ingestion is required to prevent mapping explosion
Risks
The storage device reaches its IOPS or throughput ceiling, causing all disk- dependent database operations to queue behind I/O requests, driving latency from sub-millisecond to hundreds of milliseconds and degrading all database operations simultaneously.
One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.
PostgreSQL WAL (Write-Ahead Log) generation rate exceeds wal_buffers flush capacity or downstream replica/WAL archive bandwidth, causing write transactions to stall waiting for WAL flush and replication lag to grow unboundedly.
Each logical application write triggers multiple physical writes through index maintenance, WAL generation, MVCC versioning, and replication, causing actual disk IOPS to exceed the provisioned I/O ceiling while the logical write rate appears modest.
A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, redis) with the chosen architecture but applies different structural patterns; Observability Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (clickhouse, kafka) with the chosen architecture but applies different structural patterns; Observability Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, redis) with the chosen architecture but applies different structural patterns; Observability Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (clickhouse, kafka) with the chosen architecture but applies different structural patterns; Observability Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Metric Cardinality Budget Exceeded
Signal: ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning
Evolution: Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance.
Tier 2: Log Volume Spike Exceeding Consumer Throughput
Signal: Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation
Evolution: Size the log consumer for 10x normal throughput, not 1x: observability platform capacity must be planned for the incident scenario, not the steady state. Implement consumer autoscaling triggered by consumer lag metric: when Kafka consumer lag exceeds a threshold, add consumer instances automatically. Implement log sampling at the producer side for DEBUG and INFO level messages during identified spike periods : preserve all ERROR and WARN messages, sample INFO at 10%, sample DEBUG at 1%. This bounds the worst-case log volume without sacrificing diagnostic signal.
Tier 3: Alert Evaluation Fanout Amplification
Signal: Alert routing system receiving > 10,000 alert events per minute; on-call engineers reporting alert fatigue and inability to identify the root alert in notification floods; PagerDuty or equivalent showing duplicate alerts firing simultaneously for correlated failures; alert evaluation CPU dominating observability platform resource consumption
Evolution: Introduce alert grouping at the evaluation layer: alerts on the same metric name within the same time window are grouped into a single notification with a count of affected series. Implement alert inhibition rules: if a datacenter-level alert fires, suppress region-level and service-level alerts that are downstream of the same failure. Move from per-series alert rules to aggregate alert rules: "more than 10% of service instances have error rate > 5%" is a single alert, not 500 individual alerts.
Tier 4: Storage Cost Exceeding Retention Budget
Signal: Monthly storage growth rate for metrics and logs exceeding capacity plan; ClickHouse data volume > 10TB with no retention enforcement in place; Elasticsearch index size growing faster than the retention policy deletes old indices; cost reports showing observability storage as top-3 infrastructure cost item
Evolution: Implement a three-tier retention strategy: hot tier (raw data, last 7 days, full resolution, ClickHouse), warm tier (1-hour rollups, last 90 days, ClickHouse compressed), cold tier (daily rollups, 2+ years, S3 Parquet via ClickHouse external tables or Athena). Log retention follows a separate policy: raw logs retained 30 days in Elasticsearch, then archived to S3 with a query interface for compliance replay. Alert evaluation always runs against the hot tier.
Migration Path
Prometheus + Grafana stack with local time-series storage → Kafka-buffered ClickHouse ingestion with Redis-backed alert evaluation
Prometheus storage limits reached at production metric volume; ClickHouse required for sub-second aggregate queries across multi-day windows that Prometheus cannot serve; need to retain metrics beyond Prometheus 15-day default retention without capacity-limited remote write targets; cross-service correlation queries not possible in PromQL
Log shipping directly to Elasticsearch without Kafka buffer → Kafka-buffered log ingestion with backpressure and sampling controls
Elasticsearch indexing pressure causing log rejection (HTTP 429) back-pressuring into application services during incidents; log volume spikes during incidents causing Elasticsearch cluster instability; inability to replay historical logs when the indexing pipeline falls behind
Direct ClickHouse queries for alert evaluation on every alert tick → TimescaleDB continuous aggregates as pre-computed alert evaluation views
Alert evaluation latency > 1s causing missed alert firing windows; ClickHouse CPU saturated by alert evaluation scans competing with dashboard queries; alert evaluation queries scanning full column segments without partition pruning due to high cardinality in alert rule label selectors
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 4 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.