Rule-based disposition: any dimension at its most severe tier caps this at “concerns” or worse. Never an averaged score.
- Operational Readiness: Observability Platform requires high operational expertise at 'experienced backend team' level. Current readiness estimate is 36%, critical gaps must be resolved before adoption. Consider starting with a simpler scenario and evolving toward this one.
Architecture Review: Observability Platform
A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume.
Evidence Confidence
Moderate
strong
Executive Summary
Observability Platform carries moderate operational readiness (81% evidence confidence). 0 architectural strengths identified, 6 operational risks to manage. Primary concern: Disk I/O Saturation. Requires Advanced operational maturity.
Readiness Rationale
Overall moderate readiness across 8 dimensions. Limited: team maturity. Strong: migration, observability, failure recovery.
Key Concerns
- !Disk I/O Saturation
- !Hot Partition
Key Strengths
- +Architecture is well-defined for the analytics pipeline problem profile
8
Assessments
3
Tradeoffs
6
Sections
12
Recommendations
Readiness Assessments
8Governance Posture
5Structural boundary and anti-pattern compliance: whether this architecture's topology violates documented governance policies. Distinct from operational readiness (below), which asks whether the team and infrastructure are prepared to run it.
5 governance policy matches and 0 anti-pattern matches put Observability Platform's governance posture at concerning risk. Resilience is limited; burden is extreme.
5
violations
0
anti-patterns
Governance Violations
Resilience
Blast radius: contained
53%
resilience score
Coupling Risks
- ·ClickHouse mutation blocking concurrent inserts: a background schema change or U
Resilience Gaps
- △4 high-exposure risk nodes increase blast radius
Operational Burden
operational burden
100%
burden index
Complexity Drivers
- ⚙6 architecture patterns increase configuration surface
- ⚙Cardinality explosion from unbounded label dimensions: a metric emitted with a u
- ⚙Log volume spike during incident overwhelming Elasticsearch indexing throughput:
Observability Burden
- ◎clickhouse: requires dedicated monitoring instrumentation
- ◎elasticsearch: requires dedicated monitoring instrumentation
- ◎kafka: requires dedicated monitoring instrumentation
- ◎redis: requires dedicated monitoring instrumentation
Recovery Complexity
- ⟳2 risk propagation path(s) complicate failure recovery
Maturity
Required
AdvancedEstimated
AdvancedGap
No GapThe architecture's required maturity (advanced) aligns with or is below the estimated team capability.
Operational Readiness
7Adoption readiness: whether the team, infrastructure, and observability are prepared to run this architecture safely. Distinct from governance posture (above), which asks whether the topology itself violates architectural boundaries.
Observability Platform requires high operational expertise at 'experienced backend team' level. Current readiness estimate is 36%, critical gaps must be resolved before adoption. Consider starting with a simpler scenario and evolving toward this one.
Readiness Score
36%
Blocking Prerequisites
4
Complexity
High
Confidence
Strong
Assessment derived from scenario knowledge, advisor output, topology analysis, and 7 prerequisite checks.
Prerequisite Checklist (4 blocking, 3 non-blocking)
team
Team at 'experienced backend team' maturity level
This scenario is rated 'experienced backend team' complexity. Engineers with 2+ years of production backend experience, including database tuning and monitoring.
Gap signal: Team frequently reaches for external help during incidents or struggles to debug multi-system issues independently.
process
Failure mode awareness and runbooks
The team must understand the 6 documented failure modes for this scenario: disk_io_saturation, hot_partition, wal_saturation, slow_consumer. Each should have a documented detection procedure and runbook.
Gap signal: The team has no documented runbooks for the scenario's failure modes or cannot name them without reference material.
monitoring
Production-grade observability stack
The scenario requires real-time metrics, structured logging, and distributed tracing on all critical components. Alerting must be configured before going live.
Gap signal: No dashboards exist for the critical path metrics in the scenario.
infrastructure
Minimum team maturity: Experienced Backend Team
This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
Gap signal: The requirement 'Minimum team maturity: Experienced Backend Team' is not yet in place.
infrastructure
Runbooks and alerting for high-severity risks
4 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
Gap signal: The requirement 'Runbooks and alerting for high-severity risks' is not yet in place.
infrastructure
Event stream operations expertise
This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
Gap signal: The requirement 'Event stream operations expertise' is not yet in place.
infrastructure
Mitigation for 2 high-risk topology node(s)
Nodes with high or critical risk exposure: Time-Series Metrics, Write-Heavy Transactional. Each requires documented mitigation before production deployment.
Gap signal: No mitigation strategy is documented for the high-risk nodes in the topology.
Infrastructure Requirements
ClickHouse
medium burdenColumn-oriented OLAP database engineered for sub-second analytical queries on billions of rows, with vectorized execution, aggressive compression, and
Managed: ClickHouse Cloud, Altinity.Cloud
Elasticsearch
high burdenDistributed full-text search and analytics engine built on Apache Lucene, designed for near-real-time indexing, complex search queries, and log analyt
Managed: Elastic Cloud (Elastic.co), Amazon OpenSearch Service, Elastic Cloud on Kubernetes (ECK)
Apache Kafka
high burdenDistributed event streaming platform designed for high-throughput, fault-tolerant, ordered, and durable log-based messaging between producers and cons
Managed: Amazon MSK (Managed Streaming for Kafka), Confluent Cloud, Azure Event Hubs (Kafka-compatible), Redpanda Cloud
Redis
low burdenIn-memory key-value store with optional persistence, supporting strings, hashes, lists, sets, sorted sets, and pub/sub.
Managed: Amazon ElastiCache for Redis, Google Cloud Memorystore, Azure Cache for Redis, Redis Cloud, Upstash
TimescaleDB
medium burdenPostgreSQL extension that adds time-series-specific capabilities: automatic time-based partitioning (hypertables), columnar compression on cold chunks
Managed: Timescale Cloud, Amazon RDS (PostgreSQL + TimescaleDB extension), Supabase (TimescaleDB extension available)
Observability Requirements
Monitor generic risk probe signals
Seed 'Disk I/O Saturation Risk Probe' identifies 2 metrics relevant to disk_io_saturation.
Seed 'Disk I/O Saturation Risk Probe' identifies 2 metrics relevant to disk_io_saturation.
Track Disk I/O Saturation exposure
Disk I/O Saturation has high exposure and affects 1 component. Affects 1 node. (Time-Series Metrics)
Disk I/O Saturation has high exposure and affects 1 component. Affects 1 node. (Time-Series Metrics)
Track Hot Partition exposure
Hot Partition has high exposure and affects 0 components. Affects 0 nodes
Hot Partition has high exposure and affects 0 components. Affects 0 nodes
Track WAL Saturation exposure
WAL Saturation has high exposure and affects 1 component. Affects 1 node. (Write-Heavy Transactional)
WAL Saturation has high exposure and affects 1 component. Affects 1 node. (Write-Heavy Transactional)
Track Write Amplification Cascade exposure
Write Amplification Cascade has high exposure and affects 0 components. Affects 0 nodes
Write Amplification Cascade has high exposure and affects 0 components. Affects 0 nodes
ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHo
This signal indicates the architecture is approaching 'Tier 1: Metric Cardinality Budget Exceeded'. Likely bottleneck: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness.
Tier 1: Metric Cardinality Budget Exceeded
Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput met
This signal indicates the architecture is approaching 'Tier 2: Log Volume Spike Exceeding Consumer Throughput'. Likely bottleneck: Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods.
Tier 2: Log Volume Spike Exceeding Consumer Throughput
Alert routing system receiving > 10,000 alert events per minute; on-call engineers reporting alert fatigue and inability
This signal indicates the architecture is approaching 'Tier 3: Alert Evaluation Fanout Amplification'. Likely bottleneck: High-cardinality alert rules evaluating independently for every label combination, producing alert notification floods that are operationally equivalent to no alerting.
Tier 3: Alert Evaluation Fanout Amplification
Readiness Action Plan
Satisfy: Team at 'experienced backend team' maturity level
Effort: 1–4 weeks depending on current state · Unblocks: Adoption of Observability Platform
Satisfy: Failure mode awareness and runbooks
Effort: 1–4 weeks depending on current state · Unblocks: Adoption of Observability Platform
Satisfy: Production-grade observability stack
Effort: 1–4 weeks depending on current state · Unblocks: Adoption of Observability Platform
Satisfy: Mitigation for 2 high-risk topology node(s)
Effort: 1–4 weeks depending on current state · Unblocks: Adoption of Observability Platform
Instrument all critical path components with metrics and alerting
Effort: 1–2 weeks · Unblocks: Safe production adoption and incident response
Validate adoption in a staging environment before production
Effort: 2–4 weeks for thorough staging validation · Unblocks: Production confidence and rollback preparedness
Mitigate risk: Disk I/O Saturation
Effort: 1–3 weeks · Unblocks: Reduces 'Disk I/O Saturation' from blocking adoption
Mitigate risk: Hot Partition
Effort: 1–3 weeks · Unblocks: Reduces 'Hot Partition' from blocking adoption
Go Signals
- ✓Team has hands-on experience with all 5 referenced technologies.
- ✓All scenario failure modes have documented runbooks and alerting coverage.
- ✓A staging environment that mirrors production load has been tested successfully.
No-Go Signals
- ✗Team cannot explain or debug any of Observability Platform's documented failure modes.
- ✗No observability baseline exists for the critical components.
- ✗Top risk is unmitigated: 'Disk I/O Saturation', do not proceed without addressing this.
Critical Gaps
- This scenario has high operational complexity, teams without deep production experience will struggle to operate it safely.
Team Requirements
ClickHouse operations
Required level: proficient
Team can explain ClickHouse's failure modes, tune configuration parameters under load, and recover from common operational issues.
Elasticsearch operations
Required level: proficient
Team can explain Elasticsearch's failure modes, tune configuration parameters under load, and recover from common operational issues.
Apache Kafka operations
Required level: proficient
Team can explain Apache Kafka's failure modes, tune configuration parameters under load, and recover from common operational issues.
Redis operations
Required level: proficient
Team can explain Redis's failure modes, tune configuration parameters under load, and recover from common operational issues.
TimescaleDB operations
Required level: proficient
Team can explain TimescaleDB's failure modes, tune configuration parameters under load, and recover from common operational issues.
Readiness assessment is derived from structured scenario and topology knowledge. It provides an evidence-grounded baseline, not a substitute for an actual team capability review or infrastructure audit. Validate each item against your specific environment.
Architectural Tradeoffs
3Recommendations
12Monitor: Disk I/O Saturation
risk_monitoringThe storage device reaches its IOPS or throughput ceiling, causing all disk- dependent database operations to queue behind I/O requests, driving latency from sub-millisecond to hundreds of milliseconds and degrading all database operations simultaneously.
Affects 1 node. (Time-Series Metrics)
Monitor: Hot Partition
risk_monitoringOne partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.
Affects 0 nodes
Implement: Monitor generic risk probe signals
observabilitySeed 'Disk I/O Saturation Risk Probe' identifies 2 metrics relevant to disk_io_saturation.
Metrics to instrument: error_rate, p95_latency_ms
Prometheus + Grafana stack with local time-series storage → Kafka-buffered ClickHouse ingestion with Redis-backed alert evaluation
migration_planningTrigger: Prometheus storage limits reached at production metric volume; ClickHouse required for sub-second aggregate queries across multi-day windows that Prometheus cannot serve; need to retain metrics beyond Prometheus 15-day default retention without capacity-limited remote write targets; cross-service correlation queries not possible in PromQL. Migrate from 'Prometheus + Grafana stack with local time-series storage' to 'Kafka-buffered ClickHouse ingestion with Redis-backed alert evaluation'. Migrate dashboards domain by domain (infrastructure first, then application metrics) rather than all at once. Run Prometheus and ClickHouse in parallel for 30 days, validating that dashboard values match before switching alert evaluation to ClickHouse.
Prometheus metric naming conventions and label schemas must be preserved exactly in ClickHouse: a schema transformation at migration time breaks all existing Grafana dashboards; Alert rules expressed in PromQL must be rewritten for the new alert evaluation engine: the migration window requires running both systems in parallel and validating alert equivalence
Log shipping directly to Elasticsearch without Kafka buffer → Kafka-buffered log ingestion with backpressure and sampling controls
migration_planningTrigger: Elasticsearch indexing pressure causing log rejection (HTTP 429) back-pressuring into application services during incidents; log volume spikes during incidents causing Elasticsearch cluster instability; inability to replay historical logs when the indexing pipeline falls behind. Migrate from 'Log shipping directly to Elasticsearch without Kafka buffer' to 'Kafka-buffered log ingestion with backpressure and sampling controls'. Set Kafka log topic retention to 24 hours minimum. Deploy the Kafka-to-Elasticsearch consumer with auto-scaling enabled before switching log producers to write to Kafka. Validate consumer throughput at 10x normal volume with a load test before production cutover.
Introducing Kafka adds infrastructure complexity for the log shipping agents: every application must be updated to write to Kafka instead of directly to Elasticsearch; Kafka retention must be set to at least the Elasticsearch indexing catch-up time under worst-case incident volume: insufficient retention causes log loss if indexing falls behind more than the retention window
Prepare runbook for: Burst Traffic Cold Cache Stampede
simulation_preparednessSimulation demonstrates critical degradation of redis, postgresql
Without a runbook, recovery from this failure mode will be ad-hoc
Prepare runbook for: CQRS Projection Lag Expansion
simulation_preparednessSimulation demonstrates critical degradation of postgresql, kafka
Without a runbook, recovery from this failure mode will be ad-hoc
Plan evolution: OLTP Analytics Queries → OLTP + OLAP Separation
evolution_planningEvolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)
Migration complexity: medium. Rollback: always.
Plan evolution: Single Cache Layer → Distributed Cache
evolution_planningEvolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)
Migration complexity: medium. Rollback: complex.
Cache-outage database fallback load
caching'Observability Platform' includes a cache in its topology. If the cache becomes unavailable, the primary database receives the cache's full request load until the cache recovers.
Capacity-plan the primary database for this fallback load, not only for the steady-state cached load.
Cache invalidation ownership
cachingCache invalidation for Observability Platform is event-driven: kafka refreshes or invalidates redis. This couples cache freshness to consumer lag on that event stream, not to the primary write path directly.
If the event-stream consumer falls behind, the cache serves stale data until it catches up -- monitor consumer lag as a cache-freshness signal, not only a backlog signal.
Monitor threshold: Tier 1: Metric Cardinality Budget Exceeded
scaling_monitoringSignal: ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning
Bottleneck: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness. Evolution: Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance.
Scaling Pressure Signals
8ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning
Threshold
Tier 1: Metric Cardinality Budget Exceeded
Likely Bottleneck
Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness
Recommended Evolution
Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance.
Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation
Threshold
Tier 2: Log Volume Spike Exceeding Consumer Throughput
Likely Bottleneck
Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods
Recommended Evolution
Size the log consumer for 10x normal throughput, not 1x: observability platform capacity must be planned for the incident scenario, not the steady state. Implement consumer autoscaling triggered by consumer lag metric: when Kafka consumer lag exceeds a threshold, add consumer instances automatically. Implement log sampling at the producer side for DEBUG and INFO level messages during identified spike periods : preserve all ERROR and WARN messages, sample INFO at 10%, sample DEBUG at 1%. This bounds the worst-case log volume without sacrificing diagnostic signal.
Alert routing system receiving > 10,000 alert events per minute; on-call engineers reporting alert fatigue and inability to identify the root alert in notification floods; PagerDuty or equivalent showing duplicate alerts firing simultaneously for correlated failures; alert evaluation CPU dominating observability platform resource consumption
Threshold
Tier 3: Alert Evaluation Fanout Amplification
Likely Bottleneck
High-cardinality alert rules evaluating independently for every label combination, producing alert notification floods that are operationally equivalent to no alerting
Recommended Evolution
Introduce alert grouping at the evaluation layer: alerts on the same metric name within the same time window are grouped into a single notification with a count of affected series. Implement alert inhibition rules: if a datacenter-level alert fires, suppress region-level and service-level alerts that are downstream of the same failure. Move from per-series alert rules to aggregate alert rules: "more than 10% of service instances have error rate > 5%" is a single alert, not 500 individual alerts.
Monthly storage growth rate for metrics and logs exceeding capacity plan; ClickHouse data volume > 10TB with no retention enforcement in place; Elasticsearch index size growing faster than the retention policy deletes old indices; cost reports showing observability storage as top-3 infrastructure cost item
Threshold
Tier 4: Storage Cost Exceeding Retention Budget
Likely Bottleneck
No tiered retention strategy: all data retained at full resolution indefinitely rather than rolled up and archived to lower-cost storage
Recommended Evolution
Implement a three-tier retention strategy: hot tier (raw data, last 7 days, full resolution, ClickHouse), warm tier (1-hour rollups, last 90 days, ClickHouse compressed), cold tier (daily rollups, 2+ years, S3 Parquet via ClickHouse external tables or Athena). Log retention follows a separate policy: raw logs retained 30 days in Elasticsearch, then archived to S3 with a query interface for compliance replay. Alert evaluation always runs against the hot tier.
ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning
Threshold
Escalation trigger: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness
Likely Bottleneck
Tier 1: Metric Cardinality Budget Exceeded
Recommended Evolution
Monitor: error_rate, p95_latency_ms
Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation
Threshold
Escalation trigger: Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods
Likely Bottleneck
Tier 2: Log Volume Spike Exceeding Consumer Throughput
Recommended Evolution
Monitor: error_rate, p95_latency_ms
Alert routing system receiving > 10,000 alert events per minute; on-call engineers reporting alert fatigue and inability to identify the root alert in notification floods; PagerDuty or equivalent showing duplicate alerts firing simultaneously for correlated failures; alert evaluation CPU dominating observability platform resource consumption
Threshold
Escalation trigger: High-cardinality alert rules evaluating independently for every label combination, producing alert notification floods that are operationally equivalent to no alerting
Likely Bottleneck
Tier 3: Alert Evaluation Fanout Amplification
Recommended Evolution
Monitor: error_rate, p95_latency_ms
Monthly storage growth rate for metrics and logs exceeding capacity plan; ClickHouse data volume > 10TB with no retention enforcement in place; Elasticsearch index size growing faster than the retention policy deletes old indices; cost reports showing observability storage as top-3 infrastructure cost item
Threshold
Escalation trigger: No tiered retention strategy: all data retained at full resolution indefinitely rather than rolled up and archived to lower-cost storage
Likely Bottleneck
Tier 4: Storage Cost Exceeding Retention Budget
Recommended Evolution
Monitor: error_rate, p95_latency_ms
Migration Readiness
12Migration Stages
3Prometheus + Grafana stack with local time-series storage → Kafka-buffered ClickHouse ingestion with Redis-backed alert evaluation
infoMigration trigger: Prometheus storage limits reached at production metric volume; ClickHouse required for sub-second aggregate queries across multi-day windows that Prometheus cannot serve; need to retain metrics beyond Prometheus 15-day default retention without capacity-limited remote write targets; cross-service correlation queries not possible in PromQL
Log shipping directly to Elasticsearch without Kafka buffer → Kafka-buffered log ingestion with backpressure and sampling controls
infoMigration trigger: Elasticsearch indexing pressure causing log rejection (HTTP 429) back-pressuring into application services during incidents; log volume spikes during incidents causing Elasticsearch cluster instability; inability to replay historical logs when the indexing pipeline falls behind
Direct ClickHouse queries for alert evaluation on every alert tick → TimescaleDB continuous aggregates as pre-computed alert evaluation views
infoMigration trigger: Alert evaluation latency > 1s causing missed alert firing windows; ClickHouse CPU saturated by alert evaluation scans competing with dashboard queries; alert evaluation queries scanning full column segments without partition pruning due to high cardinality in alert rule label selectors
Risks
9Prometheus metric naming conventions and label schemas must
warningPrometheus metric naming conventions and label schemas must be preserved exactly in ClickHouse: a schema transformation at migration time breaks all existing Grafana dashboards
Alert rules expressed in PromQL must be rewritten for the ne
warningAlert rules expressed in PromQL must be rewritten for the new alert evaluation engine: the migration window requires running both systems in parallel and validating alert equivalence
Introducing Kafka adds infrastructure complexity for the log
warningIntroducing Kafka adds infrastructure complexity for the log shipping agents: every application must be updated to write to Kafka instead of directly to Elasticsearch
Kafka retention must be set to at least the Elasticsearch in
warningKafka retention must be set to at least the Elasticsearch indexing catch-up time under worst-case incident volume: insufficient retention causes log loss if indexing falls behind more than the retention window
Continuous aggregate refresh interval determines the minimum
warningContinuous aggregate refresh interval determines the minimum alert detection latency: a 1-minute refresh interval means alerts cannot fire on sub-minute anomalies
TimescaleDB continuous aggregate must be defined for every m
warningTimescaleDB continuous aggregate must be defined for every metric queried by alert rules: adding new alert rules may require adding new continuous aggregates
Projection lag creates a read-after-write window where users
criticalProjection lag creates a read-after-write window where users see stale data after their own writes. Mitigation: Route immediate post-write reads to the write store (session-scoped write token); accept eventual consistency only for non-user-initiated reads
↗ direct-db-to-cqrsProjection rebuild after schema change can take hours or day
criticalProjection rebuild after schema change can take hours or days on large datasets. Mitigation: Design blue/green projection deployment: build new projection in parallel before switching traffic; test rebuild time in staging
↗ direct-db-to-cqrsCross-service workflows that previously used database transa
criticalCross-service workflows that previously used database transactions now require Saga orchestration. Mitigation: Design idempotent event handlers; implement compensating transactions for every multi-step workflow; test failure injection in staging
↗ modular-monolith-to-event-driven