Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Observability Platform: triggered by 'Prometheus storage limits reached at production metric volum'. E-Commerce Order Platform: triggered by 'Payment provider timeout causing full checkout rollback and '.

Migration Step 2

Both scenarios define a migration step at this stage. Observability Platform: triggered by 'Elasticsearch indexing pressure causing log rejection (HTTP '. E-Commerce Order Platform: triggered by 'Product search p99 > 1s; faceted navigation (category + pric'.

Migration Step 3

Both scenarios define a migration step at this stage. Observability Platform: triggered by 'Alert evaluation latency > 1s causing missed alert firing wi'. E-Commerce Order Platform: triggered by 'Email/push notification provider timeouts causing checkout l'.

ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning

Tier 1: Metric Cardinality Budget Exceeded: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness. Recommended evolution: Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance. .

Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation

Tier 2: Log Volume Spike Exceeding Consumer Throughput: Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods. Recommended evolution: Size the log consumer for 10x normal throughput, not 1x: observability platform capacity must be planned for the incident scenario, not the steady state. Implement consumer autoscaling triggered by consumer lag metric: when Kafka consumer lag exceeds a threshold, add consumer instances automatically. Implement log sampling at the producer side for DEBUG and INFO level messages during identified spike periods : preserve all ERROR and WARN messages, sample INFO at 10%, sample DEBUG at 1%. This bounds the worst-case log volume without sacrificing diagnostic signal. .

PostgreSQL pg_locks showing high RowExclusiveLock contention on inventory_items for specific sku_ids; checkout p99 > 2s for contended SKUs; deadlock errors appearing in application logs during sale events; effective checkout throughput for hot SKUs well below per-request checkout latency would predict

Tier 1: Flash Sale Inventory Contention: Concurrent saga checkout attempts competing for the same inventory row via row-level locking. Recommended evolution: Introduce a per-SKU checkout serialization queue at the application layer : all concurrent checkout requests for the same SKU are queued and processed serially, converting lock contention into queue latency. Alternatively, use PostgreSQL advisory locks with non-blocking trylock: requests that cannot acquire the lock immediately return a "sold out" response rather than queuing. For very high flash sale volumes, pre-allocate inventory slots (reserve N slots per sale event, each slot is a row with one reservation) to spread lock contention across N rows instead of one. .

Checkout p99 tracking payment provider p99 almost linearly; connection pool utilization on the payment service rising during payment provider slowdowns; circuit breaker trip events appearing in payment service metrics; saga timeout events correlated with payment provider latency spikes

Tier 2: Payment Provider Latency Amplifying Checkout Latency: Checkout saga holding a database connection and an inventory reservation open for the duration of the payment provider call: payment latency directly amplifies connection pool pressure. Recommended evolution: Decouple the payment step from the synchronous checkout saga: reserve inventory and create the order record synchronously, then process payment asynchronously. The customer receives an "order confirmed, payment processing" state immediately; the payment step runs as a separate saga step triggered by an event. This reduces the synchronous checkout latency to the inventory reservation time, not the payment provider round-trip time. .