Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'Notification provider latency (SendGrid, FCM) adding 200–500'. Observability Platform: triggered by 'Prometheus storage limits reached at production metric volum'.

Migration Step 2

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'Product requirement to add mobile push notifications and SMS'. Observability Platform: triggered by 'Elasticsearch indexing pressure causing log rejection (HTTP '.

Migration Step 3

Both scenarios define a migration step at this stage. Notification Delivery Platform: triggered by 'User complaints about notification volume increasing; spam c'. Observability Platform: triggered by 'Alert evaluation latency > 1s causing missed alert firing wi'.

RabbitMQ queue depth for email channel growing > 100k messages; SendGrid 429 responses visible in delivery worker logs; email delivery p95 latency > 2 minutes; delivery worker retry thread pool saturated; dead-letter queue receiving messages from retry exhaustion despite provider being available

Tier 1: Provider Rate Limit Backlog: Delivery worker retry policy not aligned with provider rate limit reset window; workers retrying before the rate limit has reset, accumulating failed attempts. Recommended evolution: Implement provider-aware retry backoff: on 429 response, parse the Retry-After header and schedule the next retry attempt at exactly that time, not on a fixed exponential backoff schedule; add per-provider circuit breakers that open after 5 consecutive 429s and attempt a probe request at the retry-after interval; scale email consumer replicas to process the backlog faster when the rate limit window resets .

Kafka consumer lag for notification event consumers growing; notification delivery volume much higher than upstream business event volume (ratio > 5:1); RabbitMQ aggregate message rate across all channel queues elevated; one upstream event type (e.g., new_comment) accounting for disproportionate share of notification volume

Tier 2: Notification Fan-Out Amplification: Upstream event fan-out generating more notification sends per event than expected; or a notification rule misconfiguration triggering notifications for every event regardless of user preference. Recommended evolution: Audit notification fan-out ratio per upstream event type; add fan-out cost metrics (notifications_generated per upstream event) as a monitored SLA; implement notification preference filtering before fan-out: only generate delivery attempts for users with the corresponding notification type enabled; implement a notification aggregation layer that batches multiple low-priority events into digest notifications rather than individual sends .

ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning

Tier 1: Metric Cardinality Budget Exceeded: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness. Recommended evolution: Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance. .

Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation

Tier 2: Log Volume Spike Exceeding Consumer Throughput: Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods. Recommended evolution: Size the log consumer for 10x normal throughput, not 1x: observability platform capacity must be planned for the incident scenario, not the steady state. Implement consumer autoscaling triggered by consumer lag metric: when Kafka consumer lag exceeds a threshold, add consumer instances automatically. Implement log sampling at the producer side for DEBUG and INFO level messages during identified spike periods : preserve all ERROR and WARN messages, sample INFO at 10%, sample DEBUG at 1%. This bounds the worst-case log volume without sacrificing diagnostic signal. .