Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Observability Platform: triggered by 'Prometheus storage limits reached at production metric volum'. Gaming Backend Platform: triggered by 'Single game server instance running at connection ceiling; h'.

Migration Step 2

Both scenarios define a migration step at this stage. Observability Platform: triggered by 'Elasticsearch indexing pressure causing log rejection (HTTP '. Gaming Backend Platform: triggered by 'Game server instances blocking on PostgreSQL writes at end-o'.

Migration Step 3

Both scenarios define a migration step at this stage. Observability Platform: triggered by 'Alert evaluation latency > 1s causing missed alert firing wi'. Gaming Backend Platform: triggered by 'PostgreSQL event log table approaching 100B rows; reconnect '.

ClickHouse part merge frequency increasing; dashboard queries timing out on metrics with high label cardinality; ClickHouse system.metrics showing active_parts count elevated; new metric instrumentation causing sudden storage growth disproportionate to fleet size; query_log showing metrics queries scanning full column segments without pruning

Tier 1: Metric Cardinality Budget Exceeded: Unbounded label cardinality generating millions of distinct time series that exceed ClickHouse part merge capacity and query planner pruning effectiveness. Recommended evolution: Enforce a cardinality budget at ingestion: before a metric is accepted, evaluate the distinct value count of each label dimension against a per-dimension limit (e.g., max 100 distinct values for any single label key). Reject or rewrite metrics that exceed the budget: rewrite user_id labels to user_cohort or drop them entirely. Implement a cardinality analysis dashboard showing the top 10 highest-cardinality metric series sorted by storage cost. ClickHouse distributed table partitioning by metric name reduces the impact of a single high-cardinality metric on global query performance. .

Kafka log topic consumer lag growing > 1 million messages during incident periods; Elasticsearch indexing throughput metrics showing queue buildup; incident post-mortems noting that relevant log records were not available in the search interface during the incident; log consumer memory pressure from unbounded batch accumulation

Tier 2: Log Volume Spike Exceeding Consumer Throughput: Log Kafka consumer sized for normal throughput; unable to drain the spike volume produced during incident-driven log floods. Recommended evolution: Size the log consumer for 10x normal throughput, not 1x: observability platform capacity must be planned for the incident scenario, not the steady state. Implement consumer autoscaling triggered by consumer lag metric: when Kafka consumer lag exceeds a threshold, add consumer instances automatically. Implement log sampling at the producer side for DEBUG and INFO level messages during identified spike periods : preserve all ERROR and WARN messages, sample INFO at 10%, sample DEBUG at 1%. This bounds the worst-case log volume without sacrificing diagnostic signal. .

Game server instance file descriptor count approaching OS limit (typically 65k open connections); WebSocket accept latency increasing; new connection establishment p99 > 200ms; CPU on game server instances > 70% during peak concurrent player count

Tier 1: WebSocket Connection Ceiling per Instance: Single game server instance WebSocket connection count limit; OS-level fd_max or application-level connection accept queue saturation. Recommended evolution: Increase OS fd_max to 512k and application connection accept queue depth; tune SO_REUSEPORT to allow multiple accept threads per socket; add game server instances and update consistent hashing ring; the affinity layer automatically routes new game rooms to the new instances as the ring expands: existing rooms are unaffected .

Redis command throughput > 500k/second; Redis CPU > 60%; per-tick Redis write latency p99 > 5ms (above the acceptable state sync threshold); game tick rate visibly dropping below target (30 ticks/second falling to 20) under load

Tier 2: Redis Game State Write Amplification: Game state serialization to Redis per tick producing more writes than expected; unoptimized state struct serialization writing entire state blob on any field change. Recommended evolution: Implement delta state serialization: only changed fields are written to Redis per tick using HSET with only the modified keys, not full state replacement; profile Redis command distribution per game tick to identify specific state fields with high churn; consider moving ephemeral per-tick state (player positions, projectile states) to local server memory with only durable state (scores, inventory changes) written to Redis .