Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Feature computation logic duplicated across 3+ model serving'. Gaming Backend Platform: triggered by 'Single game server instance running at connection ceiling; h'.

Migration Step 2

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Regulatory or reproducibility requirement to re-run model pr'. Gaming Backend Platform: triggered by 'Game server instances blocking on PostgreSQL writes at end-o'.

Migration Step 3

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Semantic similarity search recall insufficient from BM25 ful'. Gaming Backend Platform: triggered by 'PostgreSQL event log table approaching 100B rows; reconnect '.

Feature store p99 rising from < 5ms to > 50ms immediately following a model deployment or pod scaling event; Cassandra or PostgreSQL feature store read QPS spiking 10–100x simultaneously with serving fleet restart; feature store connection pool exhaustion visible in application logs during the cold start window; Redis cache hit rate dropping to < 10% for the first 60 seconds after deployment

Tier 1: Cache Cold Start Latency Spike: Simultaneous cache cold start across all serving pods after deployment, converting sequential feature requests into a thundering herd against the backing feature store. Recommended evolution: Implement pre-warming: before a new model version receives traffic, a warm-up job runs inference requests for a representative sample of entity IDs, populating the Redis cache before the pod enters the serving fleet. Use probabilistic early cache refresh (fetch from backing store before TTL expiry when remaining TTL < 20% under high request rate) to prevent simultaneous TTL expiry on hot features. Stagger deployment rollout: deploy 10% of pods, wait for cache warm, then proceed to the next 10%. .

Model performance metrics (precision, recall, AUC) degrading without a corresponding input distribution shift; ClickHouse drift dashboard showing feature value distributions served online diverging from training population distributions; explicit skew audit (comparing offline training feature values against replayed online feature values for the same entity at the same timestamp) showing systematic differences in specific features

Tier 2: Training-Serving Skew Detection: Feature computation logic divergence between the offline training pipeline and the online serving pipeline: a feature transformation applied in training is not applied identically in serving. Recommended evolution: Enforce a single feature computation function registered per feature in a shared feature registry, executed by both the online pipeline and the offline training pipeline. The two paths must use the same code, not independently maintained implementations. Implement a skew monitoring job that computes a random sample of features using both paths and alerts on distribution divergence above a threshold. .

Game server instance file descriptor count approaching OS limit (typically 65k open connections); WebSocket accept latency increasing; new connection establishment p99 > 200ms; CPU on game server instances > 70% during peak concurrent player count

Tier 1: WebSocket Connection Ceiling per Instance: Single game server instance WebSocket connection count limit; OS-level fd_max or application-level connection accept queue saturation. Recommended evolution: Increase OS fd_max to 512k and application connection accept queue depth; tune SO_REUSEPORT to allow multiple accept threads per socket; add game server instances and update consistent hashing ring; the affinity layer automatically routes new game rooms to the new instances as the ring expands: existing rooms are unaffected .

Redis command throughput > 500k/second; Redis CPU > 60%; per-tick Redis write latency p99 > 5ms (above the acceptable state sync threshold); game tick rate visibly dropping below target (30 ticks/second falling to 20) under load

Tier 2: Redis Game State Write Amplification: Game state serialization to Redis per tick producing more writes than expected; unoptimized state struct serialization writing entire state blob on any field change. Recommended evolution: Implement delta state serialization: only changed fields are written to Redis per tick using HSET with only the modified keys, not full state replacement; profile Redis command distribution per game tick to identify specific state fields with high churn; consider moving ephemeral per-tick state (player positions, projectile states) to local server memory with only durable state (scores, inventory changes) written to Redis .