Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Feature computation logic duplicated across 3+ model serving'. E-Commerce Order Platform: triggered by 'Payment provider timeout causing full checkout rollback and '.

Migration Step 2

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Regulatory or reproducibility requirement to re-run model pr'. E-Commerce Order Platform: triggered by 'Product search p99 > 1s; faceted navigation (category + pric'.

Migration Step 3

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Semantic similarity search recall insufficient from BM25 ful'. E-Commerce Order Platform: triggered by 'Email/push notification provider timeouts causing checkout l'.

Feature store p99 rising from < 5ms to > 50ms immediately following a model deployment or pod scaling event; Cassandra or PostgreSQL feature store read QPS spiking 10–100x simultaneously with serving fleet restart; feature store connection pool exhaustion visible in application logs during the cold start window; Redis cache hit rate dropping to < 10% for the first 60 seconds after deployment

Tier 1: Cache Cold Start Latency Spike: Simultaneous cache cold start across all serving pods after deployment, converting sequential feature requests into a thundering herd against the backing feature store. Recommended evolution: Implement pre-warming: before a new model version receives traffic, a warm-up job runs inference requests for a representative sample of entity IDs, populating the Redis cache before the pod enters the serving fleet. Use probabilistic early cache refresh (fetch from backing store before TTL expiry when remaining TTL < 20% under high request rate) to prevent simultaneous TTL expiry on hot features. Stagger deployment rollout: deploy 10% of pods, wait for cache warm, then proceed to the next 10%. .

Model performance metrics (precision, recall, AUC) degrading without a corresponding input distribution shift; ClickHouse drift dashboard showing feature value distributions served online diverging from training population distributions; explicit skew audit (comparing offline training feature values against replayed online feature values for the same entity at the same timestamp) showing systematic differences in specific features

Tier 2: Training-Serving Skew Detection: Feature computation logic divergence between the offline training pipeline and the online serving pipeline: a feature transformation applied in training is not applied identically in serving. Recommended evolution: Enforce a single feature computation function registered per feature in a shared feature registry, executed by both the online pipeline and the offline training pipeline. The two paths must use the same code, not independently maintained implementations. Implement a skew monitoring job that computes a random sample of features using both paths and alerts on distribution divergence above a threshold. .

PostgreSQL pg_locks showing high RowExclusiveLock contention on inventory_items for specific sku_ids; checkout p99 > 2s for contended SKUs; deadlock errors appearing in application logs during sale events; effective checkout throughput for hot SKUs well below per-request checkout latency would predict

Tier 1: Flash Sale Inventory Contention: Concurrent saga checkout attempts competing for the same inventory row via row-level locking. Recommended evolution: Introduce a per-SKU checkout serialization queue at the application layer : all concurrent checkout requests for the same SKU are queued and processed serially, converting lock contention into queue latency. Alternatively, use PostgreSQL advisory locks with non-blocking trylock: requests that cannot acquire the lock immediately return a "sold out" response rather than queuing. For very high flash sale volumes, pre-allocate inventory slots (reserve N slots per sale event, each slot is a row with one reservation) to spread lock contention across N rows instead of one. .

Checkout p99 tracking payment provider p99 almost linearly; connection pool utilization on the payment service rising during payment provider slowdowns; circuit breaker trip events appearing in payment service metrics; saga timeout events correlated with payment provider latency spikes

Tier 2: Payment Provider Latency Amplifying Checkout Latency: Checkout saga holding a database connection and an inventory reservation open for the duration of the payment provider call: payment latency directly amplifies connection pool pressure. Recommended evolution: Decouple the payment step from the synchronous checkout saga: reserve inventory and create the order record synchronously, then process payment asynchronously. The customer receives an "order confirmed, payment processing" state immediately; the payment step runs as a separate saga step triggered by an event. This reduces the synchronous checkout latency to the inventory reservation time, not the payment provider round-trip time. .