Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Two-Sided Marketplace Platform: triggered by 'Checkout failures from payment provider unavailability causi'. ML Feature Serving Platform: triggered by 'Feature computation logic duplicated across 3+ model serving'.

Migration Step 2

Both scenarios define a migration step at this stage. Two-Sided Marketplace Platform: triggered by 'Listing search p99 > 1s; faceted navigation (category + pric'. ML Feature Serving Platform: triggered by 'Regulatory or reproducibility requirement to re-run model pr'.

Migration Step 3

Both scenarios define a migration step at this stage. Two-Sided Marketplace Platform: triggered by 'Domain teams stepping on each other's schema migrations; dat'. ML Feature Serving Platform: triggered by 'Semantic similarity search recall insufficient from BM25 ful'.

Redis cache miss spike visible in monitoring; PostgreSQL query rate spiking for listing reads despite stable write volume; p99 listing API latency > 500ms during traffic spike events

Tier 1: Viral Listing Thundering Herd: Cache TTL expiry on hot listings during peak traffic: all concurrent requests bypass cache simultaneously. Recommended evolution: Implement staggered TTL jitter on listing cache entries; use probabilistic early refresh (refresh before TTL expiry when remaining TTL < 20% and request rate is high); implement single-flight/request coalescing at the application layer to collapse concurrent cache misses into a single database read .

Saga compensation events appearing in order event log; checkout p99 > 2s; pg_locks showing contended rows on inventory_reservations table; idempotency key conflicts increasing in payment service logs

Tier 2: Checkout Saga Contention: Concurrent checkout transactions competing for the same inventory rows; saga timeout thresholds too aggressive. Recommended evolution: Increase inventory reservation table partition count; tune saga step timeout to 2x the observed p99 for each step under load; implement a per-listing checkout serialization queue to prevent N concurrent sagas competing for the same inventory .

Feature store p99 rising from < 5ms to > 50ms immediately following a model deployment or pod scaling event; Cassandra or PostgreSQL feature store read QPS spiking 10–100x simultaneously with serving fleet restart; feature store connection pool exhaustion visible in application logs during the cold start window; Redis cache hit rate dropping to < 10% for the first 60 seconds after deployment

Tier 1: Cache Cold Start Latency Spike: Simultaneous cache cold start across all serving pods after deployment, converting sequential feature requests into a thundering herd against the backing feature store. Recommended evolution: Implement pre-warming: before a new model version receives traffic, a warm-up job runs inference requests for a representative sample of entity IDs, populating the Redis cache before the pod enters the serving fleet. Use probabilistic early cache refresh (fetch from backing store before TTL expiry when remaining TTL < 20% under high request rate) to prevent simultaneous TTL expiry on hot features. Stagger deployment rollout: deploy 10% of pods, wait for cache warm, then proceed to the next 10%. .

Model performance metrics (precision, recall, AUC) degrading without a corresponding input distribution shift; ClickHouse drift dashboard showing feature value distributions served online diverging from training population distributions; explicit skew audit (comparing offline training feature values against replayed online feature values for the same entity at the same timestamp) showing systematic differences in specific features

Tier 2: Training-Serving Skew Detection: Feature computation logic divergence between the offline training pipeline and the online serving pipeline: a feature transformation applied in training is not applied identically in serving. Recommended evolution: Enforce a single feature computation function registered per feature in a shared feature registry, executed by both the online pipeline and the offline training pipeline. The two paths must use the same code, not independently maintained implementations. Implement a skew monitoring job that computes a random sample of features using both paths and alerts on distribution divergence above a threshold. .