Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Feature computation logic duplicated across 3+ model serving'. Healthcare Records Platform: triggered by 'HIPAA audit requirement exposed during external security rev'.

Migration Step 2

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Regulatory or reproducibility requirement to re-run model pr'. Healthcare Records Platform: triggered by 'FHIR events being published to Kafka but corresponding clini'.

Migration Step 3

Both scenarios define a migration step at this stage. ML Feature Serving Platform: triggered by 'Semantic similarity search recall insufficient from BM25 ful'. Healthcare Records Platform: triggered by 'Facility acquisition or merger; compliance requirement for d'.

Feature store p99 rising from < 5ms to > 50ms immediately following a model deployment or pod scaling event; Cassandra or PostgreSQL feature store read QPS spiking 10–100x simultaneously with serving fleet restart; feature store connection pool exhaustion visible in application logs during the cold start window; Redis cache hit rate dropping to < 10% for the first 60 seconds after deployment

Tier 1: Cache Cold Start Latency Spike: Simultaneous cache cold start across all serving pods after deployment, converting sequential feature requests into a thundering herd against the backing feature store. Recommended evolution: Implement pre-warming: before a new model version receives traffic, a warm-up job runs inference requests for a representative sample of entity IDs, populating the Redis cache before the pod enters the serving fleet. Use probabilistic early cache refresh (fetch from backing store before TTL expiry when remaining TTL < 20% under high request rate) to prevent simultaneous TTL expiry on hot features. Stagger deployment rollout: deploy 10% of pods, wait for cache warm, then proceed to the next 10%. .

Model performance metrics (precision, recall, AUC) degrading without a corresponding input distribution shift; ClickHouse drift dashboard showing feature value distributions served online diverging from training population distributions; explicit skew audit (comparing offline training feature values against replayed online feature values for the same entity at the same timestamp) showing systematic differences in specific features

Tier 2: Training-Serving Skew Detection: Feature computation logic divergence between the offline training pipeline and the online serving pipeline: a feature transformation applied in training is not applied identically in serving. Recommended evolution: Enforce a single feature computation function registered per feature in a shared feature registry, executed by both the online pipeline and the offline training pipeline. The two paths must use the same code, not independently maintained implementations. Implement a skew monitoring job that computes a random sample of features using both paths and alerts on distribution divergence above a threshold. .

Audit log table growing at > 500K rows/day; INSERT p99 on audit_log > 20ms; autovacuum unable to keep up with dead tuple accumulation from UPDATE operations on the audit log's index pages

Tier 1: Audit Log Write Throughput: Audit log receiving one row per record access creates I/O contention with clinical record writes on the same PostgreSQL primary. Recommended evolution: Partition the audit_log table by month using PostgreSQL declarative partitioning; child partitions allow VACUUM to operate on bounded table segments without scanning the entire history; index each partition independently to keep index size proportional to partition row count rather than total log size .

pg_locks showing RowExclusiveLock waits on clinical_records or encounter_notes during shift-change peak hours; write p99 > 100ms; occasional deadlock errors in application logs correlated with concurrent addenda writes to the same encounter

Tier 2: Concurrent Encounter Write Lock Contention: Multiple clinical staff members writing addenda to the same encounter simultaneously, or two processes updating encounter status concurrently. Recommended evolution: Implement optimistic locking with an encounter version column; reject concurrent writes with a conflict error and require the client to reload and retry; this eliminates lock waits by failing fast rather than waiting; ensure the application presents a clear conflict resolution UI: in a clinical context, silent overwrites of concurrent edits are a patient safety risk, not just a data integrity issue .