Architecture Review: ML Feature Serving Platform

Vector embeddings become semantically stale when source document content changes but the stored embedding is not regenerated: causing semantic search and RAG retrieval to return outdated, incorrect, or misleading results without any error signal, silently degrading the quality of AI-backed features.

Affects 3 nodes. (AI Embedding Lookup, Qdrant, Vector Similarity Search)

When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.

Affects 1 node. (Read-Heavy API Backend)

Seed 'Embedding Drift Risk Probe' identifies 2 metrics relevant to embedding_drift.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: Feature computation logic duplicated across 3+ model serving services with no coordination; training-serving skew discovered after model deployment causing unexplained model performance regression; feature computation latency dominating inference latency for complex features; inability to enforce feature versioning across model versions. Migrate from 'Features computed inline in the inference service with no shared feature store' to 'Centralized feature store with Redis cache and Cassandra backing store'. Identify the 10 most reused features across models and migrate those first. Validate that feature values from the central store match inline-computed values before migrating any model serving path to use the store.

Initial feature store population requires backfilling historical feature values for all entity IDs: backfill jobs must be idempotent and resumable to handle interruptions; Migrating from inline computation to feature store lookups requires model serving code changes across all inference services simultaneously or a dual-lookup period

Trigger: Regulatory or reproducibility requirement to re-run model predictions at a historical point in time; training data contamination investigation requiring knowledge of what features were available at training label creation time; model backtesting requiring consistent feature snapshots across historical time ranges. Migrate from 'Latest-value-only feature store with no temporal history' to 'Point-in-time feature store with training-time lookup support'. Add computed_at and valid_until columns to the feature store schema first. Start writing temporal feature records in parallel with the existing latest-value records. Migrate training jobs one at a time to use point-in-time lookups, validating that model performance does not change from the temporal semantics change.

Point-in-time storage requires retaining every feature update with its timestamp rather than overwriting the latest value: storage volume increases by the number of feature updates per entity per time window; Existing training pipelines that read "current" feature values must be updated to read "feature value as-of label timestamp": this changes training job semantics and requires validation

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Feature store p99 rising from < 5ms to > 50ms immediately following a model deployment or pod scaling event; Cassandra or PostgreSQL feature store read QPS spiking 10–100x simultaneously with serving fleet restart; feature store connection pool exhaustion visible in application logs during the cold start window; Redis cache hit rate dropping to < 10% for the first 60 seconds after deployment

Bottleneck: Simultaneous cache cold start across all serving pods after deployment, converting sequential feature requests into a thundering herd against the backing feature store. Evolution: Implement pre-warming: before a new model version receives traffic, a warm-up job runs inference requests for a representative sample of entity IDs, populating the Redis cache before the pod enters the serving fleet. Use probabilistic early cache refresh (fetch from backing store before TTL expiry when remaining TTL < 20% under high request rate) to prevent simultaneous TTL expiry on hot features. Stagger deployment rollout: deploy 10% of pods, wait for cache warm, then proceed to the next 10%.

Signal: Model performance metrics (precision, recall, AUC) degrading without a corresponding input distribution shift; ClickHouse drift dashboard showing feature value distributions served online diverging from training population distributions; explicit skew audit (comparing offline training feature values against replayed online feature values for the same entity at the same timestamp) showing systematic differences in specific features

Bottleneck: Feature computation logic divergence between the offline training pipeline and the online serving pipeline: a feature transformation applied in training is not applied identically in serving. Evolution: Enforce a single feature computation function registered per feature in a shared feature registry, executed by both the online pipeline and the offline training pipeline. The two paths must use the same code, not independently maintained implementations. Implement a skew monitoring job that computes a random sample of features using both paths and alerts on distribution divergence above a threshold.