Use ML Feature Serving Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for ML Feature Serving Platform. Traceable to YAML knowledge entities.
Context
ML feature serving sits at the intersection of two workloads with opposing operational constraints. The online serving path demands sub-5ms p99 latency for multi-feature lookups at high concurrency (model inference calls feature store 50–200 times per prediction request, and predictions are user-facing with strict latency budgets). The offline training path demands point-in-time correctness: the feature values retrieved for any training example must exactly match what the model would have seen at the time the label was generated, otherwise the model trains on features that did not exist at prediction time (training-serving skew). These two paths are often served from the same logical store but require completely different consistency and latency guarantees that cannot be simultaneously optimized on a single storage tier. Primary operational risks include: Training-serving skew from cache-only feature serving: if online inference reads features from Redis and offline training reads from PostgreSQL, any discrepancy in feature computation logic, normalization, or time window definition between the two paths introduces systematic skew that is invisible until model performance degrades. Skew is not an edge case; it is the expected outcome of any system where online and offline feature computation paths are maintained separately.; Feature pipeline version mismatch on model rollout: when a new model version is deployed alongside a new feature pipeline version, there is a window during which the old model reads features computed by the new pipeline (or vice versa). Without feature version tagging on every served value and strict version pinning in the inference client, model inference during the transition window runs on mismatched feature distributions.; Qdrant vector index staleness after embedding model upgrade: when the embedding model is upgraded (e.g., from text-embedding-ada-002 to a newer model), all existing vectors in Qdrant are incompatible with new-model embeddings. Queries mixing old-model embeddings with new-model vectors produce undefined similarity results without error: the system silently returns wrong results until a full re-indexing completes. Full re-indexing of millions of vectors takes hours to days..
Decision
We will adopt the **ML Feature Serving Platform** architecture pattern. This is a expert-complexity architecture appropriate for teams at platform engineering team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A low-latency feature serving platform for ML model inference, providing both batch (offline) and real-time (online) feature access with strict training-serving consistency. Redis serves the hot feature cache with p99 latency targets below 5ms for online inference requests; PostgreSQL provides point-in-time feature lookups for offline training jobs with temporal consistency guarantees; Cassandra stores high-cardinality feature entities at write scale beyond PostgreSQL's single-primary ceiling; Kafka streams feature computation events from online feature pipelines to update the cache; Qdrant stores vector features for embedding-based model inputs and similarity lookups; ClickHouse serves feature analytics and drift monitoring across training dataset populations. Feature versioning is first-class: every feature value is tagged with a pipeline_version and computed_at timestamp to support model reproducibility and training-serving skew diagnosis. Core technology stack: redis, postgresql, kafka, qdrant, cassandra.
Accepted Tradeoffs
- ⚠Redis hot feature cache achieves p99 < 5ms for online inference but requires a cache warming strategy for every model deployment and every fleet scaling event: uncached serving falls back to Cassandra with 20–50ms latency, which violates model serving SLAs for latency-sensitive endpoints
- ⚠Point-in-time feature correctness for offline training requires storing feature values with their computation timestamp and supporting time-travel queries (what was the value of feature X for entity Y at time T): this significantly complicates the feature store schema and doubles storage for every feature update compared to a latest-value-only store
- ⚠Cassandra provides high write throughput and horizontal scale for high-cardinality feature entities (per-user, per-product features at millions of distinct IDs), but Cassandra's eventual consistency model means a feature written by the online pipeline may not be immediately visible to the inference path: the consistency window must be measured and validated against model serving requirements
- ⚠Qdrant vector similarity search provides sub-millisecond approximate nearest-neighbor lookup for embedding-based features, but approximate indexes (HNSW) have a recall tradeoff: index parameters (ef_construction, m) tuned for recall at index build time may be insufficient for recall at query time if the query distribution shifts after deployment
- ⚠ClickHouse feature analytics enables drift detection across training population statistics but requires a separate data flow from the serving path: features served online must be logged to ClickHouse for drift monitoring, adding a write side-effect to every inference request that must not be in the synchronous latency path
Risks
Vector embeddings become semantically stale when source document content changes but the stored embedding is not regenerated: causing semantic search and RAG retrieval to return outdated, incorrect, or misleading results without any error signal, silently degrading the quality of AI-backed features.
When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.
In LSM-tree storage engines (Cassandra, RocksDB, LevelDB), a single logical read may require checking multiple immutable SSTables across multiple levels of compaction before the most recent version of a row is found: multiplying I/O by the number of levels checked and producing latency spikes on reads that cross many levels.
An HNSW or IVF vector index built on a corpus grows increasingly inaccurate as new vectors are inserted and the index is not rebuilt: because HNSW's greedy graph construction assumes a representative distribution during build time, and IVF's cluster centroids become stale: degrading recall for queries about recently added content.
A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; ML Feature Serving Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (clickhouse, kafka) with the chosen architecture but applies different structural patterns; ML Feature Serving Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; ML Feature Serving Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (clickhouse, kafka) with the chosen architecture but applies different structural patterns; ML Feature Serving Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Cache Cold Start Latency Spike
Signal: Feature store p99 rising from < 5ms to > 50ms immediately following a model deployment or pod scaling event; Cassandra or PostgreSQL feature store read QPS spiking 10–100x simultaneously with serving fleet restart; feature store connection pool exhaustion visible in application logs during the cold start window; Redis cache hit rate dropping to < 10% for the first 60 seconds after deployment
Evolution: Implement pre-warming: before a new model version receives traffic, a warm-up job runs inference requests for a representative sample of entity IDs, populating the Redis cache before the pod enters the serving fleet. Use probabilistic early cache refresh (fetch from backing store before TTL expiry when remaining TTL < 20% under high request rate) to prevent simultaneous TTL expiry on hot features. Stagger deployment rollout: deploy 10% of pods, wait for cache warm, then proceed to the next 10%.
Tier 2: Training-Serving Skew Detection
Signal: Model performance metrics (precision, recall, AUC) degrading without a corresponding input distribution shift; ClickHouse drift dashboard showing feature value distributions served online diverging from training population distributions; explicit skew audit (comparing offline training feature values against replayed online feature values for the same entity at the same timestamp) showing systematic differences in specific features
Evolution: Enforce a single feature computation function registered per feature in a shared feature registry, executed by both the online pipeline and the offline training pipeline. The two paths must use the same code, not independently maintained implementations. Implement a skew monitoring job that computes a random sample of features using both paths and alerts on distribution divergence above a threshold.
Tier 3: Qdrant Vector Index Rebuild Under Embedding Model Upgrade
Signal: Embedding model upgrade planned or in progress; Qdrant query recall metrics below threshold for new-model query vectors; similarity search results visibly degraded for new-model queries against old-model index; full re-indexing job running for hours against Qdrant collection
Evolution: Maintain two Qdrant collections per embedding model version (v1 and v2) simultaneously during embedding model transitions. Serve queries from the new-model collection as soon as re-indexing is complete, then deprecate the old collection. This requires that embedding model version is a first-class routing parameter in the feature serving path, not an implicit global. Re-indexing must be triggered and validated before the model serving path is pointed at the new collection.
Tier 4: Feature Store Write Throughput Ceiling for Online Pipeline
Signal: Cassandra feature store write p99 > 20ms under online pipeline update rate; Kafka online feature computation consumer consistently lagging behind the event stream; feature freshness SLA violations (features served are older than the required freshness window); Cassandra compaction backlog growing under sustained high-frequency feature updates
Evolution: Tune Cassandra compaction strategy to TWCS (TimeWindowCompactionStrategy) for time-ordered feature updates: this reduces compaction amplification for append-heavy feature pipelines. Introduce write batching at the feature pipeline layer: group feature updates by partition key and batch-write using logged batches to reduce coordinator round-trips. For the highest-throughput feature types, evaluate ScyllaDB as a drop-in Cassandra replacement with significantly higher write throughput at equivalent hardware.
Migration Path
Features computed inline in the inference service with no shared feature store → Centralized feature store with Redis cache and Cassandra backing store
Feature computation logic duplicated across 3+ model serving services with no coordination; training-serving skew discovered after model deployment causing unexplained model performance regression; feature computation latency dominating inference latency for complex features; inability to enforce feature versioning across model versions
Latest-value-only feature store with no temporal history → Point-in-time feature store with training-time lookup support
Regulatory or reproducibility requirement to re-run model predictions at a historical point in time; training data contamination investigation requiring knowledge of what features were available at training label creation time; model backtesting requiring consistent feature snapshots across historical time ranges
Text similarity search via PostgreSQL full-text tsvector → Qdrant vector similarity search with pre-computed embeddings
Semantic similarity search recall insufficient from BM25 full-text scoring; user query matching requiring understanding of synonyms, paraphrases, or cross-language similarity; feature retrieval for embedding-based ranking models requiring approximate nearest-neighbor search with p99 < 10ms
Operational Requirements
- Minimum team maturity: Platform Engineering Team: This scenario has expert operational complexity. It is recommended for Platform Engineering Team teams or higher.
- Runbooks and alerting for high-severity risks: 3 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.