Blast Radius Analysis · ML Feature Serving Platform

Stale Vector Index

partialContained

data quality failure · isolated propagation

Blast Radius

1/21 nodes

Time to Detect

Without retrieval quality benchmarks, stale index degradation can persist for months undetected. Detection requires periodic evaluation of recall@K against a labeled test set that includes recent documents. A decline in recall from 95% to 85% is not visible in error rate or latency metrics: only in retrieval quality metrics.

Preventive Mitigations

Confidence

Strong

Impacted Components

Immediate

Qdrant

supporting component · Directly connected to failure mode 'Stale Vector Index' via risk propagation path in the topology.

Failure Cascade

Qdrant

Stale Vector Index (isolated propagation) directly affects these components.

Severity at this step: partial

Detection Signals

Alert

Recovery time estimate: Full index rebuild duration: minutes (small corpus, <1M vectors) to hours (large corpus, 100M+ vectors) depending on corpus size and compute available. Quality improvement is immediate upon swap to the rebuilt index.

Mitigation Checklist(1 preventive, 2 reactive)

Use a vector database with incremental indexingpreventshigh

Managed vector databases (Pinecone, Weaviate, Milvus) handle incremental index updates internally, rebuilding or merging index segments without requiring application-level scheduling. Offloads the maintenance problem.

Scheduled full index rebuildmedium

Rebuild the vector index from scratch on a regular schedule (weekly or monthly depending on corpus change rate). Building from the full current corpus produces optimal graph structure and centroid placement. Requires offline rebuild with atomic swap to the new index (similar to index rotation for inverted indexes).

Continuous recall quality monitoringmedium

Maintain a labeled test set of (query, relevant_document) pairs. Run recall@K evaluation against this test set daily. Alert when recall drops below a threshold (e.g., recall@10 < 90%). Detects degradation before users notice.

Affected Systems

Workloads

AI Embedding LookupAnalytics Heavy (OLAP)Read-Heavy API Backend

Technologies

Qdrant

Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.