Embedding Drift
criticalContaineddata quality failure · isolated propagation
Blast Radius
14%
3/21 nodes
Time to Detect
Without active retrieval quality monitoring, embedding drift can persist for weeks or months undetected. Detection requires: (1) tracking embedding_generated_at timestamp per document and alerting when content_updated_at > embedding_generated_at, or (2) continuous evaluation of retrieval quality against a labeled benchmark set.
Preventive Mitigations
2
Confidence
StrongImpacted Components
Vector Similarity Search
architecture pattern · Directly connected to failure mode 'Embedding Drift' via risk propagation path in the topology.
Qdrant
supporting component · Directly connected to failure mode 'Embedding Drift' via risk propagation path in the topology.
AI Embedding Lookup
workload · Directly connected to failure mode 'Embedding Drift' via risk propagation path in the topology.
Failure Cascade
AI Embedding Lookup, Qdrant, Vector Similarity Search
Embedding Drift (isolated propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: Re-embedding duration depends on collection size and model API throughput limits. At OpenAI's default rate limits, re-embedding 100,000 documents takes 2–4 hours depending on document length. Self-hosted models on GPU can process faster. A complete model migration for millions of documents may take days.
Mitigation Checklist(2 preventive, 1 reactive)
When upgrading the embedding model, create a new index for the new model version. Re-embed all documents into the new index. Switch query routing to the new index only after re-embedding is complete. Never mix vectors from different model versions in the same index.
Subscribe to database change events (via CDC or application hooks) for the source content table. When a document is updated, publish an re-embedding task to a queue. The embedding worker processes the queue, regenerates the vector, and upserts into the vector index.
Store embedding_model_version and embedding_generated_at alongside every stored vector. On content update, mark the embedding as stale (embedding_stale=true). A background worker processes the stale queue, re-embeds, and updates the vector. Alert if the stale queue depth exceeds a threshold.
Affected Systems
Workloads
Technologies
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.