Post-Mortem Framework · Data Quality: Embedding Drift

Embedding Drift

SEV-2, Significant Impact

Isolated propagation · data quality · Affects 1 scenario(s)

Severity Classification

Classified as CRITICAL based on failure mode severity.

Propagation Chain

Origin component

Embedding Drift begins at the source component. Trigger: Source document content edited or deleted without triggering re-embedding.

Immediate (T+0) · Signal: Alert

Blast Radius

All queries against the affected portion of the vector index return lower-quality results. For RAG systems, this translates to lower-quality LLM responses: which may not surface as errors but as subtly wrong, outdated, or off-topic answers. If the embedding model change is not handled, the entire index is corrupted : every query's results are a mix of old and new vector spaces.

Contributing Factors

Workload: Ai Embedding Lookupoperational

AI embedding lookup workloads are directly vulnerable to embedding drift when source content changes without triggering re-embedding, silently degrading retrieval quality without any error signal.

Trigger Condition: Source document content edited or deleted without triggeringoperational

This operational trigger enables Embedding Drift: Source document content edited or deleted without triggering re-embedding

Trigger Condition: Embedding model upgraded to an incompatible versionoperational

This operational trigger enables Embedding Drift: Embedding model upgraded to an incompatible version

Trigger Condition: Embedding pipeline failure: documents processed after the mooperational

This operational trigger enables Embedding Drift: Embedding pipeline failure: documents processed after the model change were embedded with a different model than documents processed before, creating a mixed-model index

Remediation Plan

ImmediateIdentify scope: how many documents have content_updated_at > embedding_generated

Identify scope: how many documents have content_updated_at > embedding_generated_at?

Effort: Minutes to hours (on-call response)

ImmediateFor model version change: build a new index with the new model; do not query unt

For model version change: build a new index with the new model; do not query until complete

Effort: Minutes to hours (on-call response)

ImmediatePrioritize re-embedding by last-accessed timestamp (most-read documents first)

Prioritize re-embedding by last-accessed timestamp (most-read documents first)

Effort: Minutes to hours (on-call response)

Short-TermTrack embedding freshness metadata

Store embedding_model_version and embedding_generated_at alongside every stored vector. On content update, mark the embedding as stale (embedding_stale=true). A background worker processes the stale queue, re-embeds, and updates the vector. Alert if the stale queue depth exceeds a threshold.

Effort: 1 day to 1 week

Short-TermCDC-triggered re-embedding pipeline

Subscribe to database change events (via CDC or application hooks) for the source content table. When a document is updated, publish an re-embedding task to a queue. The embedding worker processes the queue, regenerates the vector, and upserts into the vector index.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermModel version isolation

When upgrading the embedding model, create a new index for the new model version. Re-embed all documents into the new index. Switch query routing to the new index only after re-embedding is complete. Never mix vectors from different model versions in the same index.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.