Post-Mortem Framework · Data Quality: Stale Vector Index

Stale Vector Index

SEV-3, Limited Impact

Isolated propagation · data quality · Affects 1 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity.

Propagation Chain

Origin component

Stale Vector Index begins at the source component. Trigger: Continuous vector insertion without index rebuilds over weeks or months.

Immediate (T+0) · Signal: Alert

Blast Radius

Degraded recall for queries about recently added or underrepresented content. Queries about well-represented original content continue to perform well. The degradation is proportional to the fraction of the corpus that is "new" relative to the build-time distribution.

Contributing Factors

Trigger Condition: Continuous vector insertion without index rebuilds over weekoperational

This operational trigger enables Stale Vector Index: Continuous vector insertion without index rebuilds over weeks or months

Trigger Condition: Addition of content from a substantially new domain not reproperational

This operational trigger enables Stale Vector Index: Addition of content from a substantially new domain not represented in the original corpus

Trigger Condition: Corpus composition changes significantly (e.g., 50% of old voperational

This operational trigger enables Stale Vector Index: Corpus composition changes significantly (e.g., 50% of old vectors deleted, new categories added)

Remediation Plan

ImmediateMeasure current recall@K using the labeled test set to quantify degradation

Measure current recall@K using the labeled test set to quantify degradation

Effort: Minutes to hours (on-call response)

ImmediateTrigger a full index rebuild during off-peak hours

Trigger a full index rebuild during off-peak hours

Effort: Minutes to hours (on-call response)

ImmediateAfter rebuild, re-measure recall@K to confirm improvement

After rebuild, re-measure recall@K to confirm improvement

Effort: Minutes to hours (on-call response)

Short-TermScheduled full index rebuild

Rebuild the vector index from scratch on a regular schedule (weekly or monthly depending on corpus change rate). Building from the full current corpus produces optimal graph structure and centroid placement. Requires offline rebuild with atomic swap to the new index (similar to index rotation for inverted indexes).

Effort: 1 day to 1 week

Short-TermContinuous recall quality monitoring

Maintain a labeled test set of (query, relevant_document) pairs. Run recall@K evaluation against this test set daily. Alert when recall drops below a threshold (e.g., recall@10 < 90%). Detects degradation before users notice.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermUse a vector database with incremental indexing

Managed vector databases (Pinecone, Weaviate, Milvus) handle incremental index updates internally, rebuilding or merging index segments without requiring application-level scheduling. Offloads the maintenance problem.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.