Architecture Review: AI Retrieval-Augmented Generation Platform

When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.

Affects 1 node. (Redis)

When total memory demand from a process or the entire host exceeds available physical RAM plus swap, the Linux OOM killer terminates one or more processes to reclaim memory, causing immediate connection loss, data corruption risk if in-flight writes are lost, and process restart overhead.

Affects 1 node. (AI Embedding Lookup)

Seed 'Thundering Herd (Cache Stampede) Risk Probe' identifies 2 metrics relevant to thundering_herd.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: LLM responses requiring more factual accuracy or domain-specific context; context window limitations requiring selective document retrieval; user queries returning hallucinated answers that could be grounded with retrieval. Migrate from 'LLM application with no retrieval augmentation (prompt-only context)' to 'PostgreSQL + pgvector for semantic retrieval with manual embedding generation'. Start with synchronous embedding generation on write and exact ANN search. Introduce asynchronous pipeline and approximate indexes once baseline retrieval quality and query volume are understood.

Embedding model selection is a significant decision: dimension size affects index size, query latency, and migration cost if the model is changed later; Naive cosine similarity without metadata filtering returns semantically related but contextually wrong results (e.g., wrong tenant, wrong time range)

Trigger: Document ingestion p99 > 500ms due to embedding API call in write path; embedding API rate limits blocking document writes during traffic spikes. Migrate from 'Synchronous embedding generation on write path' to 'Asynchronous embedding pipeline via Kafka consumer'. Decouple embedding generation from document writes before the write path latency becomes user-visible. The Kafka-backed pipeline also provides natural rate limiting against embedding API quotas.

Asynchronous pipeline introduces a retrieval lag window: must be communicated in product UX (e.g., "indexing in progress"); Consumer failure requires replay from Kafka offset: embedding API idempotency must be verified

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Retrieval quality metrics (MRR, NDCG) declining despite stable query volume; users reporting irrelevant context being surfaced; pgvector IVFFlat probes set below recommended value for current document count

Bottleneck: IVFFlat index not rebuilt after significant document additions; or probes too low for current index size. Evolution: Schedule periodic index rebuilds triggered by document count growth (e.g., rebuild at 2x the document count present at last index build); increase ivfflat.probes to improve recall at cost of query latency; evaluate HNSW for recall-critical workloads

Signal: PostgreSQL process memory > 8GB; OOM killer events on the database host; vector query p99 latency increasing as shared_buffers evicts vector index pages; pg_stat_bgwriter showing high buffers_clean rate

Bottleneck: Vector index (HNSW or large IVFFlat) and embedding storage competing with relational data for shared_buffers. Evolution: Increase PostgreSQL shared_buffers to 40% of available RAM; move vector tables to a dedicated tablespace on NVMe; partition large vector tables by document category to reduce per-query index scan range; evaluate dedicated pgvector replica for query isolation