Use AI Retrieval-Augmented Generation Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for AI Retrieval-Augmented Generation Platform. Traceable to YAML knowledge entities.
Context
LLM-based applications require retrieval of semantically relevant context before generation. Storing embeddings in a dedicated vector database while relational metadata lives in a separate OLTP store creates synchronization complexity and cross-store query latency. Using PostgreSQL with pgvector consolidates both concerns in one operational unit while supporting hybrid queries (vector similarity + SQL predicate filtering). The embedding generation pipeline must be asynchronous to prevent document ingestion latency from blocking the write path. Primary operational risks include: IVFFlat index staleness: pgvector IVFFlat indexes are not updated incrementally; the index must be rebuilt after significant document additions, or recall degrades silently; Memory pressure from HNSW index load: HNSW indexes are loaded into shared memory on first query; large vector dimensions (1536+) with millions of vectors can exhaust PostgreSQL shared_buffers; Embedding pipeline slow consumer: Kafka consumer processing embedding generation API calls falls behind document ingestion rate, causing newly added documents to be unsearchable for extended periods.
Decision
We will adopt the **AI Retrieval-Augmented Generation Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A Retrieval-Augmented Generation (RAG) architecture that combines vector similarity search for semantic document retrieval with relational metadata filtering, using PostgreSQL with pgvector as the unified store for both embeddings and structured data. Redis provides a semantic cache to avoid redundant embedding model inference and reduce vector index query load for repeated or similar queries. Kafka manages the asynchronous embedding generation pipeline that keeps the vector index current as source documents are added or updated. Core technology stack: postgresql, redis, kafka.
Accepted Tradeoffs
- ⚠pgvector consolidates embeddings and metadata in one database but trades raw ANN query throughput against dedicated vector databases (Pinecone, Weaviate) for very large indexes (> 10M vectors)
- ⚠IVFFlat provides faster queries than exact search but requires probes tuning to balance recall vs. latency; wrong probes configuration silently degrades retrieval quality without errors
- ⚠HNSW provides better recall than IVFFlat at query time but requires full index rebuild on any parameter change and consumes significantly more memory
- ⚠Semantic caching reduces embedding API costs and retrieval latency but introduces a potential quality regression if cache hit threshold (cosine similarity) is tuned too liberally
- ⚠Asynchronous embedding pipeline improves write throughput but means newly ingested documents are unavailable for retrieval until the pipeline processes them: RAG freshness has a floor latency
Risks
When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.
When total memory demand from a process or the entire host exceeds available physical RAM plus swap, the Linux OOM killer terminates one or more processes to reclaim memory, causing immediate connection loss, data corruption risk if in-flight writes are lost, and process restart overhead.
A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.
Dead tuples from UPDATE and DELETE operations accumulate in PostgreSQL heap pages and index pages when autovacuum cannot reclaim them fast enough, causing table and index storage to grow well beyond the live data size and degrading query performance through wasted I/O on dead pages.
Alternatives Considered
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; AI Retrieval-Augmented Generation Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; AI Retrieval-Augmented Generation Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; AI Retrieval-Augmented Generation Platform is a better fit for the identified workload profile.
Content Management Platform shares core technology (postgresql, redis) with the chosen architecture but applies different structural patterns; AI Retrieval-Augmented Generation Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Vector Index Recall Degradation
Signal: Retrieval quality metrics (MRR, NDCG) declining despite stable query volume; users reporting irrelevant context being surfaced; pgvector IVFFlat probes set below recommended value for current document count
Evolution: Schedule periodic index rebuilds triggered by document count growth (e.g., rebuild at 2x the document count present at last index build); increase ivfflat.probes to improve recall at cost of query latency; evaluate HNSW for recall-critical workloads
Tier 2: PostgreSQL Memory Pressure from Vector Operations
Signal: PostgreSQL process memory > 8GB; OOM killer events on the database host; vector query p99 latency increasing as shared_buffers evicts vector index pages; pg_stat_bgwriter showing high buffers_clean rate
Evolution: Increase PostgreSQL shared_buffers to 40% of available RAM; move vector tables to a dedicated tablespace on NVMe; partition large vector tables by document category to reduce per-query index scan range; evaluate dedicated pgvector replica for query isolation
Tier 3: Embedding Pipeline Backlog
Signal: Kafka consumer group lag growing for the embedding generation consumer; document ingestion reporting "indexing pending" status for > 5 minutes; embedding API rate limit errors in consumer logs
Evolution: Increase embedding consumer parallelism (capped at Kafka partition count); batch documents per embedding API call to improve inference efficiency; implement priority queuing to index recent documents ahead of backlog
Tier 4: Scale Ceiling for pgvector
Signal: pgvector ANN query p99 > 100ms at > 20M vectors with HNSW; PostgreSQL unable to serve concurrent relational and vector queries without I/O contention; index rebuild duration > 4 hours
Evolution: Evaluate dedicated vector database (Qdrant, Weaviate) for the vector search path while retaining PostgreSQL for relational metadata; implement a hybrid query layer that fetches candidate IDs from the vector store and hydrates with PostgreSQL metadata
Migration Path
LLM application with no retrieval augmentation (prompt-only context) → PostgreSQL + pgvector for semantic retrieval with manual embedding generation
LLM responses requiring more factual accuracy or domain-specific context; context window limitations requiring selective document retrieval; user queries returning hallucinated answers that could be grounded with retrieval
Synchronous embedding generation on write path → Asynchronous embedding pipeline via Kafka consumer
Document ingestion p99 > 500ms due to embedding API call in write path; embedding API rate limits blocking document writes during traffic spikes
Single pgvector index serving all document types → Partitioned vector indexes per document namespace or tenant
Index scan range too large for per-query latency targets; tenant isolation requirements demand separate vector spaces; different document types requiring different embedding models or dimensions
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 2 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.