Use Search-Heavy Content Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Search-Heavy Content Platform. Traceable to YAML knowledge entities.
Context
Content platforms, job boards, e-commerce catalogs, and media libraries require full-text search with faceted navigation that PostgreSQL full-text search cannot serve at scale. Elasticsearch provides the query capabilities: inverted indexes, relevance scoring, aggregation buckets: but adds a second system that must stay consistent with the PostgreSQL source of truth. A stale index serves outdated results; an over-indexed catalog degrades write throughput and index segment count. The indexing pipeline must be decoupled from the write path to prevent search indexing latency from degrading transactional writes. Primary operational risks include: Elasticsearch index segment explosion: high write rates with frequent mapping changes produce too many segments, causing query performance degradation until a force merge completes; Thundering herd on cache miss: a trending content item that expires from Redis simultaneously causes N search requests to bypass cache and hit Elasticsearch directly; Replication lag silently serves stale search results: CDC consumer falling behind makes search results diverge from PostgreSQL source of truth without user-visible indication.
Decision
We will adopt the **Search-Heavy Content Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'intermediate' operational maturity.
Rationale
A content platform architecture centered on Elasticsearch for full-text search, faceted navigation, and ranked results, with PostgreSQL as the transactional source of truth and Redis for session management and hot content caching. WAL-based CDC maintains index freshness by streaming PostgreSQL changes into Elasticsearch asynchronously. The core tension is between search index freshness, query performance, and index maintenance cost under high write volume. The primary architectural strength is: Redis caching absorbs repeated read requests at the edge, reducing database load and latency for high read-to-write ratio workloads by orders of magnitude. Redis caching absorbs repeated read requests at the edge, reducing database load and latency for high read-to-write ratio workloads by orders of magnitude. Key trade-off: Introduces eventual consistency: stale reads possible within TTL window. Operational note: Cache-aside (lazy population) is the dominant integration pattern. Evidence: Cache hit rates of 80–95% observed in production read-heavy APIs. Core technology stack: elasticsearch, postgresql, redis.
Architectural Strengths
- ✓Redis caching absorbs repeated read requests at the edge, reducing database load and latency for high read-to-write ratio workloads by orders of magnitude
- ✓Redis distributed locks (via SET NX EX or Redlock) prevent thundering herd by ensuring only one caller repopulates a cache entry…
- ✓Read-heavy APIs benefit directly from Redis as a caching tier that absorbs repeated identical reads and provides sub-millisecond…
- ✓Search-heavy workloads cache popular queries and their result sets, absorbing the majority of search traffic from cache and…
- ✓CQRS separates the write model (normalized, ACID) from the read model; materialized views implement the read model by…
Accepted Tradeoffs
- ⚠Elasticsearch adds significant operational complexity (cluster management, shard sizing, JVM tuning, snapshot/restore) that is absent in a PostgreSQL-only stack
- ⚠WAL CDC introduces a consistency gap between the write path and the search index: the size of this gap (typically 1–10 seconds) must be communicated to users
- ⚠Redis caching for search results reduces Elasticsearch query load but requires explicit invalidation when indexed content changes: cache poisoning is the failure mode
- ⚠Full reindexing during mapping changes is a multi-hour operation for large catalogs and requires a blue/green index alias strategy to avoid downtime
- ⚠Elasticsearch facet aggregations are expensive and do not cache by default: high-cardinality facets on large indexes cause disproportionate cluster load
Risks
One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.
When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.
Dead tuples from UPDATE and DELETE operations accumulate in PostgreSQL heap pages and index pages when autovacuum cannot reclaim them fast enough, causing table and index storage to grow well beyond the live data size and degrading query performance through wasted I/O on dead pages.
Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (postgresql, redis) with the chosen architecture but applies different structural patterns; Search-Heavy Content Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (postgresql) with the chosen architecture but applies different structural patterns; Search-Heavy Content Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (postgresql, redis) with the chosen architecture but applies different structural patterns; Search-Heavy Content Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (postgresql, redis) with the chosen architecture but applies different structural patterns; Search-Heavy Content Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Index Freshness Degradation
Signal: Elasticsearch index CDC consumer lag > 10s; search results showing items that no longer exist or missing recently published items; CDC connector health dashboard showing processing rate below write rate
Evolution: Increase Elasticsearch bulk indexer thread count; tune bulk index batch size and flush interval; profile CDC connector bottleneck (network vs Elasticsearch write throughput vs mapping complexity)
Tier 2: Search Cluster Heap Pressure
Signal: Elasticsearch JVM heap usage > 75% sustained; GC pause events visible in cluster logs; query p99 latency spikes during GC; cluster health showing yellow (unassigned shards during GC recovery)
Evolution: Increase Elasticsearch heap to 50% of node RAM (max 30GB for ZGC); reduce shard count to keep per-shard document count < 50M; disable dynamic mapping and explicitly define all field types; move to doc values for all non-analyzed fields
Tier 3: Hot Shard Imbalance
Signal: Elasticsearch node stats showing one shard handling > 3x the query/index operations of others; hot-spotted shard's node CPU > 80% while others are idle
Evolution: Enable shard-level routing with custom routing hash; review document routing key selection; for write-heavy scenarios, increase primary shard count and reindex with a new shard allocation
Tier 4: Full Reindex Requirement
Signal: Index mapping change required (new field type, changed analyzer); full catalog reindex estimated > 4 hours; active searches against index during reindex causing performance degradation
Evolution: Implement blue/green index alias strategy before this event occurs; new index is built under an alias while the old index serves traffic; alias is atomically flipped on reindex completion
Migration Path
PostgreSQL full-text search (tsvector) serving all search queries → Elasticsearch for full-text and faceted search, PostgreSQL as source of truth
Search query p99 > 500ms on full-text queries; faceted navigation requires aggregation over more than 5 dimensions simultaneously; relevance ranking quality insufficient for product requirements
Synchronous dual-write (application writes to PostgreSQL then Elasticsearch) → Asynchronous CDC-based indexing pipeline (PostgreSQL → WAL CDC → Kafka → Elasticsearch)
Application write latency increasing due to Elasticsearch indexing latency in the synchronous path; Elasticsearch unavailability causing application write failures
Single Elasticsearch cluster serving all query types → Separate read-optimized and write-optimized Elasticsearch indexes
High write throughput (> 10k documents/min) causing segment refresh to degrade query latency; bulk indexing jobs competing with user search queries for heap
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 2 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.
- Elasticsearch: scenario uses Elasticsearch as a primary datastore: Elasticsearch is a search index, not a source of truth: add a durable primary store and sync to ES