Architecture Review: Search-Heavy Content Platform

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 0 nodes

When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.

Affects 1 node. (Redis). 1 mitigation identified

Seed 'Thundering Herd (Cache Stampede) Risk Probe' identifies 2 metrics relevant to thundering_herd.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: Search query p99 > 500ms on full-text queries; faceted navigation requires aggregation over more than 5 dimensions simultaneously; relevance ranking quality insufficient for product requirements. Migrate from 'PostgreSQL full-text search (tsvector) serving all search queries' to 'Elasticsearch for full-text and faceted search, PostgreSQL as source of truth'. Design the Elasticsearch index mapping and shard count before the first document is indexed. Consult the expected document count at 2-year growth before setting primary shard count. Initial sync from PostgreSQL to Elasticsearch must run before the CDC pipeline takes over incremental updates.

Dual-write period (writing to both PostgreSQL and Elasticsearch) requires careful consistency management; Elasticsearch shard sizing and index design decisions made at creation time are expensive to change later

Trigger: Application write latency increasing due to Elasticsearch indexing latency in the synchronous path; Elasticsearch unavailability causing application write failures. Migrate from 'Synchronous dual-write (application writes to PostgreSQL then Elasticsearch)' to 'Asynchronous CDC-based indexing pipeline (PostgreSQL → WAL CDC → Kafka → Elasticsearch)'. The CDC pipeline decouples search availability from write path availability. Elasticsearch downtime no longer causes write failures. Communicate the indexing lag expectation to product stakeholders before the migration.

CDC pipeline introduces indexing lag: search results will lag writes by 1–30 seconds; CDC setup requires PostgreSQL logical replication, adding WAL overhead

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Evolution from Unified Read/Write Database → CQRS with Separate Read Projections

Migration complexity: high. Rollback: complex.

Signal: Elasticsearch index CDC consumer lag > 10s; search results showing items that no longer exist or missing recently published items; CDC connector health dashboard showing processing rate below write rate

Bottleneck: CDC consumer or Elasticsearch bulk indexer not keeping pace with PostgreSQL write rate. Evolution: Increase Elasticsearch bulk indexer thread count; tune bulk index batch size and flush interval; profile CDC connector bottleneck (network vs Elasticsearch write throughput vs mapping complexity)

Signal: Elasticsearch JVM heap usage > 75% sustained; GC pause events visible in cluster logs; query p99 latency spikes during GC; cluster health showing yellow (unassigned shards during GC recovery)

Bottleneck: Large aggregation queries or high document count per shard exceeding JVM heap budget. Evolution: Increase Elasticsearch heap to 50% of node RAM (max 30GB for ZGC); reduce shard count to keep per-shard document count < 50M; disable dynamic mapping and explicitly define all field types; move to doc values for all non-analyzed fields