Architecture Review: Search-Heavy Content Platform

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 0 nodes

When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.

Affects 2 nodes. (Read-Heavy API Backend, Redis). 1 mitigation identified

Seed 'Cache Stampede (Dog-Pile) Risk Probe' identifies 2 metrics relevant to cache_stampede.

Metrics to instrument: error_rate, p95_latency_ms

'Search-Heavy Content Platform' is flagged vulnerable to cache stampede, but no carried pattern documents a mutex lock, probabilistic early expiry, or TTL jitter mechanism. This does not mean the mechanism is absent in a real deployment, only that no such mitigation is recorded in this scenario's structured knowledge yet.

Without one of these mechanisms, a hot-key expiry can produce a database load spike; see cache_stampede for the full mitigation list.

Trigger: Search query p99 > 500ms on full-text queries; faceted navigation requires aggregation over more than 5 dimensions simultaneously; relevance ranking quality insufficient for product requirements. Migrate from 'PostgreSQL full-text search (tsvector) serving all search queries' to 'Elasticsearch for full-text and faceted search, PostgreSQL as source of truth'. Design the Elasticsearch index mapping and shard count before the first document is indexed. Consult the expected document count at 2-year growth before setting primary shard count. Initial sync from PostgreSQL to Elasticsearch must run before the CDC pipeline takes over incremental updates.

Dual-write period (writing to both PostgreSQL and Elasticsearch) requires careful consistency management; Elasticsearch shard sizing and index design decisions made at creation time are expensive to change later

Trigger: Application write latency increasing due to Elasticsearch indexing latency in the synchronous path; Elasticsearch unavailability causing application write failures. Migrate from 'Synchronous dual-write (application writes to PostgreSQL then Elasticsearch)' to 'Asynchronous CDC-based indexing pipeline (PostgreSQL → WAL CDC → Kafka → Elasticsearch)'. The CDC pipeline decouples search availability from write path availability. Elasticsearch downtime no longer causes write failures. Communicate the indexing lag expectation to product stakeholders before the migration.

CDC pipeline introduces indexing lag: search results will lag writes by 1–30 seconds; CDC setup requires PostgreSQL logical replication, adding WAL overhead

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Evolution from Unified Read/Write Database → CQRS with Separate Read Projections

Migration complexity: high. Rollback: complex.

'Search-Heavy Content Platform' includes a cache in its topology. If the cache becomes unavailable, the primary database receives the cache's full request load until the cache recovers.

Capacity-plan the primary database for this fallback load, not only for the steady-state cached load.

Signal: Elasticsearch index CDC consumer lag > 10s; search results showing items that no longer exist or missing recently published items; CDC connector health dashboard showing processing rate below write rate

Bottleneck: CDC consumer or Elasticsearch bulk indexer not keeping pace with PostgreSQL write rate. Evolution: Increase Elasticsearch bulk indexer thread count; tune bulk index batch size and flush interval; profile CDC connector bottleneck (network vs Elasticsearch write throughput vs mapping complexity)