Architecture Review: Content Management Platform

When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.

Affects 1 node. (Redis)

When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.

Affects 1 node. (Read-Heavy API Backend)

Seed 'Thundering Herd (Cache Stampede) Risk Probe' identifies 2 metrics relevant to thundering_herd.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: Content read API p99 > 100ms during peak traffic; PostgreSQL read IOPS exceeding 80% of provisioned capacity; read replica falling behind on heavy read workloads. Migrate from 'PostgreSQL primary serving all content reads directly (no caching)' to 'Redis cache-aside for published content with explicit publish invalidation'. Warm the cache on publish by writing the new content value into Redis as part of the publish transaction, not by deleting the key and waiting for the first read to repopulate. This eliminates the miss window entirely.

Cache invalidation logic must be tested against every content state transition in the editorial workflow: a missed invalidation on schedule-to-published transition will serve stale content indefinitely until the TTL expires; Cache population must be synchronous with the publish operation from the editorial team's perspective: if the first reader after publish always gets a cache miss and slow database response, editors will report that "the site is slow after publish"

Trigger: Full-text search p99 > 500ms; inability to implement faceted filtering (filter by category, date range, author simultaneously) with acceptable PostgreSQL query performance; relevance ranking quality insufficient for user-facing search. Migrate from 'PostgreSQL full-text search (tsvector, GIN index)' to 'Elasticsearch with incremental indexing via CDC or outbox'. Use the outbox pattern or PostgreSQL LISTEN/NOTIFY to trigger incremental Elasticsearch indexing on content change, rather than polling. This reduces indexing lag from polling interval to near-real-time and avoids the database overhead of constant polling queries.

Elasticsearch indexing lag creates a window where newly published content is not yet searchable; this window must be communicated to editorial teams as a search latency SLA (e.g., "new content appears in search within 60 seconds"); Elasticsearch schema mapping changes require a full re-index: plan schema evolution strategy before adding facet fields that may need to change

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Evolution from Unified Read/Write Database → CQRS with Separate Read Projections

Migration complexity: high. Rollback: complex.

Signal: PostgreSQL pg_stat_statements showing > 10 distinct query patterns with high call counts from the content list API path; database queries per second growing linearly with API request rate for content list endpoints (should be sub-linear with proper batch fetching); p99 for content list API > 200ms during moderate traffic

Bottleneck: ORM-level N+1 patterns in content relationship traversal: author fetch, category fetch, related content fetch as independent queries per article. Evolution: Audit every content API response with query logging enabled and count queries per request for each content type; implement eager loading for all included relationships (JOIN for 1:1, IN-clause batch for 1:N); validate that content list endpoints produce a fixed number of queries regardless of list size (O(1) queries, not O(N)); add a query count assertion to integration tests for content list endpoints to prevent regression

Signal: PostgreSQL read replica CPU spike correlated exactly with publish events; Redis cache hit rate dropping to near 0% immediately after publish for popular content; content API p99 spiking from < 20ms to > 500ms during the 1–3 second window after a high-traffic content item is published

Bottleneck: Cache key deletion on publish triggering simultaneous cache miss stampede for all concurrent readers of popular content. Evolution: Implement cache-aside with probabilistic early expiration (PER): before the cache TTL expires, a fraction of reads proactively refresh the cache value while other reads continue serving the cached value; this eliminates the hard expiry boundary that causes simultaneous misses; alternatively, on publish, write the new content value directly into the cache key before invalidating the old one (update-in-place rather than delete-and-miss) to eliminate the invalidation gap