Architecture Review: Social Feed Platform

When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.

Affects 1 node. (Redis)

Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.

Affects 1 node. (Event Streaming)

Seed 'Fanout Amplification Risk Probe' identifies 2 metrics relevant to fanout_amplification.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: PostgreSQL timeline read query p99 > 500ms; query plan for "SELECT posts WHERE user_id IN (following_list) ORDER BY created_at DESC LIMIT 50" showing sequential scan or index merge; feed read traffic accounting for > 40% of database CPU. Migrate from 'Monolithic feed built on PostgreSQL timeline queries' to 'Redis pre-materialized feed with Kafka async fan-out workers'. Introduce the Kafka fan-out worker and Redis feed writer in shadow mode for 2–4 weeks before cutting feed reads from PostgreSQL to Redis. Use a 5% traffic canary to validate feed correctness (compare PostgreSQL and Redis results) before full cutover. The outbox pattern on post writes must be in place before Kafka fan-out workers are added.

Redis feed hydration requires a backfill job to populate feed lists for all existing users from historical post data; backfill must be idempotent and rate-limited to avoid Redis write saturation; During the migration period, double-writes (to both PostgreSQL timeline query path and Redis fan-out path) are required; consistency between the two paths must be validated before cutting over read traffic

Trigger: Fan-out worker queue lag during celebrity post events > 5 minutes; Redis write throughput spiking to > 80% of capacity during these events; fan-out cost for top 100 accounts measured at > 1000x median fan-out cost. Migrate from 'Uniform fan-out-on-write for all accounts' to 'Hybrid fan-out model (fan-out-on-write for <10k followers, fan-out-on-read for high-follower accounts)'. Implement fan-out-on-read as a supplementary merge layer on top of the existing fan-out-on-write path, not a replacement. Feed reads for users following celebrity accounts fetch the celebrity's recent posts directly from PostgreSQL (via Redis cache) and merge them into the pre-materialized feed list at read time.

Fan-out-on-read for celebrity accounts requires feed merge logic at read time (combine pre-materialized follower feed with recent celebrity posts); this merge adds latency and complexity to the feed read path; Threshold for routing (follower count) must be maintained as a dynamic config; hardcoding it causes operational problems when accounts cross the threshold

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Kafka consumer group lag for fan-out worker group growing steadily; feed propagation latency (time from post write to follower feed update) exceeding 30s p95; Redis write rate on feed keys elevated but not saturated; post activity rate normal

Bottleneck: Fan-out worker pool undersized for burst post activity or a celebrity post creating a sustained high-fan-out event. Evolution: Add fan-out worker replicas; implement fan-out cost routing: route high-follower-count fan-out events to a dedicated high-cost worker pool with separate Kafka consumer group and Redis write quota; use follower count threshold (e.g., >50k followers) as the routing decision. Monitor fan-out cost per post as a first-class metric.

Signal: Redis memory utilization > 75%; eviction rate rising; cache miss rate on feed reads increasing; cold feed read fallback queries appearing in PostgreSQL slow query log; feed read p99 > 100ms despite Redis being online

Bottleneck: Redis feed list storage approaching memory limit; feed items being evicted before TTL; or feed list cap set too high for available memory. Evolution: Reduce feed list cap from current value toward 100–150 items; increase Redis cluster capacity or shard feed keys by user_id range across multiple Redis primaries; implement tiered feed storage: hot recent items in Redis, older items fetched from PostgreSQL on demand with explicit product UX affordance