Use Social Feed Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Social Feed Platform. Traceable to YAML knowledge entities.
Context
Social feed platforms have an extreme read/write asymmetry: most users read their feed far more often than they post, but a single post from a high-follower account may require writing to millions of follower feed lists simultaneously. Synchronous fan-out on write is acceptable when follower counts are bounded, but breaks under celebrity accounts where a single post triggers 10–50 million Redis writes. The result is fan-out amplification: the write cost of one action grows with the fame of the actor, not the frequency of the action. The architecture must prevent celebrity posts from becoming an operational incident while keeping feed read latency under 50ms p99 for all users at any traffic level. Primary operational risks include: Celebrity fan-out storm: a single post from an account with 5M+ followers triggers 5M+ Redis LPUSH operations across fan-out worker pool; if worker pool is sized for average-case fan-out, this creates a sustained backlog that delays all feed updates for 10–30 minutes until the backlog drains; Hot Kafka partition on celebrity user_id: all activity events for a celebrity account hash to the same Kafka partition (partitioned by actor_id), serializing fan-out processing for that user behind a single consumer; p99 fan-out latency for followers of that account becomes bounded by single-partition throughput; Redis memory pressure from feed materialization: at 100M users each with a feed list capped at 200 items (20B average post ID pointers at 8 bytes = 16GB), a smaller-than-expected cap or a cache eviction misconfiguration causes cold feed reads to cascade into PostgreSQL via the outbox event log.
Decision
We will adopt the **Social Feed Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth. Core technology stack: postgresql, redis, kafka, rabbitmq.
Accepted Tradeoffs
- ⚠Fan-out-on-write gives O(1) feed reads at the cost of O(followers) write amplification per post; this is operationally correct for the 99th percentile of user accounts but wrong for the top 0.01%: the hybrid model adds routing complexity to get both properties
- ⚠Redis pre-materialized feed lists require a cap (typically 200–500 items) to bound memory growth; items older than the cap require a fallback read from PostgreSQL, making ancient-timeline reads slower by design
- ⚠Kafka fan-out workers provide backpressure isolation between the write path and feed materialization, but introduce feed propagation latency of 1–10 seconds under normal load; this is acceptable for asynchronous social content but requires user communication in product UX
- ⚠Outbox pattern on post writes ensures no fan-out event is lost on service failure, but adds ~2ms to post write latency and requires outbox table cleanup to prevent bloat under high post volume
- ⚠Read replicas for social graph queries reduce primary read load but introduce replica lag risk; fan-out workers reading follower lists from a lagged replica will produce incomplete fan-out for recently acquired followers
Risks
When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.
Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.
One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.
Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.
A single inbound request triggers N downstream calls, amplifying load on downstream services by the fanout factor. At sustained inbound rates, the amplified load overwhelms downstream services that appear correctly sized for direct traffic.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Social Feed Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Social Feed Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Social Feed Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Social Feed Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Fan-Out Worker Queue Backlog
Signal: Kafka consumer group lag for fan-out worker group growing steadily; feed propagation latency (time from post write to follower feed update) exceeding 30s p95; Redis write rate on feed keys elevated but not saturated; post activity rate normal
Evolution: Add fan-out worker replicas; implement fan-out cost routing: route high-follower-count fan-out events to a dedicated high-cost worker pool with separate Kafka consumer group and Redis write quota; use follower count threshold (e.g., >50k followers) as the routing decision. Monitor fan-out cost per post as a first-class metric.
Tier 2: Redis Feed Memory Ceiling
Signal: Redis memory utilization > 75%; eviction rate rising; cache miss rate on feed reads increasing; cold feed read fallback queries appearing in PostgreSQL slow query log; feed read p99 > 100ms despite Redis being online
Evolution: Reduce feed list cap from current value toward 100–150 items; increase Redis cluster capacity or shard feed keys by user_id range across multiple Redis primaries; implement tiered feed storage: hot recent items in Redis, older items fetched from PostgreSQL on demand with explicit product UX affordance
Tier 3: Social Graph PostgreSQL Read Pressure
Signal: PostgreSQL replica lag > 10s during peak fan-out periods; follower list queries appearing in pg_stat_activity with wait_event = Lock; read replica CPU > 70%; fan-out worker follower fetch latency rising; incorrect fan-out events (missing recent followers) appearing in feed correctness monitoring
Evolution: Materialize hot follower lists in Redis (TTL 60s) to absorb fan-out worker read volume; route all fan-out follower reads through Redis cache-aside before touching PostgreSQL replica; add a dedicated read replica for fan-out worker social graph reads, isolated from timeline API read replicas
Tier 4: Kafka Hot Partition Saturation
Signal: Kafka partition consumer lag concentrated on 2–3 partitions; fan-out latency for specific high-follower accounts disproportionately high vs. median; Kafka broker CPU uneven across partitions; single partition throughput ceiling (~100MB/s) visibly constraining fan-out for specific actor IDs
Evolution: Re-partition fan-out topic by a composite key (actor_id + post_id hash bucket) to distribute celebrity writes across multiple partitions; alternatively, use a separate Kafka topic for high-fan-out events with wider partition count; ensure downstream fan-out workers deduplicate or order correctly when a single post may be processed by multiple partitions
Migration Path
Monolithic feed built on PostgreSQL timeline queries → Redis pre-materialized feed with Kafka async fan-out workers
PostgreSQL timeline read query p99 > 500ms; query plan for "SELECT posts WHERE user_id IN (following_list) ORDER BY created_at DESC LIMIT 50" showing sequential scan or index merge; feed read traffic accounting for > 40% of database CPU
Uniform fan-out-on-write for all accounts → Hybrid fan-out model (fan-out-on-write for <10k followers, fan-out-on-read for high-follower accounts)
Fan-out worker queue lag during celebrity post events > 5 minutes; Redis write throughput spiking to > 80% of capacity during these events; fan-out cost for top 100 accounts measured at > 1000x median fan-out cost
Single Redis primary for all feed data → Redis Cluster with feed keys sharded by user_id range
Redis memory utilization approaching 80% of a single node; Redis CPU > 60% sustained on feed write operations; single-node Redis becoming a reliability risk for the entire feed system
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 4 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.