Fan-Out Write Amplification
SEV-3, Limited ImpactFan-Out propagation · capacity · Affects 0 scenario(s)
Severity Classification
Classified as PARTIAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Fan-Out Write Amplification begins at the source component. Trigger: High-follower user (celebrity account) posts content in a fan-out-on-write timeline system.
Immediate (T+0) · Signal: Queue Depth
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
1–5 minutes via write latency monitoring on the primary write path (synchronous fan-out) or queue depth monitoring on the fan-out delivery queue (asynchronous fan-out). The diagnostic signature: a single entity write event correlating with a queue depth spike that takes minutes to drain. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
Downstream write saturation affects all writes to the fan-out targets (search, cache, analytics), not only the fan-out writes from the triggering event. If the fan-out exhausts the downstream store's write IOPS (e.g., saturating Redis pipeline throughput), all writes to that store are delayed. In synchronous fan-out, the write response latency for the original write path spikes, degrading all user-facing writes during the high-fan-out event window.
Contributing Factors
This operational trigger enables Fan-Out Write Amplification: High-follower user (celebrity account) posts content in a fan-out-on-write timeline system
This operational trigger enables Fan-Out Write Amplification: Entity update triggers notification writes to all subscribers when subscriber count grows beyond expected scale
This operational trigger enables Fan-Out Write Amplification: Bulk update operation on a widely-referenced entity triggers cascading invalidations across caches, search indexes, and derived stores
Remediation Plan
Identify the triggering entity (celebrity user, widely-referenced entity) causing fan-out via queue depth monitoring
Effort: Minutes to hours (on-call response)
If synchronous fan-out is causing write latency spikes, temporarily disable non-critical downstream writes (analytics, audit log) to shed load
Effort: Minutes to hours (on-call response)
Monitor the async fan-out queue depth and confirm it is draining (consumer throughput > new item ingestion rate)
Effort: Minutes to hours (on-call response)
Move all fan-out writes to an async queue (Kafka topic, SQS queue). Configure the fan-out consumer with explicit rate limits per destination system (max 20,000 timeline writes/second to Redis, max 5,000 to Elasticsearch). Add a "post delivery status" indicator so users see "post is being delivered to followers" during high-fan-out events rather than a silent delay. This is honest about the eventual-consistency nature of fan-out and prevents destination system saturation from cascading back to the primary write path.
Effort: 1 day to 1 week
For fan-out workloads where the same entity is written multiple times in rapid succession (e.g., a frequently-updated user profile), coalesce multiple updates into a single fan-out delivery by buffering writes for 100–500ms before fan-out. If 10 updates arrive in 200ms, deliver only the most recent state to downstream systems. Reduces fan-out volume for high-frequency update patterns by collapsing consecutive updates into a single downstream write.
Effort: 1 day to 1 week
Configure alerts for: queue depth, latency spike, disk saturation. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Route writes through two code paths based on the publisher's follower count. For publishers with <threshold followers (e.g., 10,000): use fan-out-on-write, precomputing timeline entries for all followers at write time. For publishers with >threshold followers (celebrities): use fan-out-on-read, computing the timeline by querying the celebrity's posts at read time and merging with the pre-computed non-celebrity feed. This bounds the maximum fan-out at the write path to threshold * write_rate, eliminating pathological amplification. Twitter's architecture uses this approach with a threshold of approximately 300,000 followers.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.