Fanout Amplification
partialSummary
A single inbound request triggers N downstream calls, amplifying load on downstream services by the fanout factor. At sustained inbound rates, the amplified load overwhelms downstream services that appear correctly sized for direct traffic.
Description
Fanout amplification occurs when a single operation at one layer causes multiple operations at the next layer. The amplification factor compounds at each hop: a 10x fanout at layer A feeding a 5x fanout at layer B produces 50x amplification on layer B's dependencies.
N+1 query pattern: the most common variant. An API endpoint returns a list of N objects, each requiring one additional query to resolve a relationship. Fetching 50 posts returns 50 queries for authors. At 1000 req/s, 50k author queries per second hit the user service: a service that only sees 1k req/s of direct traffic and is sized accordingly.
GraphQL N+1: GraphQL resolvers execute independently by default. A query for users with their orders and order items resolves: 1 query for users, N queries for each user's orders, M queries for each order's items. DataLoader batches and deduplicates resolver calls within a single request, but only if implemented.
Scatter-gather: a request is fanned out to all K shards simultaneously to find the record. Each shard handles one sub-query. At 1000 req/s with 20 shards, each shard handles 20k sub-queries/s. If one shard is slow, the entire response is delayed (slowest-shard problem).
Recursive resolution: a service resolves a record that references another record, which references another. Without depth limits, recursive lookups grow unboundedly. Knowledge graph and category tree traversals are common triggers.
Write fanout: fan-out on write amplifies writes. A single post by a user with 100k followers triggers 100k cache write operations. At 100 posts/min, that is 10M cache writes/min: potentially overwhelming Redis.
Characteristics
Triggers
- ·N+1 query pattern: returning a list and resolving each item's relationship individually
- ·GraphQL resolvers without DataLoader batching
- ·Scatter-gather query routing to all shards without result short-circuit
- ·Unbounded recursive resolution without depth limits
- ·Fan-out on write for high-follower-count users
Detection Signals
Mitigation Strategies
Collect all N item IDs from a list response, issue a single batch query to the downstream service (SELECT WHERE id IN (...)), and map results back to the list. DataLoader (originally from Facebook/GraphQL) implements this automatically per request: it accumulates lookups within a single request tick and issues one batched query.
Replace N single-item queries with one query that returns all N items. For ORM-based N+1, use eager loading (SELECT_IN or JOIN fetch strategy). For microservices, expose a batch endpoint (GET /users?ids=1,2,3) and call it once per list response.
Within a single request, cache already-resolved items by ID. If the same author appears in 30 posts in a feed, only resolve them once. Request-scoped memoization avoids the redundant downstream calls without changing the API.
Set a maximum recursion depth (e.g., 5 hops for graph traversal, 3 levels for nested category resolution) and return a truncated result beyond that depth rather than continuing to recurse.
Recovery Steps
- 1.Use distributed tracing (Jaeger, Zipkin) to identify the amplification point: look for spans with high child count
- 2.Check downstream service metrics: is the call rate N× higher than the upstream request rate?
- 3.Temporarily reduce inbound rate to the upstream service to relieve downstream pressure
- 4.Identify the specific query or resolver responsible for N+1 pattern
- 5.Implement batch loading or join query as the permanent fix
Estimated recovery time: Immediate relief by reducing upstream inbound rate or adding downstream capacity. Permanent fix requires code change and deployment: hours to days.
Affected Systems
Patterns
Technologies
Basis
Common production failure with well-documented pattern (N+1) and mitigations (DataLoader, batch loading)
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
Fan-out on write multiplies each post event into one write per follower; at high follower counts this produces write amplification that can saturate the write path for popular accounts.
Full relationship →Used In Architecture Scenarios
Event-Driven System
A multi-channel notification delivery architecture that accepts upstream business events (order placed, payment received, comment posted, threshold alert triggered) and routes them to per-channel delivery workers (push via FCM/APNs, email via SendGrid, SMS via Twilio, in-app via WebSocket). Kafka carries raw business events from upstream producers. RabbitMQ handles per-channel fan-out with separate exchanges and queues per delivery channel, isolating email queue backlog from push notification delivery. PostgreSQL provides durable notification state tracking (sent, failed, bounced, suppressed). Redis enforces per-user rate limiting (notification frequency caps to prevent fatigue) and stores deduplication tokens to prevent duplicate sends across retry attempts. The inbox pattern on the consumer side ensures idempotent delivery even when Kafka produces duplicate events.
Analytics Pipeline
A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume.
Event-Driven System
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth.