Post-Mortem Framework · Cascading: Fanout Amplification

Fanout Amplification

SEV-3, Limited Impact

Fan-Out propagation · cascading · Affects 3 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 3 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Fanout Amplification begins at the source component. Trigger: N+1 query pattern: returning a list and resolving each item's relationship individually.

Immediate (T+0) · Signal: Latency Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

Milliseconds to seconds for downstream CPU or connection saturation signals. Root cause identification (tracing to the amplification point) typically requires distributed tracing and may take minutes to hours in investigation. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Downstream of dependents (fan-out)

Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.

Within minutes of initial failure · Signal: Multiple services reporting elevated error rates

Blast Radius

Downstream services receive N× the inbound request rate. Services sized for direct traffic saturate CPU, connections, or I/O. If downstream services are shared (e.g., a user service called by multiple upstream services), all upstream callers degrade simultaneously. The blast radius expands with amplification factor: a 100x amplification means one upstream service can alone saturate a downstream service at 1% of its own traffic rate.

Contributing Factors

Trigger Condition: N+1 query pattern: returning a list and resolving each item'operational

This operational trigger enables Fanout Amplification: N+1 query pattern: returning a list and resolving each item's relationship individually

Trigger Condition: GraphQL resolvers without DataLoader batchingoperational

This operational trigger enables Fanout Amplification: GraphQL resolvers without DataLoader batching

Trigger Condition: Scatter-gather query routing to all shards without result shoperational

This operational trigger enables Fanout Amplification: Scatter-gather query routing to all shards without result short-circuit

Remediation Plan

ImmediateUse distributed tracing (Jaeger, Zipkin) to identify the amplification point: lo

Use distributed tracing (Jaeger, Zipkin) to identify the amplification point: look for spans with high child count

Effort: Minutes to hours (on-call response)

ImmediateCheck downstream service metrics: is the call rate N× higher than the upstream r

Check downstream service metrics: is the call rate N× higher than the upstream request rate?

Effort: Minutes to hours (on-call response)

ImmediateTemporarily reduce inbound rate to the upstream service to relieve downstream pr

Temporarily reduce inbound rate to the upstream service to relieve downstream pressure

Effort: Minutes to hours (on-call response)

Short-TermImplement batch loading with DataLoader pattern

Collect all N item IDs from a list response, issue a single batch query to the downstream service (SELECT WHERE id IN (...)), and map results back to the list. DataLoader (originally from Facebook/GraphQL) implements this automatically per request: it accumulates lookups within a single request tick and issues one batched query.

Effort: 1 day to 1 week

Short-TermRedesign query to use join or batch endpoint

Replace N single-item queries with one query that returns all N items. For ORM-based N+1, use eager loading (SELECT_IN or JOIN fetch strategy). For microservices, expose a batch endpoint (GET /users?ids=1,2,3) and call it once per list response.

Effort: 1 day to 1 week

Short-TermCache responses at the resolver level to deduplicate repeated lookups

Within a single request, cache already-resolved items by ID. If the same author appears in 30 posts in a feed, only resolve them once. Request-scoped memoization avoids the redundant downstream calls without changing the API.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: latency spike, cpu saturation, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermEliminate cross-scenario Fanout Amplification exposure

Fanout Amplification affects 3 architecture scenarios (Notification Delivery Platform, Observability Platform, Social Feed Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.