Cross-Shard Query Degradation
SEV-2, Significant ImpactFan-Out propagation · sharding · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Cross-Shard Query Degradation begins at the source component. Trigger: Admin, reporting, or analytics queries without a shard-key predicate.
Immediate (T+0) · Signal: Alert
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Immediate with query-level instrumentation that tags a query as cross-shard and logs its fan-out count. Without that instrumentation, the symptom is elevated latency for specific features or background jobs, which can take hours to days to trace back to a sharding fan-out pattern rather than a single slow query. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
Each cross-shard query holds N connection-pool slots simultaneously; concurrent cross-shard queries can exhaust the pool and starve unrelated single-shard queries that would otherwise be fast. Merging large fan-out result sets in application memory can trigger OOM on the application process, taking down every request it was serving, not just the cross-shard one. A background reporting job that fans out to all shards can by itself consume enough connections and shard CPU to degrade production transaction throughput.
Contributing Factors
Analytics-heavy workloads that require aggregate queries across all shards are particularly vulnerable to cross-shard query degradation as shard count grows.
This operational trigger enables Cross-Shard Query Degradation: Admin, reporting, or analytics queries without a shard-key predicate
This operational trigger enables Cross-Shard Query Degradation: A new feature querying on a non-shard-key attribute (email lookup in a user-id sharded system)
This operational trigger enables Cross-Shard Query Degradation: Search queries on text fields not covered by the shard key or a global index
Remediation Plan
Identify the specific queries triggering cross-shard fan-out through query-level logging
Effort: Minutes to hours (on-call response)
Classify each: acceptable infrequent admin query, or high-frequency production query?
Effort: Minutes to hours (on-call response)
For high-frequency queries: build a global secondary index or lookup service
Effort: Minutes to hours (on-call response)
Maintain a global index, or a dedicated lookup table, mapping the non-shard-key attribute (email, in a user-id sharded system) to shard id plus primary key. Route the query through the lookup first, then fetch from the one correct shard. Converts a scatter-gather into two single-shard lookups. The cost is index or service maintenance and consistency lag between the primary write and the index update.
Effort: 1 day to 1 week
Accept fan-out for genuinely infrequent cross-shard queries, but isolate their resource use: a separate connection pool so they cannot starve single-shard transactional queries, and a circuit breaker that caps concurrent fan-out queries during a query storm. This does not reduce tail latency, it contains the blast radius.
Effort: 1 day to 1 week
Configure alerts for: alert, queue depth. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Replicate data from every shard into a single non-sharded analytics database (ClickHouse, Snowflake, BigQuery) and route aggregate and reporting queries there instead of at the production shards. This is the right fix once cross-shard queries are routinely aggregating across all shards rather than doing a targeted lookup on one attribute; it decouples analytics load from OLTP load entirely.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.