Hot Shard (Unbalanced Write Distribution)
SEV-2, Significant ImpactIsolated propagation · sharding · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Hot Shard (Unbalanced Write Distribution) begins at the source component. Trigger: Sequential or timestamp-based shard key chosen for a write-heavy table.
Immediate (T+0) · Signal: Alert
Blast Radius
Writes routed to the hot shard slow down and may error under load; writes to other shards are unaffected. If the hot shard's replicas fall behind, reads served from those replicas return staler data than reads against any other shard's replicas, and this staleness gap widens with sustained write pressure. Recovery operations (failover, backup, resharding) on the hot shard take longer because it holds more data and a higher ongoing write rate than its siblings.
Contributing Factors
DynamoDB partitions data by partition key hash; sequential or low-cardinality partition keys cause hot partitions that exceed per-partition throughput limits and receive 400 ProvisionedThroughputExceededException responses.
High-throughput OLTP workloads are vulnerable to hot shard problems when the shard key is a sequential ID or timestamp, concentrating all new writes on the highest-range shard.
This operational trigger enables Hot Shard (Unbalanced Write Distribution): Sequential or timestamp-based shard key chosen for a write-heavy table
This operational trigger enables Hot Shard (Unbalanced Write Distribution): A viral or high-activity entity (a seller, a shared resource) generating outsized write volume on one shard
This operational trigger enables Hot Shard (Unbalanced Write Distribution): A low-cardinality shard key that cannot produce uniform hash distribution across the configured shard count
Remediation Plan
Measure per-shard write rate to confirm which shard (or shards) are hot
Effort: Minutes to hours (on-call response)
Identify the cause: is the shard key sequential, or is one entity dominating?
Effort: Minutes to hours (on-call response)
For a sequential key: plan a migration to hash-based sharding (significant effort)
Effort: Minutes to hours (on-call response)
Replace a sequential shard key with a UUID v4 or an application-generated random id. Hash distribution of random values is uniform across shards, which removes the monotonic-key hotspot entirely. The cost is a shard-key migration, and any code that relied on key ordering for range queries loses that property.
Effort: 1 day to 1 week
Track write rate per shard as a first-class metric and alert when any shard exceeds roughly 2x the cluster mean. This does not fix skew, but it turns a silent throughput ceiling into a visible signal early enough to rebalance before the hot shard becomes the binding constraint.
Effort: 1 day to 1 week
Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
For an entity expected to generate outsized write volume, append a suffix to spread its writes across N virtual shards: shard_key = f"{entity_id}_{random.randint(0, N)}". Writes for that entity spread across N shards instead of one. The cost lands on reads, which must fan out across all N virtual shards and merge to reconstruct the entity's full write history.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.