Hot Shard (Unbalanced Write Distribution)

A sharded database where a small number of shards absorb disproportionately more writes than the rest, because the shard key does not distribute uniformly under the real workload. The hot shard hits its own write ceiling while its siblings sit idle, so the cluster's aggregate write throughput never rises to meet demand. Sharding added operational surface without adding write capacity for that workload.

Sharding divides write load across N independent databases by a shard key. It only delivers 1/N write load per shard if the shard key distributes writes uniformly, and shard-key choice is the whole story here; hot_partition covers the general mapping problem across partitioned systems, this entity is specifically about what shard-key choice does to write throughput in a sharded database.

The dominant cause is a monotonically increasing shard key: an auto-increment id or a timestamp under range-based sharding. Every new row's key is greater than every existing key, so every write lands on the single highest-range shard while every other shard goes read-only cold. This is not a corner case, it is the default outcome of sharding a sequential key by range, and it is the primary reason to shard write-heavy tables by hash or by a well-distributed business key instead.

A second cause is a high-cardinality key with a skewed real-world distribution: a marketplace seller with orders of magnitude more orders than the median, a shared resource many users write through. The key hashes fine in principle, but in practice one hash value carries far more than 1/N of the traffic.

A third cause is a low-cardinality key: a shard key with only a few dozen distinct values sharded across a thousand shards leaves most shards empty and concentrates all traffic on the handful of shards that own the popular values.

What the hot shard suffers, specifically. Write throughput on the hot shard hits the same ceiling a non-sharded system would hit; for that workload, sharding bought nothing. Because the shard also carries disproportionate write volume, its WAL (or binlog) generation rate rises with it, and if replica apply falls behind the shard's own WAL throughput, its replicas lag more than the cluster average, so reads routed to those replicas return staler data than reads against a cold shard's replicas. Sustained high write volume on one shard also increases buffer-cache churn there (recently written pages compete with the working set for cache), which can raise read latency on that shard even for queries that do not touch the hot rows. And because more writes route through the same small key range or set of hot rows, the hot shard is also the shard most likely to show lock_contention.