Hot Partition
SEV-2, Significant ImpactIsolated propagation · capacity · Affects 9 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. This failure mode appears in 9 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Hot Partition begins at the source component. Trigger: Partition key is low-cardinality or highly skewed (a user_id where one user dominates traffic).
Immediate (T+0) · Signal: Disk Saturation
Blast Radius
Traffic routed to the hot partition degrades in throughput and latency; the other partitions are unaffected, so the failure is partial. In Kafka, consumer lag grows on the hot partition while the rest drain normally, and downstream consumers of its data fall behind. If the hot partition is a database shard, its entities throttle while cold-shard entities stay fast. The partial failure can still reach a large fraction of users when the hot key is a popular resource everyone touches.
Contributing Factors
Marketplace workloads are highly susceptible to hot partitions: viral listings, celebrity sellers, and flash sales concentrate enormous traffic on a small number of items or sellers, overwhelming the shards or database rows that store their data.
This operational trigger enables Hot Partition: Partition key is low-cardinality or highly skewed (a user_id where one user dominates traffic)
This operational trigger enables Hot Partition: Monotonically increasing key (timestamp, auto-increment id) concentrating new writes on the highest range partition
This operational trigger enables Hot Partition: A temporal hot key: a product, event, or resource that attracts disproportionate traffic during a surge
Mitigation Gaps
Add 'consistent hashing' to the runbook. Consistent hashing distributes data across nodes using a hash ring, ensuring that load is spread uniformly across all nodes regardless of key distribution. Virtual nodes further smooth out variance, reducing the likelihood of any single node becoming a hot partition.
Remediation Plan
Identify the hot partition: compare per-partition throughput and lag across the fleet
Effort: Minutes to hours (on-call response)
Identify the hot key(s): aggregate rates by key on the hot partition
Effort: Minutes to hours (on-call response)
Determine whether the skew is many-key (add partitions / hash) or single-key (salt or dedicate)
Effort: Minutes to hours (on-call response)
Append a small random prefix or suffix so one hot key spreads across N partitions: salted_key = original_key + "_" + random(0, N). This is the only fix for a single dominant key. The cost moves to reads, which must fan out to all N salted keys and merge, and it breaks per-key ordering, so it suits write-heavy or independent-event keys more than ordered streams.
Effort: 1 day to 1 week
More partitions spread moderate, many-key skew more evenly. In Kafka partition count can only grow, and existing messages keep their assignment. This helps when several keys are hot; it does nothing for a single dominant key, which still maps to one partition.
Effort: 1 day to 1 week
Identify high-volume keys in advance (system accounts, top-N products) and route them to a dedicated high-throughput topic or shard instead of the shared pool. The cost is key-classification logic in the producer or router and a separate consumer path.
Effort: 1 day to 1 week
Configure alerts for: disk saturation, queue depth, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Replace the hot key with a composite (user_id + time_bucket) so one logical entity spreads across partitions over time. It reduces concentration at the cost of ordered-per-entity consumption, since consumers must correlate across buckets.
Effort: 1–4 sprints
Hot Partition affects 9 architecture scenarios (Analytics Data Platform, Geospatial Tracking Platform, IoT Telemetry Ingestion Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.