Hot Partition
criticalSummary
One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.
Description
Partitioning distributes load by mapping each key to a partition. It only balances load if the traffic per key is roughly even, and the mapping scheme decides how skew shows up.
Hash versus range partitioning. Range partitioning keeps keys ordered, which makes range scans cheap, but a monotonically increasing key (a timestamp, an auto-increment id) sends every new write to the single highest-range partition while the historical partitions go cold. That is the classic hot-latest-partition, and it is intrinsic to range partitioning of sequential keys. Hash partitioning removes that hotspot by scattering adjacent keys across partitions, but it pays for it by destroying range locality: a range scan now has to touch every partition. And hashing does not save you from one dominant key, because every request for that key hashes to the same partition no matter how many partitions exist.
Skew across many keys versus one key. If the skew is spread (a few dozen keys each somewhat hotter than the median), adding partitions or switching to hash partitioning spreads them out and helps. If a single key dominates (a celebrity account, a system user, the top product in a flash sale), no partitioning scheme fixes it on its own, because that key is indivisible under the scheme. The only remedy is to split the key deliberately: append a small random salt so one logical key becomes N physical keys spread across partitions. That works, but it moves the cost to reads, which must now fan out to all N salted keys and merge, and to bookkeeping, since the system must track which keys are salted and to what cardinality. It also breaks per-key ordering.
How it manifests by system. In Kafka a skewed producer key overloads one partition, and because a consumer group assigns each partition to exactly one consumer, the hot partition's lag cannot be spread to the idle consumers: consumer lag climbs on it alone. In a sharded key-value store like DynamoDB, a hot key concentrates reads and writes on one physical partition; adaptive capacity redistributes throughput toward busy partitions and can isolate a hot item onto its own partition, so uneven distribution across keys is absorbed automatically now, but a single hot key is still bounded by the per-partition throughput ceiling and will throttle (ProvisionedThroughputExceededException) once it hits it. In Redis Cluster every operation on a slot lands on the one node owning it, and since a Redis node is single-threaded, a hot slot saturates that node's single core while the rest of the cluster is idle.
The write-distribution version of this in a sharded database is detailed in hot_shard_unbalanced_writes; the read-side fan-out cost of avoiding a hot key by querying every shard is detailed in cross_shard_query_degradation.
Characteristics
Triggers
- ·Partition key is low-cardinality or highly skewed (a user_id where one user dominates traffic)
- ·Monotonically increasing key (timestamp, auto-increment id) concentrating new writes on the highest range partition
- ·A temporal hot key: a product, event, or resource that attracts disproportionate traffic during a surge
- ·System or service-account keys that generate orders of magnitude more events than regular keys
- ·Celebrity or trending content generating viral traffic concentrated on specific keys
Detection Signals
Mitigation Strategies
Append a small random prefix or suffix so one hot key spreads across N partitions: salted_key = original_key + "_" + random(0, N). This is the only fix for a single dominant key. The cost moves to reads, which must fan out to all N salted keys and merge, and it breaks per-key ordering, so it suits write-heavy or independent-event keys more than ordered streams.
More partitions spread moderate, many-key skew more evenly. In Kafka partition count can only grow, and existing messages keep their assignment. This helps when several keys are hot; it does nothing for a single dominant key, which still maps to one partition.
Replace the hot key with a composite (user_id + time_bucket) so one logical entity spreads across partitions over time. It reduces concentration at the cost of ordered-per-entity consumption, since consumers must correlate across buckets.
Identify high-volume keys in advance (system accounts, top-N products) and route them to a dedicated high-throughput topic or shard instead of the shared pool. The cost is key-classification logic in the producer or router and a separate consumer path.
Recovery Steps
- 1.Identify the hot partition: compare per-partition throughput and lag across the fleet
- 2.Identify the hot key(s): aggregate rates by key on the hot partition
- 3.Determine whether the skew is many-key (add partitions / hash) or single-key (salt or dedicate)
- 4.Apply the matching fix and roll it out in stages
- 5.Monitor per-partition metrics after the change to confirm the load redistributed
Estimated recovery time: Hours to days for structural fixes (partition-count increase, key-schema change). Immediate tactical relief is possible by adding consumers or raising provisioned throughput on the hot partition, but neither removes the underlying skew.
Affected Systems
Patterns
Technologies
Basis
Well-documented failure in partitioned systems with clear per-partition observability; the hash-versus-range tradeoff and the single-hot-key salting remedy are standard, and the per-system behavior (Kafka consumer assignment, DynamoDB adaptive capacity and per-partition limits, Redis single-threaded nodes) is documented.
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
Consistent hashing distributes data across nodes using a hash ring, ensuring that load is spread uniformly across all nodes regardless of key distribution. Virtual nodes further smooth out variance, reducing the likelihood of any single node becoming a hot partition.
Tradeoffs
- ·Consistent hashing prevents range queries: scan operations require scatter-gather across all nodes
- ·Uniform distribution assumes uniform access patterns: does not help if 90% of requests target the same logical entity
- ·Adding nodes with consistent hashing moves O(K/N) keys (K keys, N nodes): lower disruption than rehashing
Marketplace workloads are highly susceptible to hot partitions: viral listings, celebrity sellers, and flash sales concentrate enormous traffic on a small number of items or sellers, overwhelming the shards or database rows that store their data.
Tradeoffs
- ·Write sharding for viral items requires a read aggregation step: adds latency for total inventory reads
- ·Pre-sharding hot keys requires identifying them before the spike: reactive resharding is too slow
- ·Redis-based inventory counters sacrifice ACID guarantees: requires reconciliation with the relational database
Sharding distributes data across partitions, but poor shard key selection concentrates traffic on a small number of shards. A hot partition receives disproportionate load, becomes a bottleneck, and degrades performance for all data on that shard.
Tradeoffs
- ·Hash-based shard keys (good distribution) eliminate range query capability: all range queries become scatter-gather
- ·Detecting hot partitions requires per-shard metrics: aggregate metrics mask the problem
- ·Resharding to fix a hot partition is expensive: requires data migration across shards
Used In Architecture Scenarios
Analytics Pipeline
An OLAP-oriented analytics architecture that ingests operational changes from PostgreSQL via WAL-based CDC into Kafka, then routes them to a columnar analytics store (ClickHouse or Snowflake) for product analytics, business intelligence, and operational reporting. The CQRS separation ensures analytical queries never degrade transactional write performance, and materialized views provide pre-aggregated query acceleration for the most expensive analytical patterns.
Realtime Collaboration
A real-time location tracking platform for moving entities: vehicles, delivery couriers, field workers, and assets: receiving position updates at 1–30 second intervals from thousands of simultaneously tracked objects. Redis geospatial indexes (GEOADD / GEOSEARCH) serve sub-10ms proximity queries against the live position surface; PostgreSQL stores durable entity metadata and historical location records; TimescaleDB stores high-frequency time-series location data with automatic hypertable chunking and retention policies; Kafka streams location events to downstream consumers (dispatch systems, customer tracking apps, analytics pipelines). Geofence evaluation runs at ingestion time: each incoming position update is tested against active geofences for the entity's region, and geofence entry/exit events are emitted as Kafka messages to downstream consumers.
Write-Heavy Application
A high-rate device telemetry ingestion architecture designed for millions of devices emitting metrics at 1–60 second intervals. Kafka absorbs device writes as an ingestion buffer, decoupling device-facing ingest endpoints from the storage write path so that downstream storage pressure never propagates back to devices. TimescaleDB provides time-series storage with automatic chunk partitioning by time range, native compression, and continuous aggregate views for rollup queries. ClickHouse serves as the OLAP layer for device fleet analytics queries. Redis caches last-known device state (current readings per device) for real-time alerting queries that must not scan historical storage. Backpressure on the Kafka consumer side prevents storage write throughput from being overwhelmed by burst ingestion events from device reconnect storms.
Multi-Tenant SaaS
A multi-tenant SaaS architecture where multiple customers are served from a shared deployment, with PostgreSQL row-level security providing logical tenant isolation, Redis delivering per-tenant caching, and connection pooling managing the aggregate connection demand across tenant workloads. Tenant isolation, resource fairness, and operational simplicity are the three competing forces this architecture must balance.
Analytics Pipeline
A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume.
Search-Heavy Application
A content platform architecture centered on Elasticsearch for full-text search, faceted navigation, and ranked results, with PostgreSQL as the transactional source of truth and Redis for session management and hot content caching. WAL-based CDC maintains index freshness by streaming PostgreSQL changes into Elasticsearch asynchronously. The core tension is between search index freshness, query performance, and index maintenance cost under high write volume.
Event-Driven System
A social activity feed architecture where user actions (posts, likes, comments, follows) fan out asynchronously to follower timelines. Redis stores hot feed data as pre-materialized lists per user, enabling O(1) timeline reads for the 99th percentile of users. Kafka carries fan-out work to async workers that write to follower Redis keys. PostgreSQL is the durable store for the social graph, posts, and user content. The system uses a hybrid fan-out model: fan-out-on-write for users with fewer than ~10,000 followers (low fan-out cost), fan-out-on-read for high-follower celebrity accounts where pre-materialized fan-out would saturate workers and Redis write bandwidth.
Event-Driven System
A video and audio streaming architecture where content ingestion triggers an async multi-variant transcoding pipeline, CDN delivery handles 95%+ of playback traffic, and Cassandra absorbs the write volume of per-user viewing history. Kafka decouples upload events from transcoding workers; MinIO stores raw and encoded assets; Redis maintains playback session state and view counters. The architecture must handle upload spikes without blocking delivery, and cache cold starts without cascading database load.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.