Kafka Consumer Group Rebalancing Storm
partialSummary
When consumer group members crash, restart, or are deployed in rapid succession, Kafka triggers continuous partition rebalancing that prevents any consumer from accumulating enough stable assignment time to make meaningful progress. During the rebalance window all consumption is paused, and if restarts occur faster than the rebalance completes, throughput drops to near-zero while messages accumulate in the partition backlog.
Description
Kafka consumer group rebalancing is a stop-the-world operation for the group. When any member joins or leaves, the group coordinator marks the group as rebalancing and revokes all partition assignments from all members simultaneously. The rebalance protocol then reassigns partitions and waits for all members to acknowledge the new assignment before consumption resumes. For a group of 20 consumers with the default rebalance timeout (session.timeout.ms=10000), a single rebalance takes 5–15 seconds. During this window, zero messages are processed.
A rebalancing storm occurs when restarts happen at a higher frequency than the rebalance completion time. Common triggers: a rolling deployment restarts consumers one at a time every 10 seconds; each restart triggers a rebalance that takes 15 seconds; the next restart begins before the previous rebalance completes, keeping the group in a perpetual rebalancing state for the full deployment duration. For a 20-instance deployment with 10-second rolling interval and 15-second rebalance time, total consumption is paused for the entire 3-minute deployment window.
OOM-triggered restarts amplify the problem further. If consumers are processing large messages and hitting memory limits, the OOM kill-restart cycle can recur every 20–30 seconds. Each restart resets the rebalance timer. Consumer group lag grows at the full producer write rate. For a topic with 1 MB/s write throughput, a 5-minute rebalancing storm accumulates 300 MB of unprocessed messages. Depending on topic retention configuration, this lag may exceed the retention window, causing message loss if the storm persists.
Static group membership (group.instance.id, introduced in Kafka 2.3) is the primary architectural defense. A consumer with a static group ID does not trigger a rebalance on restart if it reconnects before the session timeout expires. This eliminates the restart-rebalance cycle for rolling deployments and most OOM scenarios, at the cost of requiring unique stable identity per consumer instance.
Characteristics
Triggers
- ·Rolling deployment restarting consumers faster than rebalance timeout (e.g., 10-second restart interval vs 15-second rebalance)
- ·Consumer OOM crash loop with <30 seconds between restarts
- ·Network partition between consumers and Kafka broker causing repeated session timeouts
- ·Kubernetes liveness probe misconfiguration triggering unnecessary consumer pod restarts
- ·Consumer application startup time exceeds session.timeout.ms (consumer joins group before it is ready to poll)
Detection Signals
Mitigation Strategies
Assign each consumer instance a unique stable group.instance.id (e.g., hostname, pod name, or UUID persisted across restarts). A static member that rejoins the group within the session timeout does not trigger a rebalance; the broker re-assigns its previous partitions directly. For rolling deployments this eliminates the rebalance storm entirely. Requires Kafka 2.3+ and careful instance identity management in containerized environments.
Enable cooperative-sticky assignor (partition.assignment.strategy= CooperativeStickyAssignor) so that rebalances only revoke partitions that need to move, rather than revoking all partitions from all members. Most consumers continue processing during the rebalance; only the members gaining or losing partitions pause. Eliminates stop-the-world rebalance impact for all members not involved in the reassignment. Requires Kafka client 2.4+.
Increase session.timeout.ms to 30–45 seconds (from default 10s) and set heartbeat.interval.ms to session.timeout.ms / 3. This gives a crashed consumer more time to reconnect before the group triggers a rebalance. Trades slower detection of genuinely dead consumers for fewer spurious rebalances on slow restarts. Not appropriate for latency-sensitive consumers that require fast partition reassignment on failure.
Recovery Steps
- 1.Check consumer group state via kafka-consumer-groups.sh --describe --group <group>; look for "PreparingRebalance" or "CompletingRebalance" state
- 2.Identify which consumer instances are repeatedly joining and leaving (log pattern "Member ... has left" followed by "Member ... has joined")
- 3.Stop all consumer instances simultaneously rather than rolling, to allow a single clean rebalance
- 4.Wait for full group rebalance to complete (all partitions assigned, state = Stable)
- 5.Restart all consumer instances simultaneously with static group membership configured
- 6.Monitor consumer lag metric to confirm it is decreasing after stable assignment
Estimated recovery time: 5–15 minutes to achieve stable group state after stopping all instances and restarting cleanly. Lag drain time depends on consumer throughput relative to accumulated backlog; plan for 1 minute of drain time per 60 MB of accumulated lag at typical consumer throughput.
Affected Systems
Patterns
Technologies
Basis
Precisely documented Kafka-specific failure mode with exact protocol mechanics; static group membership as prevention is an established Kafka engineering pattern with well-understood configuration parameters