Kafka Consumer Group Rebalancing Storm
SEV-3, Limited ImpactFan-Out propagation · messaging · Affects 0 scenario(s)
Severity Classification
Classified as PARTIAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Kafka Consumer Group Rebalancing Storm begins at the source component. Trigger: Rolling deployment restarting consumers faster than rebalance timeout (e.g., 10-second restart interval vs 15-second rebalance).
Immediate (T+0) · Signal: Queue Depth
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
1–3 minutes via Kafka consumer group lag monitoring (kafka_consumer_group_lag metric). Log pattern detection of frequent "Rebalancing..." log lines within 30 seconds. Consumer group lag growing at the full producer write rate is the primary signal. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
All partitions assigned to the affected consumer group experience message accumulation during the rebalancing window. Downstream consumers of the processed output experience data gaps or delays. If the consumer group feeds a real-time pipeline (fraud detection, event aggregation), downstream systems may generate incorrect or incomplete results for data arriving during the storm window. At sufficient lag, offset retention expiry causes permanent message loss.
Contributing Factors
This operational trigger enables Kafka Consumer Group Rebalancing Storm: Rolling deployment restarting consumers faster than rebalance timeout (e.g., 10-second restart interval vs 15-second rebalance)
This operational trigger enables Kafka Consumer Group Rebalancing Storm: Consumer OOM crash loop with <30 seconds between restarts
This operational trigger enables Kafka Consumer Group Rebalancing Storm: Network partition between consumers and Kafka broker causing repeated session timeouts
Remediation Plan
Check consumer group state via kafka-consumer-groups.sh --describe --group <group>; look for "PreparingRebalance" or "CompletingRebalance" state
Effort: Minutes to hours (on-call response)
Identify which consumer instances are repeatedly joining and leaving (log pattern "Member ... has left" followed by "Member ... has joined")
Effort: Minutes to hours (on-call response)
Stop all consumer instances simultaneously rather than rolling, to allow a single clean rebalance
Effort: Minutes to hours (on-call response)
Assign each consumer instance a unique stable group.instance.id (e.g., hostname, pod name, or UUID persisted across restarts). A static member that rejoins the group within the session timeout does not trigger a rebalance; the broker re-assigns its previous partitions directly. For rolling deployments this eliminates the rebalance storm entirely. Requires Kafka 2.3+ and careful instance identity management in containerized environments.
Effort: 1 day to 1 week
Enable cooperative-sticky assignor (partition.assignment.strategy= CooperativeStickyAssignor) so that rebalances only revoke partitions that need to move, rather than revoking all partitions from all members. Most consumers continue processing during the rebalance; only the members gaining or losing partitions pause. Eliminates stop-the-world rebalance impact for all members not involved in the reassignment. Requires Kafka client 2.4+.
Effort: 1 day to 1 week
Increase session.timeout.ms to 30–45 seconds (from default 10s) and set heartbeat.interval.ms to session.timeout.ms / 3. This gives a crashed consumer more time to reconnect before the group triggers a rebalance. Trades slower detection of genuinely dead consumers for fewer spurious rebalances on slow restarts. Not appropriate for latency-sensitive consumers that require fast partition reassignment on failure.
Effort: 1 day to 1 week
Configure alerts for: queue depth, alert, log errors. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.