Zombie Consumer Holding Partition Assignment

degraded

Summary

A Kafka consumer becomes a zombie when its processing thread is paused (long GC, application deadlock, blocking external call) while its heartbeat thread continues to run in the background, signaling liveness to the broker. The broker considers the consumer healthy and does not trigger a rebalance. Messages assigned to the zombie's partitions accumulate unprocessed. The consumer group lag grows silently until max.poll.interval.ms (default 5 minutes) is exceeded and the broker finally forces a rebalance: 5 minutes of guaranteed message accumulation per event.

Description

Kafka's consumer liveness model has two separate mechanisms: heartbeat sending (controlled by heartbeat.interval.ms, background thread) and poll calling (controlled by max.poll.interval.ms, application thread). A consumer is considered alive by the broker as long as heartbeats are received within session.timeout.ms. However, a consumer is only considered to be making progress if it calls poll() within max.poll.interval.ms. These two checks operate independently.

The zombie scenario exploits this separation. A consumer's processing thread enters a 4-minute GC pause (common in JVM consumers with large heap sizes and infrequent full GC). The heartbeat thread is a separate Java Thread and continues to send heartbeats every 3 seconds. The broker receives heartbeats and marks the consumer as healthy. During the 4-minute GC pause, the consumer processes zero messages. Its partition's consumer group lag grows at the full producer write rate. After the GC pause ends and the application thread resumes, it calls poll() and the broker: which has been counting the time since the last poll : has not yet exceeded max.poll.interval.ms (default 300 seconds). No rebalance occurs. The consumer resumes processing but is now 4 minutes behind.

Application-level deadlocks create a more persistent variant. If the processing thread deadlocks (circular lock acquisition, blocking on a downstream service with no timeout), the consumer never calls poll() again. After max.poll.interval.ms (300 seconds by default), the broker detects the missed poll and forces the consumer to leave the group. This triggers a rebalance. But during the 300-second window, the assigned partitions are stalled. For a topic with 5 MB/s write throughput, 300 seconds of stall accumulates 1.5 GB of lag. Depending on topic retention, this lag may persist for hours.

External API call hangs are the most common production trigger. A consumer calls an HTTP endpoint without a timeout. The endpoint is experiencing degraded availability and the TCP connection is established but responses are slow. The consumer's processing thread is blocked in the HTTP call. It does not call poll(). After max.poll.interval.ms, the partition is reassigned. The new assignee begins at the committed offset: which is behind the stall point: and reprocesses messages from that point, potentially causing duplicate processing.

Characteristics

Propagationlinear

Time to detect5–10 minutes via consumer group lag monitoring (alert when lag increases by >10,000 messages without throughput). The diagnostic signature: consumer group shows partitions assigned to a specific consumer ID with zero messages-consumed rate but non-zero consumer group heartbeat. The kafka-consumer-groups.sh --describe output shows CURRENT-OFFSET not advancing on specific partitions.

Blast radiusAll partitions assigned to the zombie consumer are stalled for the duration of the zombie state (up to max.poll.interval.ms). Downstream consumers of the processed output experience data gaps or delays. If the consumer is responsible for triggering time-sensitive actions (fraud alerts, real-time notifications, SLA-bound processing), the stall window represents guaranteed SLA breach for all messages arriving during that window.

Triggers

·JVM stop-the-world GC pause exceeding session.timeout.ms (if GC is >10 seconds) or close to max.poll.interval.ms
·Downstream HTTP or database call in the processing path with no timeout configured
·Application-level deadlock between processing threads (lock A, lock B in opposing order)
·Consumer processes an unusually large batch of messages that takes longer than max.poll.interval.ms to handle
·Thread pool saturation in the consumer application (all threads blocked, no thread available to call poll())

Detection Signals

queue depthalertlog errors

Mitigation Strategies

Set timeouts on all external calls in the processing pathpreventscomplexity: low

Configure explicit read/write timeouts on every HTTP client, database connection, and external service call within the consumer processing path. Example: HTTPClient with connectTimeout=2s, readTimeout=10s. If the external call exceeds the timeout, throw an exception that the consumer handles as a retriable or dead-letter error. This prevents indefinite blocking of the processing thread. The timeout value should be <max.poll.interval.ms / max_batch_size.

Reduce max.poll.interval.ms for latency-sensitive consumerscomplexity: low

Set max.poll.interval.ms=30000 (30 seconds) for consumers where a 5-minute zombie window is unacceptable. This forces faster broker detection of stalled consumers. Ensure that the actual maximum processing time per poll() batch is well below 30 seconds (measure p99 processing time in production and set max.poll.interval.ms to 3x the p99). Reduce max.poll.records to lower batch size if processing time is too variable.

JVM GC tuning for large-heap consumerscomplexity: medium

Switch from CMS/Serial GC to G1GC or ZGC for JVM Kafka consumers with heap >8 GB. G1GC limits stop-the-world pauses to <200ms in most cases; ZGC limits pauses to <10ms. For consumers experiencing long GC pauses, reduce heap size and use off-heap storage for large intermediate data structures. Monitor GC pause time via JVM GC logs and alert if any GC pause exceeds session.timeout.ms / 2.

Recovery Steps

1.Run kafka-consumer-groups.sh --describe --group <group> and identify partitions with zero message-consumed rate
2.Check consumer application logs for GC pause events, deadlock thread dumps, or blocked external call timeouts
3.If the consumer is in a GC pause or temporary block, wait for it to recover (poll() will resume and lag will drain)
4.If the consumer is deadlocked or permanently blocked, kill and restart it: this triggers a rebalance and reassigns its partitions
5.After partition reassignment, monitor that the new assignee begins processing and lag decreases
6.Review and add timeouts to all external calls in the consumer before redeploying

Estimated recovery time: 5–10 minutes for the broker to detect the zombie via max.poll.interval.ms expiry and trigger a rebalance. After rebalance, lag drain time depends on consumer throughput vs accumulated lag; plan for 1 minute drain per 60 seconds of stall at equal producer/consumer throughput.

Affected Systems

Patterns

competing consumerspublisher subscriberevent sourcing

Technologies

kafka

Basis

Zombie consumer mechanics are precisely specified in the Kafka consumer group protocol documentation; heartbeat/poll separation is a defined behavior of the KafkaConsumer API; max.poll.interval.ms default of 300 seconds is the documented Kafka default