Zombie Consumer Holding Partition Assignment
SEV-4, Near-MissLinear propagation · messaging · Affects 0 scenario(s)
Severity Classification
Classified as DEGRADED based on failure mode severity.
Propagation Chain
Origin component
Zombie Consumer Holding Partition Assignment begins at the source component. Trigger: JVM stop-the-world GC pause exceeding session.timeout.ms (if GC is >10 seconds) or close to max.poll.interval.ms.
Immediate (T+0) · Signal: Queue Depth
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
5–10 minutes via consumer group lag monitoring (alert when lag increases by >10,000 messages without throughput). The diagnostic signature: consumer group shows partitions assigned to a specific consumer ID with zero messages-consumed rate but non-zero consumer group heartbeat. The kafka-consumer-groups.sh --describe output shows CURRENT-OFFSET not advancing on specific partitions. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Blast Radius
All partitions assigned to the zombie consumer are stalled for the duration of the zombie state (up to max.poll.interval.ms). Downstream consumers of the processed output experience data gaps or delays. If the consumer is responsible for triggering time-sensitive actions (fraud alerts, real-time notifications, SLA-bound processing), the stall window represents guaranteed SLA breach for all messages arriving during that window.
Contributing Factors
This operational trigger enables Zombie Consumer Holding Partition Assignment: JVM stop-the-world GC pause exceeding session.timeout.ms (if GC is >10 seconds) or close to max.poll.interval.ms
This operational trigger enables Zombie Consumer Holding Partition Assignment: Downstream HTTP or database call in the processing path with no timeout configured
This operational trigger enables Zombie Consumer Holding Partition Assignment: Application-level deadlock between processing threads (lock A, lock B in opposing order)
Remediation Plan
Run kafka-consumer-groups.sh --describe --group <group> and identify partitions with zero message-consumed rate
Effort: Minutes to hours (on-call response)
Check consumer application logs for GC pause events, deadlock thread dumps, or blocked external call timeouts
Effort: Minutes to hours (on-call response)
If the consumer is in a GC pause or temporary block, wait for it to recover (poll() will resume and lag will drain)
Effort: Minutes to hours (on-call response)
Configure explicit read/write timeouts on every HTTP client, database connection, and external service call within the consumer processing path. Example: HTTPClient with connectTimeout=2s, readTimeout=10s. If the external call exceeds the timeout, throw an exception that the consumer handles as a retriable or dead-letter error. This prevents indefinite blocking of the processing thread. The timeout value should be <max.poll.interval.ms / max_batch_size.
Effort: 1 day to 1 week
Set max.poll.interval.ms=30000 (30 seconds) for consumers where a 5-minute zombie window is unacceptable. This forces faster broker detection of stalled consumers. Ensure that the actual maximum processing time per poll() batch is well below 30 seconds (measure p99 processing time in production and set max.poll.interval.ms to 3x the p99). Reduce max.poll.records to lower batch size if processing time is too variable.
Effort: 1 day to 1 week
Switch from CMS/Serial GC to G1GC or ZGC for JVM Kafka consumers with heap >8 GB. G1GC limits stop-the-world pauses to <200ms in most cases; ZGC limits pauses to <10ms. For consumers experiencing long GC pauses, reduce heap size and use off-heap storage for large intermediate data structures. Monitor GC pause time via JVM GC logs and alert if any GC pause exceeds session.timeout.ms / 2.
Effort: 1 day to 1 week
Configure alerts for: queue depth, alert, log errors. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.