Relationship · Vulnerable To
Source: Workload·Target: Failure Mode
Summary
Event streaming workloads produce events faster than consumers can process them during spikes, accumulating a consumer group lag that grows unboundedly if consumer throughput cannot recover to exceed producer throughput.
Evidence
- ·Kafka consumer group lag (latest offset - committed offset) measures unconsumed events
- ·A producer spiking to 500,000 events/second with a consumer processing 100,000/second accumulates 400,000 events/second of lag
- ·Kafka's consumer lag metric is accessible via kafka-consumer-groups.sh and JMX/Prometheus exporter
- ·Uber documented consumer lag events as a primary cause of delayed trip confirmations during peak hours
- ·LinkedIn's monitoring infrastructure treats consumer lag as a P1 operational metric
Operational Context
- ·Alert when consumer lag exceeds your processing SLA: e.g., alert if lag > 5 minutes of production volume
- ·Horizontal scale consumers (add consumer group members up to partition count) to recover during spikes
- ·Consumer lag recovery requires consumer throughput > producer throughput for the entire recovery period
Tradeoffs
- ·Kafka retains events by time or size: a slow consumer cannot force Kafka to slow down producers
- ·Adding consumer group members beyond partition count does not help: extra consumers sit idle
- ·Exactly-once processing (Kafka transactions) reduces max consumer throughput by ~30% vs at-least-once
Evidence grounding
Grounded, 5 supporting itemsQueue backlog accumulation in Kafka consumer groups is one of the most common operational incidents in event streaming systems. Consumer lag monitoring is a first-class operational concern in every Kafka deployment.