Relationship · Introduces Risk
Source: Failure Mode·Target: Failure Mode
Summary
A slow consumer processing messages below the producer rate causes queue backlog to accumulate. If processing speed does not recover, backlog grows unboundedly, eventually causing either message loss (if the queue has a depth limit) or indefinite processing delay.
Evidence
- ·{"Little's Law": 'L = λW: if arrival rate (λ) exceeds service rate, queue length (L) grows without bound'}
- ·RabbitMQ queue depth metric (messages_ready) directly measures slow consumer impact
- ·Kafka consumer lag (partition offset difference) is the direct measure of slow consumer impact
- ·SQS ApproximateNumberOfMessagesNotVisible metric tracks in-flight messages held by slow consumers
- ·DataDog's SQS monitoring documentation identifies slow consumer as the primary cause of ApproximateAgeOfOldestMessage growth
Operational Context
- ·Monitor consumer processing rate alongside producer rate: the gap directly predicts lag growth trajectory
- ·{'Identify the slow consumer bottleneck': 'CPU-bound processing, I/O-bound downstream calls, or lock contention'}
- ·Add consumer replicas (horizontally scale) up to the partition/queue concurrency limit for the fastest recovery
Tradeoffs
- ·Increasing consumer parallelism helps only up to the partition count (Kafka) or queue message visibility limit (SQS)
- ·Speeding up per-message processing (optimization) has a multiplicative effect on throughput recovery
- ·Backlog recovery after a slow consumer incident may require priority processing of newest messages first
Evidence grounding
Grounded, 5 supporting itemsSlow consumer as the primary cause of queue backlog is fundamental queuing theory (Little's Law) and extensively documented in Kafka, RabbitMQ, and SQS operational guides.