Queue Backlog Accumulation
criticalContainedcapacity failure · linear propagation
Blast Radius
12%
2/17 nodes
Time to Detect
Seconds with consumer group lag monitoring and rate-of-change alerting. Without active lag monitoring: minutes to hours, depending on whether SLA violations on downstream systems are the first signal. By the time a user- facing SLA violation is observed, lag may have been growing for 30+ minutes.
Preventive Mitigations
1
Confidence
StrongImpacted Components
Slow Consumer
operational risk · Directly connected to failure mode 'Queue Backlog Accumulation' via risk propagation path in the topology.
Event Streaming
workload · Directly connected to failure mode 'Queue Backlog Accumulation' via risk propagation path in the topology.
Failure Cascade
Event Streaming, Slow Consumer
Queue Backlog Accumulation (linear propagation) directly affects these components.
Severity at this step: critical
Detection Signals
Recovery time estimate: Time to drain depends on consumer throughput headroom above producer rate. If consumer capacity is 2× producer rate, a 24-hour lag drains in 24 hours. Root cause fix + scale-up can reduce drain time to hours. Permanent data loss from retention expiry requires separate assessment and reconciliation.
Mitigation Checklist(1 preventive, 5 reactive)
Instrument consumer processing time by stage. If the bottleneck is a database call, add a connection pool, fix a missing index, or batch the reads. If it is an external API, add a circuit breaker and process API-failure messages asynchronously with a dead-letter queue.
Monitor consumer lag and apply backpressure to producers when lag exceeds a safe threshold. Producers throttle their write rate or shed non-critical messages. This prevents lag from growing to retention expiry.
Messages that fail processing repeatedly (poison pills) block queue progress. Route messages that fail 3+ times to a dead-letter queue for manual inspection and replay, preventing them from blocking the backlog from draining.
Backpressure prevents queue backlog accumulation by signaling producers to slow or pause ingestion when the consumer is approaching capacity, ensuring the queue depth stays bounded rather than growing without limit.
Add consumer instances (up to the partition count for Kafka consumers). For Kafka, one consumer per partition is the maximum parallelism; if all partitions are consumed and lag still grows, partition count must increase or per-message processing time must decrease. For RabbitMQ, consumers can scale beyond queue count.
If a consumer falls behind, it must be able to catch up before retention expires. Set retention.ms to at least 3× the expected maximum lag duration. For a consumer that may fall behind by 24 hours, set retention to 72+ hours.
Affected Systems
Workloads
Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.