Post-Mortem Framework · Capacity: Slow Consumer

Slow Consumer

SEV-3, Limited Impact

Isolated propagation · capacity · Affects 10 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity. This failure mode appears in 10 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Slow Consumer begins at the source component. Trigger: JVM Full GC pause on one consumer exceeding session.timeout.ms (default 45s).

Immediate (T+0) · Signal: Queue Depth

Blast Radius

Without cooperative rebalancing, the blast radius is the entire consumer group: all partitions stop consuming during each rebalance event. With cooperative rebalancing, only the affected partitions are reassigned and others continue. Lag accumulates on the slow consumer's partitions; downstream systems receiving data from those partitions fall further behind. If the slow consumer causes repeated rebalances (GC loop), the effective throughput of the entire group drops significantly.

Contributing Factors

Trigger Condition: JVM Full GC pause on one consumer exceeding session.timeout.operational

This operational trigger enables Slow Consumer: JVM Full GC pause on one consumer exceeding session.timeout.ms (default 45s)

Trigger Condition: Downstream database or API degraded only for the specific cooperational

This operational trigger enables Slow Consumer: Downstream database or API degraded only for the specific consumer instance's connection

Trigger Condition: CPU throttling on the container hosting one consumer instancoperational

This operational trigger enables Slow Consumer: CPU throttling on the container hosting one consumer instance (K8s CPU limits)

Mitigation Gaps

MEDIUMKnown mitigator 'backpressure' not in runbook

Add 'backpressure' to the runbook. Backpressure prevents slow consumers from falling further behind by signaling producers to pause, giving the consumer time to drain its backlog before new messages arrive.

Remediation Plan

ImmediateIdentify the slow consumer instance: compare per-instance metrics or per-partiti

Identify the slow consumer instance: compare per-instance metrics or per-partition lag to find the outlier

Effort: Minutes to hours (on-call response)

ImmediateCheck JVM GC metrics for the slow instance: full GC frequency and duration

Check JVM GC metrics for the slow instance: full GC frequency and duration

Effort: Minutes to hours (on-call response)

ImmediateIf GC is the cause: restart the consumer instance (eviction forces partition rea

If GC is the cause: restart the consumer instance (eviction forces partition reassignment to faster instances)

Effort: Minutes to hours (on-call response)

Short-TermEnable cooperative incremental rebalancing

Configure partition.assignment.strategy = CooperativeStickyAssignor in all consumers. Cooperative rebalancing only revokes and reassigns affected partitions instead of stopping all consumers. A single slow consumer's partitions are reassigned without pausing the rest of the group.

Effort: 1 day to 1 week

Short-TermTune JVM GC to prevent pauses exceeding session.timeout.ms

Configure G1GC or ZGC with a max pause time target below heartbeat timeout: -XX:MaxGCPauseMillis=500 and session.timeout.ms=10000. ZGC achieves <10ms pauses for most heap sizes. Alternatively, increase session.timeout.ms to 120,000ms to tolerate longer GC pauses, accepting slower eviction of truly failed consumers.

Effort: 1 day to 1 week

Short-TermConfigure max.poll.interval.ms to match actual processing time

If processing time per poll batch is legitimately long (e.g., batch calls to an external API), set max.poll.interval.ms to the maximum acceptable processing time. Default is 5 minutes; if processing takes 8 minutes per batch, the consumer will be evicted. Increasing this value prevents false evictions but delays detection of truly stuck consumers.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: queue depth, alert, latency spike. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermEliminate cross-scenario Slow Consumer exposure

Slow Consumer affects 10 architecture scenarios (AI Retrieval-Augmented Generation Platform, Analytics Data Platform, Developer Tools Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.