DBRaven
Failure Mode · operational

GC Pressure

partial

Summary

JVM garbage collection runs frequently and reclaims little heap, or produces stop-the-world pauses, causing latency spikes in Java and Scala services (Kafka brokers, Elasticsearch, Cassandra, HBase).

Description

GC pressure and GC pauses are distinct but related failure modes. GC pressure occurs when the GC runs continuously but the heap is not reclaimed fast enough to keep up with object allocation: the JVM spends an increasing fraction of CPU time on collection rather than application work. GC pause occurs when the JVM must stop all application threads (stop-the-world) to perform a collection phase: threads freeze for the pause duration, and in-flight requests accumulate.

G1GC (the default collector since JDK 9): performs concurrent marking but requires stop-the-world phases for certain operations. Under heap pressure (heap utilization > 80%), G1 may initiate Full GC, which is entirely stop-the-world and can pause JVM threads for 100–500ms on large heaps. This is long enough to trigger upstream timeouts and health check failures.

ZGC and Shenandoah (JDK 11+): designed for sub-millisecond pauses even on multi-gigabyte heaps. They achieve this by doing most work concurrently with application threads. On Kafka brokers running JDK 17+, ZGC is the recommended collector.

Elasticsearch is particularly GC-sensitive: long GC pauses cause nodes to appear unavailable to the cluster coordinator, triggering shard relocation storms that compound the original problem. Cassandra: GC pauses exceeding the Phi Accrual failure detector threshold cause the node to be marked as DOWN by peers, triggering unnecessary re-reads and hints.

Common triggers at the application level: - Returning very large result sets that require all objects to be live

simultaneously (the entire list in memory during serialization)

- Unbounded cache growth inside the JVM heap (in-process caches without

eviction policy)

- String interning abuse or large byte array allocations without pooling

Characteristics

Propagationisolated
Time to detectGC pauses are detectable in microseconds via JVM GC logs. Application-level detection (latency spike at p99) occurs within seconds of a long pause. GC pressure (high GC CPU, low allocation headroom) may take minutes to manifest as visible latency degradation.
Blast radiusGC pressure is isolated to a single JVM instance. However, stop-the-world pauses can cause the instance to appear unavailable to load balancers or cluster coordinators, triggering failover or shard rebalancing that increases load on remaining healthy instances. On Elasticsearch or Cassandra clusters, GC-induced apparent unavailability of one node can trigger secondary effects across the cluster.

Triggers

  • ·Heap utilization exceeds 80% causing G1GC to run Full GC
  • ·Object allocation rate exceeds GC collection throughput rate
  • ·Large query result sets loaded into memory simultaneously
  • ·In-process cache growing unboundedly without eviction policy
  • ·JVM heap sized too small for workload data volume

Detection Signals

latency spikecpu saturation

Mitigation Strategies

Upgrade to ZGC or Shenandoah for heap sizes above 4GBpreventscomplexity: low

ZGC and Shenandoah collectors perform most work concurrently with application threads, achieving sub-millisecond pauses at heap sizes of 8–32GB. Enable with -XX:+UseZGC (JDK 15+ for production readiness). Requires JDK 11+ (Shenandoah) or JDK 15+ (ZGC production).

Tune heap size to maintain allocation headroomcomplexity: low

Set -Xmx to leave sufficient headroom for live set + allocation buffer. A heap at 85% utilization during normal operation has no headroom for spikes. Rule of thumb: live set should not exceed 50–60% of Xmx.

Enable GC logging and set up pause-duration alertingcomplexity: low

-Xlog:gc*:file=gc.log:time,level,tags captures all GC events with timestamps. Parse logs for pause durations exceeding your SLO (e.g., alert on pauses > 200ms). GC log analysis is the primary diagnostic tool for GC pressure investigation.

Reduce object allocation rate at application levelpreventscomplexity: high

Profile with async-profiler or JFR to identify allocation hotspots. Common fixes: streaming large result sets instead of buffering, object pooling for frequent allocations, reducing intermediate collection copies in hot paths.

Recovery Steps

  1. 1.Check JVM GC logs for pause duration and frequency: grep 'GC pause' gc.log | tail -50
  2. 2.Check heap utilization: JMX, Prometheus JVM metrics, or jstat -gcutil <pid>
  3. 3.If heap is > 80% utilized under load, increase Xmx or add replicas to reduce per-instance load
  4. 4.If using G1GC and heap > 4GB, evaluate migration to ZGC in next deployment
  5. 5.Identify the allocation hotspot using async-profiler: record for 60s under load, examine allocation flame graph

Estimated recovery time: GC pauses resolve in milliseconds to seconds on their own. GC pressure (sustained high GC overhead) requires intervention: increasing heap, reducing load, or restarting the instance. Adding a replica reduces per-instance allocation rate and is the fastest operational relief.

Affected Systems

Patterns

competing consumers

Technologies

kafkaelasticsearchcassandra

Basis

Well-documented JVM operational failure; GC tuning for Kafka and Elasticsearch is extensively covered in vendor documentation