Blast Radius Analysis · AI Retrieval-Augmented Generation Platform

Memory Pressure and OOM Kill

criticalContained

capacity failure · fan out propagation

Blast Radius

1/12 nodes

Time to Detect

The OOM event itself is instantaneous and detectable immediately via kernel logs and process monitoring. Pre-OOM memory pressure is detectable 5–30 minutes before the event with available memory monitoring and swap utilisation trending. Without proactive memory monitoring, OOM events are detected only when applications begin reporting connection errors.

Preventive Mitigations

Confidence

Strong

Impacted Components

Immediate

AI Embedding Lookup

workload · Directly connected to failure mode 'Memory Pressure and OOM Kill' via risk propagation path in the topology.

Failure Cascade

AI Embedding Lookup

Memory Pressure and OOM Kill (fan out propagation) directly affects these components.

Severity at this step: critical

Detection Signals

Memory PressureDisk SaturationError Rate SpikeLog Errors

Recovery time estimate: 30 seconds to 5 minutes for process restart. WAL crash recovery may add 1–10 minutes depending on shared_buffers size and checkpoint age. Redis AOF replay: seconds to minutes. Application connection pool re-establishment: 10–60 seconds.

Mitigation Checklist(4 preventive, 0 reactive)

Set work_mem based on max_connections, not arbitrary large valuepreventsmedium

work_mem × concurrent_complex_queries_per_connection × max_connections must fit within available RAM after shared_buffers. Formula: work_mem = (available_RAM - shared_buffers) / (max_connections × 0.1). For 32GB host with 8GB shared_buffers and 200 connections: work_mem = 24GB / 20 = 1.2GB. Override per-query for known-expensive analytical queries.

Enable HugeTLB and use memory overcommit controlspreventsmedium

Set vm.overcommit_memory = 2 and vm.overcommit_ratio = 80 to prevent the kernel from committing more virtual memory than 80% of physical RAM. Processes that attempt to allocate beyond this will fail at allocation time (ENOMEM) rather than being OOM-killed at access time. PostgreSQL handles this gracefully.

Set shared_buffers to 25% of available RAM, not total RAMpreventslow

PostgreSQL recommendation: shared_buffers = 25% of RAM. Leave 75% for OS page cache, work_mem allocations, and other processes. On a 32GB host: shared_buffers = 8GB. This is not the aggressive setting some guides recommend (40–50%): those figures assume the database is the only process on the host.

Set container memory limits and database maxmemory below host RAMpreventslow

In containerised environments (Kubernetes, Docker), set memory limits for the database container. Redis respects maxmemory configuration. PostgreSQL requires container-level limits combined with shared_buffers tuning. Prevents the database from consuming host memory to the OOM boundary.

Affected Systems

Workloads

AI Embedding Lookup

Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.