Memory Pressure and OOM Kill
SEV-2, Significant ImpactFan-Out propagation · capacity · Affects 2 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 2 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Memory Pressure and OOM Kill begins at the source component. Trigger: shared_buffers set to >30% of total RAM without accounting for other memory consumers.
Immediate (T+0) · Signal: Memory Pressure
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
The OOM event itself is instantaneous and detectable immediately via kernel logs and process monitoring. Pre-OOM memory pressure is detectable 5–30 minutes before the event with available memory monitoring and swap utilisation trending. Without proactive memory monitoring, OOM events are detected only when applications begin reporting connection errors. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
An OOM kill of the database process immediately severs all client connections. All applications that depend on the database receive connection errors. For PostgreSQL, in-flight transactions that had not committed are rolled back via crash recovery. For Redis without persistence, all cached data is lost, causing a full cache miss storm on recovery. If the OOM condition is not addressed, the process will be OOM-killed again shortly after restarting.
Contributing Factors
AI embedding lookup workloads require the entire vector index to reside in RAM for acceptable latency. When the index size grows beyond available memory, the OS begins paging the HNSW graph to disk, causing query latency to degrade from milliseconds to seconds and eventually OOM-killing the process.
This operational trigger enables Memory Pressure and OOM Kill: shared_buffers set to >30% of total RAM without accounting for other memory consumers
This operational trigger enables Memory Pressure and OOM Kill: work_mem × max_connections × average_concurrent_queries exceeds available RAM
This operational trigger enables Memory Pressure and OOM Kill: Connection count spike (autoscaling without connection pooling) × per-connection overhead exceeds RAM
Remediation Plan
Confirm OOM event: check dmesg for OOM killer log entries and identify victim process
Effort: Minutes to hours (on-call response)
If database process was killed and has not restarted automatically: start the process
Effort: Minutes to hours (on-call response)
Monitor crash recovery: PostgreSQL will replay WAL from last checkpoint (check logs for progress)
Effort: Minutes to hours (on-call response)
PostgreSQL recommendation: shared_buffers = 25% of RAM. Leave 75% for OS page cache, work_mem allocations, and other processes. On a 32GB host: shared_buffers = 8GB. This is not the aggressive setting some guides recommend (40–50%): those figures assume the database is the only process on the host.
Effort: 1 day to 1 week
work_mem × concurrent_complex_queries_per_connection × max_connections must fit within available RAM after shared_buffers. Formula: work_mem = (available_RAM - shared_buffers) / (max_connections × 0.1). For 32GB host with 8GB shared_buffers and 200 connections: work_mem = 24GB / 20 = 1.2GB. Override per-query for known-expensive analytical queries.
Effort: 1 day to 1 week
Set vm.overcommit_memory = 2 and vm.overcommit_ratio = 80 to prevent the kernel from committing more virtual memory than 80% of physical RAM. Processes that attempt to allocate beyond this will fail at allocation time (ENOMEM) rather than being OOM-killed at access time. PostgreSQL handles this gracefully.
Effort: 1 day to 1 week
Configure alerts for: memory pressure, disk saturation, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Memory Pressure and OOM Kill affects 2 architecture scenarios (AI Retrieval-Augmented Generation Platform, Geospatial Tracking Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.