Post-Mortem Framework · Capacity: Batch Job Resource Starvation

Batch Job Resource Starvation

SEV-3, Limited Impact

Fan-Out propagation · capacity · Affects 0 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.

Propagation Chain

Origin component

Batch Job Resource Starvation begins at the source component. Trigger: Batch job starts concurrently with peak OLTP traffic window.

Immediate (T+0) · Signal: Latency Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

2–5 minutes via p99 latency alert on OLTP endpoints. Immediately if connection pool utilization monitoring is in place (alert at >80% pool occupancy). The signature is a sharp latency increase at a predictable clock time correlated with batch job start, which makes retrospective diagnosis straightforward. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Downstream of dependents (fan-out)

Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.

Within minutes of initial failure · Signal: Multiple services reporting elevated error rates

Blast Radius

OLTP query latency spikes across all endpoints that touch the shared database, not only the tables the batch job reads. Connection pool exhaustion is a global resource: once the pool fills, all new requests queue regardless of their query complexity. If the application uses a single database cluster for all services, batch contention in one service can degrade unrelated services sharing the cluster.

Contributing Factors

Trigger Condition: Batch job starts concurrently with peak OLTP traffic windowoperational

This operational trigger enables Batch Job Resource Starvation: Batch job starts concurrently with peak OLTP traffic window

Trigger Condition: Scheduled export or aggregation query acquires >20% of the soperational

This operational trigger enables Batch Job Resource Starvation: Scheduled export or aggregation query acquires >20% of the shared connection pool

Trigger Condition: Batch job issues a full-table sequential scan, evicting hot operational

This operational trigger enables Batch Job Resource Starvation: Batch job issues a full-table sequential scan, evicting hot pages from the buffer pool

Remediation Plan

ImmediateIdentify active batch queries consuming connections via SELECT pid, query, state

Identify active batch queries consuming connections via SELECT pid, query, state, query_start FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start ASC

Effort: Minutes to hours (on-call response)

ImmediateTerminate the longest-running batch queries using SELECT pg_terminate_backend(pi

Terminate the longest-running batch queries using SELECT pg_terminate_backend(pid) for batch PIDs

Effort: Minutes to hours (on-call response)

ImmediateVerify OLTP connection queue drains by monitoring pool utilization: should drop

Verify OLTP connection queue drains by monitoring pool utilization: should drop below 50% within 30 seconds

Effort: Minutes to hours (on-call response)

Short-TermSeparate connection pools for batch and OLTP workloads

Create a dedicated connection pool (PgBouncer or application-level) for batch jobs, capped at 5–10 connections. OLTP pool retains its full allocation. Batch jobs are blocked from borrowing from the OLTP pool. This is the most effective and lowest-risk mitigation: it requires no database changes, only application-level pool configuration.

Effort: 1 day to 1 week

Short-TermQuery resource limits and workload isolation

Vanilla PostgreSQL has no native per-query CPU quota. Bound batch queries per role with statement_timeout, lock_timeout, and a capped work_mem set via ALTER ROLE, and run batch through a separate connection pool with a low connection limit. Isolate CPU and I/O with OS-level cgroups on the batch worker, or move batch/analytics to a dedicated read replica so it never shares CPU with OLTP. (Greenplum resource groups and MySQL 8.0 Resource Groups offer CPU controls, but standard PostgreSQL does not.)

Effort: 1 day to 1 week

Short-TermBatch job scheduling outside peak OLTP windows

Schedule all batch jobs during off-peak hours (02:00–06:00 local time) when OLTP traffic is at <10% of peak. Combined with a batch job circuit breaker that aborts jobs if OLTP p99 exceeds 200ms during the batch window, this prevents runaway jobs from degrading production during edge cases.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: latency spike, connection exhaustion, cpu saturation. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermArchitecture review for Batch Job Resource Starvation resilience

Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.

Effort: 1–2 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.