Post-Mortem Framework · Capacity: Noisy Neighbor

Noisy Neighbor

SEV-4, Near-Miss

Fan-Out propagation · capacity · Affects 1 scenario(s)

Severity Classification

Classified as DEGRADED based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.

Propagation Chain

Origin component

Noisy Neighbor begins at the source component. Trigger: Tenant issues expensive analytical query on a shared transactional database.

Immediate (T+0) · Signal: Latency Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

Latency spikes are visible within seconds. Identifying the noisy neighbor as the root cause requires correlating victim latency spikes with a co-tenant's resource spike: typically requires infrastructure-level metrics (node CPU, disk I/O) correlated with per-pod or per-tenant metrics. This investigation may take minutes to hours. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Downstream of dependents (fan-out)

Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.

Within minutes of initial failure · Signal: Multiple services reporting elevated error rates

Blast Radius

All co-tenants on the shared resource experience degradation proportional to the resource saturation caused by the noisy neighbor. In a fully saturated shared database, all tenants experience degraded query performance. In a Kubernetes node with CPU saturation, all pods on that node are affected. The blast radius is bounded by the shared resource boundary.

Contributing Factors

Trigger Condition: Tenant issues expensive analytical query on a shared transacoperational

This operational trigger enables Noisy Neighbor: Tenant issues expensive analytical query on a shared transactional database

Trigger Condition: Pod without CPU or memory limits bursts and starves co-locatoperational

This operational trigger enables Noisy Neighbor: Pod without CPU or memory limits bursts and starves co-located pods

Trigger Condition: High-I/O workload saturates shared disk without per-tenant Ioperational

This operational trigger enables Noisy Neighbor: High-I/O workload saturates shared disk without per-tenant I/O throttling

Mitigation Gaps

MEDIUMKnown mitigator 'tenant_isolation' not in runbook

Add 'tenant isolation' to the runbook. Tenant isolation enforces resource boundaries that prevent a single tenant's workload from impacting shared infrastructure used by other tenants.

Remediation Plan

ImmediateIdentify the noisy tenant: check node CPU, disk I/O, and per-pod/per-tenant metr

Identify the noisy tenant: check node CPU, disk I/O, and per-pod/per-tenant metrics simultaneously

Effort: Minutes to hours (on-call response)

ImmediateCorrelate victim latency spike timestamps with noisy tenant resource spike times

Correlate victim latency spike timestamps with noisy tenant resource spike timestamps

Effort: Minutes to hours (on-call response)

ImmediateFor immediate relief: kill or throttle the offending workload (DELETE noisy pod,

For immediate relief: kill or throttle the offending workload (DELETE noisy pod, kill database query)

Effort: Minutes to hours (on-call response)

Short-TermSet CPU and memory limits on all Kubernetes pods

CPU limits cap a pod's burst capacity at the node level. Memory limits cause OOM kill instead of unbounded memory growth. Setting requests = limits creates a Guaranteed QoS class: the pod receives guaranteed CPU and is not throttled by noisy neighbors.

Effort: 1 day to 1 week

Short-TermIsolate high-impact workloads to dedicated node pools

Move analytical workloads, batch jobs, or high-CPU services to dedicated node pools using Kubernetes node selectors and taints/tolerations. This eliminates co-tenancy between workloads with different resource profiles.

Effort: 1 day to 1 week

Short-TermApply database-level statement timeouts

PostgreSQL statement_timeout kills queries that exceed a time limit. This prevents a single expensive query from holding shared I/O for an extended period. Set conservatively per workload type (e.g., 5s for OLTP, 300s for analytics connections).

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: latency spike, cpu saturation, disk saturation. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermUse separate database instances or clusters per tenant tier

Move high-value or latency-sensitive tenants to dedicated database instances. Accepts higher operational cost for isolation guarantees. The standard progression for SaaS multi-tenancy as usage grows.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.