DBRaven
Failure Mode · capacity

Noisy Neighbor

degraded

Summary

A co-located workload consumes shared resources (CPU, I/O, memory, network bandwidth) at the expense of other tenants on the same host, database instance, or cluster, causing latency degradation that is invisible in the victim's own metrics but visible in shared infrastructure metrics.

Description

Noisy neighbor is a multi-tenant resource contention problem. The critical property is that the victim workload is not doing anything wrong: it is operating within its own expected parameters. The cause is a co-tenant consuming a shared resource without limits.

Multi-tenant database: one tenant issues an analytical query (full table scan, complex aggregation) on a shared PostgreSQL instance. The query saturates shared_buffers and disk I/O. Other tenants on the same instance experience elevated query latency and connection delays: not because their queries are expensive, but because shared I/O is saturated. PostgreSQL provides no per-tenant resource isolation (statement_timeout helps, but is coarse).

Kubernetes without resource limits: a pod with no CPU limits bursts to consume all available CPU on the node when it has a traffic spike. Other pods on the same node are throttled by the CPU scheduler, experiencing latency spikes correlated with the noisy pod's activity: not their own. Setting CPU limits on all pods prevents this; the trade-off is that a pod cannot burst beyond its limit even if the node has spare capacity.

AWS EBS gp2 volumes: gp2 uses a credit-based I/O system. A volume bursts up to 3000 IOPS but accrues credits at a rate proportional to provisioned storage. Volumes that burst frequently drain their credit pool. This affects only the volume's own workload: not neighboring volumes. The noisy neighbor effect is within multi-tenant storage systems (NAS, SAN, shared NVMe) where I/O operations are shared.

Cloud instance type: c5.large instances share physical cores with other instances via hyperthreading and CPU time-slicing. A CPU-bound neighbor on the same physical host increases CPU steal time: the instance receives less CPU time than scheduled, causing latency spikes that appear as CPU saturation from the victim's perspective.

Shared Redis cluster: one application floods a Redis cluster with large keys or high-frequency SET operations. Redis being single-threaded means all other clients on the same Redis instance experience increased command latency during the flood.

Characteristics

Propagationfan out
Time to detectLatency spikes are visible within seconds. Identifying the noisy neighbor as the root cause requires correlating victim latency spikes with a co-tenant's resource spike: typically requires infrastructure-level metrics (node CPU, disk I/O) correlated with per-pod or per-tenant metrics. This investigation may take minutes to hours.
Blast radiusAll co-tenants on the shared resource experience degradation proportional to the resource saturation caused by the noisy neighbor. In a fully saturated shared database, all tenants experience degraded query performance. In a Kubernetes node with CPU saturation, all pods on that node are affected. The blast radius is bounded by the shared resource boundary.

Triggers

  • ·Tenant issues expensive analytical query on a shared transactional database
  • ·Pod without CPU or memory limits bursts and starves co-located pods
  • ·High-I/O workload saturates shared disk without per-tenant I/O throttling
  • ·Neighbor instance on shared physical host increases CPU steal time
  • ·Large-key operations on shared Redis cluster blocking command pipeline

Detection Signals

latency spikecpu saturationdisk saturation

Mitigation Strategies

Set CPU and memory limits on all Kubernetes podspreventscomplexity: low

CPU limits cap a pod's burst capacity at the node level. Memory limits cause OOM kill instead of unbounded memory growth. Setting requests = limits creates a Guaranteed QoS class: the pod receives guaranteed CPU and is not throttled by noisy neighbors.

Isolate high-impact workloads to dedicated node poolspreventscomplexity: medium

Move analytical workloads, batch jobs, or high-CPU services to dedicated node pools using Kubernetes node selectors and taints/tolerations. This eliminates co-tenancy between workloads with different resource profiles.

Apply database-level statement timeoutscomplexity: low

PostgreSQL statement_timeout kills queries that exceed a time limit. This prevents a single expensive query from holding shared I/O for an extended period. Set conservatively per workload type (e.g., 5s for OLTP, 300s for analytics connections).

Use separate database instances or clusters per tenant tierpreventscomplexity: high

Move high-value or latency-sensitive tenants to dedicated database instances. Accepts higher operational cost for isolation guarantees. The standard progression for SaaS multi-tenancy as usage grows.

Recovery Steps

  1. 1.Identify the noisy tenant: check node CPU, disk I/O, and per-pod/per-tenant metrics simultaneously
  2. 2.Correlate victim latency spike timestamps with noisy tenant resource spike timestamps
  3. 3.For immediate relief: kill or throttle the offending workload (DELETE noisy pod, kill database query)
  4. 4.For Kubernetes: add resource limits to the noisy pod before restarting
  5. 5.For database: terminate offending query via pg_terminate_backend; add statement_timeout
  6. 6.Long-term: isolate noisy workload to dedicated infrastructure

Estimated recovery time: Immediate once the noisy workload is terminated or throttled: seconds. Permanent isolation requires infrastructure changes: hours to days.

Affected Systems

Patterns

connection poolingsharding

Technologies

postgresqlredis

Basis

Classic cloud and multi-tenant operational failure; extensively documented in Kubernetes, AWS, and database tuning literature

Run This Failure

Blast radius analysis for this failure mode within each scenario that carries it.

Related Architecture Knowledge

Inbound: affects this entity

MitigatesPattern
tenant isolation
Draft · unverified

Tenant isolation enforces resource boundaries that prevent a single tenant's workload from impacting shared infrastructure used by other tenants.

Full relationship →

Used In Architecture Scenarios