Noisy Neighbor
degradedSummary
A co-located workload consumes shared resources (CPU, I/O, memory, network bandwidth) at the expense of other tenants on the same host, database instance, or cluster, causing latency degradation that is invisible in the victim's own metrics but visible in shared infrastructure metrics.
Description
Noisy neighbor is a multi-tenant resource contention problem. The critical property is that the victim workload is not doing anything wrong: it is operating within its own expected parameters. The cause is a co-tenant consuming a shared resource without limits.
Multi-tenant database: one tenant issues an analytical query (full table scan, complex aggregation) on a shared PostgreSQL instance. The query saturates shared_buffers and disk I/O. Other tenants on the same instance experience elevated query latency and connection delays: not because their queries are expensive, but because shared I/O is saturated. PostgreSQL provides no per-tenant resource isolation (statement_timeout helps, but is coarse).
Kubernetes without resource limits: a pod with no CPU limits bursts to consume all available CPU on the node when it has a traffic spike. Other pods on the same node are throttled by the CPU scheduler, experiencing latency spikes correlated with the noisy pod's activity: not their own. Setting CPU limits on all pods prevents this; the trade-off is that a pod cannot burst beyond its limit even if the node has spare capacity.
AWS EBS gp2 volumes: gp2 uses a credit-based I/O system. A volume bursts up to 3000 IOPS but accrues credits at a rate proportional to provisioned storage. Volumes that burst frequently drain their credit pool. This affects only the volume's own workload: not neighboring volumes. The noisy neighbor effect is within multi-tenant storage systems (NAS, SAN, shared NVMe) where I/O operations are shared.
Cloud instance type: c5.large instances share physical cores with other instances via hyperthreading and CPU time-slicing. A CPU-bound neighbor on the same physical host increases CPU steal time: the instance receives less CPU time than scheduled, causing latency spikes that appear as CPU saturation from the victim's perspective.
Shared Redis cluster: one application floods a Redis cluster with large keys or high-frequency SET operations. Redis being single-threaded means all other clients on the same Redis instance experience increased command latency during the flood.
Characteristics
Triggers
- ·Tenant issues expensive analytical query on a shared transactional database
- ·Pod without CPU or memory limits bursts and starves co-located pods
- ·High-I/O workload saturates shared disk without per-tenant I/O throttling
- ·Neighbor instance on shared physical host increases CPU steal time
- ·Large-key operations on shared Redis cluster blocking command pipeline
Detection Signals
Mitigation Strategies
CPU limits cap a pod's burst capacity at the node level. Memory limits cause OOM kill instead of unbounded memory growth. Setting requests = limits creates a Guaranteed QoS class: the pod receives guaranteed CPU and is not throttled by noisy neighbors.
Move analytical workloads, batch jobs, or high-CPU services to dedicated node pools using Kubernetes node selectors and taints/tolerations. This eliminates co-tenancy between workloads with different resource profiles.
PostgreSQL statement_timeout kills queries that exceed a time limit. This prevents a single expensive query from holding shared I/O for an extended period. Set conservatively per workload type (e.g., 5s for OLTP, 300s for analytics connections).
Move high-value or latency-sensitive tenants to dedicated database instances. Accepts higher operational cost for isolation guarantees. The standard progression for SaaS multi-tenancy as usage grows.
Recovery Steps
- 1.Identify the noisy tenant: check node CPU, disk I/O, and per-pod/per-tenant metrics simultaneously
- 2.Correlate victim latency spike timestamps with noisy tenant resource spike timestamps
- 3.For immediate relief: kill or throttle the offending workload (DELETE noisy pod, kill database query)
- 4.For Kubernetes: add resource limits to the noisy pod before restarting
- 5.For database: terminate offending query via pg_terminate_backend; add statement_timeout
- 6.Long-term: isolate noisy workload to dedicated infrastructure
Estimated recovery time: Immediate once the noisy workload is terminated or throttled: seconds. Permanent isolation requires infrastructure changes: hours to days.
Affected Systems
Patterns
Technologies
Basis
Classic cloud and multi-tenant operational failure; extensively documented in Kubernetes, AWS, and database tuning literature
Run This Failure
Blast radius analysis for this failure mode within each scenario that carries it.
Related Architecture Knowledge
Inbound: affects this entity
Tenant isolation enforces resource boundaries that prevent a single tenant's workload from impacting shared infrastructure used by other tenants.
Full relationship →