Tenant Noisy Neighbor
SEV-2, Significant ImpactFan-Out propagation · multi tenancy · Affects 3 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point. This failure mode appears in 3 known architecture scenarios, indicating widespread relevance.
Propagation Chain
Origin component
Tenant Noisy Neighbor begins at the source component. Trigger: Large tenant bulk data import or batch migration without rate limiting.
Immediate (T+0) · Signal: Alert
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
30–120 seconds with per-tenant query rate and resource consumption metrics. Without per-tenant monitoring, the symptom appears as general platform degradation with no obvious single cause: investigation time extends to hours. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
All tenants sharing the affected resource tier. In shared-schema PostgreSQL, one tenant's buffer cache pollution affects all tenants on that primary instance. In connection pool exhaustion, all tenants on that pool endpoint are blocked. Severity is proportional to the resource consumed relative to total capacity. A tenant consuming 10% of capacity causes minor degradation; one consuming 80% may cause cascading failures across all affected tenants.
Contributing Factors
Marketplace mixed workloads are vulnerable to tenant noisy neighbor when high-volume sellers concentrate write activity on shared infrastructure, degrading performance for other sellers.
This operational trigger enables Tenant Noisy Neighbor: Large tenant bulk data import or batch migration without rate limiting
This operational trigger enables Tenant Noisy Neighbor: Analytics or reporting query on production database without query timeout
This operational trigger enables Tenant Noisy Neighbor: Application bug causing connection leak in one tenant's integration
Mitigation Gaps
Add 'tenant isolation' to the runbook. Tenant isolation partitions resources between tenants so that one tenant's workload cannot consume resources allocated to others, eliminating the noisy neighbor problem.
Remediation Plan
Identify the noisy tenant using per-tenant query logs, pg_stat_activity, or tenant-tagged metrics
Effort: Minutes to hours (on-call response)
Terminate or rate-limit the noisy tenant's queries immediately
Effort: Minutes to hours (on-call response)
Apply statement_timeout and connection limits to the identified tenant
Effort: Minutes to hours (on-call response)
Apply per-tenant connection limits and statement_timeout (PostgreSQL: SET statement_timeout = '30s') on each tenant's connection. Long-running queries are automatically terminated before they saturate resources. Combine with per-tenant query QPS limits enforced at the connection proxy layer.
Effort: 1 day to 1 week
Route reporting and analytics queries to a read replica dedicated to analytics workloads. The primary and other replicas serving production traffic are isolated from the buffer cache pollution caused by large analytical scans.
Effort: 1 day to 1 week
Configure alerts for: alert, queue depth. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Identify tenants above a size or activity threshold and migrate them to dedicated database instances. This is the tiered isolation model: most tenants on shared, power users on dedicated. Eliminates the noisy neighbor problem for the most impactful tenants.
Effort: 1–4 sprints
Tenant Noisy Neighbor affects 3 architecture scenarios (API Gateway Platform, Developer Tools Platform, Multi-Tenant SaaS Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.
Effort: 1–3 months
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.