Architecture Review: Multi-Tenant SaaS Platform

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 1 node. (Sharding)

All database connections in the pool are in use; new requests queue and then time out, causing cascading latency and errors across all dependent services.

Affects 1 node. (Redis). 1 mitigation identified

Seed 'Read Hotspot Saturation' identifies 3 metrics relevant to hot_partition. Execution preview confirms this risk manifests under modelled load.

Metrics to instrument: partition_qps, p99_latency_ms, cache_hit_rate

Trigger: Adding a second paying customer; first audit or security review; team growing beyond sole developer who holds mental model of tenant data boundaries. Migrate from 'Single-tenant PostgreSQL with per-user row filtering in application code' to 'Multi-tenant PostgreSQL with row-level security policies'. Define a standard RLS policy template at database setup time. Use automated testing to verify cross-tenant data isolation for all major query paths before onboarding any paying customers.

RLS policies must be applied to all existing and future tables: missing a table creates a data exposure risk; RLS with complex policies has measurable query planning overhead; test policy performance on large tables before enabling

Trigger: Database read load growing disproportionately with tenant count; per-tenant read p99 degrading despite stable individual tenant query volumes. Migrate from 'Shared schema multi-tenant with no caching' to 'Shared schema + Redis with tenant-namespaced cache keys'. Namespace all cache keys with tenant_id as prefix. Set Redis maxmemory-policy to allkeys-lru and size the cache for the aggregate working set of the top 10 tenants rather than all tenants combined.

Cache key namespacing mistakes cause cross-tenant reads: must be validated in tests; Redis eviction policy must account for large tenants evicting small tenants' hot keys

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Evolution from Unified Read/Write Database → CQRS with Separate Read Projections

Migration complexity: high. Rollback: complex.

Signal: PgBouncer pool wait queue > 0 during peak hours; application errors reporting "connection pool exhausted" or pool timeout; pool utilization > 90%

Bottleneck: Aggregate tenant connection demand exceeding PgBouncer pool_size. Evolution: Implement per-tenant connection quotas at the application layer before hitting the pool; increase pool_size incrementally; identify top-N connection consumers by tenant and implement connection reuse within tenant request handlers

Signal: pg_stat_activity shows one tenant's queries dominating query runtime; other tenants reporting p99 latency regression while their own query counts are stable; PostgreSQL shared_buffers cache eviction rate increasing

Bottleneck: Single large tenant displacing other tenants' working sets from shared buffer cache. Evolution: Implement pg_cgroups or connection-level resource groups if available; consider database-per-tenant for the top N largest tenants while keeping the shared schema for smaller tenants (hybrid isolation model)