Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. Developer Tools Platform: triggered by 'First noisy neighbor incident where one tenant's CI burst de'. API Gateway Platform: triggered by 'PostgreSQL hot path query p99 > 2ms under sustained request '.

Migration Step 2

Both scenarios define a migration step at this stage. Developer Tools Platform: triggered by 'Kafka publish failures rolling back pipeline completion tran'. API Gateway Platform: triggered by 'Rate limit enforcement allowing requests above the configure'.

Migration Step 3

Both scenarios define a migration step at this stage. Developer Tools Platform: triggered by 'Log search returning results from other tenants' pipelines d'. API Gateway Platform: triggered by 'First Redis instance crash causing 100% gateway error rate f'.

Redis job queue depth > 1000 correlated with a single tenant identifier; other tenants reporting p99 job start time > 60 seconds; tenant-level queue metrics showing one tenant holding > 50% of in-flight worker slots

Tier 1: Job Queue Tenant Noisy Neighbor: Shared Redis queue with shared worker pool allowing one tenant to monopolize available capacity. Recommended evolution: Implement per-tenant queue lanes in Redis (separate key namespaces per tenant, e.g., jobs:{tenant_id}:{priority}); implement a weighted fair scheduler at the worker dispatch layer that reads from tenant queues in round-robin order with priority weighting; cap the number of concurrently executing jobs per tenant to the tenant's quota, not to the total available worker count .

DDL migration duration > 10s on pipeline_runs, jobs, or artifacts tables; migration deployment causing timeout errors for active CI pipeline API calls during the deployment window; pg_locks showing AccessExclusiveLock held by ALTER TABLE statement

Tier 2: PostgreSQL Schema Migration Lock: High-volume tables requiring locking DDL changes during deployments with concurrent tenant activity. Recommended evolution: Adopt zero-downtime migration patterns exclusively: add columns with nullable defaults first (no table lock in PostgreSQL 11+), then backfill, then add constraints via NOT VALID followed by VALIDATE CONSTRAINT in a separate transaction; use pg_repack for table rewrites; never run concurrent index creation without CONCURRENTLY on any table with > 1M rows .

Redis command latency p99 > 0.5ms; gateway hot path p99 exceeding 2ms with Redis as the bottleneck (not upstream service); Redis CPU > 60% sustained; Lua script execution visible in SLOWLOG at > 0.1ms frequency

Tier 1: Redis Rate Limit Throughput: Single Redis instance processing all rate limit Lua scripts serially for all tenants across all gateway replicas. Recommended evolution: Shard rate limit counters across Redis Cluster nodes by hashing tenant_id to a cluster slot; this distributes Lua script execution across nodes proportional to tenant count; ensure tenant_id-keyed counters use hash tags ({tenant_id}) so all keys for a tenant land on the same slot and Lua scripts can operate on them atomically; do not use Redis Cluster without testing Lua script compatibility against your cluster topology first .

Tenant reports that rate limit increase takes > 30 seconds to take effect across all gateway replicas; configuration change audit log shows primary PostgreSQL write completing, but gateway replicas still routing to old backend endpoints beyond the expected cache TTL window

Tier 2: Configuration Propagation Latency: Local in-process cache TTL too long, or cache invalidation signal (Redis pub/sub or Kafka) not reaching all replicas. Recommended evolution: Implement configuration change notification via Redis pub/sub: PostgreSQL configuration writes also publish a config_invalidated event to a Redis channel; each gateway replica subscribes to this channel and flushes the affected local cache key on receipt; this reduces propagation latency from TTL duration to sub-second pub/sub delivery without eliminating the local cache that protects Redis from per-request configuration lookups .