Architecture Review: API Gateway Platform

When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.

Affects 1 node. (Read-Heavy API Backend)

When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.

Affects 1 node. (Redis)

Seed 'Cache Stampede (Dog-Pile) Risk Probe' identifies 2 metrics relevant to cache_stampede.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: PostgreSQL hot path query p99 > 2ms under sustained request load; connection pool exhaustion on the configuration database during traffic spikes; gateway horizontal scaling causing proportional growth in PostgreSQL connection demand. Migrate from 'Per-request PostgreSQL configuration lookup on the hot path' to 'Local in-process configuration cache with Redis pub/sub invalidation'. Start with a 10-second TTL on routing rules and 30-second TTL on rate limits. Monitor cache hit rate; it should be > 99.9% under steady state. A low hit rate indicates the cache is being evicted faster than it is being populated, which requires heap sizing investigation.

Local cache introduces consistency window: test and document the maximum staleness window for each configuration type (rate limits, routing rules, API key validity) before switching to cached paths; Cache warm-up latency on gateway startup causes cold start period where every request falls through to PostgreSQL: implement background prefetch of all active tenant configurations at startup before accepting traffic

Trigger: Rate limit enforcement allowing requests above the configured limit during concurrent burst traffic; rate limit anomalies found during load testing where measured allowed rate exceeds configured limit by > 10%. Migrate from 'INCR + EXPIRE as separate Redis commands for rate limiting' to 'Atomic Lua script implementing sliding window rate limiting'. The atomic INCR/TTL Lua script is a minimal 8–12 line script. Implement the simplest version first (fixed window), validate correctness, then migrate to sliding window if the fixed window burst behavior is unacceptable to tenants.

Lua script must be tested against the Redis version running in production: some Redis Cluster configurations restrict KEYS access within Lua scripts; validate that hash tags route all tenant keys to the same slot before deploying; Script complexity must be kept minimal: a Lua script that takes > 0.1ms to execute under load should be profiled and simplified; complex rate limit algorithms (token bucket with multi-level hierarchy) should be validated against Redis latency benchmarks before replacing the simpler approach

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Redis command latency p99 > 0.5ms; gateway hot path p99 exceeding 2ms with Redis as the bottleneck (not upstream service); Redis CPU > 60% sustained; Lua script execution visible in SLOWLOG at > 0.1ms frequency

Bottleneck: Single Redis instance processing all rate limit Lua scripts serially for all tenants across all gateway replicas. Evolution: Shard rate limit counters across Redis Cluster nodes by hashing tenant_id to a cluster slot; this distributes Lua script execution across nodes proportional to tenant count; ensure tenant_id-keyed counters use hash tags ({tenant_id}) so all keys for a tenant land on the same slot and Lua scripts can operate on them atomically; do not use Redis Cluster without testing Lua script compatibility against your cluster topology first

Signal: Tenant reports that rate limit increase takes > 30 seconds to take effect across all gateway replicas; configuration change audit log shows primary PostgreSQL write completing, but gateway replicas still routing to old backend endpoints beyond the expected cache TTL window

Bottleneck: Local in-process cache TTL too long, or cache invalidation signal (Redis pub/sub or Kafka) not reaching all replicas. Evolution: Implement configuration change notification via Redis pub/sub: PostgreSQL configuration writes also publish a config_invalidated event to a Redis channel; each gateway replica subscribes to this channel and flushes the affected local cache key on receipt; this reduces propagation latency from TTL duration to sub-second pub/sub delivery without eliminating the local cache that protects Redis from per-request configuration lookups