Use API Gateway Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for API Gateway Platform. Traceable to YAML knowledge entities.
Context
An API gateway sits in the critical path of every request from every API consumer of every tenant. At 100,000 requests per second, a 1ms added latency in the gateway hot path is 100 CPU-seconds per second of added overhead downstream. Redis atomic operations (INCR, Lua scripts for sliding window rate limiting) deliver sub-millisecond rate limit decisions; PostgreSQL configuration lookups on the hot path would be an order of magnitude slower and would create a configuration database bottleneck. The challenge is keeping Redis rate limit state consistent across N gateway replicas under concurrent increment operations, while ensuring that configuration changes published by one gateway replica are visible to all others without a restart or cache invalidation storm. A tenant that misconfigures their rate limit to 0 must not cause all their consumers to fail silently: misconfiguration must be caught at write time, not at request time. Primary operational risks include: Distributed rate limit counter race condition: a sliding window rate limiter implemented as Redis INCR across N gateway replicas races between the INCR and TTL SET operations if not wrapped in a Lua script; without atomicity, a burst of requests hitting different replicas simultaneously can each see the counter as below the limit before any replica's increment is visible to the others, allowing the true request rate to exceed the limit by up to N times the per-window limit; Configuration cache staleness causing wrong routing: gateway replicas cache tenant routing rules and rate limit configuration in local memory; a tenant updating their rate limit or changing a backend endpoint URL will not take effect on replicas that have cached the old configuration until the cache TTL expires; in a worst case, a tenant reducing their rate limit to handle backend capacity reduction sees the higher limit enforced for up to the cache TTL after the change; Redis failure causing total gateway outage: if rate limit state lives exclusively in Redis and Redis becomes unavailable, the gateway must choose between fail-open (allow all requests, potentially overwhelming backend services) and fail-closed (reject all requests, causing 503 for all tenants); neither is acceptable without a circuit breaker that implements a safe fallback policy.
Decision
We will adopt the **API Gateway Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart. Core technology stack: redis, postgresql, kafka.
Accepted Tradeoffs
- ⚠Redis Lua scripts for atomic rate limiting eliminate the INCR/TTL race condition but Lua script execution blocks other Redis commands on the same slot during execution; at very high request rates (> 500K/s), Lua script lock duration becomes measurable and should be profiled; WAIT-based Lua scripts are an anti-pattern at this throughput
- ⚠Local in-process configuration cache provides sub-millisecond routing decisions but creates a consistency window between configuration updates and enforcement across replicas; the window is bounded by cache TTL (typically 5–30 seconds) and must be communicated to tenants as part of the platform SLA: "rate limit changes take effect within 30 seconds" is an explicit operational contract, not a bug
- ⚠Publishing API usage events to Kafka on the hot path adds per-request Kafka produce latency; async fire-and-forget produce (with batching) reduces hot path impact but means usage events can be lost if the gateway process crashes between producing and Kafka acknowledging; the tradeoff between hot path latency (synchronous) and usage event durability (asynchronous) must be an explicit decision, not an accident
- ⚠Bulkhead isolation per tenant (dedicated Redis key namespaces, dedicated Kafka consumer groups for webhook delivery) prevents cross-tenant interference but increases the number of active Redis connections and Kafka consumer group members proportional to tenant count; at 10,000 tenants this is a connection management scaling challenge
Risks
When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.
When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.
All database connections in the pool are in use; new requests queue and then time out, causing cascading latency and errors across all dependent services.
In a multi-tenant system, one tenant's high resource consumption: query load, connection count, write rate, or storage I/O: degrades database or service performance for all other tenants sharing the same infrastructure, violating the implicit isolation guarantee that a shared-infrastructure SaaS product implies.
When a downstream service begins rate limiting requests from an upstream service, the upstream's retry logic with insufficient backoff amplifies the request rate : exceeding the rate limit further and potentially pushing the rejection downstream to other upstream callers, producing a cascade of rate-limited retries across the call graph.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; API Gateway Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; API Gateway Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; API Gateway Platform is a better fit for the identified workload profile.
Content Management Platform shares core technology (postgresql, redis) with the chosen architecture but applies different structural patterns; API Gateway Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Redis Rate Limit Throughput
Signal: Redis command latency p99 > 0.5ms; gateway hot path p99 exceeding 2ms with Redis as the bottleneck (not upstream service); Redis CPU > 60% sustained; Lua script execution visible in SLOWLOG at > 0.1ms frequency
Evolution: Shard rate limit counters across Redis Cluster nodes by hashing tenant_id to a cluster slot; this distributes Lua script execution across nodes proportional to tenant count; ensure tenant_id-keyed counters use hash tags ({tenant_id}) so all keys for a tenant land on the same slot and Lua scripts can operate on them atomically; do not use Redis Cluster without testing Lua script compatibility against your cluster topology first
Tier 2: Configuration Propagation Latency
Signal: Tenant reports that rate limit increase takes > 30 seconds to take effect across all gateway replicas; configuration change audit log shows primary PostgreSQL write completing, but gateway replicas still routing to old backend endpoints beyond the expected cache TTL window
Evolution: Implement configuration change notification via Redis pub/sub: PostgreSQL configuration writes also publish a config_invalidated event to a Redis channel; each gateway replica subscribes to this channel and flushes the affected local cache key on receipt; this reduces propagation latency from TTL duration to sub-second pub/sub delivery without eliminating the local cache that protects Redis from per-request configuration lookups
Tier 3: Kafka Usage Event Throughput
Signal: Kafka producer batch queue filling faster than it can be flushed; usage event lag on the billing consumer > 5 minutes; Kafka broker I/O saturation during peak request periods; gateway producer retries visible in producer metrics
Evolution: Tune Kafka producer batch.size and linger.ms for usage events to maximize batching efficiency (linger.ms = 5, batch.size = 65536 is a reasonable starting point); ensure Kafka topic partition count for usage events matches the maximum billing consumer parallelism; usage events can tolerate at-least-once delivery with deduplication on consumer side: set acks = 1 (not all) for usage events to reduce produce latency at the cost of broker failure durability
Tier 4: Redis Availability and Failover
Signal: Redis Sentinel or Cluster failover taking > 30 seconds; gateway error rate spiking to 100% during failover window; rate limit counters reset post-failover causing burst allowance across the tenant fleet simultaneously
Evolution: Deploy Redis Sentinel with at least 3 sentinel nodes for automatic failover with < 30s promotion time; implement a circuit breaker in the gateway for Redis unavailability: fail-open with local approximate rate limiting (leaky bucket in process memory) during the failover window; ensure Redis AOF persistence is enabled with appendfsync = everysec to minimize counter loss on failover
Migration Path
Per-request PostgreSQL configuration lookup on the hot path → Local in-process configuration cache with Redis pub/sub invalidation
PostgreSQL hot path query p99 > 2ms under sustained request load; connection pool exhaustion on the configuration database during traffic spikes; gateway horizontal scaling causing proportional growth in PostgreSQL connection demand
INCR + EXPIRE as separate Redis commands for rate limiting → Atomic Lua script implementing sliding window rate limiting
Rate limit enforcement allowing requests above the configured limit during concurrent burst traffic; rate limit anomalies found during load testing where measured allowed rate exceeds configured limit by > 10%
Single Redis instance with no persistence → Redis Sentinel with AOF persistence and gateway-side failover circuit breaker
First Redis instance crash causing 100% gateway error rate for the full recovery period; tenant SLA requirements for gateway availability > 99.9%; rate limit counter reset post-restart causing coordinated burst across all tenants
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 5 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.