Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. IoT Telemetry Ingestion Platform: triggered by 'PostgreSQL write p99 > 100ms at sustained device fleet load;'. API Gateway Platform: triggered by 'PostgreSQL hot path query p99 > 2ms under sustained request '.

Migration Step 2

Both scenarios define a migration step at this stage. IoT Telemetry Ingestion Platform: triggered by 'Alerting system query latency > 500ms due to TimescaleDB que'. API Gateway Platform: triggered by 'Rate limit enforcement allowing requests above the configure'.

Migration Step 3

Both scenarios define a migration step at this stage. IoT Telemetry Ingestion Platform: triggered by 'Multi-device aggregate queries (fleet-wide max/min/avg over '. API Gateway Platform: triggered by 'First Redis instance crash causing 100% gateway error rate f'.

TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load

Tier 1: TimescaleDB Write Throughput Ceiling: TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration). Recommended evolution: Tune TimescaleDB chunk_time_interval to match write cadence (smaller chunks = faster compression, lower WAL amplification per chunk); enable native compression on chunks older than 1 hour to reduce on-disk footprint; add a dedicated NVMe volume for WAL separate from data directory; consider TimescaleDB multi-node for horizontal write distribution across data nodes .

Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate elevated; TimescaleDB write thread pool fully saturated; Redis last-known-value update latency acceptable but historical storage significantly behind real-time; device reconnect event visible in device authentication logs correlating with lag spike

Tier 2: Kafka Consumer Lag from Reconnect Storm: Kafka consumer pool sized for steady-state throughput, not burst from device reconnect storm; insufficient storage writer parallelism for burst absorption. Recommended evolution: Pre-scale storage writer consumer replicas before anticipated high-risk windows (maintenance events, regional failovers); implement burst-aware consumer scaling using consumer group lag as the autoscale signal; tune Kafka consumer max.poll.records to batch storage INSERTs into TimescaleDB for higher per-consumer throughput (target 500–1000 rows per INSERT batch rather than single-row inserts) .

Redis command latency p99 > 0.5ms; gateway hot path p99 exceeding 2ms with Redis as the bottleneck (not upstream service); Redis CPU > 60% sustained; Lua script execution visible in SLOWLOG at > 0.1ms frequency

Tier 1: Redis Rate Limit Throughput: Single Redis instance processing all rate limit Lua scripts serially for all tenants across all gateway replicas. Recommended evolution: Shard rate limit counters across Redis Cluster nodes by hashing tenant_id to a cluster slot; this distributes Lua script execution across nodes proportional to tenant count; ensure tenant_id-keyed counters use hash tags ({tenant_id}) so all keys for a tenant land on the same slot and Lua scripts can operate on them atomically; do not use Redis Cluster without testing Lua script compatibility against your cluster topology first .

Tenant reports that rate limit increase takes > 30 seconds to take effect across all gateway replicas; configuration change audit log shows primary PostgreSQL write completing, but gateway replicas still routing to old backend endpoints beyond the expected cache TTL window

Tier 2: Configuration Propagation Latency: Local in-process cache TTL too long, or cache invalidation signal (Redis pub/sub or Kafka) not reaching all replicas. Recommended evolution: Implement configuration change notification via Redis pub/sub: PostgreSQL configuration writes also publish a config_invalidated event to a Redis channel; each gateway replica subscribes to this channel and flushes the affected local cache key on receipt; this reduces propagation latency from TTL duration to sub-second pub/sub delivery without eliminating the local cache that protects Redis from per-request configuration lookups .