Architecture Review: IoT Telemetry Ingestion Platform

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 0 nodes

Each logical application write triggers multiple physical writes through index maintenance, WAL generation, MVCC versioning, and replication, causing actual disk IOPS to exceed the provisioned I/O ceiling while the logical write rate appears modest.

Affects 0 nodes

Seed 'WAL Saturation Risk Probe' identifies 2 metrics relevant to wal_saturation.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: PostgreSQL write p99 > 100ms at sustained device fleet load; device write errors spiking during reconnect events (devices cannot block on write failure); time-range partition pruning queries slow due to too many manual partitions; need for native rollup views without manual aggregation jobs. Migrate from 'Direct device writes to PostgreSQL with time-range partitioning' to 'Kafka ingestion buffer + TimescaleDB consumer writers'. Deploy Kafka ingest endpoint alongside the existing PostgreSQL direct write path. Run both in parallel for 2 weeks with a 10% traffic canary to TimescaleDB, validating that data arrives correctly and rollup views are accurate before migrating 100% of device traffic to the Kafka path.

Kafka introduces an ingestion buffer that makes device data available in storage with 1–5 second lag; real-time alerting systems reading from TimescaleDB must tolerate this latency or migrate to Redis last-known-value reads; TimescaleDB continuous aggregate refresh policy must be configured before production use; unconfigured refresh means rollup views are never updated

Trigger: Alerting system query latency > 500ms due to TimescaleDB query execution on current time chunk under write pressure; alert evaluation falling behind schedule; real-time dashboard queries competing with batch analytics queries on same TimescaleDB instance. Migrate from 'TimescaleDB as sole query layer for both real-time and historical queries' to 'Redis last-known-value cache for real-time queries + TimescaleDB for historical queries'. Write to Redis last-known-value in the Kafka consumer before writing to TimescaleDB. This ensures Redis is always as current as TimescaleDB or more current. The alerting system migrates to Redis reads incrementally, one alert rule at a time, with comparison validation against TimescaleDB during the transition period.

Redis last-known-value may lag TimescaleDB by up to Kafka consumer latency (1–5s); alerting systems must tolerate this latency window and must not treat a stale Redis value as ground truth for historical analysis; Redis key schema for device state must be defined once and treated as a contract; changes to the schema require coordinated deploy of both the Kafka consumer (writer) and alerting system (reader)

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load

Bottleneck: TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration). Evolution: Tune TimescaleDB chunk_time_interval to match write cadence (smaller chunks = faster compression, lower WAL amplification per chunk); enable native compression on chunks older than 1 hour to reduce on-disk footprint; add a dedicated NVMe volume for WAL separate from data directory; consider TimescaleDB multi-node for horizontal write distribution across data nodes

Signal: Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate elevated; TimescaleDB write thread pool fully saturated; Redis last-known-value update latency acceptable but historical storage significantly behind real-time; device reconnect event visible in device authentication logs correlating with lag spike

Bottleneck: Kafka consumer pool sized for steady-state throughput, not burst from device reconnect storm; insufficient storage writer parallelism for burst absorption. Evolution: Pre-scale storage writer consumer replicas before anticipated high-risk windows (maintenance events, regional failovers); implement burst-aware consumer scaling using consumer group lag as the autoscale signal; tune Kafka consumer max.poll.records to batch storage INSERTs into TimescaleDB for higher per-consumer throughput (target 500–1000 rows per INSERT batch rather than single-row inserts)