Use IoT Telemetry Ingestion Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for IoT Telemetry Ingestion Platform. Traceable to YAML knowledge entities.
Context
IoT telemetry platforms have an asymmetric write profile: ingest volume is determined entirely by the device fleet size and sampling rate, not by user demand. A fleet of 2M devices sampling every 5 seconds produces 400,000 metric points per second: sustained, with no natural off-peak period. Devices are embedded hardware: they cannot back off on write rate, cannot retry with intelligence, and frequently go offline and reconnect in bursts. The architecture must absorb device reconnect storms (1M devices reconnecting simultaneously after a network outage) without dropping data, must handle cardinality explosion (each new device adds unique time-series identifiers), and must provide sub-second last-known-value lookups for alerting while simultaneously serving multi-day historical queries for device fleet analytics. Primary operational risks include: Device reconnect storm: a regional network outage causing 500k devices to go offline simultaneously results in a burst reconnect event where all devices resume uploading queued telemetry at once; Kafka consumer lag spikes to tens of millions of messages, TimescaleDB write throughput saturates, and chunk autovacuum falls behind, degrading all query performance while the backlog drains; TimescaleDB chunk bloat from late-arriving data: offline devices reconnecting after 72+ hours of backlog insert into a time chunk that was already compressed, forcing TimescaleDB to decompress the chunk for the insert; at high device count, simultaneous late-arriving inserts across many old chunks can saturate I/O and stall all writes; Redis last-known-value cardinality explosion: at 5M unique device IDs with one Redis key per device, a naive key scheme consumes 5M+ keys; Redis memory for device state alone exceeds expected cluster sizing if key TTL is not enforced and devices are never retired from the active key space.
Decision
We will adopt the **IoT Telemetry Ingestion Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A high-rate device telemetry ingestion architecture designed for millions of devices emitting metrics at 1–60 second intervals. Kafka absorbs device writes as an ingestion buffer, decoupling device-facing ingest endpoints from the storage write path so that downstream storage pressure never propagates back to devices. TimescaleDB provides time-series storage with automatic chunk partitioning by time range, native compression, and continuous aggregate views for rollup queries. ClickHouse serves as the OLAP layer for device fleet analytics queries. Redis caches last-known device state (current readings per device) for real-time alerting queries that must not scan historical storage. Backpressure on the Kafka consumer side prevents storage write throughput from being overwhelmed by burst ingestion events from device reconnect storms. Core technology stack: kafka, timescaledb, redis, postgresql, clickhouse.
Accepted Tradeoffs
- ⚠Kafka ingestion buffer absorbs device reconnect storms but introduces an end-to-end latency floor of 1–5 seconds between device write and storage availability; real-time alerting systems that require sub-second device state queries must read from Redis last-known-value cache, not from TimescaleDB
- ⚠TimescaleDB time-based partitioning is operationally excellent for time-range queries and retention policies (drop old chunks atomically), but schema changes require touching all active chunks; a column addition to the metrics table is a multi-chunk DDL operation that must be carefully staged
- ⚠ClickHouse columnar storage gives 10–50x query speedup for fleet-wide analytics (aggregate metrics across all devices over 30 days) but is a separate storage system requiring a replication pipeline from TimescaleDB; this pipeline adds operational complexity and introduces query staleness of several minutes for ClickHouse-backed views
- ⚠Continuous aggregate views in TimescaleDB provide 1-minute, 1-hour, and 1-day rollups automatically, eliminating the need for a separate rollup job, but the refresh policy introduces a known refresh lag window during which rollup queries return stale data for the most recent time bucket
- ⚠Rate limiting at the Kafka consumer layer prevents storage write saturation but means that during a device reconnect storm, the last-known-value Redis cache is updated before the historical record is persisted; alerts may fire on current state while TimescaleDB still shows the pre-outage value for that device
Risks
One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.
Each logical application write triggers multiple physical writes through index maintenance, WAL generation, MVCC versioning, and replication, causing actual disk IOPS to exceed the provisioned I/O ceiling while the logical write rate appears modest.
PostgreSQL WAL (Write-Ahead Log) generation rate exceeds wal_buffers flush capacity or downstream replica/WAL archive bandwidth, causing write transactions to stall waiting for WAL flush and replication lag to grow unboundedly.
The storage device reaches its IOPS or throughput ceiling, causing all disk- dependent database operations to queue behind I/O requests, driving latency from sub-millisecond to hundreds of milliseconds and degrading all database operations simultaneously.
Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; IoT Telemetry Ingestion Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (clickhouse, kafka) with the chosen architecture but applies different structural patterns; IoT Telemetry Ingestion Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; IoT Telemetry Ingestion Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (clickhouse, kafka) with the chosen architecture but applies different structural patterns; IoT Telemetry Ingestion Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: TimescaleDB Write Throughput Ceiling
Signal: TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load
Evolution: Tune TimescaleDB chunk_time_interval to match write cadence (smaller chunks = faster compression, lower WAL amplification per chunk); enable native compression on chunks older than 1 hour to reduce on-disk footprint; add a dedicated NVMe volume for WAL separate from data directory; consider TimescaleDB multi-node for horizontal write distribution across data nodes
Tier 2: Kafka Consumer Lag from Reconnect Storm
Signal: Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate elevated; TimescaleDB write thread pool fully saturated; Redis last-known-value update latency acceptable but historical storage significantly behind real-time; device reconnect event visible in device authentication logs correlating with lag spike
Evolution: Pre-scale storage writer consumer replicas before anticipated high-risk windows (maintenance events, regional failovers); implement burst-aware consumer scaling using consumer group lag as the autoscale signal; tune Kafka consumer max.poll.records to batch storage INSERTs into TimescaleDB for higher per-consumer throughput (target 500–1000 rows per INSERT batch rather than single-row inserts)
Tier 3: Late-Arriving Data Chunk Decompression Cascade
Signal: TimescaleDB I/O saturation visible in disk throughput metrics during specific consumer lag drain periods; chunk decompression operations appearing in TimescaleDB logs (decompress_chunk); write latency spiking for historical time ranges (not current time chunk); device backlog replay operations (devices offline >1 hour) correlating with I/O spikes
Evolution: Implement a late-data ingest path separate from the real-time ingest path: late data (> 2 hours old by device timestamp) routes to a dedicated consumer that writes to a separate TimescaleDB hypertable with relaxed compression policy; this isolates late-data decompression I/O from the real-time write path; add monitoring alert when device timestamp delta vs. wall clock > 2 hours
Tier 4: Cardinality Explosion and Redis Memory Saturation
Signal: Redis memory utilization > 80%; Redis INFO keyspace showing active device key count significantly exceeding expected active fleet size; Redis eviction rate positive (LRU evictions occurring on device state keys); last-known-value read miss rate rising; alerting system false-positives from missing device state
Evolution: Enforce TTL on all device state Redis keys (set TTL = max expected device reporting interval * 3, e.g., for 60s devices: TTL = 180s); implement a device lifecycle event in Kafka (device_decommissioned) that explicitly deletes Redis keys; shard Redis by device_id hash range across cluster nodes if memory requirement after TTL enforcement still exceeds single-node capacity
Migration Path
Direct device writes to PostgreSQL with time-range partitioning → Kafka ingestion buffer + TimescaleDB consumer writers
PostgreSQL write p99 > 100ms at sustained device fleet load; device write errors spiking during reconnect events (devices cannot block on write failure); time-range partition pruning queries slow due to too many manual partitions; need for native rollup views without manual aggregation jobs
TimescaleDB as sole query layer for both real-time and historical queries → Redis last-known-value cache for real-time queries + TimescaleDB for historical queries
Alerting system query latency > 500ms due to TimescaleDB query execution on current time chunk under write pressure; alert evaluation falling behind schedule; real-time dashboard queries competing with batch analytics queries on same TimescaleDB instance
TimescaleDB for both ingest storage and analytics queries → TimescaleDB for hot storage + ClickHouse for fleet analytics
Multi-device aggregate queries (fleet-wide max/min/avg over 30-day windows) consuming > 30% of TimescaleDB CPU; analytics query p99 > 10s; analytics users and device ingest writers competing for I/O on the same TimescaleDB node; product requirement for fleet-wide queries that require full column scans
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 5 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.