IoT Telemetry Ingestion Platform

Summary

A high-rate device telemetry ingestion architecture designed for millions of devices emitting metrics at 1–60 second intervals. Kafka absorbs device writes as an ingestion buffer, decoupling device-facing ingest endpoints from the storage write path so that downstream storage pressure never propagates back to devices. TimescaleDB provides time-series storage with automatic chunk partitioning by time range, native compression, and continuous aggregate views for rollup queries. ClickHouse serves as the OLAP layer for device fleet analytics queries. Redis caches last-known device state (current readings per device) for real-time alerting queries that must not scan historical storage. Backpressure on the Kafka consumer side prevents storage write throughput from being overwhelmed by burst ingestion events from device reconnect storms.

Problem Statement

IoT telemetry platforms have an asymmetric write profile: ingest volume is determined entirely by the device fleet size and sampling rate, not by user demand. A fleet of 2M devices sampling every 5 seconds produces 400,000 metric points per second: sustained, with no natural off-peak period. Devices are embedded hardware: they cannot back off on write rate, cannot retry with intelligence, and frequently go offline and reconnect in bursts. The architecture must absorb device reconnect storms (1M devices reconnecting simultaneously after a network outage) without dropping data, must handle cardinality explosion (each new device adds unique time-series identifiers), and must provide sub-second last-known-value lookups for alerting while simultaneously serving multi-day historical queries for device fleet analytics.