IoT Telemetry Ingestion Platform

Topology Notes

• ·Devices publish telemetry via an ingest endpoint (HTTP or MQTT over TLS). The endpoint is stateless and immediately produces to Kafka without synchronous storage writes. This means device write acknowledgment is decoupled from storage durability: devices receive an ACK when Kafka accepts the message, not when TimescaleDB commits the row. This is the correct tradeoff for embedded devices that cannot block on storage latency.
• ·Kafka topics are partitioned by device_id hash to order events within a device's stream. Partition count should be set at 3–5x the expected peak consumer concurrency to allow consumer scaling without topic re-partitioning. Avoid partitioning by device manufacturer or device type: synchronized device sampling would create hot partitions.
• ·TimescaleDB Kafka consumers batch INSERT rows using COPY or multi-row INSERT for efficiency. Target batch sizes of 500–2000 rows per INSERT based on row width. Each consumer also writes the latest value per device_id to Redis using a pipeline SET with TTL before committing the Kafka offset: ensuring Redis is updated atomically with storage acknowledgment from the consumer's perspective.
• ·Redis stores last-known device state as a hash per device_id with fields for timestamp, each metric key, and a staleness flag. TTL is set to 3x the device's expected reporting interval. The alerting system reads from Redis exclusively for current-state checks; it reads from TimescaleDB only for threshold trending (e.g., "has this device's temperature been rising for 10 consecutive minutes").
• ·ClickHouse receives replicated data via a separate export pipeline (TimescaleDB logical replication or periodic export job). ClickHouse is never in the hot write path for device ingest. Fleet-wide analytics queries (p99 CPU across all devices in region X for the past 7 days) run exclusively against ClickHouse. No device ingest consumer writes directly to ClickHouse.