Skip to content
DBRaven
Concerns

Rule-based disposition: any dimension at its most severe tier caps this at “concerns” or worse. Never an averaged score.

  • Operational Readiness: IoT Telemetry Ingestion Platform requires high operational expertise at 'experienced backend team' level. Current readiness estimate is 34%, critical gaps must be resolved before adoption. Consider starting with a simpler scenario and evolving toward this one.
Full ReviewModerate Readinessdraft

Architecture Review: IoT Telemetry Ingestion Platform

A high-rate device telemetry ingestion architecture designed for millions of devices emitting metrics at 1–60 second intervals. Kafka absorbs device writes as an ingestion buffer, decoupling device-facing ingest endpoints from the storage write path so that downstream storage pressure never propagates back to devices. TimescaleDB provides time-series storage with automatic chunk partitioning by time range, native compression, and continuous aggregate views for rollup queries. ClickHouse serves as the OLAP layer for device fleet analytics queries. Redis caches last-known device state (current readings per device) for real-time alerting queries that must not scan historical storage. Backpressure on the Kafka consumer side prevents storage write throughput from being overwhelmed by burst ingestion events from device reconnect storms.

Evidence Confidence

Moderate

strong

Executive Summary

IoT Telemetry Ingestion Platform carries moderate operational readiness (81% evidence confidence). 0 architectural strengths identified, 6 operational risks to manage. Primary concern: Hot Partition. Requires Advanced operational maturity.

Readiness Rationale

Overall moderate readiness across 8 dimensions. Limited: team maturity. Strong: migration, observability, failure recovery.

Key Concerns

  • !Hot Partition
  • !Write Amplification Cascade

Key Strengths

  • +Architecture is well-defined for the write heavy application problem profile

8

Assessments

3

Tradeoffs

6

Sections

12

Recommendations

Readiness Assessments

8

Governance Posture

7

Structural boundary and anti-pattern compliance: whether this architecture's topology violates documented governance policies. Distinct from operational readiness (below), which asks whether the team and infrastructure are prepared to run it.

Concerning

7 governance policy matches and 0 anti-pattern matches put IoT Telemetry Ingestion Platform's governance posture at concerning risk. Resilience is limited; burden is extreme.

7

violations

0

anti-patterns

Governance Violations

Resilience

limited

Blast radius: contained

40%

resilience score

Consistency Risks

  • ·Device reconnect storm: a regional network outage causing 500k devices to go off

Resilience Gaps

  • 5 high-exposure risk nodes increase blast radius

Operational Burden

extreme

operational burden

100%

burden index

Complexity Drivers

  • 5 architecture patterns increase configuration surface
  • Device reconnect storm: a regional network outage causing 500k devices to go off
  • TimescaleDB chunk bloat from late-arriving data: offline devices reconnecting af

Observability Burden

  • clickhouse: requires dedicated monitoring instrumentation
  • kafka: requires dedicated monitoring instrumentation
  • postgresql: requires dedicated monitoring instrumentation
  • redis: requires dedicated monitoring instrumentation

Recovery Complexity

  • 3 risk propagation path(s) complicate failure recovery

Maturity

Required

Advanced

Estimated

Advanced

Gap

No Gap

The architecture's required maturity (advanced) aligns with or is below the estimated team capability.

Operational Readiness

7

Adoption readiness: whether the team, infrastructure, and observability are prepared to run this architecture safely. Distinct from governance posture (above), which asks whether the topology itself violates architectural boundaries.

Not Readywrite heavy application

IoT Telemetry Ingestion Platform requires high operational expertise at 'experienced backend team' level. Current readiness estimate is 34%, critical gaps must be resolved before adoption. Consider starting with a simpler scenario and evolving toward this one.

Readiness Score

34%

Blocking Prerequisites

4

Complexity

High

Confidence

Strong

Assessment derived from scenario knowledge, advisor output, topology analysis, and 7 prerequisite checks.

Prerequisite Checklist (4 blocking, 3 non-blocking)

blocking

team

Team at 'experienced backend team' maturity level

This scenario is rated 'experienced backend team' complexity. Engineers with 2+ years of production backend experience, including database tuning and monitoring.

Gap signal: Team frequently reaches for external help during incidents or struggles to debug multi-system issues independently.

blocking

process

Failure mode awareness and runbooks

The team must understand the 6 documented failure modes for this scenario: hot_partition, write_amplification_cascade, wal_saturation, slow_consumer. Each should have a documented detection procedure and runbook.

Gap signal: The team has no documented runbooks for the scenario's failure modes or cannot name them without reference material.

blocking

monitoring

Production-grade observability stack

The scenario requires real-time metrics, structured logging, and distributed tracing on all critical components. Alerting must be configured before going live.

Gap signal: No dashboards exist for the critical path metrics in the scenario.

infrastructure

Minimum team maturity: Experienced Backend Team

This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.

Gap signal: The requirement 'Minimum team maturity: Experienced Backend Team' is not yet in place.

infrastructure

Runbooks and alerting for high-severity risks

5 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.

Gap signal: The requirement 'Runbooks and alerting for high-severity risks' is not yet in place.

infrastructure

Event stream operations expertise

This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.

Gap signal: The requirement 'Event stream operations expertise' is not yet in place.

blocking

infrastructure

Mitigation for 4 high-risk topology node(s)

Nodes with high or critical risk exposure: Time-Series Metrics, Event Streaming, Write-Heavy Transactional, Slow Consumer. Each requires documented mitigation before production deployment.

Gap signal: No mitigation strategy is documented for the high-risk nodes in the topology.

Infrastructure Requirements

ClickHouse

medium burden

Column-oriented OLAP database engineered for sub-second analytical queries on billions of rows, with vectorized execution, aggressive compression, and

Managed: ClickHouse Cloud, Altinity.Cloud

Apache Kafka

high burden

Distributed event streaming platform designed for high-throughput, fault-tolerant, ordered, and durable log-based messaging between producers and cons

Managed: Amazon MSK (Managed Streaming for Kafka), Confluent Cloud, Azure Event Hubs (Kafka-compatible), Redpanda Cloud

PostgreSQL

medium burden

ACID-compliant relational database with strong consistency, JSONB support, full-text search, and mature replication.

Managed: Amazon RDS for PostgreSQL, Amazon Aurora PostgreSQL, Google Cloud SQL for PostgreSQL, Azure Database for PostgreSQL, Supabase, Neon

Redis

low burden

In-memory key-value store with optional persistence, supporting strings, hashes, lists, sets, sorted sets, and pub/sub.

Managed: Amazon ElastiCache for Redis, Google Cloud Memorystore, Azure Cache for Redis, Redis Cloud, Upstash

TimescaleDB

medium burden

PostgreSQL extension that adds time-series-specific capabilities: automatic time-based partitioning (hypertables), columnar compression on cold chunks

Managed: Timescale Cloud, Amazon RDS (PostgreSQL + TimescaleDB extension), Supabase (TimescaleDB extension available)

Observability Requirements

Monitor generic risk probe signals

Seed 'WAL Saturation Risk Probe' identifies 2 metrics relevant to wal_saturation.

Seed 'WAL Saturation Risk Probe' identifies 2 metrics relevant to wal_saturation.

Monitor queue backlog signals

Seed 'Queue Consumer Backlog' identifies 4 metrics relevant to queue_backlog_accumulation.

Seed 'Queue Consumer Backlog' identifies 4 metrics relevant to queue_backlog_accumulation.

Track Hot Partition exposure

Hot Partition has high exposure and affects 0 components. Affects 0 nodes

Hot Partition has high exposure and affects 0 components. Affects 0 nodes

Track Write Amplification Cascade exposure

Write Amplification Cascade has high exposure and affects 0 components. Affects 0 nodes

Write Amplification Cascade has high exposure and affects 0 components. Affects 0 nodes

Track WAL Saturation exposure

WAL Saturation has high exposure and affects 1 component. Affects 1 node. (Write-Heavy Transactional)

WAL Saturation has high exposure and affects 1 component. Affects 1 node. (Write-Heavy Transactional)

TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; Tim

This signal indicates the architecture is approaching 'Tier 1: TimescaleDB Write Throughput Ceiling'. Likely bottleneck: TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration).

Tier 1: TimescaleDB Write Throughput Ceiling

Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate ele

This signal indicates the architecture is approaching 'Tier 2: Kafka Consumer Lag from Reconnect Storm'. Likely bottleneck: Kafka consumer pool sized for steady-state throughput, not burst from device reconnect storm; insufficient storage writer parallelism for burst absorption.

Tier 2: Kafka Consumer Lag from Reconnect Storm

TimescaleDB I/O saturation visible in disk throughput metrics during specific consumer lag drain periods; chunk decompre

This signal indicates the architecture is approaching 'Tier 3: Late-Arriving Data Chunk Decompression Cascade'. Likely bottleneck: Compressed chunk decompression triggered by late-arriving device data; at high device count, simultaneous decompression of many chunks saturates I/O.

Tier 3: Late-Arriving Data Chunk Decompression Cascade

Readiness Action Plan

Criticalteam

Satisfy: Team at 'experienced backend team' maturity level

Effort: 1–4 weeks depending on current state · Unblocks: Adoption of IoT Telemetry Ingestion Platform

Criticalprocess

Satisfy: Failure mode awareness and runbooks

Effort: 1–4 weeks depending on current state · Unblocks: Adoption of IoT Telemetry Ingestion Platform

Criticalmonitoring

Satisfy: Production-grade observability stack

Effort: 1–4 weeks depending on current state · Unblocks: Adoption of IoT Telemetry Ingestion Platform

Criticalinfrastructure

Satisfy: Mitigation for 4 high-risk topology node(s)

Effort: 1–4 weeks depending on current state · Unblocks: Adoption of IoT Telemetry Ingestion Platform

Highmonitoring

Instrument all critical path components with metrics and alerting

Effort: 1–2 weeks · Unblocks: Safe production adoption and incident response

Highprocess

Validate adoption in a staging environment before production

Effort: 2–4 weeks for thorough staging validation · Unblocks: Production confidence and rollback preparedness

Mediuminfrastructure

Mitigate risk: Hot Partition

Effort: 1–3 weeks · Unblocks: Reduces 'Hot Partition' from blocking adoption

Mediuminfrastructure

Mitigate risk: Write Amplification Cascade

Effort: 1–3 weeks · Unblocks: Reduces 'Write Amplification Cascade' from blocking adoption

Go Signals

  • Team has hands-on experience with all 5 referenced technologies.
  • All scenario failure modes have documented runbooks and alerting coverage.
  • A staging environment that mirrors production load has been tested successfully.

No-Go Signals

  • Team cannot explain or debug any of IoT Telemetry Ingestion Platform's documented failure modes.
  • No observability baseline exists for the critical components.
  • Top risk is unmitigated: 'Hot Partition', do not proceed without addressing this.

Critical Gaps

  • This scenario has high operational complexity, teams without deep production experience will struggle to operate it safely.

Team Requirements

ClickHouse operations

Required level: proficient

Team can explain ClickHouse's failure modes, tune configuration parameters under load, and recover from common operational issues.

Apache Kafka operations

Required level: proficient

Team can explain Apache Kafka's failure modes, tune configuration parameters under load, and recover from common operational issues.

PostgreSQL operations

Required level: proficient

Team can explain PostgreSQL's failure modes, tune configuration parameters under load, and recover from common operational issues.

Redis operations

Required level: proficient

Team can explain Redis's failure modes, tune configuration parameters under load, and recover from common operational issues.

TimescaleDB operations

Required level: proficient

Team can explain TimescaleDB's failure modes, tune configuration parameters under load, and recover from common operational issues.

Readiness assessment is derived from structured scenario and topology knowledge. It provides an evidence-grounded baseline, not a substitute for an actual team capability review or infrastructure audit. Validate each item against your specific environment.

Architectural Tradeoffs

3

Recommendations

12
High

Monitor: Hot Partition

risk_monitoring

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 0 nodes

High

Monitor: Write Amplification Cascade

risk_monitoring

Each logical application write triggers multiple physical writes through index maintenance, WAL generation, MVCC versioning, and replication, causing actual disk IOPS to exceed the provisioned I/O ceiling while the logical write rate appears modest.

Affects 0 nodes

High

Implement: Monitor generic risk probe signals

observability

Seed 'WAL Saturation Risk Probe' identifies 2 metrics relevant to wal_saturation.

Metrics to instrument: error_rate, p95_latency_ms

Moderate

Direct device writes to PostgreSQL with time-range partitioning → Kafka ingestion buffer + TimescaleDB consumer writers

migration_planning

Trigger: PostgreSQL write p99 > 100ms at sustained device fleet load; device write errors spiking during reconnect events (devices cannot block on write failure); time-range partition pruning queries slow due to too many manual partitions; need for native rollup views without manual aggregation jobs. Migrate from 'Direct device writes to PostgreSQL with time-range partitioning' to 'Kafka ingestion buffer + TimescaleDB consumer writers'. Deploy Kafka ingest endpoint alongside the existing PostgreSQL direct write path. Run both in parallel for 2 weeks with a 10% traffic canary to TimescaleDB, validating that data arrives correctly and rollup views are accurate before migrating 100% of device traffic to the Kafka path.

Kafka introduces an ingestion buffer that makes device data available in storage with 1–5 second lag; real-time alerting systems reading from TimescaleDB must tolerate this latency or migrate to Redis last-known-value reads; TimescaleDB continuous aggregate refresh policy must be configured before production use; unconfigured refresh means rollup views are never updated

Moderate

TimescaleDB as sole query layer for both real-time and historical queries → Redis last-known-value cache for real-time queries + TimescaleDB for historical queries

migration_planning

Trigger: Alerting system query latency > 500ms due to TimescaleDB query execution on current time chunk under write pressure; alert evaluation falling behind schedule; real-time dashboard queries competing with batch analytics queries on same TimescaleDB instance. Migrate from 'TimescaleDB as sole query layer for both real-time and historical queries' to 'Redis last-known-value cache for real-time queries + TimescaleDB for historical queries'. Write to Redis last-known-value in the Kafka consumer before writing to TimescaleDB. This ensures Redis is always as current as TimescaleDB or more current. The alerting system migrates to Redis reads incrementally, one alert rule at a time, with comparison validation against TimescaleDB during the transition period.

Redis last-known-value may lag TimescaleDB by up to Kafka consumer latency (1–5s); alerting systems must tolerate this latency window and must not treat a stale Redis value as ground truth for historical analysis; Redis key schema for device state must be defined once and treated as a contract; changes to the schema require coordinated deploy of both the Kafka consumer (writer) and alerting system (reader)

Moderate

Prepare runbook for: Burst Traffic Cold Cache Stampede

simulation_preparedness

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Moderate

Prepare runbook for: Connection Pool Exhaustion with Horizontal User Scale

simulation_preparedness

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Moderate

Plan evolution: OLTP Analytics Queries → OLTP + OLAP Separation

evolution_planning

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Moderate

Plan evolution: Single Cache Layer → Distributed Cache

evolution_planning

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Moderate

Cache-outage database fallback load

caching

'IoT Telemetry Ingestion Platform' includes a cache in its topology. If the cache becomes unavailable, the primary database receives the cache's full request load until the cache recovers.

Capacity-plan the primary database for this fallback load, not only for the steady-state cached load.

Moderate

Cache invalidation ownership

caching

Cache invalidation for IoT Telemetry Ingestion Platform is event-driven: kafka refreshes or invalidates redis. This couples cache freshness to consumer lag on that event stream, not to the primary write path directly.

If the event-stream consumer falls behind, the cache serves stale data until it catches up -- monitor consumer lag as a cache-freshness signal, not only a backlog signal.

Low

Monitor threshold: Tier 1: TimescaleDB Write Throughput Ceiling

scaling_monitoring

Signal: TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load

Bottleneck: TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration). Evolution: Tune TimescaleDB chunk_time_interval to match write cadence (smaller chunks = faster compression, lower WAL amplification per chunk); enable native compression on chunks older than 1 hour to reduce on-disk footprint; add a dedicated NVMe volume for WAL separate from data directory; consider TimescaleDB multi-node for horizontal write distribution across data nodes

Scaling Pressure Signals

8

TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load

Threshold

Tier 1: TimescaleDB Write Throughput Ceiling

Likely Bottleneck

TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration)

Recommended Evolution

Tune TimescaleDB chunk_time_interval to match write cadence (smaller chunks = faster compression, lower WAL amplification per chunk); enable native compression on chunks older than 1 hour to reduce on-disk footprint; add a dedicated NVMe volume for WAL separate from data directory; consider TimescaleDB multi-node for horizontal write distribution across data nodes

Evidence:kafka-consumer-lag-cascadepartition-hotspot-amplification

Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate elevated; TimescaleDB write thread pool fully saturated; Redis last-known-value update latency acceptable but historical storage significantly behind real-time; device reconnect event visible in device authentication logs correlating with lag spike

Threshold

Tier 2: Kafka Consumer Lag from Reconnect Storm

Likely Bottleneck

Kafka consumer pool sized for steady-state throughput, not burst from device reconnect storm; insufficient storage writer parallelism for burst absorption

Recommended Evolution

Pre-scale storage writer consumer replicas before anticipated high-risk windows (maintenance events, regional failovers); implement burst-aware consumer scaling using consumer group lag as the autoscale signal; tune Kafka consumer max.poll.records to batch storage INSERTs into TimescaleDB for higher per-consumer throughput (target 500–1000 rows per INSERT batch rather than single-row inserts)

Evidence:kafka-consumer-lag-cascadepartition-hotspot-amplification

TimescaleDB I/O saturation visible in disk throughput metrics during specific consumer lag drain periods; chunk decompression operations appearing in TimescaleDB logs (decompress_chunk); write latency spiking for historical time ranges (not current time chunk); device backlog replay operations (devices offline >1 hour) correlating with I/O spikes

Threshold

Tier 3: Late-Arriving Data Chunk Decompression Cascade

Likely Bottleneck

Compressed chunk decompression triggered by late-arriving device data; at high device count, simultaneous decompression of many chunks saturates I/O

Recommended Evolution

Implement a late-data ingest path separate from the real-time ingest path: late data (> 2 hours old by device timestamp) routes to a dedicated consumer that writes to a separate TimescaleDB hypertable with relaxed compression policy; this isolates late-data decompression I/O from the real-time write path; add monitoring alert when device timestamp delta vs. wall clock > 2 hours

Evidence:kafka-consumer-lag-cascadepartition-hotspot-amplification

Redis memory utilization > 80%; Redis INFO keyspace showing active device key count significantly exceeding expected active fleet size; Redis eviction rate positive (LRU evictions occurring on device state keys); last-known-value read miss rate rising; alerting system false-positives from missing device state

Threshold

Tier 4: Cardinality Explosion and Redis Memory Saturation

Likely Bottleneck

Redis key space growing unboundedly as devices are added without corresponding key expiry; inactive/retired devices retaining Redis keys indefinitely

Recommended Evolution

Enforce TTL on all device state Redis keys (set TTL = max expected device reporting interval * 3, e.g., for 60s devices: TTL = 180s); implement a device lifecycle event in Kafka (device_decommissioned) that explicitly deletes Redis keys; shard Redis by device_id hash range across cluster nodes if memory requirement after TTL enforcement still exceeds single-node capacity

Evidence:kafka-consumer-lag-cascadepartition-hotspot-amplification

TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load

Threshold

Escalation trigger: TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration)

Likely Bottleneck

Tier 1: TimescaleDB Write Throughput Ceiling

Recommended Evolution

Monitor: error_rate, p95_latency_ms, queue_depth

Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate elevated; TimescaleDB write thread pool fully saturated; Redis last-known-value update latency acceptable but historical storage significantly behind real-time; device reconnect event visible in device authentication logs correlating with lag spike

Threshold

Escalation trigger: Kafka consumer pool sized for steady-state throughput, not burst from device reconnect storm; insufficient storage writer parallelism for burst absorption

Likely Bottleneck

Tier 2: Kafka Consumer Lag from Reconnect Storm

Recommended Evolution

Monitor: error_rate, p95_latency_ms, queue_depth

TimescaleDB I/O saturation visible in disk throughput metrics during specific consumer lag drain periods; chunk decompression operations appearing in TimescaleDB logs (decompress_chunk); write latency spiking for historical time ranges (not current time chunk); device backlog replay operations (devices offline >1 hour) correlating with I/O spikes

Threshold

Escalation trigger: Compressed chunk decompression triggered by late-arriving device data; at high device count, simultaneous decompression of many chunks saturates I/O

Likely Bottleneck

Tier 3: Late-Arriving Data Chunk Decompression Cascade

Recommended Evolution

Monitor: error_rate, p95_latency_ms, queue_depth

Redis memory utilization > 80%; Redis INFO keyspace showing active device key count significantly exceeding expected active fleet size; Redis eviction rate positive (LRU evictions occurring on device state keys); last-known-value read miss rate rising; alerting system false-positives from missing device state

Threshold

Escalation trigger: Redis key space growing unboundedly as devices are added without corresponding key expiry; inactive/retired devices retaining Redis keys indefinitely

Likely Bottleneck

Tier 4: Cardinality Explosion and Redis Memory Saturation

Recommended Evolution

Monitor: error_rate, p95_latency_ms, queue_depth

Migration Readiness

12

Migration Stages

3
Stage

Direct device writes to PostgreSQL with time-range partitioning → Kafka ingestion buffer + TimescaleDB consumer writers

info

Migration trigger: PostgreSQL write p99 > 100ms at sustained device fleet load; device write errors spiking during reconnect events (devices cannot block on write failure); time-range partition pruning queries slow due to too many manual partitions; need for native rollup views without manual aggregation jobs

Stage

TimescaleDB as sole query layer for both real-time and historical queries → Redis last-known-value cache for real-time queries + TimescaleDB for historical queries

info

Migration trigger: Alerting system query latency > 500ms due to TimescaleDB query execution on current time chunk under write pressure; alert evaluation falling behind schedule; real-time dashboard queries competing with batch analytics queries on same TimescaleDB instance

Stage

TimescaleDB for both ingest storage and analytics queries → TimescaleDB for hot storage + ClickHouse for fleet analytics

info

Migration trigger: Multi-device aggregate queries (fleet-wide max/min/avg over 30-day windows) consuming > 30% of TimescaleDB CPU; analytics query p99 > 10s; analytics users and device ingest writers competing for I/O on the same TimescaleDB node; product requirement for fleet-wide queries that require full column scans

!

Risks

9
Risk

Kafka introduces an ingestion buffer that makes device data

warning

Kafka introduces an ingestion buffer that makes device data available in storage with 1–5 second lag; real-time alerting systems reading from TimescaleDB must tolerate this latency or migrate to Redis last-known-value reads

Risk

TimescaleDB continuous aggregate refresh policy must be conf

warning

TimescaleDB continuous aggregate refresh policy must be configured before production use; unconfigured refresh means rollup views are never updated

Risk

Redis last-known-value may lag TimescaleDB by up to Kafka co

warning

Redis last-known-value may lag TimescaleDB by up to Kafka consumer latency (1–5s); alerting systems must tolerate this latency window and must not treat a stale Redis value as ground truth for historical analysis

Risk

Redis key schema for device state must be defined once and t

warning

Redis key schema for device state must be defined once and treated as a contract; changes to the schema require coordinated deploy of both the Kafka consumer (writer) and alerting system (reader)

Risk

ClickHouse replication pipeline from TimescaleDB adds operat

warning

ClickHouse replication pipeline from TimescaleDB adds operational overhead; pipeline failures mean ClickHouse data is stale, and analytics users may not notice without explicit staleness indicators

Risk

ClickHouse query semantics differ from PostgreSQL/TimescaleD

warning

ClickHouse query semantics differ from PostgreSQL/TimescaleDB; analytics queries written for TimescaleDB may require rewrite to use ClickHouse-native functions

Risk

Projection lag creates a read-after-write window where users

critical

Projection lag creates a read-after-write window where users see stale data after their own writes. Mitigation: Route immediate post-write reads to the write store (session-scoped write token); accept eventual consistency only for non-user-initiated reads

direct-db-to-cqrs
Risk

Projection rebuild after schema change can take hours or day

critical

Projection rebuild after schema change can take hours or days on large datasets. Mitigation: Design blue/green projection deployment: build new projection in parallel before switching traffic; test rebuild time in staging

direct-db-to-cqrs
Risk

Cross-service workflows that previously used database transa

critical

Cross-service workflows that previously used database transactions now require Saga orchestration. Mitigation: Design idempotent event handlers; implement compensating transactions for every multi-step workflow; test failure injection in staging

modular-monolith-to-event-driven

Review Sections

6

Referenced Intelligence

Architecture Review: IoT Telemetry Ingestion Platform: DBRaven