Compare Scenarios

Migration Step 1

Both scenarios define a migration step at this stage. IoT Telemetry Ingestion Platform: triggered by 'PostgreSQL write p99 > 100ms at sustained device fleet load;'. Audit and Compliance Platform: triggered by 'Compliance audit finding that audit records were modified af'.

Migration Step 2

Both scenarios define a migration step at this stage. IoT Telemetry Ingestion Platform: triggered by 'Alerting system query latency > 500ms due to TimescaleDB que'. Audit and Compliance Platform: triggered by 'Compliance report generation taking > 5 minutes against Post'.

Migration Step 3

Both scenarios define a migration step at this stage. IoT Telemetry Ingestion Platform: triggered by 'Multi-device aggregate queries (fleet-wide max/min/avg over '. Audit and Compliance Platform: triggered by 'SIEM consumer lag causing it to fall behind retention window'.

TimescaleDB write latency p99 > 50ms for batch INSERT operations; pg_stat_activity showing wait events on WAL flush; TimescaleDB active chunk autovacuum running continuously; Kafka consumer group lag for storage writers growing steadily at baseline (non-storm) load

Tier 1: TimescaleDB Write Throughput Ceiling: TimescaleDB single-node write throughput ceiling (~50k–100k rows/second depending on row width and chunk size configuration). Recommended evolution: Tune TimescaleDB chunk_time_interval to match write cadence (smaller chunks = faster compression, lower WAL amplification per chunk); enable native compression on chunks older than 1 hour to reduce on-disk footprint; add a dedicated NVMe volume for WAL separate from data directory; consider TimescaleDB multi-node for horizontal write distribution across data nodes .

Kafka consumer group lag jumping from baseline (<100k) to >10M messages within minutes; Kafka broker disk write rate elevated; TimescaleDB write thread pool fully saturated; Redis last-known-value update latency acceptable but historical storage significantly behind real-time; device reconnect event visible in device authentication logs correlating with lag spike

Tier 2: Kafka Consumer Lag from Reconnect Storm: Kafka consumer pool sized for steady-state throughput, not burst from device reconnect storm; insufficient storage writer parallelism for burst absorption. Recommended evolution: Pre-scale storage writer consumer replicas before anticipated high-risk windows (maintenance events, regional failovers); implement burst-aware consumer scaling using consumer group lag as the autoscale signal; tune Kafka consumer max.poll.records to batch storage INSERTs into TimescaleDB for higher per-consumer throughput (target 500–1000 rows per INSERT batch rather than single-row inserts) .

PostgreSQL write p99 > 20ms with low connection count; pg_stat_activity showing transactions serialized on the same partition's chain-tip read; ingestion throughput plateauing well below hardware limits; auto_explain showing sequential scan on audit_events for "SELECT hash FROM audit_events ORDER BY id DESC LIMIT 1"

Tier 1: Integrity Chain Write Serialization: Per-partition chain-tip read before each insert serializing concurrent audit writers. Recommended evolution: Introduce partition-level chain sequence tables: a single row per partition tracking the current chain tip with an advisory lock, eliminating the full table read. Alternatively, shard the integrity chain by source system or tenant, accepting per-shard chains rather than a single global chain. Use PostgreSQL INSERT ... RETURNING with sequence-assigned IDs to eliminate the pre-insert read entirely, deferring chain hash computation to an async integrity sealer that appends hashes in order without blocking the write path. .

Compliance investigator queries returning in > 30s; PostgreSQL showing high sequential scan counts on audit_events partitions; investigator-facing API p99 > 10s; pg_stat_statements showing actor_id-scoped queries without partition pruning in the query plan

Tier 2: Actor Query Full-Partition Scan: Missing secondary index table for actor_id and resource_id lookup paths across time-partitioned audit data. Recommended evolution: Build a secondary index table audit_events_by_actor(actor_id, event_time, event_id) populated synchronously on insert. Accept the additional write per event as the cost of O(log n) actor-scoped queries. Alternatively, route actor-scoped queries to ClickHouse where columnar storage makes actor_id filters efficient without a secondary B-tree index. .