Architecture Review: Geospatial Tracking Platform

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 0 nodes

Each logical application write triggers multiple physical writes through index maintenance, WAL generation, MVCC versioning, and replication, causing actual disk IOPS to exceed the provisioned I/O ceiling while the logical write rate appears modest.

Affects 0 nodes

Seed 'Disk I/O Saturation Risk Probe' identifies 2 metrics relevant to disk_io_saturation.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: PostGIS proximity query p99 > 100ms under concurrent fleet tracking; PostgreSQL table size for location_history exceeding 500GB; VACUUM unable to keep pace with high-frequency insert + update pattern on the live position table; live position queries competing with historical analytics queries on the same table. Migrate from 'PostgreSQL with PostGIS extension for both live position queries and historical storage' to 'Redis geospatial index for live positions, TimescaleDB for historical time-series'. Migrate historical data to TimescaleDB first, validating time-range query performance. Migrate the live position surface to Redis second, running dual-write for 2 weeks. Only decommission PostGIS proximity queries after Redis latency is validated under production fleet write rate.

Redis does not persist position data durably by default: AOF or RDB must be configured before Redis becomes the live position surface; Migration requires maintaining both PostgreSQL PostGIS and Redis in sync during the transition period; a dual-write period is required to validate Redis proximity query correctness

Trigger: Location update API p99 > 200ms correlated with geofence count growth; geofence evaluation CPU dominating ingestion service profiling; requirement to independently scale ingestion throughput from geofence evaluation capacity. Migrate from 'Synchronous geofence evaluation in the HTTP write handler' to 'Kafka-based asynchronous geofence evaluation consumer'. The geofence evaluation consumer must be idempotent: position updates may be replayed from Kafka on consumer restart. Implement idempotency using position update event_id as the deduplication key in the geofence event emission logic.

Asynchronous evaluation means geofence events are emitted with a lag relative to the position update: dispatch systems expecting synchronous geofence confirmation must be updated to consume geofence events from Kafka rather than the API response; Geofence evaluation consumer lag must be monitored; a stalled consumer means geofence events stop being emitted silently, which dispatch systems may not detect immediately

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Redis used_memory > 75% of maxmemory; Redis evictions appearing in INFO stats; GEOSEARCH returning stale or missing entity positions; Redis OOM errors in application logs during fleet expansion events; proximity query latency increasing above 10ms baseline

Bottleneck: Redis memory exhausted by unbounded geospatial entity growth without entity expiry or cleanup. Evolution: Implement entity-scoped Redis key TTL tied to the last received update timestamp. Entities that have not sent a position update in > 5 minutes are expired from Redis automatically (Redis EXPIRE on the ZSET entry using a per-entity auxiliary key pattern, since ZSET members do not support per-member TTL natively). Alternatively, introduce a background reconciliation job that removes entities from the live position surface after an inactivity threshold. Shard the geospatial index across multiple Redis instances by geographic region using consistent hashing on the region key.

Signal: TimescaleDB write p99 > 50ms; WAL volume > 200MB/minute sustained; disk I/O utilization > 80% on TimescaleDB data volume; chunk creation log entries during fleet expansion events correlated with write latency spikes; TimescaleDB worker queue depth growing during ingestion bursts

Bottleneck: TimescaleDB hypertable chunk write throughput saturated by high-frequency location update volume; chunk creation DDL causing write stalls during expansion. Evolution: Tune TimescaleDB chunk_time_interval to match the ingestion rate: smaller chunks (1-hour intervals instead of 1-day) reduce per-chunk write volume but increase chunk creation frequency. Use timescaledb-parallel-copy for bulk historical ingestion. Move the TimescaleDB WAL to a dedicated NVMe volume. Introduce write batching at the application layer: buffer 500ms of position updates per entity and write as a single multi-row INSERT, reducing the per-update overhead from N single-row INSERTs to N/batch_size batch INSERTs.