Full ReviewModerate Readinessdraft

Architecture Review: Event-Driven Analytics Pipeline

A streaming architecture that captures database changes via WAL-based CDC, publishes them to an event stream (Kafka), and routes them to analytics consumers. Decouples the write path from the read path while maintaining a durable, replayable event log.

Evidence Confidence

Limited

moderate

Executive Summary

Event-Driven Analytics Pipeline: moderate operational readiness (73% evidence confidence). 0 architectural strengths identified, 1 operational risk to manage. Primary concern: Replication Lag Cascade. Requires Advanced operational maturity.

Readiness Rationale

Overall moderate readiness across 8 dimensions. Limited: observability, team maturity. Strong: operational, topology resilience, failure recovery.

Key Concerns

!Replication Lag Cascade

Key Strengths

+Architecture is well-defined for the event driven system problem profile

Assessments

Tradeoffs

Sections

Recommendations

Readiness Assessments

Architectural Tradeoffs

Recommendations

Moderate

Direct database queries serving analytics workloads → Polling-based ETL from read replica to analytics database

migration_planning

Trigger: OLTP query performance degrading due to analytics query interference. Migrate from 'Direct database queries serving analytics workloads' to 'Polling-based ETL from read replica to analytics database'. Polling-based ETL is a reasonable first step before committing to stream-based CDC infrastructure.

Read replica lag degrades analytics freshness during heavy write periods; ETL polling interval creates minimum latency floor for analytics data

Moderate

Polling-based ETL from read replica → WAL CDC → Kafka → analytics consumers

migration_planning

Trigger: Sub-minute analytics latency required; ETL scheduling overhead becoming operational burden; need for replayable event history. Migrate from 'Polling-based ETL from read replica' to 'WAL CDC → Kafka → analytics consumers'. This transition is a significant infrastructure investment. It delivers replayability, decoupling, and sub-second analytics latency in exchange for substantial operational complexity.

Kafka operational complexity significantly higher than ETL polling; CDC setup requires PostgreSQL logical replication configuration and careful slot monitoring

Moderate

Prepare runbook for: Burst Traffic Cold Cache Stampede

simulation_preparedness

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

◎ burst-traffic-cold-cache-stampede

Moderate

Prepare runbook for: Connection Pool Exhaustion with Horizontal User Scale

simulation_preparedness

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

◎ connection-pool-growth-with-user-scale

Moderate

Plan evolution: OLTP Analytics Queries → OLTP + OLAP Separation

evolution_planning

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

↗ oltp-analytics-to-separated

Moderate

Plan evolution: PostgreSQL → Partitioned PostgreSQL

evolution_planning

Evolution from Single-Node PostgreSQL → Partitioned PostgreSQL

Migration complexity: high. Rollback: rarely.

↗ postgresql-to-partitioned

Low

Monitor threshold: Tier 1: CDC Slot Lag

scaling_monitoring

Signal: pg_replication_slots shows growing pg_wal_lsn delta for CDC slot; PostgreSQL WAL directory growing faster than expected

Bottleneck: Debezium / CDC connector not keeping up with write volume. Evolution: Increase CDC connector parallelism; review filtered topics vs full-table CDC; monitor slot lag as a first-class SLA

Low

Monitor threshold: Tier 2: Kafka Consumer Lag

scaling_monitoring

Signal: Kafka consumer group lag growing; analytics dashboards increasingly stale; consumer CPU and network I/O near ceiling

Bottleneck: Insufficient consumer parallelism or insufficient Kafka partitions. Evolution: Increase topic partition count (note: keyed messages lose ordering when partitions added); add consumer replicas up to partition count

Scaling Pressure Signals

↑

pg_replication_slots shows growing pg_wal_lsn delta for CDC slot; PostgreSQL WAL directory growing faster than expected

Threshold

Tier 1: CDC Slot Lag

Likely Bottleneck

Debezium / CDC connector not keeping up with write volume

Recommended Evolution

Increase CDC connector parallelism; review filtered topics vs full-table CDC; monitor slot lag as a first-class SLA

↑

Kafka consumer group lag growing; analytics dashboards increasingly stale; consumer CPU and network I/O near ceiling

Threshold

Tier 2: Kafka Consumer Lag

Likely Bottleneck

Insufficient consumer parallelism or insufficient Kafka partitions

Recommended Evolution

Increase topic partition count (note: keyed messages lose ordering when partitions added); add consumer replicas up to partition count

Evidence:kafka-consumer-lag-cascadepartition-hotspot-amplification

↑

Analytics consumers failing deserialization; event count drops for specific topics; schema registry (if in use) reports compatibility violations

Threshold

Tier 3: Schema Evolution Crisis

Likely Bottleneck

Unmanaged DDL changes breaking Avro or JSON schema contracts

Recommended Evolution

Adopt schema registry with backward-compatible evolution policy; enforce schema review as part of migration deployment

↑

pg_replication_slots shows growing pg_wal_lsn delta for CDC slot; PostgreSQL WAL directory growing faster than expected

Threshold

Escalation trigger: Debezium / CDC connector not keeping up with write volume

Likely Bottleneck

Tier 1: CDC Slot Lag

Recommended Evolution

Monitor:

↑

Kafka consumer group lag growing; analytics dashboards increasingly stale; consumer CPU and network I/O near ceiling

Threshold

Escalation trigger: Insufficient consumer parallelism or insufficient Kafka partitions

Likely Bottleneck

Tier 2: Kafka Consumer Lag

Recommended Evolution

Monitor:

↑

Analytics consumers failing deserialization; event count drops for specific topics; schema registry (if in use) reports compatibility violations

Threshold

Escalation trigger: Unmanaged DDL changes breaking Avro or JSON schema contracts

Likely Bottleneck

Tier 3: Schema Evolution Crisis

Recommended Evolution

Monitor:

Migration Readiness

→

Migration Stages

Stage

Direct database queries serving analytics workloads → Polling-based ETL from read replica to analytics database

info

Migration trigger: OLTP query performance degrading due to analytics query interference

Stage

Polling-based ETL from read replica → WAL CDC → Kafka → analytics consumers

info

Migration trigger: Sub-minute analytics latency required; ETL scheduling overhead becoming operational burden; need for replayable event history

Risks

Risk

Read replica lag degrades analytics freshness during heavy w

warning

Read replica lag degrades analytics freshness during heavy write periods

Risk

ETL polling interval creates minimum latency floor for analy

warning

ETL polling interval creates minimum latency floor for analytics data

Risk

Kafka operational complexity significantly higher than ETL p

warning

Kafka operational complexity significantly higher than ETL polling

Risk

CDC setup requires PostgreSQL logical replication configurat

warning

CDC setup requires PostgreSQL logical replication configuration and careful slot monitoring

Risk

Cross-service workflows that previously used database transa

critical

Cross-service workflows that previously used database transactions now require Saga orchestration. Mitigation: Design idempotent event handlers; implement compensating transactions for every multi-step workflow; test failure injection in staging

↗ modular-monolith-to-event-driven

Risk

Consumer lag silently accumulates: a lagging consumer is not

critical

Consumer lag silently accumulates: a lagging consumer is not a failed consumer. Mitigation: Alert on consumer lag rate-of-change, not absolute depth; implement dead letter queues with alerting

↗ modular-monolith-to-event-driven

Risk

Missing partition for current time window causes all INSERTs

critical

Missing partition for current time window causes all INSERTs to fail with 'no partition of relation found'. Mitigation: Create partitions 7-30 days in advance; alert when next partition does not exist before its time window opens

↗ postgresql-to-partitioned

Risk

Historical data migration batch failures can leave partial d

critical

Historical data migration batch failures can leave partial data in partitioned table. Mitigation: Validate row counts and checksums per partition before dropping old table; keep old table for 30+ days after cutover

↗ postgresql-to-partitioned

Risk

Consumer lag accumulation goes undetected without monitoring

critical

Consumer lag accumulation goes undetected without monitoring: messages expire from Kafka retention before processing. Mitigation: Deploy consumer lag alerting before migrating any consumer; alert on lag rate-of-change, not absolute depth

↗ rabbitmq-to-kafka

Risk

Read-after-write violations are invisible to monitoring but

critical

Read-after-write violations are invisible to monitoring but visible to users: 'my change disappeared'. Mitigation: Track write LSN per user session; route reads to primary until replica confirms that LSN; accept primary load increase

↗ single-region-to-multi-region

Review Sections

Referenced Intelligence

kafkapostgresql◎ burst-traffic-cold-cache-stampede◎ connection-pool-growth-with-user-scale◎ cqrs-projection-lag-expansion◎ cross-region-stale-read-window◎ distributed-cache-invalidation-failure◎ event-replay-storm-recovery◎ kafka-consumer-lag-cascade◎ multi-tenant-noisy-neighbor◎ partition-hotspot-amplification◎ postgresql-replication-lag-surge◎ query-cost-without-indexes◎ read-amplification-n-plus-one-queries◎ redis-cache-collapse-stampede◎ retry-storm-amplification◎ split-brain-during-network-partition◎ storage-bloat-without-archiving◎ storage-cost-compounding-without-retention◎ write-heavy-bulk-import-saturation↗ modular-monolith-to-event-driven↗ oltp-analytics-to-separated↗ postgresql-to-partitioned↗ rabbitmq-to-kafka↗ single-region-to-multi-region◈ architecture-evolution◈ auditability◈ btree-indexing◈ cache-invalidation◈ cap-theorem◈ consistency-models◈ cqrs-operational◈ event-sourcing◈ eventual-consistency◈ kafka-consumer-lag◈ multi-tenancy◈ normalization◈ oltp-vs-olap◈ partition-hotspots◈ query-planning◈ queue-backlog◈ replication-lag◈ vector-databases◈ write-amplification