Use Event-Driven Analytics Pipeline as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Event-Driven Analytics Pipeline. Traceable to YAML knowledge entities.
Context
Operational databases accumulate write-heavy OLTP traffic that cannot simultaneously serve analytics queries without degrading transactional performance. A separate analytics path is required that consumes database changes asynchronously, provides replayable history, and allows analytics consumers to scale independently of the primary database. Primary operational risks include: Stale replication slot: an inactive Kafka consumer causes PostgreSQL to retain all WAL since slot creation, filling disk; Schema evolution breaks CDC consumers: DDL changes (column renames, drops) produce incompatible events; Consumer lag accumulation: slow analytics consumers fall behind and cannot catch up under sustained write load.
Decision
We will adopt the **Event-Driven Analytics Pipeline** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
A streaming architecture that captures database changes via WAL-based CDC, publishes them to an event stream (Kafka), and routes them to analytics consumers. Decouples the write path from the read path while maintaining a durable, replayable event log. Core technology stack: postgresql, kafka.
Accepted Tradeoffs
- ⚠WAL-based CDC adds replication overhead to the primary PostgreSQL instance
- ⚠Kafka is high-operational-complexity infrastructure (partition sizing, ISR, retention, consumer groups)
- ⚠Analytics consumers receive eventual consistency: lag between write and read is inherent to the architecture
- ⚠Schema evolution must be planned as a first-class operational concern, not an afterthought
Risks
Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Event-Driven Analytics Pipeline is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Event-Driven Analytics Pipeline is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Event-Driven Analytics Pipeline is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Event-Driven Analytics Pipeline is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: CDC Slot Lag
Signal: pg_replication_slots shows growing pg_wal_lsn delta for CDC slot; PostgreSQL WAL directory growing faster than expected
Evolution: Increase CDC connector parallelism; review filtered topics vs full-table CDC; monitor slot lag as a first-class SLA
Tier 2: Kafka Consumer Lag
Signal: Kafka consumer group lag growing; analytics dashboards increasingly stale; consumer CPU and network I/O near ceiling
Evolution: Increase topic partition count (note: keyed messages lose ordering when partitions added); add consumer replicas up to partition count
Tier 3: Schema Evolution Crisis
Signal: Analytics consumers failing deserialization; event count drops for specific topics; schema registry (if in use) reports compatibility violations
Evolution: Adopt schema registry with backward-compatible evolution policy; enforce schema review as part of migration deployment
Migration Path
Direct database queries serving analytics workloads → Polling-based ETL from read replica to analytics database
OLTP query performance degrading due to analytics query interference
Polling-based ETL from read replica → WAL CDC → Kafka → analytics consumers
Sub-minute analytics latency required; ETL scheduling overhead becoming operational burden; need for replayable event history
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Apache Kafka: scenario uses Kafka for event streaming or CDC: Set min.insync.replicas=2 with acks=all; monitor consumer lag as primary health signal
- Apache Kafka: scenario has team_maturity below senior: Kafka operational complexity requires dedicated expertise: consider MSK or Confluent Cloud to reduce ops burden