Batch ETL → Streaming CDC Pipeline
HighReplacing nightly or hourly batch ETL jobs with a continuous CDC pipeline that captures database writes from the source WAL, publishes change events to Kafka, and delivers updates to analytics systems within seconds: at the cost of significantly higher operational complexity, schema evolution management, and permanent consumer lag monitoring.
Topology Changes
From
Nightly Batch ETL Jobs
To
Continuous CDC Streaming Pipeline
Topology Mutations
Kafka serves as the durable event bus between the CDC source connector and stream processing consumers. Each source table maps to a Kafka topic. Events are retained for a configurable retention period (typically 7–30 days), enabling consumer replay from any point in history.
Operational Impact
Kafka consumer lag becomes a critical operational metric. Unmonitored consumer lag accumulates silently: a stalled consumer can fall hours or days behind without any user-visible error until the analytics data is investigated.
A stream processing framework enriches, aggregates, and transforms CDC events before writing to the analytics destination. Flink provides exactly-once semantics end-to-end with transactional sinks. Spark Streaming provides micro-batch processing with simpler operational model but higher latency (30s–2min).
Operational Impact
Stream processor checkpoint failures leave the processor in an unknown state: must restart from last checkpoint. Checkpoint storage must be persistent and monitored. Checkpoint failure silently pauses processing without immediate user-visible consequence.
Nightly or hourly cron-driven extraction jobs (Airflow DAGs, custom scripts, or managed ETL) are decommissioned once streaming pipeline has proven accuracy and stability over a parallel run period.
Operational Impact
Batch job failure detection (a morning alert when the 2AM job failed) is replaced by continuous consumer lag monitoring. The failure mode shifts from periodic discrete job failures to continuous degradation that requires alert threshold tuning.
Debezium reads the PostgreSQL WAL via a logical replication slot and publishes row-level change events (INSERT, UPDATE, DELETE) to Kafka topics. Each event includes the before and after row state, operation type, and transaction metadata.
Operational Impact
The replication slot on PostgreSQL holds WAL segments until Debezium acknowledges them. If Debezium falls behind or its Kafka consumer lags, WAL accumulates on the source database disk: this can fill the disk and crash the database.
SQL extraction queries that periodically read all changed rows are replaced by event stream consumption. The analytics pipeline now reacts to each individual row change as it happens rather than polling for bulk changes on a schedule.
Operational Impact
Out-of-order event delivery is possible: a DELETE event may arrive before the corresponding INSERT if they came from different Kafka partitions. Stream processors must handle this using event-time windowing and watermarks.
Migration Stages
Deploy Kafka cluster. Configure Debezium connector for the source PostgreSQL database : create a replication slot, configure wal_level=logical, and start the connector. Validate that CDC events are flowing to Kafka topics: verify event format, field names, and schema representation. Monitor replication slot WAL size from day one.
Implement the Flink or Spark Streaming application that transforms CDC events into the target analytics schema. Run it in parallel with the existing batch job: both pipelines write to separate destinations (or use separate target tables). Compare output for parity.
Wire the stream processor to write to the production analytics destination. Run batch and streaming outputs in parallel for the same time window. Compare row counts, aggregate values, and spot-check individual records. Identify and resolve discrepancies before any traffic is cut over.
Run batch and streaming in parallel for 2–4 weeks. Build confidence in streaming accuracy across full business cycles (month-end, peak traffic periods, weekend load). Tune Flink watermarks and late-event handling. Validate that consumer lag stays bounded during peak source write periods.
Switch analytics dashboards and downstream consumers to the streaming-backed data. Deprecate the batch job: keep it on standby but do not run it in production. Monitor consumer lag, checkpoint health, and data freshness SLA continuously.
After 30+ days of stable streaming operation, remove the batch ETL job. Update runbooks to reflect streaming operational model: consumer lag monitoring replaces batch job execution monitoring.
Migration Risks
The Debezium replication slot holds PostgreSQL WAL until Debezium commits its offset. If Debezium's Kafka consumer lags: due to Kafka slowness, backpressure, or Debezium failure: the replication slot retains WAL indefinitely. Unmonitored WAL accumulation can fill the source database disk, causing a full database outage.
Mitigation
Alert on replication slot retained_bytes exceeding 5GB. Configure a maximum WAL retention limit (max_slot_wal_keep_size in PostgreSQL 13+) to prevent unbounded accumulation. If the slot must be dropped to recover disk space, the CDC pipeline must be rebuilt from a full snapshot: plan for this as a recovery procedure.
Source table schema changes (new column added, column renamed, type changed) break the CDC event schema unless schema evolution is handled via a schema registry. Debezium emits events using the table's schema at the time of the change: downstream consumers must handle schema evolution gracefully.
Mitigation
Use Avro serialization with Confluent Schema Registry (or equivalent). Register schemas for all CDC event types before changes are applied. Configure Debezium to use Avro serialization. Test schema evolution in staging before any production source schema change.
Events from the same transaction may arrive on different Kafka partitions if the stream is partitioned by table rather than by transaction. A record updated in a transaction with a record in a different table may arrive at the consumer out of order relative to the other table's event.
Mitigation
Partition Kafka topics by the business entity key (user_id, order_id) so that related events for the same entity arrive on the same partition and are processed in order. Use Flink event-time processing with watermarks to handle any residual out-of-order delivery.
Coupling Changes
Analytics pipeline is decoupled from the batch schedule: data flows continuously rather than in discrete windows
Consequence
Analytics data freshness is bounded by pipeline latency (seconds) rather than batch schedule (hours)
Source database and CDC pipeline are now coupled via the replication slot: source database disk health depends on pipeline health
Consequence
CDC pipeline failure directly threatens source database operational stability if WAL accumulates unchecked
CDC events encode the source table schema: source schema changes propagate to the event stream
Consequence
Source schema changes must be coordinated with the CDC pipeline to avoid breaking downstream consumers
Consistency Model Changes
- ·Analytics data is eventually consistent with source: streaming lag of 1–30 seconds under normal load
- ·Exactly-once semantics require deliberate design: Flink transactional sinks and Kafka exactly-once producer configuration
- ·Most production streaming pipelines operate at-least-once with idempotent writes to the analytics destination
- ·Batch jobs produced a complete, consistent snapshot at a point in time; streaming produces a continuous approximate view that may reflect partial transactions
Rollback Risks
- ·Reverting to batch ETL after decommissioning requires rebuilding the extraction scripts and Airflow DAGs from documentation
- ·Analytics consumers that have been redesigned around streaming freshness SLAs will not tolerate reverting to nightly batch
- ·If the Debezium replication slot is dropped during rollback, CDC history is lost: the pipeline must be rebuilt from a full table snapshot