Summary
Kafka is the standard downstream target for WAL-based CDC pipelines: Debezium captures database WAL records and publishes them to Kafka topics, which downstream consumers process to maintain derived data stores, caches, and event-driven services.
Evidence
- ·Debezium (Red Hat) captures PostgreSQL, MySQL, and MongoDB WAL and publishes to Kafka topics
- ·Confluent Platform bundles Kafka Connect with Debezium connectors as a first-class CDC product
- ·LinkedIn's Databus (2013) was an early WAL-to-Kafka CDC system that inspired the Debezium architecture
- ·Kafka's log compaction enables CDC topics to serve as full-state snapshots for new consumers
- ·Netflix uses WAL-to-Kafka CDC to maintain Elasticsearch indexes in sync with their primary PostgreSQL databases
Operational Context
- ·Debezium replication slot on PostgreSQL must be monitored: a lagging or disconnected Debezium causes replication slot WAL accumulation on the primary
- ·{'Kafka topic naming convention for CDC': 'dbname.schemaname.tablename (Debezium default)'}
- ·Consumer groups on CDC topics must handle schema evolution: Confluent Schema Registry with AVRO provides compatibility
Tradeoffs
- ·Debezium replication slot holds WAL until consumed: disconnected Debezium can fill primary disk
- ·CDC events are row-level operations: application-level event semantics require transformation in a stream processor
- ·Ordering guarantees are per-partition only: cross-table ordering requires careful partition key strategy
Evidence grounding
Grounded, 5 supporting itemsThe Debezium + Kafka CDC pattern is the de facto standard for CDC-to-streaming, documented and used in production at hundreds of companies. Kafka's durability and consumer group model makes it the natural CDC sink.