DBRaven
Start Review →
Architecture Evolution Path

RabbitMQ → Kafka

High

Migrating from RabbitMQ's broker-centric push model to Kafka's log-based pull model to gain event replay, independent consumer scaling, fan-out without queue proliferation, and durable event retention: at the cost of higher operational complexity.

Topology Changes

Architecture Diff
+2 added1 modified
RabbitMQ Message Queue1 changes
→
Kafka Event Stream3 changes
Unchanged
Added
Removed
Modified

From

RabbitMQ Message Queue

→
4 mutations

To

Kafka Event Stream

Topology Mutations

Technology ReplacedMessage Broker

RabbitMQ broker replaced by Kafka cluster (typically 3-node minimum)

Operational Impact

Kafka requires ZooKeeper or KRaft coordination; broker failure tolerance requires 3+ brokers; replication factor ≥2

Component AddedSchema Registry

Confluent Schema Registry or Apicurio added to enforce message schema evolution

Operational Impact

Producers and consumers must register schemas; backward incompatible schema changes are rejected at publish time

Dependency Made AsyncConsumer Groups

RabbitMQ competing consumers replaced by Kafka consumer groups with independent offset tracking

Operational Impact

Consumer rebalances cause processing pauses; partition count determines maximum consumer parallelism

Component AddedConsumer Lag Monitor

Dedicated consumer lag monitoring required (Kafka Manager, Burrow, or Prometheus kafka_consumer_group_lag)

Operational Impact

Without lag monitoring, slow consumers accumulate silently until retention period expires and messages are lost

Migration Stages

1
Kafka Cluster Provisioning2-3 weeks

Deploy 3-node Kafka cluster with replication factor 2-3. Configure retention policies per topic. Set up Schema Registry. Establish topic naming conventions. Deploy consumer lag monitoring before any consumers are connected.

Medium risk·Rollback possible
2
Dual Publish2-3 weeks

Producers publish to both RabbitMQ and Kafka simultaneously. Consumers still read from RabbitMQ. Validate Kafka message delivery, offset ordering, and schema registry registration. Monitor for publish latency increase.

Medium risk·Rollback possible
3
Consumer Migration: New Consumers First2-4 weeks

New consumer groups connecting for the first time read from Kafka only. Existing consumers continue reading RabbitMQ. Validate new consumer group lag, error rates, and processing semantics.

Medium risk·Rollback possible
4
Existing Consumer Migration4-8 weeks

Migrate existing consumer groups from RabbitMQ to Kafka one by one. Each migration requires validating that Kafka consumer semantics match RabbitMQ consumer behavior (especially around error handling and retry logic).

High risk·Rollback possible
5
RabbitMQ Decommission2 weeks

Stop dual publishing. Drain remaining RabbitMQ queues. Remove RabbitMQ infrastructure after confirming no consumers remain active.

Medium risk·No rollback after this stage

Migration Risks

operationalCritical

Consumer lag accumulation goes undetected without monitoring: messages expire from Kafka retention before processing

Mitigation

Deploy consumer lag alerting before migrating any consumer; alert on lag rate-of-change, not absolute depth

consistencyWarning

Kafka at-least-once delivery semantics differ from RabbitMQ: duplicate messages are possible under consumer failure

Mitigation

Make all consumer handlers idempotent; use idempotency keys in message payloads; test duplicate processing in staging

operationalWarning

Partition rebalance during consumer deployment pauses processing for all partitions in the group

Mitigation

Use Kafka's cooperative-sticky assignor; implement static membership for stable consumer groups

couplingWarning

Schema registry creates a shared dependency: schema changes require consumer coordination

Mitigation

Enforce backward-compatible schema evolution only; use schema compatibility checks in CI

Coupling Changes

consumer couplingDecreases

Consumer groups advance independently: a slow consumer does not block other consumer groups

Consequence

Consumer groups can be added without touching producer code or creating new queues

schema couplingIncreases

Schema Registry enforces shared schema: all producers and consumers share a schema contract

Consequence

Schema incompatibility is caught at publish time: coordinated schema evolution is required

operational couplingIncreases

Kafka cluster health affects all producers and consumers simultaneously

Consequence

Kafka broker failure affects the entire event stream: cluster redundancy and replication factor become critical

Consistency Model Changes

  • ·Kafka guarantees at-least-once delivery: consumer logic must handle duplicate messages
  • ·Exactly-once semantics available but require transactional producers and idempotent consumers
  • ·Message ordering guaranteed within a partition: cross-partition ordering not guaranteed
  • ·Consumer lag introduces an eventual consistency window between producer commit and consumer processing

Rollback Risks

  • ·Messages consumed from Kafka cannot be un-consumed: offset advancement is permanent
  • ·If dual publish is stopped before full consumer migration, RabbitMQ consumers will miss Kafka-only messages
  • ·RabbitMQ queue configuration and bindings must be preserved during migration for rapid rollback