DBRaven
Start Review →
Migration Playbook

RabbitMQ Message Queue→Kafka Event Stream

High complexityMedium riskComplex Rollback

Duration Estimate

1–3 months

Migration Stages

5

Blocking Prerequisites

3

Confidence

Strong

Readiness Checklist(3 blocking, 3 advisory)

blocking
Team

Team at senior maturity level

This migration requires senior engineering maturity. The team should have hands-on experience with the relevant technologies and proven ability to execute complex system changes safely.

Guidance: Build team skills through learning modules before attempting this migration.

blocking
Monitoring

Production observability baseline established

Comprehensive metrics, logs, and traces must be in place before the migration starts. You need a clear baseline to detect regressions during and after each migration stage.

Guidance: Instrument key metrics (latency p99, error rate, throughput, resource utilization) on all affected components.

Process

Rollback plan documented and tested

Rollback feasibility for this migration is 'complex'. Rollback is possible but may be complex, document the exact rollback steps and test them in a non-production environment.

Guidance: Run a full dry-run in a staging environment that mirrors production load characteristics.

blocking
Data

Data backup and validation strategy in place

This migration involves technology changes that affect data storage or access patterns. Ensure full backups are current and a data validation strategy is defined to confirm data integrity at each stage boundary.

Guidance: Perform a full backup before starting. Run data validation checksums after each stage.

Infrastructure

'kafka' provisioned and validated in target environment

'kafka' must be available, configured, and load-tested in the target environment before the migration stage that introduces it.

Guidance: Deploy 'kafka' with production-equivalent configuration. Run a smoke test under representative load.

Process

Migration trigger condition confirmed

Confirm that the actual trigger for this migration is present. Common triggers: Fan-out requirement: multiple consumers need the same event; RabbitMQ requires N queues or exchanges; Event replay needed: past events must be reprocessable for new consumers or data recovery; Consumer independence: consumer groups must advance at different rates without blocking each other.

Guidance: Validate the trigger with production metrics or benchmarks before committing to the migration timeline.

Migration Stages

1Kafka Cluster ProvisioningMedium2-3 weeks

Deploy 3-node Kafka cluster with replication factor 2-3. Configure retention policies per topic. Set up Schema Registry. Establish topic naming conventions. Deploy consumer lag monitoring before any consumers are connected.

Key tasks

  • ·Deploy 3-node Kafka cluster with replication factor 2-3. Configure retention policies per topic. Set up Schema Registry. Establish topic naming conventions. Deploy consumer lag monitoring before any consumers are connected.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Kafka Cluster Provisioning' validated in production with no regression in key metrics

Rollback: possible at this stage

2Dual PublishMedium2-3 weeks

Producers publish to both RabbitMQ and Kafka simultaneously. Consumers still read from RabbitMQ. Validate Kafka message delivery, offset ordering, and schema registry registration. Monitor for publish latency increase.

Key tasks

  • ·Producers publish to both RabbitMQ and Kafka simultaneously. Consumers still read from RabbitMQ. Validate Kafka message delivery, offset ordering, and schema registry registration. Monitor for publish latency increase.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Dual Publish' validated in production with no regression in key metrics

Rollback: possible at this stage

3Consumer Migration: New Consumers FirstMedium2-4 weeks

New consumer groups connecting for the first time read from Kafka only. Existing consumers continue reading RabbitMQ. Validate new consumer group lag, error rates, and processing semantics.

Key tasks

  • ·New consumer groups connecting for the first time read from Kafka only. Existing consumers continue reading RabbitMQ. Validate new consumer group lag, error rates, and processing semantics.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Consumer Migration: New Consumers First' validated in production with no regression in key metrics

Rollback: possible at this stage

4Existing Consumer MigrationHigh4-8 weeks

Migrate existing consumer groups from RabbitMQ to Kafka one by one. Each migration requires validating that Kafka consumer semantics match RabbitMQ consumer behavior (especially around error handling and retry logic).

Key tasks

  • ·Migrate existing consumer groups from RabbitMQ to Kafka one by one. Each migration requires validating that Kafka consumer semantics match RabbitMQ consumer behavior (especially around error handling and retry logic).
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Existing Consumer Migration' validated in production with no regression in key metrics

Rollback: possible at this stage

5RabbitMQ DecommissionMedium2 weeks

Stop dual publishing. Drain remaining RabbitMQ queues. Remove RabbitMQ infrastructure after confirming no consumers remain active.

Key tasks

  • ·Stop dual publishing. Drain remaining RabbitMQ queues. Remove RabbitMQ infrastructure after confirming no consumers remain active.
  • ·No rollback available after this stage, confirm all exit criteria before proceeding.

Exit criteria

  • ✓'RabbitMQ Decommission' validated in production with no regression in key metrics

Rollback: not available after this stage

Rollback Decision Points

After stage 4

Trigger: Any of the following in the 30 minutes after completing stage 4: p99 latency increases >50% from baseline, error rate exceeds 1%, or data consistency checks fail.

Procedure: Revert stage 4 changes. Rollback is complex, follow the documented rollback procedure for 'rabbitmq-to-kafka'. Engage senior engineering support.

Recovery: 1–4 hours (complex rollback)

After stage 5

Trigger: Final validation: confirm all success criteria are met within 24 hours of migration completion. If any success criterion fails, trigger rollback or remediation.

Procedure: If success criteria are not met within 24 hours, escalate to the architecture team. Decide between full rollback, partial rollback, or targeted remediation based on scope.

Recovery: 24–72 hours for full assessment and remediation

Success Criteria

  • ✓Target state 'Kafka Event Stream' is stable in production for 72+ hours with no regressions.
  • ✓p99 latency on all affected services is within acceptable range (≤110% of pre-migration baseline).
  • ✓Error rate is at or below pre-migration baseline.
  • ✓New components (kafka) are fully operational and monitored.
  • ✓Deprecated components (rabbitmq) are fully decommissioned with no remaining traffic.

This migration playbook is derived from structured architecture knowledge. Duration estimates and risk levels are structural approximations based on documented migration complexity and operational risk, not measured execution data. Validate all stages against your specific system constraints before executing.