DBRaven
Start Review →
Migration Playbook

Synchronous HTTP Service Dependencies→Event-Driven Architecture with Message Broker

High complexityHigh riskComplex Rollback

Duration Estimate

1–3 months

Migration Stages

6

Blocking Prerequisites

2

Confidence

Strong

Readiness Checklist(2 blocking, 2 advisory)

blocking
Team

Team at senior maturity level

This migration requires senior engineering maturity. The team should have hands-on experience with the relevant technologies and proven ability to execute complex system changes safely.

Guidance: Build team skills through learning modules before attempting this migration.

blocking
Monitoring

Production observability baseline established

Comprehensive metrics, logs, and traces must be in place before the migration starts. You need a clear baseline to detect regressions during and after each migration stage.

Guidance: Instrument key metrics (latency p99, error rate, throughput, resource utilization) on all affected components.

Process

Rollback plan documented and tested

Rollback feasibility for this migration is 'complex'. Rollback is possible but may be complex, document the exact rollback steps and test them in a non-production environment.

Guidance: Run a full dry-run in a staging environment that mirrors production load characteristics.

Process

Migration trigger condition confirmed

Confirm that the actual trigger for this migration is present. Common triggers: Synchronous call failures in downstream services cascade to callers: a slow inventory service degrades order checkout; Scaling one service is blocked by dependent services that cannot handle increased call volume; Adding a new consumer of a domain event requires changing the producing service's code and redeploying it.

Guidance: Validate the trigger with production metrics or benchmarks before committing to the migration timeline.

Migration Stages

1Classify Calls as Commands vs NotificationsLow2–3 weeks

Audit every service-to-service call. Commands (the caller requires the callee's result to continue: payment, authorization, synchronous validation) cannot be made async. Notifications (the caller does not wait for the result: send email, update cache, trigger analytics event) are candidates for async messaging. Document each call with its classification and rationale.

Key tasks

  • ·Audit every service-to-service call. Commands (the caller requires the callee's result to continue: payment, authorization, synchronous validation) cannot be made async. Notifications (the caller does not wait for the result: send email, update cache, trigger analytics event) are candidates for async messaging. Document each call with its classification and rationale.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Classify Calls as Commands vs Notifications' validated in production with no regression in key metrics

Rollback: possible at this stage

2Deploy Message Broker and ObservabilityLow2–3 weeks

Deploy Kafka cluster with consumer lag monitoring (Prometheus + Kafka exporter) from day one. Establish lag alerting thresholds. Create topics for the first migration candidate. Do not route any production traffic to the broker yet: validate operator tooling, topic configuration, and monitoring in staging.

Key tasks

  • ·Deploy Kafka cluster with consumer lag monitoring (Prometheus + Kafka exporter) from day one. Establish lag alerting thresholds. Create topics for the first migration candidate. Do not route any production traffic to the broker yet: validate operator tooling, topic configuration, and monitoring in staging.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Deploy Message Broker and Observability' validated in production with no regression in key metrics

Rollback: possible at this stage

3Implement Outbox Pattern on ProducerMedium2–4 weeks

The outbox pattern ensures event publication is atomic with the business database write. Write the event to an outbox table in the same transaction as the business mutation. A separate relay process reads the outbox and publishes to Kafka. This eliminates the dual-write inconsistency where the database write succeeds but the Kafka publish fails.

Key tasks

  • ·The outbox pattern ensures event publication is atomic with the business database write. Write the event to an outbox table in the same transaction as the business mutation. A separate relay process reads the outbox and publishes to Kafka. This eliminates the dual-write inconsistency where the database write succeeds but the Kafka publish fails.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Implement Outbox Pattern on Producer' validated in production with no regression in key metrics

Rollback: possible at this stage

4Build Idempotent ConsumersMedium2–4 weeks

Kafka delivers at-least-once. Every consumer must handle duplicate delivery without producing duplicate side effects. Implement idempotency via deduplication keys stored in the consumer's database (processed_event_ids). Test duplicate delivery explicitly in staging: replaying the same event twice must produce the same outcome.

Key tasks

  • ·Kafka delivers at-least-once. Every consumer must handle duplicate delivery without producing duplicate side effects. Implement idempotency via deduplication keys stored in the consumer's database (processed_event_ids). Test duplicate delivery explicitly in staging: replaying the same event twice must produce the same outcome.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Build Idempotent Consumers' validated in production with no regression in key metrics

Rollback: possible at this stage

5Dual-Run PeriodHigh2–4 weeks

Activate the event path while keeping the old synchronous call in place. Validate that consumers produce the same state outcomes as the synchronous path. Monitor consumer lag and event ordering. Identify any cases where event ordering is required but not guaranteed by the partition assignment.

Key tasks

  • ·Activate the event path while keeping the old synchronous call in place. Validate that consumers produce the same state outcomes as the synchronous path. Monitor consumer lag and event ordering. Identify any cases where event ordering is required but not guaranteed by the partition assignment.
  • ·Validate rollback capability at this stage boundary before proceeding.

Exit criteria

  • ✓'Dual-Run Period' validated in production with no regression in key metrics

Rollback: possible at this stage

6Remove Synchronous PathMedium1–2 weeks

After consumer parity is validated, remove the old synchronous call from the producer. The producer now only publishes the event. Monitor consumer lag closely for 30 days after removal: this is when silent consumer failures become visible.

Key tasks

  • ·After consumer parity is validated, remove the old synchronous call from the producer. The producer now only publishes the event. Monitor consumer lag closely for 30 days after removal: this is when silent consumer failures become visible.
  • ·No rollback available after this stage, confirm all exit criteria before proceeding.

Exit criteria

  • ✓'Remove Synchronous Path' validated in production with no regression in key metrics

Rollback: not available after this stage

Rollback Decision Points

After stage 5

Trigger: Any of the following in the 30 minutes after completing stage 5: p99 latency increases >50% from baseline, error rate exceeds 1%, or data consistency checks fail.

Procedure: Revert stage 5 changes. Rollback is complex, follow the documented rollback procedure for 'synchronous-to-event-driven'. Engage senior engineering support.

Recovery: 1–4 hours (complex rollback)

After stage 6

Trigger: Final validation: confirm all success criteria are met within 24 hours of migration completion. If any success criterion fails, trigger rollback or remediation.

Procedure: If success criteria are not met within 24 hours, escalate to the architecture team. Decide between full rollback, partial rollback, or targeted remediation based on scope.

Recovery: 24–72 hours for full assessment and remediation

Success Criteria

  • ✓Target state 'Event-Driven Architecture with Message Broker' is stable in production for 72+ hours with no regressions.
  • ✓p99 latency on all affected services is within acceptable range (≤110% of pre-migration baseline).
  • ✓Error rate is at or below pre-migration baseline.

This migration playbook is derived from structured architecture knowledge. Duration estimates and risk levels are structural approximations based on documented migration complexity and operational risk, not measured execution data. Validate all stages against your specific system constraints before executing.