Synchronous Service Calls → Event-Driven Messaging
HighReplacing synchronous HTTP service dependencies with an event-driven message broker so that producers publish events and consumers subscribe independently: eliminating synchronous call failure cascades at the cost of eventual consistency, idempotent consumer design, and permanent operational overhead for broker and pipeline health.
Topology Changes
From
Synchronous HTTP Service Dependencies
To
Event-Driven Architecture with Message Broker
Topology Mutations
A durable message broker is introduced as the communication substrate. Producers publish events to topics; consumers subscribe and process independently. Kafka is preferred for high-throughput append-only event streams with replay capability; RabbitMQ for task queues requiring complex routing and consumer acknowledgement control.
Operational Impact
Broker becomes a critical path component: its unavailability blocks all event-driven flows. Consumer lag monitoring is required from day one; unmonitored lag accumulates silently until consumers fall hours behind.
Direct HTTP calls from producer to consumer are replaced by event publish on the producer side and event subscription on the consumer side. The producer no longer awaits the consumer's response: it publishes and moves on.
Operational Impact
The consumer's processing may lag hours behind the producer's writes during backlog events. Any business logic that previously depended on the synchronous response must be redesigned to accept asynchronous notification.
Operations that previously completed synchronously across services are now eventually consistent. The producing service writes its state and emits an event; the consuming service processes that event at an indeterminate time in the future.
Operational Impact
Users and downstream systems may observe a window where the producer's state is updated but the consumer's state is not yet. Business processes must be designed to handle this intermediate state gracefully.
Migration Stages
Audit every service-to-service call. Commands (the caller requires the callee's result to continue: payment, authorization, synchronous validation) cannot be made async. Notifications (the caller does not wait for the result: send email, update cache, trigger analytics event) are candidates for async messaging. Document each call with its classification and rationale.
Deploy Kafka cluster with consumer lag monitoring (Prometheus + Kafka exporter) from day one. Establish lag alerting thresholds. Create topics for the first migration candidate. Do not route any production traffic to the broker yet: validate operator tooling, topic configuration, and monitoring in staging.
The outbox pattern ensures event publication is atomic with the business database write. Write the event to an outbox table in the same transaction as the business mutation. A separate relay process reads the outbox and publishes to Kafka. This eliminates the dual-write inconsistency where the database write succeeds but the Kafka publish fails.
Kafka delivers at-least-once. Every consumer must handle duplicate delivery without producing duplicate side effects. Implement idempotency via deduplication keys stored in the consumer's database (processed_event_ids). Test duplicate delivery explicitly in staging: replaying the same event twice must produce the same outcome.
Activate the event path while keeping the old synchronous call in place. Validate that consumers produce the same state outcomes as the synchronous path. Monitor consumer lag and event ordering. Identify any cases where event ordering is required but not guaranteed by the partition assignment.
After consumer parity is validated, remove the old synchronous call from the producer. The producer now only publishes the event. Monitor consumer lag closely for 30 days after removal: this is when silent consumer failures become visible.
Migration Risks
Kafka guarantees ordering only within a single partition. Events for the same entity published to different partitions may be processed out of order by consumers. An order created event may arrive after the order shipped event if the events are on different partitions.
Mitigation
Use the entity key (order_id, user_id) as the Kafka partition key. All events for the same entity will be co-located on one partition and processed in publish order. Document this requirement in every event schema definition.
Multi-step business processes that previously completed in a single synchronous call chain now require saga coordination. A failure partway through a saga leaves data in an inconsistent intermediate state until compensating transactions execute.
Mitigation
Design explicit saga state machines with compensating transactions before migrating any multi-step cross-service flow. Use a saga orchestrator (Temporal, custom state machine) rather than pure choreography for complex flows with more than 3 steps.
At-least-once delivery requires idempotent consumers: but consumer idempotency is often not tested until a duplicate delivery occurs in production. Non-idempotent consumers silently double-process events, causing data corruption or duplicate side effects.
Mitigation
Write idempotency tests as part of consumer development: send the same event twice, assert the outcome is identical to a single delivery. Add this as a required test pattern in code review checklist for all new consumers.
Coupling Changes
Producer and consumer no longer need to be simultaneously available: events buffer in the broker
Consequence
Consumer downtime no longer blocks producer writes; consumer processes the backlog when it recovers
Both producer and consumer are now coupled to the broker's availability and lag state
Consequence
Broker becomes a shared critical dependency: its operational health affects all event-driven flows simultaneously
Event schemas become a shared contract between producer and all consumers
Consequence
Schema changes require backward compatibility or coordinated consumer migration: breaking schema changes are not safe
Consistency Model Changes
- ·Producer writes are locally consistent: the outbox pattern ensures the event is durably recorded with the business write
- ·Consumer state is eventually consistent with producer state: lag window depends on broker and consumer throughput
- ·There is no distributed transaction: the producer cannot roll back a committed event if a downstream consumer rejects it
- ·Exactly-once semantics are achievable with Kafka transactional APIs and idempotent consumers but require deliberate design
Rollback Risks
- ·Reverting from event-driven back to synchronous requires re-introducing the direct call while the consumer continues processing the event backlog: dual state during transition
- ·Events already published to Kafka before rollback cannot be unpublished: consumers will process them even after rollback
- ·Outbox pattern database schema changes (outbox table, relay job) require migration rollback if the path is abandoned