Event-Driven Microservices
Three domain services: User, Order, Notification: communicating exclusively through Kafka topics using the outbox pattern. No synchronous service-to-service calls. Each service owns its own PostgreSQL database. The API Gateway handles auth and routing.
Description
This composition implements the database-per-service pattern with event-driven integration via Kafka. Services are decoupled at the data layer: no shared tables, no cross-service foreign keys, no synchronous RPC between services. All inter-service communication is asynchronous via Kafka topics.
The outbox pattern is the critical implementation detail that makes this reliable. Each service writes domain events to an outbox table in the same ACID transaction as the business write. A CDC process (Debezium or a custom poller) reads the outbox and publishes to Kafka. This guarantees that the event is published if and only if the business write commits: solving the dual-write problem without distributed transactions.
The consequence is eventual consistency: a user creation event may arrive at the Notification Service 100–500ms after the User Service committed. Services must be designed for this: idempotent consumers, at-least-once delivery handling, and no assumptions about event ordering across topics without explicit sequencing.
Use Cases
- ·Platform with distinct domain boundaries and separate team ownership per service
- ·Systems requiring independent service deployability and rollback
- ·High-volume event processing at 1M–100M events/day
- ·Systems where domain events need to fan out to multiple consumers
- ·Organizations with 3+ teams and bounded context ownership
Scale Profile
Entry Point
10k DAU or 100k events/day: justified when team size and domain complexity warrant it
Sweet Spot
100k–10M DAU, 1M–100M events/day
Scaling Ceiling
Kafka handles 1M+ messages/second with proper partitioning. Service-level saturation depends on per-service load.
Typical RPS
1k–50k RPS across all services
Architecture Nodes (9)
Web and mobile clients, third-party API consumers.
Single ingress point handling authentication (JWT validation), rate limiting per client, request routing to services, and response aggregation for multi-service read endpoints.
API Gateway rate limiting state and short-lived response cache. Keyed by client ID + endpoint. TTL: 1–60 seconds depending on endpoint sensitivity.
Owns all user identity, profile, and authentication data. Source of truth for user entities. Publishes user.created, user.updated, user.deleted events.
Private database for User Service. Includes users table and outbox table. No other service has direct access.
Owns order lifecycle: creation, payment status, fulfillment. Publishes order.created, order.paid, order.cancelled events. Consumes user-events to validate user existence.
Private database for Order Service. Includes orders and outbox tables. Stores denormalized user reference (user_id only: no user data).
Stateless event consumer. Consumes user-events and order-events. Sends emails, push notifications, and webhooks. No persistent state: relies on external delivery providers.
Durable distributed event log. Three topics: user-events (3 partitions), order-events (6 partitions), notifications (3 partitions). Retention: 7 days. Consumer groups per service.
Dependencies (10)
5 critical path edges. Failure on these directly degrades user-facing requests.
HTTPS API requests
All client traffic enters via API Gateway. JWT authentication validated at gateway before routing.
Timeout: 30s
Rate limit checks
Sliding window rate limit state stored in Redis. Incremented per request. Circuit breaks at threshold.
Timeout: 50ms
User API requests
GET/POST /users routed to User Service. Gateway strips internal headers before forwarding.
Timeout: 5s
Order API requests
GET/POST /orders routed to Order Service.
Timeout: 5s
User data reads/writes
ACID transactions to private user database. Outbox table updated in same transaction as business writes.
Timeout: 2s
Order data reads/writes
ACID transactions to private order database. Outbox table updated in same transaction as order state changes.
Timeout: 2s
user-events topic
CDC reads outbox table, publishes user domain events to Kafka. Event schema: {event_id, event_type, user_id, payload, occurred_at}.
order-events topic
CDC reads order outbox, publishes order domain events. Partitioned by order_id for ordering guarantees within an order lifecycle.
Event consumption
Notification Service subscribes to user-events and order-events. Consumer group ensures at-least-once delivery. Idempotency key prevents duplicate sends.
user-events consumption
Order Service consumes user.deleted events to mark orders for orphan handling. Saga compensation flow.
Failure Propagation
How a failure in one component cascades through the system. Each path documents the mechanism and the mitigation that breaks the chain.
Mechanism
Services continue writing to outbox tables during Kafka downtime. CDC accumulates uncommitted outbox rows. No events delivered until Kafka recovers. Notification Service consumption pauses: no sends during outage.
Mitigation
3-node Kafka cluster with replication factor 3. Min ISR = 2. Outbox backlog delivers automatically on recovery. Alert on consumer group lag.
Mechanism
User Service write and read path fails. API Gateway receives 503 for all /users endpoints. Order Service cannot validate new users (if it calls User Service synchronously): but in this architecture, it relies on events only.
Mitigation
PostgreSQL with streaming replication + Patroni automated failover. Failover window: 15–30s. Implement retry with backoff in API Gateway.
Mechanism
Notification Service crashes or slows. Consumer group lag grows. Kafka retains events per retention policy (7 days). When service recovers, it replays all lagged events: risk of duplicate sends if not idempotent.
Mitigation
Idempotency key per notification (dedup on event_id + channel). Alert on consumer lag > 10k messages or 60s. Dead letter topic for repeatedly failing events.
Scaling Transitions
Inflection points where this architecture begins to degrade and what the recommended evolution looks like.
Kafka topic partition count becomes insufficient. Consumer group rebalancing latency grows. Schema evolution across events requires coordination.
Recommended Action
Introduce Confluent Schema Registry for Avro/Protobuf event schemas. Increase partition count for high-volume topics. Implement consumer lag SLOs.
Outbox table polling creates read pressure on primary. CDC slot accumulates WAL under burst writes.
Recommended Action
Replace outbox table polling with Debezium PostgreSQL CDC connector reading WAL directly. Eliminates polling overhead.
Patterns Applied
Architectural Notes
- ·Never share a database between services: it is the single most common architectural mistake in microservice migrations. Schema coupling undoes all deployment independence.
- ·The outbox pattern is non-negotiable for reliable event publishing. Direct Kafka publish in the same request fails silently on Kafka unavailability.
- ·Idempotent consumers are required, not optional. Kafka at-least-once delivery guarantees duplicates will occur during rebalance, crash recovery, and retry.
- ·Saga compensation is complex. Before splitting to microservices, verify the domain truly has bounded contexts: premature decomposition is harder to reverse than a well-factored monolith.
Confidence
StrongEvent-driven microservices with outbox pattern is well-documented by Netflix, Uber, and major e-commerce platforms. Kafka reliability at this scale is operationally proven.