Summary
Kafka's durable, ordered, append-only log is the canonical infrastructure for an event store at scale. Topics with compaction or retention policies serve as the persistent event log that event sourcing requires.
Evidence
- ·Kafka topic partitions are ordered, append-only logs: exactly the event store structure event sourcing requires
- ·Consumer groups track offsets independently, enabling multiple projections to consume the same event stream at their own pace
- ·Log compaction preserves the latest event per key, supporting aggregate snapshot-like behavior
- ·Kafka retention can be set to "forever" (retention.ms=-1), making it a permanent event archive
- ·Confluent Schema Registry provides event schema versioning and compatibility enforcement
Operational Context
- ·Partition count determines maximum parallelism for both event writes and projection consumers
- ·Aggregate-keyed partitioning (producer key = aggregate_id) ensures all events for one aggregate land on the same partition, preserving ordering per aggregate
- ·Consumer group lag monitoring is essential: a lagging projection means read models are stale
Tradeoffs
- ·Kafka does not support optimistic concurrency at the aggregate level natively: application must enforce sequence numbers
- ·Event replay for a single aggregate requires filtering a partition by aggregate_id: not as efficient as a database query by aggregate_id
- ·At-least-once delivery requires idempotent projection handlers
Generator Relevance
The kafka + event_sourcing combination is the canonical recommendation for event-driven analytics pipelines and audit log systems at scale. Architecture generator should suggest this pair for event_streaming_workload scenarios.
Evidence grounding
Grounded, 5 supporting itemsKafka's design properties: ordered partitions, configurable retention, consumer group offset tracking: directly satisfy event sourcing requirements. Widely used in production event-sourced systems at LinkedIn, Uber, and Confluent.