Event Ordering Violation
SEV-2, Significant ImpactIsolated propagation · messaging · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Event Ordering Violation begins at the source component. Trigger: Kafka producer sends events for the same entity to multiple partitions.
Immediate (T+0) · Signal: Alert
Blast Radius
Incorrect consumer state for the affected entity. Depending on event semantics, this can appear as an incorrect balance (financial events applied out of order), an incorrect status (an order marked "delivered" before "shipped"), or an incorrect projection (a search index reflecting stale state). The blast radius is bounded to entities whose events were actually reordered; unaffected entities process correctly, because the failure is a causal-order violation for specific entities, not a system-wide ordering collapse.
Contributing Factors
Financial transaction workloads are vulnerable to event ordering violations where applying a balance credit before a balance debit produces an incorrect intermediate state.
This operational trigger enables Event Ordering Violation: Kafka producer sends events for the same entity to multiple partitions
This operational trigger enables Event Ordering Violation: Consumer fails mid-processing and re-queues an already-partially-processed batch
This operational trigger enables Event Ordering Violation: Multiple producers emitting events for the same entity without coordination
Remediation Plan
Identify affected entities by checking for state inconsistencies
Effort: Minutes to hours (on-call response)
Determine the correct event order from the producer's event log
Effort: Minutes to hours (on-call response)
Confirm the affected handlers are idempotent before replaying: a non-idempotent handler will repeat side effects on replay, not just correct state
Effort: Minutes to hours (on-call response)
Route all events for the same entity to the same Kafka partition by keying messages on the entity identifier (order_id, user_id). This works because causal order is what is actually needed, and Kafka already preserves order within a partition; a single consumer thread per partition then processes each entity's events in production order without requiring any global ordering across entities that do not depend on each other.
Effort: 1 day to 1 week
Each event carries a sequence number (event_version) assigned at the producer. The consumer stores the last processed version per entity; if an event's version is not (last_version + 1), it is rejected and parked in a dead-letter queue. This detects a broken sequence for a single-producer-per-entity setup. It does not resolve true concurrent writes from uncoordinated producers, which need a vector clock or equivalent to distinguish "arrived out of order" from "genuinely concurrent, no order exists."
Effort: 1 day to 1 week
Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Design events so applying them in any order produces the same result (commutative): emit "balance is now $350" rather than "balance increased by $50" (order-dependent). This also matters for replay safety: correcting an ordering violation means replaying events for the affected entity, and replay is only safe to run again if the handler is idempotent, since a non-idempotent handler with side effects (sending a notification, charging a card) would repeat that side effect on every replay. Not always possible for all event types, but it reduces both the blast radius of ordering violations and the risk of the recovery procedure itself.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.