Event Replay Storm Recovery
A large-scale event replay: triggered by a projection rebuild, a bug fix requiring reprocessing, or a new consumer catching up from offset zero: floods the event store and downstream systems with a burst of historical events. The replay storm saturates database write capacity, overloads downstream services not designed for burst replay, and surfaces non-idempotent handlers that produce duplicate side effects.
Event store (Kafka or EventStoreDB) + projection processor + downstream OLTP database + external API consumers
Degradation Replay
Nominal: Production Event Rate
Consumer processing at normal production event rate; no replay in progress
- ·Event consumption rate at normal production level
- ·Downstream DB write rate within normal operating range
- ·No duplicate events or double-processing in handler logs
- ·Consumers tracking near-current offset on topic partitions
- ·Downstream services processing events within their provisioned capacity
- !Normal operation: consumers processing production events in real time
Operational simulation model only, not a production forecast. Degradation stages are derived from structured operational knowledge, not measured telemetry. Do not use for capacity planning or incident response.
Run With Your Parameters
Adjust the parameters below to see how metric values shift across degradation stages. Formulas are deterministic: same inputs always produce the same output.
Simulation Parameters
Rate multiplier during dead-letter replay burst
Computed Degradation Stages
Consumer processing at normal production event rate; no replay in progress
Replay started: consumer offset reset to historical point; processing at maximum throughput
Replay write burst saturates downstream database; OLTP transactions queuing
Non-idempotent handlers have produced data corruption; external APIs triggering circuit breakers
Handlers fixed for idempotency; replay resumed with throttling; OLTP recovering
Threshold Events
Message backlog at 352,500 (3.5× threshold). Consumers processing at 4,500 msg/s vs 16,250 produce rate.
Consumer health at 60%: sustained backlog causes GC pressure and memory growth.
Interpretation
Produce rate at 5× storm rate exceeds consumer capacity (4,500 msg/s). Backlog peaks at 6,900,000 messages. Consumer health: 60%.
Producer throughput exceeds consumer capacity: backlog accumulates unboundedly
Scale consumers to at least 17 instances. Add consumer health monitoring with backpressure. Set dead-letter queue with bounded retry budget.
Parameterized simulation: not a production forecast. Values derived from deterministic formulas applied to your parameters. Do not use for capacity planning or operational decisions without validation.
Propagation Model
Replay at maximum throughput produces N× normal event rate; downstream systems receive burst they were not provisioned for
Stabilizes: Storm subsides when replay reaches current offset: returns to normal production event rate
Non-idempotent handlers create duplicate records, double-counting, or repeated API calls for each replayed event
Stabilizes: Resolved only by idempotency fix + compensating data cleanup: cannot auto-heal
Replay-driven write burst saturates database IOPS; OLTP transactions queue behind replay writes; user-facing reads degrade
Stabilizes: OLTP performance recovers once replay rate is throttled or replay completes
Recovery Patterns
Stop, fix idempotency, resume with throttling
2-8 hours for idempotency fix + data reconciliation + controlled replay- ·Delayed replay completion: business feature depending on rebuilt projection has longer wait
- ·Throttled replay impacts replay completion time proportionally
- !Data reconciliation may be incomplete: audit trail must confirm all duplicates identified
Replay via separate dedicated cluster
Replay time unchanged; OLTP cluster protected from burst- ·Requires provisioning a temporary replay cluster: cost and operational overhead
- ·Results must be merged back to production cluster
- !Merge step introduces its own data integrity risk: must be carefully validated
Operational Summary
Event replay storms occur when historical event reprocessing runs at maximum throughput, flooding downstream systems with a burst N× normal production rate. The critical failure mode is non-idempotent event handlers: replaying events creates duplicate records, double-counted aggregates, or repeated API calls that cannot auto-heal. Prevention requires idempotent handlers (event ID deduplication), dedicated replay consumer groups, and strict rate throttling relative to downstream write capacity.