Dual-Write Inconsistency Between Systems
SEV-2, Significant ImpactIsolated propagation · consistency · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Dual-Write Inconsistency Between Systems begins at the source component. Trigger: Elasticsearch write fails due to shard unavailability, mapping exception, or circuit breaker open.
Immediate (T+0) · Signal: Error Rate Spike
Blast Radius
Each dual-write failure produces one permanently diverged record. The scope of impact depends on how the secondary store is used: if Elasticsearch is the search surface, search results become permanently incorrect for the affected entity. If the secondary is a read cache, reads from the cache return stale data until the cache key expires or is manually invalidated. If the secondary is an analytics store, aggregate reports include incorrect values for all time until a full recompute is run.
Contributing Factors
This operational trigger enables Dual-Write Inconsistency Between Systems: Elasticsearch write fails due to shard unavailability, mapping exception, or circuit breaker open
This operational trigger enables Dual-Write Inconsistency Between Systems: Cache write fails due to Redis memory pressure, network timeout, or eviction of the key during the write
This operational trigger enables Dual-Write Inconsistency Between Systems: Analytics write fails due to analytics store maintenance window, schema evolution conflict, or rate limiting
Remediation Plan
Identify the time window of secondary write failures via error logs or secondary store write metrics
Effort: Minutes to hours (on-call response)
Determine the scope of diverged records by querying the primary store for records modified during the failure window
Effort: Minutes to hours (on-call response)
For each diverged record, re-apply the current primary store value to the secondary store using idempotency keys
Effort: Minutes to hours (on-call response)
Schedule a reconciliation job (every 1–6 hours) that samples recent writes to the primary store and checks whether the corresponding records exist and are current in the secondary store. For any diverged records, the job re-applies the write to the secondary store. This does not prevent dual-write failures but bounds the duration of inconsistency to the reconciliation interval. Most suitable for systems where a 1–6 hour eventual consistency window is acceptable.
Effort: 1 day to 1 week
Add an idempotency key to all primary writes. On retry, check if the record in the primary store was updated between the original attempt and the retry (compare version number or updated_at timestamp). If it was, do not overwrite with the original payload. Send the current version to the secondary store instead. This prevents retry-induced data corruption while allowing safe re-delivery to the secondary store.
Effort: 1 day to 1 week
Configure alerts for: error rate spike, log errors, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Write the outbox record to the same PostgreSQL transaction as the primary write: BEGIN; INSERT INTO primary_table ...; INSERT INTO outbox (aggregate_id, payload, status) VALUES ...; COMMIT. A separate outbox consumer reads the outbox table (or CDC stream from it) and delivers to Elasticsearch asynchronously with full retry semantics. The outbox record is marked as delivered only after Elasticsearch confirms the write. Failures are retried with exponential backoff. This decouples the primary write's success from the secondary write, eliminates dual-write inconsistency, and provides guaranteed eventual consistency.
Effort: 1–4 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.