Post-Mortem Framework · Data Consistency: Saga Compensation Cascade

Saga Compensation Cascade

SEV-2, Significant Impact

Linear propagation · consistency · Affects 0 scenario(s)

Severity Classification

Classified as CRITICAL based on failure mode severity.

Propagation Chain

Origin component

Saga Compensation Cascade begins at the source component. Trigger: Compensating service is unavailable when compensation is triggered (same failure that caused the original step to time out).

Immediate (T+0) · Signal: Alert

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

5–30 minutes via saga state monitoring (alert on sagas in "compensating" state for >5 minutes). Without explicit saga state monitoring, detection occurs when customers escalate stuck orders, typically 30–120 minutes after the incident. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Blast Radius

Each stuck saga locks business resources (inventory, payment authorization, reservation slots) for the duration of the stuck state. At scale, this produces ghost reservations that are never released, causing false inventory unavailability and customer-visible order failures. The financial impact includes charged-but-not-fulfilled payments, voided-but-already-fulfilled payments, or both, depending on which step timed out and whether compensation partially succeeded.

Contributing Factors

Trigger Condition: Compensating service is unavailable when compensation is trioperational

This operational trigger enables Saga Compensation Cascade: Compensating service is unavailable when compensation is triggered (same failure that caused the original step to time out)

Trigger Condition: Saga step timeout threshold too short for external payment ooperational

This operational trigger enables Saga Compensation Cascade: Saga step timeout threshold too short for external payment or logistics provider latency variance

Trigger Condition: Compensation transaction is not idempotent, causing retry looperational

This operational trigger enables Saga Compensation Cascade: Compensation transaction is not idempotent, causing retry logic to abort after first attempt fails

Mitigation Gaps

HIGHNo preventive mitigation documented

Identify and document at least one architectural change that prevents Saga Compensation Cascade from occurring (not just recovering from it).

Remediation Plan

ImmediateQuery the saga state store for all sagas in "compensating" state with age > 10 m

Query the saga state store for all sagas in "compensating" state with age > 10 minutes

Effort: Minutes to hours (on-call response)

ImmediateFor each stuck saga, determine which compensation steps have succeeded and which

For each stuck saga, determine which compensation steps have succeeded and which are pending

Effort: Minutes to hours (on-call response)

ImmediateManually execute compensation steps that cannot be retried automatically, using

Manually execute compensation steps that cannot be retried automatically, using the idempotency key to check if already applied

Effort: Minutes to hours (on-call response)

Short-TermMandatory idempotency keys for all compensating transactions

Every compensating transaction must accept an idempotency key (saga_id + step_id) and deduplicate on it at the target service. The target service checks if the compensation was already applied (SELECT 1 FROM compensation_log WHERE idempotency_key = $1) before executing, and returns success if it was. This makes compensation retries safe and allows the saga coordinator to retry stuck compensations indefinitely without risk of double-application.

Effort: 1 day to 1 week

Short-TermSaga timeout tuning with external provider SLA margins

Set saga step timeouts to 3x the p99 latency of the target service SLA (e.g., if payment provider p99 is 2 seconds, set step timeout to 6 seconds). Monitor external provider latency separately from saga timeout configuration. Reduces the frequency of timeout-triggered compensation without increasing end-user wait time beyond acceptable bounds for most requests.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: alert, log errors, error rate spike. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermDurable saga state with persistent compensation queue

Store all saga state transitions in a persistent store (outbox table in PostgreSQL, or a dedicated saga log). Use the outbox pattern to enqueue compensation messages: compensation steps are written to an outbox table atomically with the saga state update, then polled and delivered by a reliable outbox consumer. If the saga coordinator crashes, the outbox consumer restarts and re-delivers unacknowledged compensation messages. This makes compensation delivery at-least-once and durable across coordinator failures.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.