Post-Mortem Framework · Configuration: Schema Version Mismatch Between Services

Schema Version Mismatch Between Services

SEV-3, Limited Impact

Linear propagation · configuration · Affects 0 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity.

Propagation Chain

Origin component

Schema Version Mismatch Between Services begins at the source component. Trigger: Rolling deployment of a service introducing a new required field without a multi-phase migration.

Immediate (T+0) · Signal: Error Rate Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

1 to 10 minutes for hard errors (NullPointerException, deserialization failure) via error rate monitoring. Hours to days for silent corruption variants where data is written with incorrect default values and no exception is thrown. Schema registry compatibility violations are detected at deployment time, effectively zero-delay. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Blast Radius

Scope depends on the traffic share hitting mismatched version pairs. During a rolling deployment at 3 of 10 instances upgraded, roughly 30% of writes carry the new schema and 70% of reads by old instances encounter new-schema records. Data corruption can affect any record touched during the deployment window and persists in storage after deployment completes, potentially corrupting downstream caches, search indexes, and materialized views populated from the incorrect data.

Contributing Factors

Trigger Condition: Rolling deployment of a service introducing a new required foperational

This operational trigger enables Schema Version Mismatch Between Services: Rolling deployment of a service introducing a new required field without a multi-phase migration

Trigger Condition: Renaming a field (a breaking semantic change) via a technicaoperational

This operational trigger enables Schema Version Mismatch Between Services: Renaming a field (a breaking semantic change) via a technically compatible mechanism (add new field, deprecate old) without updating all consumers at once

Trigger Condition: Adding a new enum value that old consumers handle with a defoperational

This operational trigger enables Schema Version Mismatch Between Services: Adding a new enum value that old consumers handle with a default case producing incorrect behavior

Remediation Plan

ImmediateStop the deployment rollout to prevent additional old-service instances from wri

Stop the deployment rollout to prevent additional old-service instances from writing incorrect data

Effort: Minutes to hours (on-call response)

ImmediateIdentify the deployment time window to scope affected records

Identify the deployment time window to scope affected records

Effort: Minutes to hours (on-call response)

ImmediateQuery for records modified during that window with incorrect default values for

Query for records modified during that window with incorrect default values for the new field

Effort: Minutes to hours (on-call response)

Short-TermSchema registry with an explicit compatibility mode matching the deployment shape

Use Confluent Schema Registry or AWS Glue Schema Registry for Kafka topic schemas, or OpenAPI validation middleware for REST. Set BACKWARD (or BACKWARD_TRANSITIVE) if only new readers must tolerate old data, FORWARD (or FORWARD_TRANSITIVE) if only old readers must tolerate new data, or FULL (or FULL_TRANSITIVE) for a rolling deployment, which needs both directions at once. Choosing BACKWARD alone for a rolling deployment is the most common version of this mistake: it protects the new code but not the old code still running alongside it.

Effort: 1 day to 1 week

Short-TermMulti-phase schema migration with the expand-contract pattern

Phase 1 (expand): add the new field as optional with a safe default and deploy to all services, so every instance can read both old and new records regardless of which version wrote them. Phase 2 (migrate): backfill the field on existing data. Phase 3 (contract): make the field required and remove the old one, only once every instance is confirmed on the new version. No service ever encounters a version it cannot handle, because the schema stays additively compatible in both directions until the rollout is complete.

Effort: 1 day to 1 week

Short-TermStrict ORM field selection to prevent unintentional column overwrites

Configure ORM UPDATE statements to modify only explicitly set fields (dirty-field tracking in Hibernate, SQLAlchemy, or ActiveRecord), so queries are column-specific rather than full-row. This is what makes a schema-compatible addition also safe in practice: without it, an old service can silently overwrite a column its schema never told it about, even though the schema change itself was technically forward compatible.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: error rate spike, log errors, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermArchitecture review for Schema Version Mismatch Between Services resilience

Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.

Effort: 1–2 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.