Schema Version Mismatch Between Services

When services sharing a data schema run at different versions at once, during a rolling deployment or a failed upgrade, a reading service encounters data written by a schema version it was not built to handle. The precise question in every such encounter is which of two distinct compatibility directions the change actually needs, and most incidents trace back to a change that only satisfied one of them.

Schema evolution in distributed systems means every service sharing a schema must handle more than one version of it during the transition. There are two distinct compatibility directions, and they are not the same requirement. Backward compatibility means new code can read data written by old code: a newer service version encountering an older record must still work. Forward compatibility means old code can read data written by new code: an older service version, not yet upgraded, encountering a record a newer sibling already wrote must still work. A rolling deployment needs both simultaneously, because for its whole duration both old and new code are running and both are reading records the other version wrote. A schema change that is only backward compatible is not safe to roll out gradually; it is only safe once every reader has already upgraded.

In a rolling deployment of 10 instances, at any moment some run the new version and some run the old. Records written by new instances contain fields old instances have never seen; records read by new instances may have been written by old instances and lack fields the new version treats as required. Additive-only changes with safe defaults, the standard way to get both directions at once, are what make a change safe during this window: a new optional field with a default is backward compatible (old data still reads fine because the new reader falls back to the default) and forward compatible (new data still reads fine on the old reader because unknown fields are ignored) at the same time. A required field, or a semantic change to an existing field, breaks one direction or both.

How the failure manifests depends on the schema format. For JSON without a schema registry, the reading service's deserialization code accesses a field that does not exist and either throws a NullPointerException, returns a zero-value silently (Go structs), or applies an incorrect default. For Avro or Protobuf with a schema registry, compatibility checks catch changes that violate the configured direction before deployment, but a change that is technically compatible (an optional field with a default) can still be treated as required in application code and cause a logic error the registry cannot see, because the registry checks schema shape, not what the application code assumes about it. For database column additions, an old service reads a row without the new column and, on a subsequent update, may overwrite the new column's value with NULL if its UPDATE statement does not include that column.

The silent corruption variant is the most dangerous because both compatibility directions can look satisfied while application logic still breaks. Consider adding an is_verified boolean column to a users table, defaulting to false. The new service reads it and uses it correctly in authorization logic. The old service reads the user, does not see the column in its model, and later writes an update (changing last_seen). If the ORM constructs the UPDATE from only the fields it knows about, the column keeps its database value; that ORM behavior is what makes the change forward compatible in practice. But if the ORM maps every struct field including ones the old code never set, is_verified is silently overwritten with false, stripping verification status without either compatibility check ever firing, because the schema itself was fine, the ORM's write behavior was not.

Schema registries (Confluent Schema Registry for Kafka topics, Apicurio for OpenAPI) enforce the configured compatibility direction at publish time but cannot see application-level semantic assumptions layered on top of a technically compatible schema.