Schema Drift
SEV-2, Significant ImpactIsolated propagation · operational · Affects 0 scenario(s)
Severity Classification
Classified as CRITICAL based on failure mode severity.
Propagation Chain
Origin component
Schema Drift begins at the source component. Trigger: Deploying application code before running the corresponding database migration.
Immediate (T+0) · Signal: Alert
Blast Radius
Requests touching the drifted columns or tables fail. Drift on a widely-used table (users, orders) can fail many features simultaneously; drift on a rarely-executed code path can go undetected for weeks until a specific combination of conditions triggers the affected query.
Contributing Factors
High-throughput OLTP workloads are vulnerable to schema drift when migrations are applied in different orders across environments, causing queries to fail in production but succeed in staging.
This operational trigger enables Schema Drift: Deploying application code before running the corresponding database migration
This operational trigger enables Schema Drift: Applying migrations in a different order across environments
This operational trigger enables Schema Drift: Manual DDL in production not reflected in migration files
Remediation Plan
Compare the live schema against the expected schema from migration history
Effort: Minutes to hours (on-call response)
Identify the specific columns, tables, or constraints that have drifted
Effort: Minutes to hours (on-call response)
Create a migration that brings the schema to the expected state
Effort: Minutes to hours (on-call response)
Before deploying application code, verify every pending migration has been applied, and fail the deployment if the application's expected schema version does not match the database's current one. A CI/CD gate keeps code and schema synchronized by construction rather than by discipline.
Effort: 1 day to 1 week
On application startup, compare the live database schema against the ORM's expected schema and fail startup on mismatch. Prevents deploying code against the wrong schema entirely, rather than failing on the first request that touches the mismatch.
Effort: 1 day to 1 week
Revoke DDL privileges from application and operator database users; DDL changes go through the migration framework only, including emergency changes, which get a migration file even under time pressure. This is what checksum-based tooling cannot substitute for, since a checksum mismatch only catches tampering with an already-tracked file, not DDL that never went through the tool at all.
Effort: 1 day to 1 week
Configure alerts for: alert. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.