Post-Mortem Framework · Configuration: Configuration Drift

Configuration Drift

SEV-3, Limited Impact

Linear propagation · configuration · Affects 3 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity. This failure mode appears in 3 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Configuration Drift begins at the source component. Trigger: Manual configuration change applied directly to a production node without code update.

Immediate (T+0) · Signal: Log Errors

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

Configuration drift without tooling may take days to weeks to detect, often discovered only when a condition-dependent failure occurs. With automated drift detection (Ansible, Puppet, Terraform plan), detectable within minutes of the drift occurring. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Blast Radius

Blast radius depends on how many nodes have drifted and what configuration was changed. A single drifted node causes intermittent failures on requests routed to it. A cluster-wide drift causes uniform failures. Memory limit drift (too low) causes OOM kills. Connection limit drift causes connection exhaustion. The blast radius of the configuration change itself, multiplied by the fraction of nodes that have drifted.

Contributing Factors

Trigger Condition: Manual configuration change applied directly to a productionoperational

This operational trigger enables Configuration Drift: Manual configuration change applied directly to a production node without code update

Trigger Condition: Partial rollout of configuration change interrupted before coperational

This operational trigger enables Configuration Drift: Partial rollout of configuration change interrupted before completion

Trigger Condition: Environment-specific override not captured in configuration operational

This operational trigger enables Configuration Drift: Environment-specific override not captured in configuration management

Remediation Plan

ImmediateIdentify the scope of drift: which nodes are affected and what configuration dif

Identify the scope of drift: which nodes are affected and what configuration differs

Effort: Minutes to hours (on-call response)

ImmediateCompare actual configuration against desired state in repository using config ma

Compare actual configuration against desired state in repository using config management tool

Effort: Minutes to hours (on-call response)

ImmediateFor immediate fix: run configuration management against drifted nodes to converg

For immediate fix: run configuration management against drifted nodes to converge state

Effort: Minutes to hours (on-call response)

Short-TermImplement GitOps with automated reconciliation

Store all configuration in git. A GitOps controller (ArgoCD, Flux for Kubernetes; Ansible Tower for host-level config) continuously compares git state against running state and applies corrections. Manual changes are overwritten at the next sync cycle (typically within minutes).

Effort: 1 day to 1 week

Short-TermRun configuration management in check mode in CI and alert on drift

Run Ansible/Puppet/Chef in check/dry-run mode on a schedule (every 30 minutes). Alert when drift is detected. This does not auto-remediate but provides early detection before condition-dependent failures occur.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: log errors, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermAdopt immutable infrastructure: replace rather than mutate

Package configuration into container images or AMIs. Configuration changes produce a new image version; all instances are replaced with the new version. No in-place mutation means no drift surface. Kubernetes deployments naturally implement this.

Effort: 1–4 sprints

Long-TermEliminate cross-scenario Configuration Drift exposure

Configuration Drift affects 3 architecture scenarios (API Gateway Platform, Developer Tools Platform, Healthcare Records Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.