Configuration Drift

Production configuration diverges from the intended state through manual changes, partial rollouts, and environment-specific overrides, causing failures that are intermittent, hard to reproduce, and require cross-node comparison to diagnose.

Configuration drift occurs when the actual configuration of a running system diverges from what is documented, version-controlled, or believed to be in effect. Drift accumulates through a pattern of small changes: an incident triggers a manual adjustment directly on one server ("just for now"), the adjustment is not documented, the next deployment does not include it, and the production state and code state diverge silently.

Partial rollout drift: a configuration change is rolled out to 70% of nodes during a maintenance window but the rollout is interrupted. Now 70% of nodes run the new configuration and 30% run the old one. The system works but is in an internally inconsistent state. Failures occur only when requests hit the minority nodes.

Environment-specific override drift: a configuration override is added to fix a production-specific issue. The override is not reflected in the infrastructure-as-code repository. Six months later, the same configuration key is changed in the repository, the override is wiped, and the original issue reappears.

The insidious property of config drift: failures are condition-dependent. A node with wrong max_connections fails only under traffic spikes. The nodes with correct configuration handle the same spike fine. This makes the failure appear non-deterministic and hard to reproduce in investigation.

Detection without tooling is difficult. An engineer investigating a production incident cannot easily compare the configuration of 30 nodes. Without a source of truth (Ansible, Terraform, Puppet), there is no basis for comparison. Configuration management tools detect drift by comparing actual state against desired state defined in code: this is their primary operational value.

Immutable infrastructure eliminates drift by replacing rather than mutating running instances. A configuration change produces a new AMI or container image; all instances of the old image are terminated and replaced. No in-place mutation means no drift can accumulate. Kubernetes deployments follow this model by default.

GitOps: all configuration state is expressed in git. A controller (ArgoCD, Flux) continuously applies the git state to the cluster, reconciling drift automatically. Unauthorized manual changes are overwritten on the next sync.