Post-Mortem Framework · Data Consistency: Clock Skew

Clock Skew

SEV-3, Limited Impact

Isolated propagation · consistency · Affects 0 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity.

Propagation Chain

Origin component

Clock Skew begins at the source component. Trigger: NTP server unavailable or unreachable for an extended period.

Immediate (T+0) · Signal: Log Errors

Blast Radius

Effects are subtle and localized but real: wrong event ordering, security-relevant token rejection or acceptance, and log timestamps across nodes that are not directly comparable. In the worst case, lease skew that lets two nodes simultaneously believe they hold the same lease can corrupt whatever state that lease was coordinating.

Contributing Factors

Trigger Condition: NTP server unavailable or unreachable for an extended periodoperational

This operational trigger enables Clock Skew: NTP server unavailable or unreachable for an extended period

Trigger Condition: VM migration causing a stale clock on the resumed instanceoperational

This operational trigger enables Clock Skew: VM migration causing a stale clock on the resumed instance

Trigger Condition: Hypervisor time adjustment pushing the guest clock forward ooperational

This operational trigger enables Clock Skew: Hypervisor time adjustment pushing the guest clock forward or backward

Remediation Plan

ImmediateCheck clock offset on all nodes: chronyc tracking | grep offset

Check clock offset on all nodes: chronyc tracking | grep offset

Effort: Minutes to hours (on-call response)

ImmediateIdentify nodes with offset beyond tolerance (for example, over 100ms): these are

Identify nodes with offset beyond tolerance (for example, over 100ms): these are the affected nodes

Effort: Minutes to hours (on-call response)

ImmediateRestart ntpd or chrony on affected nodes to force resynchronization

Restart ntpd or chrony on affected nodes to force resynchronization

Effort: Minutes to hours (on-call response)

Short-TermConfigure and monitor NTP synchronization on every node

Run chrony or ntpd on every host and container host, and monitor the offset metric directly (chronyc tracking, node_exporter), alerting when it exceeds a set tolerance (for example 100ms). Use the cloud provider's internal time-sync endpoint where available (AWS EC2: the Amazon Time Sync Service) rather than an external NTP pool, since the extra network hop to an external source adds its own skew.

Effort: 1 day to 1 week

Short-TermUse a monotonic clock for duration and timeout logic

Measure elapsed time (has this operation run too long, has this lease expired locally) with a monotonic clock source (CLOCK_MONOTONIC), not wall-clock time (CLOCK_REALTIME). A monotonic clock cannot move backward from an NTP step correction, so a duration computed from it cannot go negative. Wall-clock time is still correct for anything that must be compared across nodes or persisted.

Effort: 1 day to 1 week

Short-TermAdd explicit skew tolerance in token and lease validation

Allow a configurable tolerance window (for example 5 minutes for JWTs, low tens of milliseconds for lease timeouts) in validation logic, rejecting only tokens or leases expired or future-dated beyond that window. This absorbs ordinary skew but does not fix the underlying dual-lease-holder hazard; pair it with fencing tokens wherever a lease grants exclusive write access, per leader_election.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: log errors, alert. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermUse logical clocks or Hybrid Logical Clocks for cross-node event ordering

Replace wall-clock timestamps with Lamport clocks or HLCs when ordering events across nodes matters for correctness, not just for display. HLCs keep a human-readable timestamp bounded close to physical time while adding causal ordering that does not depend on clock synchronization. CockroachDB's HLC implementation is a reference design.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.