Clock Skew

Two nodes' clocks disagree at a given moment (skew), and each node's own clock accumulates its own error over time (drift). Systems that assume synchronized, monotonically advancing wall-clock time break in specific, predictable ways when either assumption fails: wrong event ordering, early or late expiration, and leases held by two nodes at once.

Drift and skew are related but distinct, and the distinction matters for which fix applies. Drift is a property of one clock: its hardware oscillator runs slightly fast or slow, so left uncorrected its reading diverges from true time at a roughly constant rate, commonly tens of milliseconds per day on commodity hardware. Skew is a property of two clocks compared to each other: the difference between what they currently read, which can come from different drift rates, different times since last sync, or a correction applied to one but not the other. NTP (ntpd, chronyd) corrects drift on each node against a reference; skew is what remains between nodes even with NTP running everywhere, because synchronization is not instantaneous or perfect across a fleet. Standard NTP achieves roughly plus-or-minus 10ms accuracy under good network conditions; that residual is the skew a distributed system must tolerate or design around.

How NTP corrects matters as much as whether it runs. NTP can correct in two ways: slewing gradually speeds up or slows down the local clock's rate until it converges, with no discontinuity. Stepping jumps the clock directly to the corrected time, which can move it backward if the local clock was ahead. A backward step is a distinct hazard from ordinary skew: it can violate monotonicity on a single node by itself, no second node involved, so a duration measured by reading wall-clock time twice can come out negative, and a piece of code computing "has 30 seconds passed" can be fooled twice by the same jump. Duration and timeout logic should read a monotonic clock source (POSIX CLOCK_MONOTONIC, not CLOCK_REALTIME), which is guaranteed to never move backward, precisely because it is not subject to NTP step corrections. Wall-clock time is still needed for anything that must be compared across nodes or persisted as a timestamp; monotonic time is for measuring elapsed duration on one node.

Timestamp-based ordering across nodes is a skew problem. If Node A processes event X at its local t=100ms and Node B processes event Y at its local t=99ms, and the two clocks differ by 15ms, the apparent ordering, Y before X, may be backward from the true order. Systems that order events by wall-clock timestamp without a logical clock are exposed to this whenever skew exceeds the gap between the events.

Cache expiration is a skew problem at the boundary. A TTL computed on Node A expires at absolute time T. Node B's clock, reading T+15ms or T-15ms, evicts up to 15ms early or late. For short TTLs (under a second) that error is a meaningful fraction of the TTL itself.

Distributed lock and lease timeout is where skew becomes a correctness problem, not just an approximation error. A lease granted at time T for 30 seconds expires at T+30s by the coordinator's clock. If the leaseholder's clock reads 5 seconds behind the coordinator's, the coordinator revokes the lease 5 seconds before the leaseholder believes it has expired, and a new grantee can start acting as leader while the old one still believes it holds the lease: split_brain at the lease level. This is a second, independent reason lease-based coordination is structurally unsafe, alongside the unbounded-process-pause argument in leader_election: fencing tokens are the fix for both, because they make the storage layer, not the clock, the source of truth for who is allowed to act.

JWT and session token expiry is a skew problem with a standard tolerance fix. Tokens carry an exp timestamp. If the validating server's clock runs ahead of the issuing server's, valid tokens appear expired before their real expiry. RFC 7519 recommends a few minutes of tolerance in validation for exactly this reason.

Google Spanner's TrueTime bounds skew instead of tolerating it. Spanner uses GPS receivers and atomic clocks per datacenter so every node's clock uncertainty is bounded by a known interval epsilon (typically a few milliseconds, historically up to about 7ms worst case). TrueTime returns not a single timestamp but an interval guaranteed to contain the true time. Spanner's commit wait then holds a transaction's effects invisible until the earliest bound of the current TrueTime interval has passed the transaction's assigned commit timestamp, guaranteeing any later-starting transaction really did start after the earlier one committed in real time. This is external consistency bought by bounding and waiting out skew, not by making it zero, and it depends on infrastructure (GPS and atomic clock references) not available outside a handful of operators.

Virtualized environments add their own drift and skew sources on top of ordinary NTP behavior: VM live migration, hypervisor clock skew, and a suspended VM resuming with a stale clock are common causes. Cloud providers typically offer a low-latency internal time-sync endpoint (for example, AWS's Amazon Time Sync Service) specifically because reaching an external NTP pool from inside a VM adds its own latency-driven skew.

Logical clocks sidestep the whole physical-clock problem for ordering. Lamport clocks and vector clocks order events by causal relationship rather than wall-clock reading, so they are immune to drift and skew by construction, at the cost of not being human-readable wall-clock time. Hybrid Logical Clocks combine a Lamport-style causal counter with a physical-time component bounded to stay close to real time, giving both causal ordering and a timestamp still useful for humans and TTL-style logic. CockroachDB uses HLCs for exactly this combination.