DBRaven
Start Review →
Technology · key value store

etcd

3.5.x
matureCost: lowTeam: seniorLatency: single digit msDurability: strong

Summary

Distributed key-value store implementing Raft consensus for linearizable reads and writes. Designed for configuration data, distributed coordination, leader election, and service discovery. Kubernetes uses etcd as its sole backing store for all cluster state. Not suitable for high-throughput general-purpose storage or large datasets.

Primary Use Case

Distributed system coordination: storing cluster configuration, implementing leader election via lease-based locks, enabling service discovery, and providing a consistent watch API for change notification. Primarily used as infrastructure plumbing, not as an application data store.

Workload Fit

configuration storagecoordinationleader electionservice discovery

Strengths

Best for

  • ·Kubernetes control plane backing store: etcd was designed for this use case and Kubernetes is the dominant production deployment
  • ·Leader election in distributed services where the Raft consensus guarantee eliminates split-brain without application-layer tie-breaking logic
  • ·Distributed locking via TTL-based leases where lock expiry on client failure is automatic and guaranteed
  • ·Service discovery and health registration where services register their endpoints and the watch API notifies consumers of changes
  • ·Configuration management for infrastructure components where strong consistency is required and write volume is low

Excels when

  • ·Data volume is small (under 1GB of actual data) and write rate is low (under 1,000 ops/sec)
  • ·Strong consistency (linearizable reads) is a hard requirement for the data being stored
  • ·Clients need change notification via a watch API rather than polling
  • ·The deployment is Kubernetes-native and etcd is already present as the Kubernetes control plane store

Architectural advantages

  • ·Raft consensus provides linearizable reads and writes without application-layer conflict resolution: the correct value is always returned without split-brain
  • ·Watch API delivers change notifications to clients as events stream; clients do not need to poll for configuration changes or leadership transitions
  • ·MVCC transactions allow atomic compare-and-swap operations conditioned on revision numbers: enables distributed lock implementations and configuration versioning
  • ·TTL-based leases automatically expire locks and registrations when the holding process dies: no manual cleanup required after failure
  • ·3-node cluster tolerates 1 node failure; 5-node cluster tolerates 2 node failures without write unavailability

When to Avoid

Avoid when

  • ·Application requires general-purpose key-value storage with high throughput: use Redis or a purpose-built distributed cache
  • ·Dataset exceeds 8GB: etcd's MVCC architecture, compaction requirements, and startup times become unmanageable above this threshold
  • ·Write rate exceeds 1,000 ops/sec on a consistent basis: sustained writes above this level cause Raft log pressure and leader instability
  • ·The use case is application session storage, feature flags at high access rates, or event sourcing: these workloads require throughput that etcd cannot provide

Common misuses

  • ·Storing application feature flags, user preferences, or any data that changes at more than a few times per second: competing with Kubernetes control plane writes degrades cluster reliability
  • ·Using etcd for large blob storage (configuration files, certificates at scale): values near the 1.5MB limit cause request timeouts and should be stored in object storage with etcd holding only the reference
  • ·Running etcd on the same disk volume as the Kubernetes container runtime: I/O from container image pulls competes with etcd's fsync and causes leader elections

Consistency & Transactions

Consistency modelstrong
ACID compliantNo
Supports transactionsYes

Scaling

Characteristics
vertical
Operational burdenmedium
Typical read latency2 ms
Typical write latency5 ms

Read scalability

Linearizable reads go to the Raft leader: all reads are consistent but the leader is the throughput ceiling for reads. Serializable reads can be served from any follower (stale reads) but this is not the default and violates the strong consistency guarantee. Read throughput is bounded at approximately 10,000–20,000 ops/sec on a typical 3-node cluster.

Write scalability

All writes go through the Raft leader and are committed once a majority acknowledges. Write throughput is bounded by the Raft consensus round-trip and fsync latency. Typical cluster write ceiling is 1,000–2,000 ops/sec for sustained workloads. etcd is not designed to be a write-heavy store.

Failure Behavior

Known failure modes

  • ·MVCC revision accumulation: without periodic compaction, the etcd database grows unboundedly as each write creates a new revision; disk exhaustion causes the cluster to enter a read-only alarm state
  • ·Leader election jitter during network partitions: Raft elections during leader failure cause a write unavailability window of one election timeout (default: 1 second heartbeat, 5-second election timeout)
  • ·Large value writes: etcd's default maximum request size is 1.5MB; storing large configuration blobs or entire Kubernetes resource specs near this limit causes request rejections
  • ·Clock skew causing Raft election instability: nodes with clock drift may perceive spurious leader timeouts and trigger unnecessary elections, causing write availability degradation
  • ·Defragmentation blocking: running etcdctl defrag while the cluster is under load causes a brief pause on the defragmented member; on the leader, this can cause a leader election
  • ·Watch event backlog overflow: clients that cannot consume watch events fast enough cause the watch stream to be closed and the client to re-sync: generates a thundering herd of re-sync requests

Bottlenecks

  • ·Disk fsync latency is the primary write throughput limiter: every committed write requires an fsync to the WAL before the Raft acknowledgement is sent; NVMe is effectively required for production deployments
  • ·Leader bottleneck: all linearizable reads and writes go through the single Raft leader: the leader's network and CPU become the throughput ceiling for the entire cluster
  • ·MVCC history accumulation: without compaction, the database size grows with every write, increasing startup time, backup size, and defragmentation duration
  • ·Watch event fan-out: a single key change can trigger watch notifications to hundreds of clients simultaneously; at Kubernetes scale (thousands of controllers watching the same resources) this creates significant network and CPU pressure on the leader
  • ·Election timeout penalty: after leader failure, the cluster is unavailable for writes during the election timeout window; tuning heartbeat and election timeouts trades failure detection speed against false-positive election risk

Degradation patterns

  • ·MVCC store disk exhaustion: without compaction, disk fills and etcd enters read-only alarm mode: the Kubernetes control plane becomes unable to create or update resources
  • ·Slow follower falling behind: a follower that cannot keep up with the Raft log requires the leader to retain log entries indefinitely; the leader's WAL grows until the follower catches up or is evicted
  • ·Watch stream overflow: clients that process watch events slowly receive a compacted error and must perform a full re-sync; during Kubernetes rolling deployments with many watchers, this causes a re-sync storm

Recovery considerations

  • ·etcd snapshots are the primary backup mechanism; etcdctl snapshot save creates a point-in-time backup that captures all MVCC data at the current revision
  • ·Restoring a member from snapshot requires the member to rejoin the cluster with a new peer URL; restoring the entire cluster from snapshot requires bringing all members down and restoring from the same snapshot
  • ·After leader failure, surviving members elect a new leader automatically within one election timeout; no manual intervention is required unless quorum is lost

Operational Pitfalls

  • ·Using etcd as a general-purpose cache or application data store: etcd's 1,000–2,000 write ops/sec ceiling means application writes will compete with Kubernetes control plane writes and degrade cluster stability
  • ·Not running compaction and defragmentation on a schedule: revisions accumulate silently; the first sign of a problem is a read-only alarm after disk exhaustion
  • ·Not monitoring etcd disk fsync duration: etcd's write latency is dominated by WAL fsync to disk; slow disks (HDDs or overloaded NVMe) cause Raft timeouts and leader elections
  • ·Deploying etcd on shared-disk nodes where I/O contention from other workloads causes fsync latency spikes: etcd requires dedicated, low-latency storage (NVMe preferred)
  • ·Running a 2-node cluster for 'HA': a 2-node cluster requires both nodes to agree on writes (no tolerance for 1 failure); 3-node is the minimum for fault tolerance

Architecture Guidance

Common topology roles

coordination storeconfiguration backendkubernetes control plane storeservice registrydistributed lock service

Migration notes

  • ·From ZooKeeper: etcd provides a simpler operational model (single binary, REST/gRPC API, no JVM) and stronger consistency; migration requires rewriting coordination logic from ZooKeeper's ZNode API to etcd's key-value and watch API
  • ·From Consul: Consul provides service mesh and health checking in addition to key-value storage; etcd covers only the key-value and coordination use case: evaluate whether Consul's additional features justify its operational overhead
  • ·Kubernetes cluster migration to managed etcd: cloud-managed Kubernetes (EKS, GKE, AKS) hides etcd operations; teams migrating from self-managed clusters give up direct etcd access and compaction control

Advisor Guidance

Critical

When: scenario uses etcd for application data storage with write rates above 100 ops/sec

etcd is designed for configuration and coordination data at low write rates; application data storage will compete with cluster coordination traffic and cause stability degradation: use Redis or a purpose-built store

Warning

When: scenario deploys etcd without automated compaction configured

Configure auto-compaction via --auto-compaction-mode and --auto-compaction-retention; without compaction, MVCC revisions accumulate until disk exhaustion triggers a read-only alarm

Warning

When: scenario runs etcd on shared storage with other I/O-intensive workloads

etcd requires dedicated low-latency storage for WAL fsync; I/O contention from co-located workloads causes Raft election instability and write latency spikes

Comparison Factors

consistency guarantee

Linearizable reads and writes by default: strongest possible consistency guarantee for a distributed store

high

throughput ceiling

Very low by design: 1,000–10,000 ops/sec maximum; not a general-purpose data store

low

operational simplicity

Medium: single binary, simple cluster configuration, but requires disk I/O isolation, compaction scheduling, and fsync monitoring

medium

use case specificity

Narrow: best-in-class for coordination and configuration; poor fit for any other use case

high

Managed Cloud Options

Amazon EKS (managed etcd within Kubernetes control plane)Google GKE (managed etcd within Kubernetes control plane)Azure AKS (managed etcd within Kubernetes control plane)

Enables Patterns

leader electionservice discoverydistributed lockconfiguration management

Basis

etcd is the Kubernetes control plane store; its architecture and operational constraints are extensively documented by the CNCF and CoreOS/Red Hat engineering teams