Cloudflare Workers KV Global Store
Cloudflare Workers KV committed explicitly to eventual consistency across 200+ edge locations to achieve sub-millisecond reads globally, acknowledging that globally consistent KV at acceptable latency is physically impossible and documenting the consistency model prominently rather than hiding it.
Cloudflare Workers KV is a globally distributed key-value store designed for Workers (Cloudflare's edge runtime) to access configuration, feature flags, and content with sub-millisecond latency at any of 200+ global edge locations. The fundamental constraint is the speed of light: a write committed in a US data center cannot reach an edge node in Tokyo in under ~130ms round-trip due to physical propagation delay. KV makes an explicit architectural choice: strong consistency at the origin (durable storage layer), eventual consistency at the edge (read cache with ~60-second propagation delay). Reads are served from in-memory caches at edge nodes, making them extremely fast. Writes go to a centralized origin layer and propagate asynchronously. Cloudflare documented this tradeoff prominently in their developer documentation rather than obscuring it, arguing that explicit eventual consistency causes fewer bugs than implicit eventual consistency that developers discover under load.
Scale at Decision Point
Users
Serving requests across 200+ Cloudflare edge locations; billions of KV reads per day across the customer base
Data Volume
Per-account limits of 1GB storage; keys up to 512 bytes, values up to 25MB; designed for configuration and content, not large object storage
Request Rate
Cloudflare's network handles 35+ million HTTP requests per second across all products; KV represents a subset
Edge read caches at every Cloudflare PoP; centralized durable storage layer; proprietary propagation network using Cloudflare's private backbone
Architecture Evolution
Initial Architecture
Workers KV launched in 2018 as a key-value primitive for Workers scripts to access shared state without external HTTP calls. The initial design used a centralized storage layer with read-through caching at edge nodes. The primary design tension was between consistency (requiring writes to synchronize to all edge nodes before acknowledging) and availability/latency (serving reads from the nearest cache without waiting for propagation). The team chose eventual consistency explicitly after evaluating that strongly consistent distributed KV at global edge scale would require either: a) ~130ms minimum read latency for cross-continent reads (unacceptable for edge computing), or b) a globally synchronous write path that would make writes prohibitively slow.
- Reads may return stale data up to 60 seconds after a write: Workers scripts cannot assume immediate consistency after a put()
- No compare-and-swap (CAS) or transactional operations: concurrent writers can create lost update scenarios
- Value size limit of 25MB excludes large binary storage use cases
- No secondary indexes or range queries: pure key-value access only
Evolved Architecture
Workers KV uses a two-tier architecture: a durable origin storage layer (implemented on Cloudflare's internal infrastructure, reported to use a combination of SQLite and custom storage systems at the origin) and in-memory read caches at each edge PoP. Writes go synchronously to the origin; the origin acknowledges the write as durable. Propagation to edge caches happens asynchronously over Cloudflare's private backbone, typically completing within 60 seconds globally. Edge cache TTLs are set to allow reads to serve from memory without an origin round-trip. Cache invalidation uses Cloudflare's internal notification system to push invalidations to edge nodes following a write. For use cases requiring stronger consistency, Cloudflare introduced Durable Objects (2020): single-instance stateful Workers with linearizable access, at the cost of latency for cross-region access.
- 60-second eventual consistency window is fixed: no mechanism for requesting stronger consistency on specific reads
- Edge cache invalidation is best-effort: network partitions can extend stale read windows beyond 60 seconds
- Origin storage is Cloudflare-managed with no customer visibility into durability guarantees below the SLA level
- Durable Objects (strongly consistent alternative) charge per request and per storage operation, with latency proportional to physical distance from the DO location
Key Transitions
Trigger
Workers scripts needed shared state access: feature flags, A/B test configuration, localization strings: that could not be efficiently fetched from origin on every request. An origin round-trip from a Cloudflare edge node to a customer's data center adds 30-200ms depending on geography, defeating the latency benefit of edge computing. A local cache with eventual consistency was the only model that could deliver sub-millisecond reads without requiring an origin round-trip.
Before
Workers scripts had no shared state access; configuration required either environment variables (static) or origin fetch (high latency)
After
Workers KV providing sub-millisecond reads with 60-second eventual consistency
Outcome
Workers KV enabled a new class of use cases: edge-computed personalization, feature flagging, and content serving without origin round-trips. Developer adoption was strong after the eventual consistency model was clearly documented. Early adopters who assumed immediate consistency after writes encountered stale-read bugs; documentation updates and clear warning messages reduced this.
Lessons
- Explicit eventual consistency documented at launch causes fewer production bugs than implicit eventual consistency discovered under load
- Developers who understand the consistency model build correct code; developers who assume strong consistency build fragile code
- Sub-millisecond edge reads require serving from in-process memory: any network hop reintroduces latency that defeats the edge computing proposition
Trigger
A class of use cases required linearizable access: collaborative editing state, rate limiting counters, auction systems. Eventual consistency in Workers KV was fundamentally incompatible with these use cases regardless of how the eventual consistency window was tuned. The engineering team designed Durable Objects as a complementary primitive with different consistency semantics rather than attempting to strengthen Workers KV.
Before
Workers KV as the only shared state primitive; strongly consistent use cases unsupported
After
Durable Objects providing single-instance linearizable state alongside Workers KV for eventually consistent reads
Outcome
Strongly consistent use cases became possible at the edge with Durable Objects. The two primitives serve different consistency/latency tradeoff points: Workers KV for read-heavy configuration access, Durable Objects for write-coordinated state. Durable Objects were adopted for rate limiting, WebSocket coordination, and multiplayer game state.
Lessons
- A single storage primitive cannot serve all consistency requirements: offering both eventually consistent and strongly consistent primitives is architecturally correct
- Durable Objects trade global low latency for strong consistency: a DO in US-East has ~100ms access latency from Tokyo; applications must be designed with this constraint
- Single-instance consistency (Durable Objects model) is an elegant solution for coordination problems: the instance is the lock
Trigger
Customer feedback revealed that the fixed 60-second eventual consistency window was too long for deployment scenarios: a configuration update deployed via KV would be invisible to edge nodes for up to 60 seconds, creating a visible deployment propagation delay. The team explored reducing the default TTL and adding support for shorter per-key TTLs.
Before
Fixed 60-second edge cache TTL for all KV reads
After
Per-key TTL configuration; minimum effective propagation approximately 60 seconds but tunable for lower-priority caching tiers
Outcome
Deployment use cases improved with shorter TTLs. Cache hit rates in Cloudflare's edge infrastructure were monitored to ensure TTL reductions did not increase origin load beyond acceptable bounds. The fundamental ~60-second propagation floor remained due to the time required for global private backbone propagation.
Lessons
- Eventual consistency window length directly impacts deployment workflows: a 60-second propagation delay is visible to engineers during deployments
- Reducing cache TTLs increases origin read load proportionally: there is a fundamental tradeoff between consistency freshness and origin request rate
Key Lessons
Globally consistent reads at sub-millisecond latency are physically impossible: every globally distributed cache system must choose a consistency model explicitly
The speed of light imposes a minimum round-trip of ~130ms between continents. Any read served in under 1ms must come from local memory, which may be stale. Any system claiming global consistency at sub-millisecond read latency is either lying or serving a single geographic region. Cloudflare's Workers KV acknowledges this constraint openly and builds the product around the documented limitation.
Applicable when: You are designing a globally distributed cache or configuration store and need to decide on a consistency model
Document consistency semantics prominently at launch: developer confusion about consistency is a support burden that grows with user count
Workers KV's engineering team noted that the most common developer mistake was writing to KV and immediately reading back within the same request, expecting the write to be visible. This pattern is correct for strongly consistent stores (Redis, PostgreSQL) but incorrect for eventually consistent stores. Prominent documentation of the 60-second propagation window and guidance against read-after-write patterns reduced support tickets significantly after the first documentation revision.
Applicable when: You are building or operating an eventually consistent system and need to set correct developer expectations
Offering complementary primitives with different consistency guarantees is better than forcing all use cases through one consistency model
Workers KV and Durable Objects coexist because they serve different tradeoffs. Attempting to add strong consistency to Workers KV would have required either performance regression or architectural complexity that would degrade the simple eventually consistent use cases. Cloudflare's decision to build Durable Objects as a separate primitive kept both primitives simple and well-defined.
Applicable when: You are designing storage primitives for a platform and need to decide whether to support multiple consistency models
Technologies
Patterns
Failure Modes Encountered
Related Scenarios
Sources
3 sources are pending verification and have been hidden until a followable citation is available.