Leader Write Bottleneck in Replicated Systems

partial

Summary

In single-leader replication systems, all writes must pass through the leader node. Adding read replicas scales read throughput horizontally but has no effect on write throughput, which remains bounded by the leader's single-node I/O, CPU, and WAL/binlog capacity. As write-heavy workloads grow, the leader becomes the inescapable throughput ceiling for the entire cluster, and no amount of horizontal scaling of replicas can resolve it.

Description

Single-leader replication is the dominant architecture for PostgreSQL streaming replication, MySQL primary-replica, and many distributed databases (MongoDB single primary per replica set, Redis Sentinel). The design enforces strong consistency by routing all writes through one node and replicating them to followers. This provides read scalability (add more replicas) but creates a structural write bottleneck: the leader node is the only node that can commit writes, and its write throughput ceiling is determined by its hardware and configuration.

The ceiling manifests as one of several sub-limits. WAL throughput limit: PostgreSQL can sustain approximately 200–500 MB/s of WAL writes on fast NVMe (10,000–25,000 simple inserts/second on an 8-core instance). This is the hard ceiling regardless of the number of replicas. Commit rate limit: PostgreSQL synchronous commit requires one fsync per commit batch; at 100 commits/second with group commit enabled, the WAL fsync rate is typically not the bottleneck. But at 10,000 single-row commits/second without batching, fsync frequency becomes the bottleneck. CPU limit: write-heavy workloads that include complex constraint checks, trigger execution, or JSONB updates can saturate leader CPU at write rates below the I/O limit.

The failure mode is most acute in systems that outgrow their initial write throughput assumptions. A startup deploys a PostgreSQL single-leader cluster sized for 1,000 writes/second. Two years later, growth has driven writes to 8,000/second. Adding replicas helps reads scale but the leader's WAL write throughput approaches saturation. INSERT latency increases from 2ms to 20ms. Application engineers assume the database is "overloaded" and add caching, which helps reads but does nothing for the write path. The leader remains the bottleneck.

Architectural resolution requires either write distribution (sharding, partitioning) or switching to a multi-leader or leaderless replication model. Both require significant schema and application changes. This is why the leader write bottleneck is often discovered late: it is gradual, takes months or years to become severe, and the common remediation (adding replicas) does not fix it.

Characteristics

Propagationlinear

Time to detect10–30 minutes via write latency monitoring on the leader (alert on INSERT/UPDATE p99 > 10ms sustained). WAL write rate and disk I/O throughput on the leader node are the primary capacity signals. Replication lag on followers may increase as a secondary symptom when WAL generation rate exceeds replication apply capacity.

Blast radiusWrite latency degradation affects all services that issue writes to the cluster. In a microservices architecture where multiple services share a single primary database, write saturation on one service's write patterns (e.g., audit logging at high rate) degrades write latency for all other services sharing the same leader. Read throughput on replicas is unaffected.

Triggers

·Write throughput growth approaching single-leader WAL capacity (>50% of the leader''s benchmark throughput sustained for >1 hour)
·Large write batch (bulk import, end-of-period processing) temporarily exceeding leader write capacity
·Addition of write-heavy features (real-time event logging, audit trail inserts) without evaluating leader write headroom
·Leader hardware upgrade blocked by operational constraints, while write traffic continues to grow

Detection Signals

latency spikedisk saturationcpu saturationreplication lag

Mitigation Strategies

Application-level write batching and coalescingcomplexity: medium

Buffer writes in the application tier and flush in batches (100–1,000 rows per INSERT via multi-row VALUES syntax or COPY protocol). A single batch INSERT of 1,000 rows generates 1 WAL commit, 1 fsync, and approximately 10x fewer WAL bytes than 1,000 individual INSERTs. For audit logs and event records, batch collection with 50–100ms flush intervals can reduce leader write operations by 50–100x without sacrificing data durability. Requires application-side buffering with crash recovery guarantees (e.g., buffer to local disk before batch write).

Vertical scaling of the leader nodecomplexity: low

Upgrade the leader to a larger instance class with higher IOPS (NVMe SSD vs EBS gp2), more CPU cores for parallel constraint checking, and more RAM for larger shared_buffers and WAL buffers. A leader upgrade from a 8-core/32GB instance to a 32-core/128GB instance with NVMe storage can increase write throughput by 3–5x. This is the lowest-complexity path when write throughput is approaching but not at the ceiling, buying 12–24 months of headroom before sharding becomes necessary.

Horizontal write scaling via range-based shardingpreventscomplexity: high

Partition the write-heavy table across multiple independent database clusters by a sharding key (user_id range, time range, geographic region). Each shard has its own leader, distributing writes across N leaders. Write throughput scales linearly with shard count. Requires application-level shard routing, cross-shard query handling, and a shard rebalancing strategy. Introduces significant operational complexity and is appropriate only when vertical scaling headroom is exhausted.

Recovery Steps

1.Measure current write throughput and leader WAL generation rate to quantify how close to the ceiling the system is
2.Identify the top-5 write sources by table and query pattern using pg_stat_statements
3.Implement write batching for the highest-volume low-priority write paths (logging, events, metrics) within 24 hours
4.Schedule leader vertical scaling for the next maintenance window as medium-term relief
5.Begin sharding design and capacity planning if write throughput growth rate projects saturation within 6 months

Estimated recovery time: 1–2 hours for write batching to provide partial relief (observable as write latency decrease within minutes of deployment). Vertical scaling requires a maintenance window of 15–30 minutes. Sharding migration requires weeks to months of engineering work.

Affected Systems

Patterns

read replicashardingwrite ahead log cdccqrs

Technologies

postgresqlmysqlmongodbredis

Basis

Single-leader write bottleneck is an analytically deterministic constraint of single-leader replication; WAL throughput ceiling values are grounded in PostgreSQL benchmarks; the architectural resolution options are well-documented in distributed systems literature