Post-Mortem Framework · Capacity: Leader Write Bottleneck in Replicated Systems

Leader Write Bottleneck in Replicated Systems

SEV-3, Limited Impact

Linear propagation · capacity · Affects 0 scenario(s)

Severity Classification

Classified as PARTIAL based on failure mode severity.

Propagation Chain

Origin component

Leader Write Bottleneck in Replicated Systems begins at the source component. Trigger: Write throughput growth approaching single-leader WAL capacity (>50% of the leader''s benchmark throughput sustained for >1 hour).

Immediate (T+0) · Signal: Latency Spike

Downstream dependents

Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.

10–30 minutes via write latency monitoring on the leader (alert on INSERT/UPDATE p99 > 10ms sustained). WAL write rate and disk I/O throughput on the leader node are the primary capacity signals. Replication lag on followers may increase as a secondary symptom when WAL generation rate exceeds replication apply capacity. · Signal: Latency spike, connection timeout, or error rate increase on dependents

Blast Radius

Write latency degradation affects all services that issue writes to the cluster. In a microservices architecture where multiple services share a single primary database, write saturation on one service's write patterns (e.g., audit logging at high rate) degrades write latency for all other services sharing the same leader. Read throughput on replicas is unaffected.

Contributing Factors

Trigger Condition: Write throughput growth approaching single-leader WAL capacioperational

This operational trigger enables Leader Write Bottleneck in Replicated Systems: Write throughput growth approaching single-leader WAL capacity (>50% of the leader''s benchmark throughput sustained for >1 hour)

Trigger Condition: Large write batch (bulk import, end-of-period processing) teoperational

This operational trigger enables Leader Write Bottleneck in Replicated Systems: Large write batch (bulk import, end-of-period processing) temporarily exceeding leader write capacity

Trigger Condition: Addition of write-heavy features (real-time event logging, aoperational

This operational trigger enables Leader Write Bottleneck in Replicated Systems: Addition of write-heavy features (real-time event logging, audit trail inserts) without evaluating leader write headroom

Remediation Plan

ImmediateMeasure current write throughput and leader WAL generation rate to quantify how

Measure current write throughput and leader WAL generation rate to quantify how close to the ceiling the system is

Effort: Minutes to hours (on-call response)

ImmediateIdentify the top-5 write sources by table and query pattern using pg_stat_statem

Identify the top-5 write sources by table and query pattern using pg_stat_statements

Effort: Minutes to hours (on-call response)

ImmediateImplement write batching for the highest-volume low-priority write paths (loggin

Implement write batching for the highest-volume low-priority write paths (logging, events, metrics) within 24 hours

Effort: Minutes to hours (on-call response)

Short-TermApplication-level write batching and coalescing

Buffer writes in the application tier and flush in batches (100–1,000 rows per INSERT via multi-row VALUES syntax or COPY protocol). A single batch INSERT of 1,000 rows generates 1 WAL commit, 1 fsync, and approximately 10x fewer WAL bytes than 1,000 individual INSERTs. For audit logs and event records, batch collection with 50–100ms flush intervals can reduce leader write operations by 50–100x without sacrificing data durability. Requires application-side buffering with crash recovery guarantees (e.g., buffer to local disk before batch write).

Effort: 1 day to 1 week

Short-TermVertical scaling of the leader node

Upgrade the leader to a larger instance class with higher IOPS (NVMe SSD vs EBS gp2), more CPU cores for parallel constraint checking, and more RAM for larger shared_buffers and WAL buffers. A leader upgrade from a 8-core/32GB instance to a 32-core/128GB instance with NVMe storage can increase write throughput by 3–5x. This is the lowest-complexity path when write throughput is approaching but not at the ceiling, buying 12–24 months of headroom before sharding becomes necessary.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: latency spike, disk saturation, cpu saturation. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermHorizontal write scaling via range-based sharding

Partition the write-heavy table across multiple independent database clusters by a sharding key (user_id range, time range, geographic region). Each shard has its own leader, distributing writes across N leaders. Write throughput scales linearly with shard count. Requires application-level shard routing, cross-shard query handling, and a shard rebalancing strategy. Introduces significant operational complexity and is appropriate only when vertical scaling headroom is exhausted.

Effort: 1–4 sprints

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.