Buffer Pool Churn
SEV-3, Limited ImpactFan-Out propagation · caching · Affects 0 scenario(s)
Severity Classification
Classified as PARTIAL based on failure mode severity. The fan out propagation pattern increases risk of broad impact beyond the initial failure point.
Propagation Chain
Origin component
Buffer Pool Churn begins at the source component. Trigger: A single large analytical/reporting scan or batch export reads far more distinct pages than fit in the buffer pool, evicting the resident working set in one pass.
Immediate (T+0) · Signal: Latency Spike
Downstream dependents
Failure propagates to directly dependent components via synchronous calls or shared resources. Latency increases and error rates rise on affected dependencies.
Minutes, if buffer/cache hit ratio is monitored (pg_stat_database blks_hit vs blks_read, or InnoDB Innodb_buffer_pool_read_requests vs Innodb_buffer_pool_reads) alongside query latency. Without hit-ratio monitoring, the pattern is easy to misattribute to generic load variance for longer, because the correlating event, the large scan, may be a scheduled job that nobody connects to the OLTP latency spike that follows it. · Signal: Latency spike, connection timeout, or error rate increase on dependents
Downstream of dependents (fan-out)
Failure spreads to multiple downstream systems simultaneously. Retry storms may amplify load on the failing component.
Within minutes of initial failure · Signal: Multiple services reporting elevated error rates
Blast Radius
The immediate effect is scoped to whatever shared the buffer pool with the evicting scan: every OLTP query touching the tables and indexes that were resident before the scan sees a latency spike as its next access becomes a disk read instead of a memory hit. Because the buffer pool is shared across all connections, the impact fans out to every client of the instance, not just the one that ran the scan; the scan's own client may finish and disconnect while unrelated OLTP traffic is still absorbing the aftermath. Read replicas and standbys with their own buffer pools are unaffected unless the same scan pattern also runs against them.
Contributing Factors
This operational trigger enables Buffer Pool Churn: A single large analytical/reporting scan or batch export reads far more distinct pages than fit in the buffer pool, evicting the resident working set in one pass
This operational trigger enables Buffer Pool Churn: Concurrent OLTP and analytical/batch workloads on the same instance without workload isolation
This operational trigger enables Buffer Pool Churn: Buffer pool sized below the sustained hot working set (shared_buffers or innodb_buffer_pool_size undersized relative to actual access patterns)
Remediation Plan
Identify the offending scan or query (pg_stat_activity, or the MySQL process list and slow query log) running at or shortly before the latency spike began
Effort: Minutes to hours (on-call response)
Confirm the buffer/cache hit ratio dropped coincident with that scan (pg_stat_database blks_hit/blks_read, or InnoDB buffer pool hit rate from SHOW ENGINE INNODB STATUS)
Effort: Minutes to hours (on-call response)
Cancel or throttle the offending scan if it is still running
Effort: Minutes to hours (on-call response)
Run large scans, reporting queries, and batch exports against a read replica with its own buffer pool rather than the primary serving OLTP traffic. The two workloads no longer compete for the same cache, so a large scan on the replica cannot evict the primary's OLTP working set. The cost is operational: another node to provision, monitor, and keep within an acceptable replication lag for the freshness the analytical workload actually needs.
Effort: 1 day to 1 week
On InnoDB, confirm innodb_old_blocks_time and innodb_old_blocks_pct are at their protective defaults (not disabled) so single-pass scans do not get promoted into the young sublist. On PostgreSQL, the buffer ring strategy for large sequential scans, bulk reads, and VACUUM is automatic and not directly user-tunable, but confirm the operation actually triggering the churn is one the ring strategy covers rather than one that bypasses it. This mitigates the single-scan case; it does not help if the true combined working set is simply larger than the buffer pool.
Effort: 1 day to 1 week
Increase shared_buffers or innodb_buffer_pool_size so the combined OLTP and analytical working set has more room before eviction pressure begins. This does not prevent churn under an arbitrarily large scan, it only raises the threshold at which churn starts. The cost is memory: a larger buffer pool competes with OS page cache and other processes for the same host memory, and on PostgreSQL a much larger shared_buffers also increases checkpoint dirty-page volume (see checkpoint_amplification).
Effort: 1 day to 1 week
Configure alerts for: latency spike, disk saturation. Set thresholds to fire at 70% of critical level to allow response before full failure.
Effort: 1–3 days
Conduct a structured architecture review focused on preventing recurrence. Review topology for blast radius reduction, mitigation coverage, and observability gaps. Consider whether the current architecture scenario should evolve.
Effort: 1–2 sprints
This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.