Failure Modes

When a read replica diverges from the primary by silently skipping or misapplying transactions, queries to the replica return factually incorrect data with no error, no exception, and no replication error in the logs. The divergence accumulates undetected over days or weeks until a data audit reveals the discrepancy. All reads served from the diverged replica during this window are potentially incorrect, and no automated mechanism in the replication layer detects or corrects the divergence.

When frequent update and delete operations leave B-tree index pages partially filled, index scans must traverse more pages than the data density justifies, producing read amplification. Storage utilization inflates, index cache hit rates fall, and query latency for index-range scans increases progressively as the table accumulates write churn. The degradation is gradual and often unnoticed until query plans change or periodic REINDEX maintenance is missed.

When long-running batch jobs consume all available database connections or CPU capacity, they starve concurrent OLTP requests of the resources needed for interactive response times. Batch queries hold connections for minutes while scanning large datasets, causing OLTP p99 latency to spike into the seconds as requests queue behind full-table scans in the shared connection pool.

A buffer pool caches hot pages in memory so most reads never touch disk. When the working set the workload actually touches exceeds the buffer pool size, pages are evicted and re-fetched repeatedly: churn. Nothing crashes and no error is logged, but query latency degrades and read I/O rises, because reads that used to be memory hits become disk reads on a recurring basis.

When a widely-shared cached value expires or is invalidated, all concurrent requests that miss simultaneously trigger identical expensive database queries, overwhelming the origin store before any single result can be computed and cached: a positive feedback loop that can collapse the database within seconds.

A failure or degradation in one service causes increased load, held resources, or error propagation in its callers, which in turn degrade their callers, until the failure front propagates through the entire dependency graph and brings down services with no direct dependency on the original failure point.

PostgreSQL's checkpoint process periodically flushes all dirty shared buffer pages to disk, causing a predictable I/O storm at each checkpoint interval that spikes disk utilisation and elevates write transaction latency for the duration of the flush.

Two nodes' clocks disagree at a given moment (skew), and each node's own clock accumulates its own error over time (drift). Systems that assume synchronized, monotonically advancing wall-clock time break in specific, predictable ways when either assumption fails: wrong event ordering, early or late expiration, and leases held by two nodes at once.

Newly started service instances handle their first requests significantly slower than steady-state instances, causing latency spikes when load balancers route traffic to cold pods before they are warmed up.

Production configuration diverges from the intended state through manual changes, partial rollouts, and environment-specific overrides, causing failures that are intermittent, hard to reproduce, and require cross-node comparison to diagnose.

Database or network connections are acquired from the pool but never returned, causing the pool to drain slowly over hours or days until exhausted, manifesting as a gradual degradation rather than a sudden spike.

All database connections in the pool are in use; new requests queue and then time out, causing cascading latency and errors across all dependent services.

When a downstream dependency (database, external API, microservice) experiences increased latency, in-flight requests hold their thread or connection longer, causing the connection pool to exhaust even though request rate has not increased : a latency-to-throughput coupling that amplifies a downstream slowdown into an application-wide outage.

When a shared connection pool is simultaneously used by fast OLTP queries (sub-10ms) and slow analytical or batch queries (seconds to minutes), long-running queries hold connections for their full execution duration, leaving insufficient available connections for the high-rate OLTP traffic. OLTP requests queue at the pool acquisition step, driving p99 latency into the seconds even when the database itself is not overloaded and has capacity to execute additional fast queries.

A slow downstream dependency causes requests to hold connections for the full timeout duration, exhausting the connection pool and triggering retries that amplify load on the already-slow dependency in a feedback loop.

When multi-region replication falls behind during write spikes, reads directed to secondary regions silently return stale data beyond the application SLA. Unlike an outage, the system appears healthy: queries succeed, latency is normal: but users in secondary regions observe data that may be minutes or hours old, with no automatic alerting unless explicit replication lag SLO monitoring is configured.

A query that cannot be answered from one shard forces the application to scatter the query to every shard, gather the partial results, and merge them itself. Latency becomes the slowest of N shards rather than one lookup, connection pool usage becomes N slots instead of one, and the throughput benefit sharding bought for single-shard queries does not apply to this query pattern at all.

When an application repeatedly opens and closes database connections at high rate: due to missing connection pooling, misconfigured pool recycling, or serverless function architecture: the database server is overwhelmed with connection establishment overhead. PostgreSQL forks a new backend process per connection (10–50ms overhead each), and at high connection rates the server hits its max_connections limit and rejects new connections with "too many clients already", even though the database itself has idle capacity for query execution.

Two or more transactions each hold a lock the other needs, forming a cycle in the lock wait-for graph that no participant can escape on its own. The database breaks the cycle by aborting one transaction, surfacing a serialization-class error the application must catch and retry. Under sustained contention, naive immediate retries re-enter the same cycle and amplify it into a retry storm.

The storage device reaches its IOPS or throughput ceiling, causing all disk- dependent database operations to queue behind I/O requests, driving latency from sub-millisecond to hundreds of milliseconds and degrading all database operations simultaneously.

When an application writes to two systems in sequence (database then search index, database then cache, database then analytics store) and the second write fails, the two systems permanently diverge. Without an automated reconciliation mechanism, the inconsistency is silent and persistent: queries to the primary store return the correct state while queries to the secondary store return stale or missing data indefinitely. No alert fires because no individual system reports an error: the inconsistency exists only in the gap between them.

Vector embeddings become semantically stale when source document content changes but the stored embedding is not regenerated: causing semantic search and RAG retrieval to return outdated, incorrect, or misleading results without any error signal, silently degrading the quality of AI-backed features.

When a bulk ETL job reads from the production OLTP database, it competes for shared resources: buffer pool, WAL, CPU, and transaction snapshot slots : causing OLTP query latency to increase during the ETL window. Beyond direct I/O contention, an open long-running ETL transaction prevents autovacuum from advancing its horizon, causing table bloat to accumulate and dead tuple count to grow. OLTP reads slowed by bloated tables compound the initial contention, producing a degradation window that outlasts the ETL job itself.

Events emitted by a producer arrive at a consumer in a different order than they were produced, so a consumer that applies events sequentially reaches an incorrect state, such as processing a "payment confirmed" event before the "order created" event it depends on. The fix is rarely a global order; it is preserving the order of events that are actually causally related, which is a much cheaper guarantee.

When a single logical write event triggers cascading writes to multiple downstream stores: follower timelines, search indexes, analytics pipelines, notification queues: the downstream write volume can exceed the upstream write volume by orders of magnitude. A celebrity user with 10 million followers posting a single item triggers 10 million individual timeline write operations. Executed synchronously, this blocks the write response for seconds to minutes. Executed asynchronously, it fills downstream queues and can saturate storage or write throughput on downstream systems for hours.

A single inbound request triggers N downstream calls, amplifying load on downstream services by the fanout factor. At sustained inbound rates, the amplified load overwhelms downstream services that appear correctly sized for direct traffic.

JVM garbage collection runs frequently and reclaims little heap, or produces stop-the-world pauses, causing latency spikes in Java and Scala services (Kafka brokers, Elasticsearch, Cassandra, HBase).

When Redis evicts a heavily-accessed cache key due to memory pressure under the allkeys-lru eviction policy, all concurrent requests for that key simultaneously miss the cache and issue identical queries to the backing database. Unlike TTL-based thundering herd, this eviction is triggered by memory pressure rather than expiry : meaning it can affect recently-accessed keys and occurs unpredictably as memory utilization crosses eviction thresholds, not at a predictable scheduled time.

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

A sharded database where a small number of shards absorb disproportionately more writes than the rest, because the shard key does not distribute uniformly under the real workload. The hot shard hits its own write ceiling while its siblings sit idle, so the cluster's aggregate write throughput never rises to meet demand. Sharding added operational surface without adding write capacity for that workload.

The query planner chooses to use multiple single-column indexes and merge their results (bitmap AND) rather than using a single composite index, producing higher I/O and buffer cache pressure than a purpose-built composite index would: often because the correct composite index doesn't exist and the planner falls back to the suboptimal index intersection strategy.

When consumer group members crash, restart, or are deployed in rapid succession, Kafka triggers continuous partition rebalancing that prevents any consumer from accumulating enough stable assignment time to make meaningful progress. During the rebalance window all consumption is paused, and if restarts occur faster than the rebalance completes, throughput drops to near-zero while messages accumulate in the partition backlog.

When a Kafka Streams application restarts, it must replay its changelog topic to restore the local RocksDB state store to the last committed position. If the changelog has grown large due to accumulated writes, the restore process can take 10–60 minutes, during which the stream task is unavailable. For applications with high-throughput stateful processing, state restoration on restart becomes the dominant availability risk, not application crashes themselves.

A distributed cluster repeatedly cycles through leader election: each new leader is deposed shortly after taking over: causing the cluster to be unavailable for write operations for most of the storm duration.

In single-leader replication systems, all writes must pass through the leader node. Adding read replicas scales read throughput horizontally but has no effect on write throughput, which remains bounded by the leader's single-node I/O, CPU, and WAL/binlog capacity. As write-heavy workloads grow, the leader becomes the inescapable throughput ceiling for the entire cluster, and no amount of horizontal scaling of replicas can resolve it.

Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.

A transaction that holds a snapshot open far longer than it does real work pins the database's xmin horizon, so VACUUM cannot reclaim any dead tuple newer than the snapshot, anywhere in the cluster. Bloat accumulates, freezing stalls (raising transaction-ID wraparound risk), and rows the transaction modified stay locked. One forgotten transaction degrades the whole database, not just the tables it touched.

In a log-structured merge-tree (LSM) engine, sustained write rate outpaces the background compaction process that merges on-disk SSTables. The number of SSTables a read must check grows (read amplification climbs), and once the backlog crosses an engine-defined threshold, the engine deliberately slows or stops accepting writes to keep worst-case read latency bounded, rather than let read amplification grow without limit.

When REFRESH MATERIALIZED VIEW is executed without the CONCURRENTLY option, it acquires an exclusive lock on the view that blocks all concurrent SELECT queries for the full duration of the refresh. For large views that require minutes to recompute, this causes a complete read outage on the view for the refresh window. Scheduled refreshes on high-traffic views produce predictable, repeated outage intervals that are often misattributed to query plan regressions rather than the lock acquisition pattern.

When total memory demand from a process or the entire host exceeds available physical RAM plus swap, the Linux OOM killer terminates one or more processes to reclaim memory, causing immediate connection loss, data corruption risk if in-flight writes are lost, and process restart overhead.

A query executes without an appropriate index, causing a sequential scan that is orders of magnitude slower than an indexed lookup and saturates CPU and I/O for all other queries on the same database instance.

Application code issues one query to fetch N parent records, then issues N individual queries to fetch each child record, executing N+1 round trips to the database instead of 1–2, multiplying database load proportionally to the result set size.

A subset of distributed system nodes can reach each other but not another subset, splitting the cluster into groups that disagree about the current state. Partition tolerance is not optional for a system spanning more than one node; the real choice a partition forces is between consistency and availability for the duration it lasts.

A co-located workload consumes shared resources (CPU, I/O, memory, network bandwidth) at the expense of other tenants on the same host, database instance, or cluster, causing latency degradation that is invisible in the victim's own metrics but visible in shared infrastructure metrics.

API responses, message payloads, or database result sets grow far larger than expected, causing memory pressure in processing services, timeouts from serialization overhead, and failures in downstream systems that cannot handle the payload size.

When a query uses a B-tree index on a low-selectivity column (one where the filter matches most rows), the index scan reads all or most index pages plus all the corresponding heap pages, performing more total I/O than a sequential scan would. The query planner may choose the index scan based on stale statistics that underestimate the selectivity, causing a query that should take 50ms on a sequential scan to take 5 seconds on an index scan because it reads the heap via thousands of random I/O operations instead of one sequential pass.

A subset of service instances fails while others remain healthy, producing a low aggregate error rate that masks significant per-instance failures and causes consistent errors for specific request patterns or user segments.

Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.

When a downstream service begins rate limiting requests from an upstream service, the upstream's retry logic with insufficient backoff amplifies the request rate : exceeding the rate limit further and potentially pushing the rejection downstream to other upstream callers, producing a cascade of rate-limited retries across the call graph.

In LSM-tree storage engines (Cassandra, RocksDB, LevelDB), a single logical read may require checking multiple immutable SSTables across multiple levels of compaction before the most recent version of a row is found: multiplying I/O by the number of levels checked and producing latency spikes on reads that cross many levels.

A replica in a single-leader replication topology applies changes differently than the primary did, producing a permanent state difference rather than a temporary lag. Reads from the replica return data the primary never contained, not stale data that will eventually catch up, and nothing about the replica's own error state signals that anything is wrong.

Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.

When a distributed saga times out mid-execution and the compensating transactions for already-completed steps cannot be executed because the target services are also unavailable, the system is left in an irrecoverable intermediate state with no automated resolution path. The saga coordinator marks the saga as "compensating" but cannot drive it to a terminal state, resulting in indefinitely stuck business objects (orders, payments, reservations) requiring manual intervention.

The database's actual schema diverges from what the application, and the migration history that is supposed to describe the database, both assume. Unlike schema_migration_lock, which is about a migration blocking traffic while it runs, drift is what happens after: a persistent mismatch that causes query failures, ORM errors, or silent data truncation, sometimes not triggered until a rare code path runs weeks later.

An ALTER TABLE or other DDL statement takes an ACCESS EXCLUSIVE lock that conflicts with every other lock type, including a plain SELECT's ACCESS SHARE. Once that lock request is waiting, every later query on the table queues behind it too, so a DDL statement that is merely waiting, not yet running, is enough to take the table's entire traffic down.

When services sharing a data schema run at different versions at once, during a rolling deployment or a failed upgrade, a reading service encounters data written by a schema version it was not built to handle. The precise question in every such encounter is which of two distinct compatibility directions the change actually needs, and most incidents trace back to a change that only satisfied one of them.

When a table carries many secondary indexes, each INSERT or UPDATE must maintain all indexes, amplifying the write I/O by the number of indexes. At high write throughput (10,000+ inserts/second), index maintenance saturates WAL throughput, storage I/O bandwidth, or shared_buffers write capacity, causing write latency to spike from sub-millisecond to tens of milliseconds. The amplification grows linearly with index count and write rate, making this failure predictable but frequently discovered only after a traffic spike.

A single consumer instance in a Kafka consumer group processes messages significantly slower than its peers, causing partition lag to accumulate on its assigned partitions and triggering consumer group rebalances that temporarily suspend all partition consumption.

A failover mechanism promotes a new leader without confirming the old one has stopped, so two nodes simultaneously believe they hold the primary role and both accept writes. The two histories diverge, and when the partition that triggered the failover heals, one set of committed transactions must be discarded.

An HNSW or IVF vector index built on a corpus grows increasingly inaccurate as new vectors are inserted and the index is not rebuilt: because HNSW's greedy graph construction assumes a representative distribution during build time, and IVF's cluster centroids become stale: degrading recall for queries about recently added content.

Dead tuples from UPDATE and DELETE operations accumulate in PostgreSQL heap pages and index pages when autovacuum cannot reclaim them fast enough, causing table and index storage to grow well beyond the live data size and degrading query performance through wasted I/O on dead pages.

In a multi-tenant system, one tenant's high resource consumption: query load, connection count, write rate, or storage I/O: degrades database or service performance for all other tenants sharing the same infrastructure, violating the implicit isolation guarantee that a shared-infrastructure SaaS product implies.

When a shared outage, network partition, or coordinated recovery event ends, every client, connection, or worker that was waiting or blocked resumes activity at nearly the same instant, producing a synchronized burst of retries, reconnects, or requests that can overwhelm the system just as it is recovering.

PostgreSQL WAL (Write-Ahead Log) generation rate exceeds wal_buffers flush capacity or downstream replica/WAL archive bandwidth, causing write transactions to stall waiting for WAL flush and replication lag to grow unboundedly.

Each logical application write triggers multiple physical writes through index maintenance, WAL generation, MVCC versioning, and replication, causing actual disk IOPS to exceed the provisioned I/O ceiling while the logical write rate appears modest.

Two concurrent transactions each read an overlapping set of rows, each confirms a multi-row invariant still holds, then each writes to a different row. Both commit. Individually every transaction is correct; together they break the invariant that no single one could have broken alone. Snapshot isolation (what PostgreSQL calls REPEATABLE READ) does not prevent it, because the two transactions never write the same row and so never collide. Only serializability, or an invariant enforced by the database itself, closes the gap.

A Kafka consumer becomes a zombie when its processing thread is paused (long GC, application deadlock, blocking external call) while its heartbeat thread continues to run in the background, signaling liveness to the broker. The broker considers the consumer healthy and does not trigger a rebalance. Messages assigned to the zombie's partitions accumulate unprocessed. The consumer group lag grows silently until max.poll.interval.ms (default 5 minutes) is exceeded and the broker finally forces a rebalance: 5 minutes of guaranteed message accumulation per event.