Blast Radius Analysis · Gaming Backend Platform

Network Partition

criticalContained

network failure · fan out propagation

Blast Radius

0/18 nodes

Time to Detect

Connection refused or timeout errors surface immediately, within seconds. Cluster coordination systems detect node unreachability within their configured failure detection window (typically 5 to 30 seconds). Application-level health checks typically catch it within 10 to 60 seconds.

Preventive Mitigations

Confidence

Strong

Failure Cascade

(no topology nodes mapped for this failure mode)

This failure mode is referenced in the scenario knowledge but has no connected topology nodes. Impact scope is scenario-level, all components should be considered potentially affected.

Severity at this step: critical

Detection Signals

Error Rate SpikeReplication LagAlert

Recovery time estimate: Partition resolution depends on the underlying network issue. Application recovery once the partition heals is typically automatic within seconds for CP systems (Raft re-establishes its leader) and minutes for AP systems (gossip protocol convergence). Manual conflict reconciliation for AP systems with diverged writes can take hours.

Mitigation Checklist(1 preventive, 3 reactive)

Use quorum-based consensus (Raft or Paxos) for anything requiring a single leaderpreventslow

A quorum-based consensus system can only elect a leader with majority agreement, so a partition minority cannot independently elect one and stops accepting writes. etcd, CockroachDB, and Kafka in KRaft mode implement this correctly; the specific failure this prevents, and what happens without it, is detailed in split_brain.

Choose CP or AP deliberately per data store, not by inherited defaulthigh

Coordination state and financial data need CP; user activity feeds and analytics can tolerate AP. The cost of CP is minority-side unavailability during a partition; the cost of AP is divergence that must be resolved after. Pick per data store based on which cost is acceptable for that data, rather than accepting whatever a chosen technology defaults to.

Route writes to a single AZ and reads to any AZmedium

Send writes only to the primary AZ; serve reads from any AZ. On partition, the primary AZ keeps accepting writes at the cost of reads from the isolated AZ going stale or unavailable. AZ-aware load balancing (Kubernetes topology spread constraints) supports this routing.

Test partition behavior with chaos engineeringmedium

Inject network partitions in staging with tc netem or a chaos engineering tool (Chaos Monkey, Chaos Mesh) and verify the system behaves as designed: CP systems stop writes on the minority side, AP systems converge correctly once the partition heals.

Affected Systems

Workloads

High-Throughput OLTPRealtime CollaborationWrite-Heavy Transactional

Blast radius analysis is derived from structured topology and failure mode knowledge. It models structural propagation patterns, not measured production behavior. Actual incident scope depends on runtime conditions, traffic, and recovery actions in place at the time of failure.