Architecture Review: Gaming Backend Platform

A failover mechanism promotes a new leader without confirming the old one has stopped, so two nodes simultaneously believe they hold the primary role and both accept writes. The two histories diverge, and when the partition that triggered the failover heals, one set of committed transactions must be discarded.

Affects 0 nodes

A subset of distributed system nodes can reach each other but not another subset, splitting the cluster into groups that disagree about the current state. Partition tolerance is not optional for a system spanning more than one node; the real choice a partition forces is between consistency and availability for the duration it lasts.

Affects 0 nodes

Seed 'Connection Pool Pressure Under Load' identifies 4 metrics relevant to connection_exhaustion. Execution preview confirms this risk manifests under modelled load.

Metrics to instrument: active_connections, connection_wait_time_ms, p95_latency_ms

Trigger: Single game server instance running at connection ceiling; horizontal scaling needed but in-memory room state is not accessible across instances; game server crash loses all active game room state with no recovery path; need for zero-downtime game server deploys without disconnecting active players. Migrate from 'Single-server game backend with in-memory game room state' to 'Redis-backed distributed game room state with consistent hashing affinity'. Run Redis-backed state in parallel with in-memory state for 2 weeks, validating that Redis state is identical to in-memory state after every tick. Use Redis as the read source for a canary 5% of rooms before making it the write-authoritative source. In-memory state remains the fallback until Redis state correctness is validated end-to-end.

State serialization discipline must be enforced from the migration day: ad-hoc game state structs that worked in-memory may not serialize correctly to Redis; complete game state round-trip (serialize → Redis → deserialize) must be validated for every state type before in-memory state is removed; Consistent hashing ring changes during active sessions cause room routing to shift; the affinity layer must detect this and maintain existing room-to-server mappings until sessions end

Trigger: Game server instances blocking on PostgreSQL writes at end-of-session (large event batch flush causing player disconnect delays); analytics queries on game event data competing with live game state queries on same PostgreSQL instance; anti-cheat system needing to process event stream in real-time without touching game server code. Migrate from 'Post-game event publishing via direct PostgreSQL writes in game server' to 'Kafka-based post-game event streaming for analytics and anti-cheat'. Introduce the outbox pattern in the game server for post-game events: write events to a local outbox table in PostgreSQL atomically with the session close transaction, then have a relay publish to Kafka asynchronously. This eliminates the blocking Kafka publish in the game session end path.

Kafka introduces at-least-once delivery semantics; anti-cheat and analytics consumers must deduplicate by event_id; duplicate events in the anti-cheat stream can produce false positive flags; Game server must not lose post-game events on crash before Kafka publish completes; outbox pattern on the game server adds write overhead per session end

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Game server instance file descriptor count approaching OS limit (typically 65k open connections); WebSocket accept latency increasing; new connection establishment p99 > 200ms; CPU on game server instances > 70% during peak concurrent player count

Bottleneck: Single game server instance WebSocket connection count limit; OS-level fd_max or application-level connection accept queue saturation. Evolution: Increase OS fd_max to 512k and application connection accept queue depth; tune SO_REUSEPORT to allow multiple accept threads per socket; add game server instances and update consistent hashing ring; the affinity layer automatically routes new game rooms to the new instances as the ring expands: existing rooms are unaffected

Signal: Redis command throughput > 500k/second; Redis CPU > 60%; per-tick Redis write latency p99 > 5ms (above the acceptable state sync threshold); game tick rate visibly dropping below target (30 ticks/second falling to 20) under load

Bottleneck: Game state serialization to Redis per tick producing more writes than expected; unoptimized state struct serialization writing entire state blob on any field change. Evolution: Implement delta state serialization: only changed fields are written to Redis per tick using HSET with only the modified keys, not full state replacement; profile Redis command distribution per game tick to identify specific state fields with high churn; consider moving ephemeral per-tick state (player positions, projectile states) to local server memory with only durable state (scores, inventory changes) written to Redis