Use Gaming Backend Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Gaming Backend Platform. Traceable to YAML knowledge entities.
Context
Multiplayer game backends must provide consistent game state across all players in a room at p99 < 100ms: a constraint that is orders of magnitude tighter than typical API backends. State divergence between players (two players seeing different positions for the same character) is an immediate product failure that breaks the game. Connection affinity is mandatory: all players in a game room must reach the same authoritative server instance; a load balancer round-robining players to different instances causes state divergence. Network partitions must be handled gracefully: a player losing connectivity must be able to reconnect and receive a state catch-up delta, not a full re-initialization. The system must also scale matchmaking to thousands of concurrent players queuing while maintaining <5s match formation latency. Primary operational risks include: Game server split-brain on network partition: if a game server instance loses connectivity to its Redis state store but retains WebSocket connections to players, it may continue accepting game actions and advancing game state locally while Redis holds the last-known committed state; when connectivity restores, the server's local state and Redis state have diverged, with no automated merge path for deterministic game simulation; Connection affinity failure under rolling deploy: a game server rolling deploy routes new players to updated instances while existing players hold WebSocket connections to the old version; if the new game server version changes the state machine protocol, players in the same room may be running different protocol versions, causing state deserialization failures and silent state divergence; Matchmaking Redis hot key contention: matchmaking queues keyed by game mode and skill bracket result in a small number of Redis keys (e.g., "solo_ranked_platinum") receiving thousands of concurrent LPUSH/LPOP operations per second during peak hours; at high player counts, this key becomes a throughput bottleneck for match formation.
Decision
We will adopt the **Gaming Backend Platform** architecture pattern. This is a high-complexity architecture appropriate for teams at experienced backend team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
An online multiplayer game backend built around authoritative server game state synchronization. Game rooms run as distributed state machines where the server is the single source of truth for game state: client predictions are reconciled against the server state at every tick. WebSocket connections provide low-latency bidirectional communication for state deltas. Redis stores active game room state (in-flight, with sub-millisecond access), session affinity tokens (routing players to the same server instance as their game room), and matchmaking queues. PostgreSQL is the durable store for player profiles, persistent inventory, leaderboards, and achievement records. Kafka carries post-game event streams for analytics, anti-cheat processing, and achievement evaluation. NATS provides low-latency pub/sub for intra-cluster game state broadcasting when multiple game server instances must coordinate on shared state. Event sourcing records every game action as an immutable event for replay, dispute resolution, and anti-cheat audit. Core technology stack: redis, postgresql, kafka, nats.
Accepted Tradeoffs
- ⚠Authoritative server model eliminates state divergence by design but requires all player inputs to travel to the server and back before the client renders the result; client-side prediction mitigates perceived latency but adds reconciliation complexity when predictions are wrong (prediction rollback with position snapping is visible to players)
- ⚠Event sourcing records every game action for auditability and replay, but the event log grows proportionally to game session length and player count; at 100 actions/second per player across 10 players, a 30-minute session produces 1.8M events; PostgreSQL event storage and snapshot cadence must be designed to keep state reconstruction under 200ms for reconnect catch-up
- ⚠NATS for intra-cluster state broadcasting provides microsecond pub/sub latency between game server instances, but NATS is not a durable message bus: messages in transit during a NATS node restart are lost; this is acceptable for ephemeral game state deltas (the next tick replaces the lost delta) but must not carry persistent events
- ⚠Redis for active game room state provides sub-millisecond access but requires explicit state serialization discipline: all game state structs must be serializable to Redis hashes or JSON blobs; complex nested state objects that cannot be efficiently serialized create hidden per-tick Redis write amplification
- ⚠Connection affinity routing via consistent hashing on game room ID means a game server instance crash loses all active rooms on that instance simultaneously; the affinity layer must detect the crash and route reconnecting players to a new instance that can reconstruct state from Redis, introducing a reconnect latency spike of 1–3s
Risks
A failover mechanism promotes a new leader without confirming the old one has stopped, so two nodes simultaneously believe they hold the primary role and both accept writes. The two histories diverge, and when the partition that triggered the failover heals, one set of committed transactions must be discarded.
A subset of distributed system nodes can reach each other but not another subset, splitting the cluster into groups that disagree about the current state. Partition tolerance is not optional for a system spanning more than one node; the real choice a partition forces is between consistency and availability for the duration it lasts.
All database connections in the pool are in use; new requests queue and then time out, causing cascading latency and errors across all dependent services.
A distributed cluster repeatedly cycles through leader election: each new leader is deposed shortly after taking over: causing the cluster to be unavailable for write operations for most of the storm duration.
A subset of service instances fails while others remain healthy, producing a low aggregate error rate that masks significant per-instance failures and causes consistent errors for specific request patterns or user segments.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Gaming Backend Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Gaming Backend Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Gaming Backend Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Gaming Backend Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: WebSocket Connection Ceiling per Instance
Signal: Game server instance file descriptor count approaching OS limit (typically 65k open connections); WebSocket accept latency increasing; new connection establishment p99 > 200ms; CPU on game server instances > 70% during peak concurrent player count
Evolution: Increase OS fd_max to 512k and application connection accept queue depth; tune SO_REUSEPORT to allow multiple accept threads per socket; add game server instances and update consistent hashing ring; the affinity layer automatically routes new game rooms to the new instances as the ring expands: existing rooms are unaffected
Tier 2: Redis Game State Write Amplification
Signal: Redis command throughput > 500k/second; Redis CPU > 60%; per-tick Redis write latency p99 > 5ms (above the acceptable state sync threshold); game tick rate visibly dropping below target (30 ticks/second falling to 20) under load
Evolution: Implement delta state serialization: only changed fields are written to Redis per tick using HSET with only the modified keys, not full state replacement; profile Redis command distribution per game tick to identify specific state fields with high churn; consider moving ephemeral per-tick state (player positions, projectile states) to local server memory with only durable state (scores, inventory changes) written to Redis
Tier 3: Matchmaking Throughput and Latency
Signal: Match formation latency (time from queue join to match start) p95 > 10s at peak player count; matchmaking Redis key contention visible in MONITOR output; match quality degrading (skill bracket widening under pressure) to maintain formation rate; matchmaking queue depth growing despite available game server capacity
Evolution: Move matchmaking logic to a dedicated matchmaking service with its own Redis shard (separate from game room state Redis); implement bracket-level partitioning for matchmaking queues using Redis Cluster to distribute hot bracket keys; use a batch formation algorithm that processes multiple pending players per tick rather than first-in-first-out individual matching; tune skill bracket tolerance as a time-in-queue function (expand bracket after 5s, 10s, 15s waiting)
Tier 4: Event Sourcing Storage Growth and Replay Latency
Signal: PostgreSQL event log table row count growing by >1B rows per week; event replay for player reconnect catch-up taking > 1s (above game state sync SLA); snapshot creation jobs falling behind the event log growth rate; PostgreSQL vacuum falling behind on event log table due to dead tuple accumulation
Evolution: Increase snapshot frequency to every 100 events per room (from every 500); partition the event log by game_session_id and add automated session partition archival to cold storage (S3) on session completion; closed session events are not needed for real-time state reconstruction: keep only the current session's events hot in PostgreSQL; for very high action rates, consider Kafka-backed event log with PostgreSQL storing only snapshots
Migration Path
Single-server game backend with in-memory game room state → Redis-backed distributed game room state with consistent hashing affinity
Single game server instance running at connection ceiling; horizontal scaling needed but in-memory room state is not accessible across instances; game server crash loses all active game room state with no recovery path; need for zero-downtime game server deploys without disconnecting active players
Post-game event publishing via direct PostgreSQL writes in game server → Kafka-based post-game event streaming for analytics and anti-cheat
Game server instances blocking on PostgreSQL writes at end-of-session (large event batch flush causing player disconnect delays); analytics queries on game event data competing with live game state queries on same PostgreSQL instance; anti-cheat system needing to process event stream in real-time without touching game server code
Full event sourcing in PostgreSQL for all game session state → Snapshot-only persistence in PostgreSQL with Kafka for event streaming
PostgreSQL event log table approaching 100B rows; reconnect catch-up replay latency > 2s even with snapshots; vacuum pressure from event log impacting all PostgreSQL query performance; cost of PostgreSQL storage for event log becoming material
Operational Requirements
- Minimum team maturity: Experienced Backend Team: This scenario has high operational complexity. It is recommended for Experienced Backend Team teams or higher.
- Runbooks and alerting for high-severity risks: 4 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.