Realtime Collaboration Platform
WebSocket-based collaborative editing with Redis Pub/Sub for low-latency event fan-out across stateful gateway instances, PostgreSQL for durable document persistence, and Kafka for cross-region replication and external integrations. Conflict resolution via CRDT or Operational Transforms at the application layer.
Description
Real-time collaboration systems have fundamentally different operational constraints than request-response APIs: connections are long-lived (minutes to hours), state is maintained per connection in memory, and events must be delivered to all users editing the same document within 50–200ms to feel instantaneous.
The WebSocket Gateway is stateful by design: each server holds active WebSocket connections in memory. This means deployment strategies that work for stateless APIs (rolling restart, kill all instances) cause connection drops for all connected users. Graceful drain must close connections with a reconnect signal and wait for clients to reconnect to a new instance before terminating.
Redis Pub/Sub is the fan-out mechanism: each WebSocket server subscribes to channels corresponding to documents with active connections. When user A on Server 1 sends an edit, the server publishes to Redis channel doc:{document_id}. All servers subscribed to that channel push the event to their connected clients. This decouples server instances without requiring direct server-to-server communication.
Conflict resolution is not trivial. Timestamps are unreliable for ordering distributed edits. CRDT (Conflict-Free Replicated Data Types) or Operational Transforms (OT) provide deterministic merge semantics: the same set of concurrent edits always produces the same result regardless of arrival order. This is implemented at the application layer, not the infrastructure layer.
Use Cases
- ·Real-time document editing with multiple simultaneous editors (Google Docs model)
- ·Collaborative whiteboarding and diagramming tools
- ·Shared code editing environments
- ·Live dashboard and analytics with push updates
- ·Multi-player game state synchronization
Scale Profile
Entry Point
10+ concurrent users editing the same document
Sweet Spot
1k–100k concurrent WebSocket connections, hundreds of active documents
Scaling Ceiling
Redis Pub/Sub handles ~1M channels with sub-millisecond latency. Gateway memory is the practical ceiling: 100k connections * 8KB = ~800MB RAM per gateway instance.
Typical RPS
10k–1M WebSocket messages/second (fan-out included)
Architecture Nodes (6)
Web browsers and desktop clients maintaining persistent WebSocket connections. Implement local optimistic updates and reconnect-with-backoff logic.
Stateful WebSocket server. Maintains in-memory connection registry: {connection_id → {user_id, document_id, send_fn}}. Subscribes to Redis channels for each active document. Graceful shutdown must drain connections with reconnect signal.
Two roles: (1) Pub/Sub channels for document event fan-out: ephemeral, events not persisted; (2) Presence tracking with sorted sets: who is online, per-document user list with heartbeat expiry. Redis restart loses all Pub/Sub subscriptions: clients must reconnect.
Durable document persistence. Full document state read on WebSocket connect. Periodic flush from WebSocket server of accumulated edits (every 5–30 seconds). JSONB column for flexible document schema. Optimistic concurrency via version column.
Durable event log for cross-region replication, webhook delivery, and external integrations. WebSocket events published asynchronously: not on the critical path for real-time delivery. Also serves as audit trail for collaborative edit history.
REST/HTTP API for document management operations: create, delete, share, permissions. Separate from the WebSocket gateway: different scaling characteristics. Also handles initial document load before WebSocket connection.
Dependencies (7)
3 critical path edges. Failure on these directly degrades user-facing requests.
WebSocket connection
Persistent WebSocket connection. Client authenticates via JWT on connect. Gateway joins client to document channel in Redis. Full document state loaded from PostgreSQL on first connect.
Timeout: 30s
Document management
HTTP requests for CRUD operations not requiring real-time: create document, manage sharing, load document list.
Timeout: 10s
Edit event publish
On edit received from client: publish to Redis channel doc:{document_id}. All gateway instances subscribed to this channel push event to their clients within 1–5ms.
Timeout: 10ms
Edit event fan-out
Redis Pub/Sub delivers published events to all subscribed gateway instances. Each gateway pushes events to relevant WebSocket connections. Fan-out latency: 1–10ms.
Document state flush
Periodic batch flush of accumulated edits to PostgreSQL every 5–30 seconds. Version-aware upsert with optimistic concurrency. Document state on connect loaded from here.
Timeout: 5s
Edit event audit
Async publish of edit events to Kafka for audit trail, cross-region replication, and webhook delivery. Not on the critical path: does not block WebSocket delivery.
Document reads/writes
HTTP API reads and writes document metadata and content directly to PostgreSQL.
Timeout: 3s
Failure Propagation
How a failure in one component cascades through the system. Each path documents the mechanism and the mitigation that breaks the chain.
Mechanism
Redis restart or failover drops all Pub/Sub subscriptions. WebSocket gateways no longer receive events for documents. Active editors stop seeing each other's edits. Data is not lost: edits are buffered locally and flushed to PostgreSQL, but real-time sync is broken.
Mitigation
Implement Redis reconnect with re-subscription on all active channels. Client-side heartbeat detects stale connection. Redis Sentinel or Cluster for HA failover.
Mechanism
WebSocket server crash drops all in-memory connections and any edits buffered but not yet flushed to PostgreSQL. Clients must reconnect. Buffered edits since last flush are lost.
Mitigation
Reduce flush interval to 5 seconds to minimize data loss window. Client-side local storage backup of unacknowledged edits. Reconnect triggers re-sync from last server-acknowledged version.
Mechanism
PostgreSQL failure prevents new connections from loading document state and halts periodic flush. Real-time collaboration continues on currently connected sessions (in-memory state is sufficient). New sessions cannot join until database recovers.
Mitigation
PostgreSQL HA with automated failover. Alert on new connection failures. In-memory state in WebSocket server provides continuity for existing sessions.
Mechanism
Rolling deploy or Redis restart causes all clients to reconnect simultaneously. Each reconnect loads document state from PostgreSQL and re-subscribes to Redis channels. Spike in PostgreSQL read load and Redis SUBSCRIBE commands.
Mitigation
Exponential backoff with jitter on client reconnect (500ms + random 0–2000ms). Rate limit reconnections per document at gateway level. Pre-warm document cache in Redis on deploy.
Scaling Transitions
Inflection points where this architecture begins to degrade and what the recommended evolution looks like.
Memory per connection (connection state + document context) saturates single gateway instances. 100k * 16KB = ~1.6GB per instance. CPU pressure from event routing.
Recommended Action
Horizontal scale WebSocket gateways. Sticky routing optional (Redis Pub/Sub handles cross-instance fan-out regardless). Monitor connections-per-instance, not total.
Single Redis instance handles ~100k Pub/Sub channels efficiently but memory pressure grows with active channel count. Cross-region Pub/Sub latency unacceptable.
Recommended Action
Redis Cluster with Pub/Sub sharding. Dedicated Kafka topics for cross-region replication. Regional WebSocket clusters with Kafka fan-out for cross-region documents.
Patterns Applied
Architectural Notes
- ·WebSocket gateway graceful shutdown is non-negotiable in production: send close frame with reconnect reason code, wait for in-flight messages to flush, drain all connections. Hard kill causes client-side errors and data loss.
- ·CRDT or OT must be chosen before implementation: they are not interchangeable. CRDT is simpler to reason about but adds overhead per character. OT requires a central operation history but is more memory-efficient for large documents.
- ·Redis Pub/Sub is not a message queue: undelivered messages are dropped if no subscriber is active at publication time. This is correct behavior here since WebSocket servers subscribe before clients connect. Do not use Pub/Sub for events that must be delivered reliably: use Kafka for that.
- ·The flush interval (5–30s) is a durability trade-off. Shorter intervals reduce data loss on crash but increase PostgreSQL write load. Use acknowledgment-based flushing for high-value documents.
Confidence
StrongWebSocket + Redis Pub/Sub + PostgreSQL is the architecture documented by Figma, Notion, and collaborative tool engineering teams. CRDT/OT is well-researched academic and industry practice.