Slack Channel Presence and Typing Indicators
Slack replaced polling-based presence with WebSocket-driven push notifications backed by Redis heartbeats, enforcing explicitly lower durability requirements for ephemeral presence state than for durable message records: because losing a heartbeat is acceptable, losing a message is not.
Slack's presence system tracks whether users are active, idle, or away across channels spanning millions of concurrent connections. Early implementations used HTTP polling which created untenable request amplification as workspace size grew. Slack moved to a persistent WebSocket model where clients send 30-second heartbeat pings; Redis stores ephemeral presence state with TTLs slightly longer than the heartbeat interval, so missed heartbeats cause natural expiry rather than requiring explicit cleanup. Message storage uses Vitess-sharded MySQL to provide horizontal write scaling while preserving the relational semantics required for search, threading, and delivery guarantees. The key architectural insight was that presence and messages have structurally different durability requirements: presence state can be reconstructed from the next heartbeat, but a lost message cannot be recovered.
Scale at Decision Point
Users
10+ million daily active users; ~10 million simultaneous WebSocket connections at peak
Data Volume
Billions of messages across millions of channels; presence state is transient and not persisted to disk
Request Rate
Polling model generated ~1 billion HTTP requests per day before migration; WebSocket model reduced server-side fan-out dramatically
Vitess for MySQL horizontal sharding; Redis clusters for presence state; channel server fleet managing WebSocket connection routing
Architecture Evolution
Initial Architecture
HTTP short-polling where clients requested presence state updates every few seconds. Each active client generated one HTTP request per polling interval, multiplied across all channels the user was in. As workspace sizes grew, presence polling became a dominant fraction of total API traffic: the requests were low-value (most returned unchanged state) but unavoidable under a polling model. MySQL stored both messages and presence state, creating mixed-workload pressure on a single store.
- Polling generates O(users * channels * poll_interval) HTTP requests regardless of whether state changed
- Presence query latency bounded by polling interval: up to 5 seconds stale
- MySQL write contention between high-frequency presence updates and durable message writes
- No differentiation between high-durability and low-durability write paths
Evolved Architecture
Persistent WebSocket connections from every client carry bidirectional presence state. Clients send heartbeat pings on a 30-second interval; the server updates a Redis key per user with a TTL of 70 seconds. If a client disconnects or fails to heartbeat, the key expires and the user transitions to offline: no explicit cleanup required. Typing indicators are pushed directly over the WebSocket as transient events, never persisted. Message storage was migrated to Vitess-sharded MySQL to allow horizontal write scaling by workspace, providing physical isolation between large enterprise workspaces and smaller tenants. Channel servers maintain in-memory connection state for fast fan-out to WebSocket subscribers.
- WebSocket connection state is in-memory: a channel server restart requires all clients to reconnect and re-establish presence
- Cross-shard queries for workspace-spanning operations require scatter-gather at the Vitess layer
- Redis presence TTL model introduces a brief window where a client may appear online after clean disconnect
Key Transitions
Trigger
Presence polling had grown to represent the majority of Slack's HTTP traffic. At 10,000-user workspaces with multiple channels, polling intervals of 3-5 seconds generated hundreds of thousands of requests per minute per large workspace. Infrastructure cost and server-side fan-out amplification made polling unscalable beyond current user growth projections.
Before
HTTP polling for presence state every 3-5 seconds per client
After
Persistent WebSocket connections with 30-second heartbeat pings; Redis TTL-based presence expiry
Outcome
Request volume from presence dropped by over 99% on a per-user basis. Presence latency dropped from polling interval (3-5 seconds) to sub-100ms for WebSocket push events. Typing indicators became viable as a product feature because they required no polling: events pushed directly to subscribers over existing WebSocket.
Lessons
- Polling is an emergent bottleneck: it appears cheap per client but accumulates catastrophically at scale
- WebSocket presence reduces infrastructure cost by 99%+ vs polling for large workspaces
- TTL-based expiry in Redis is the correct model for ephemeral presence: it handles disconnects, crashes, and clean logouts uniformly
Trigger
Enterprise workspace growth pushed MySQL write volumes beyond what a single primary could absorb with acceptable latency. Large enterprise customers with 50,000+ users created spiky write patterns around business hours. A single viral workspace could saturate the shared MySQL primary, degrading all other tenants.
Before
Single-primary MySQL with read replicas; all workspaces share one write path
After
Vitess-sharded MySQL with workspace-level shard routing; physical isolation for large tenants
Outcome
Horizontal write scaling beyond single MySQL primary limits. Large enterprise workspaces received dedicated shards, eliminating noisy neighbor interference. Application code required minimal changes: Vitess's query routing is transparent to the application layer for single-shard operations.
Lessons
- Multi-tenant SaaS must isolate large tenants at the storage layer before a viral enterprise customer degrades the shared platform
- Vitess enables MySQL horizontal sharding with minimal application changes for single-shard access patterns
- Workspace-based sharding is a natural key for messaging platforms: most queries are scoped to a single workspace
Key Lessons
Durability requirements must be classified per data type, not per system
Presence state and message records coexist in Slack's system but have radically different durability requirements. Presence is reconstructed automatically from the next heartbeat: a 30-second loss window is acceptable. A missing message is a correctness violation. Storing both in MySQL with identical durability semantics was wasteful and constrained; separating them to Redis (ephemeral) and MySQL (durable) resolved both the performance problem and the data model mismatch.
Applicable when: Your system manages mixed-durability data where ephemeral state coexists with permanent records
WebSocket heartbeat with TTL expiry is the correct presence model at scale
The TTL-based presence model is self-healing: disconnects, crashes, network partitions, and clean logouts all result in key expiry without requiring explicit cleanup logic. The server never has to track connection lifecycle: it only needs to reset the TTL when a heartbeat arrives.
Applicable when: You are building a real-time presence or session tracking system with millions of concurrent connections
Typing indicators are only viable as a product feature because they have zero persistence cost
Typing indicators are transient WebSocket push events with no storage. If Slack had required persistence of typing events to serve them, the feature would be economically unviable: the write volume would exceed message writes by 10-100x at active typing speeds. The architectural decision to treat them as fire-and-forget events unlocked the feature entirely.
Applicable when: You are designing real-time collaborative features and evaluating whether to persist transient interaction signals
Technologies
Patterns
Failure Modes Encountered
Related Scenarios
Sources
3 sources are pending verification and have been hidden until a followable citation is available.