Discord Message Storage Migration
Discord migrated from MongoDB to Cassandra to ScyllaDB, driven by the fundamental mismatch between MongoDB's RAM-indexed storage and the channel+time range-scan pattern required for message history.
Discord's message store started on MongoDB and scaled comfortably to ~100M daily messages. By 2017, the index for a single messages collection consumed 11GB of RAM: every message lookup required the entire index in memory. Cassandra's partition-key model (channel_id, bucket) matched the access pattern directly. ScyllaDB later replaced Cassandra to eliminate Java GC pauses that caused p99 spikes under write bursts.
Scale at Decision Point
Users
~150M registered users at MongoDB to Cassandra decision point (2017)
Data Volume
~100M documents in a single MongoDB collection; ~11GB RAM for hot index
Request Rate
Millions of messages per day; exact RPS not disclosed
Single MongoDB primary with replica set; Cassandra cluster introduced in 2017
Architecture Evolution
Initial Architecture
MongoDB with a single messages collection. All messages stored as documents. MongoDB's index-based access required the entire collection index in RAM to serve efficient message history queries.
- Message collection index grew to 11GB: required fully in RAM for efficient range scans
- MongoDB BSON documents not optimized for dense time-series range access
- No native partition-by-channel model: all messages in one collection
Evolved Architecture
ScyllaDB (Cassandra-compatible) with partition key (channel_id, bucket) and clustering key (message_id/Snowflake ID). Time-range scans per channel are native to the data model. ScyllaDB eliminated the Java GC pauses present in the original Cassandra deployment.
- Cassandra/ScyllaDB does not support multi-partition transactions
- Data model migrations are more complex than SQL schema changes
- Wide rows can cause partition hotspots if bucket sizing is wrong
Key Transitions
Trigger
MongoDB collection index reached 11GB: fully in-RAM requirement made further scale economically and operationally unsustainable.
Before
Single MongoDB collection with full index in RAM
After
Cassandra cluster with (channel_id, bucket) partitioning
Outcome
Message history reads became efficient time-range scans without full index in RAM. Data model aligned with the channel+time access pattern.
Lessons
- Access pattern should drive data model design: not just storage capacity
- Dual-write migration enabled zero-downtime cutover at 100M+ message scale
Trigger
Java GC pauses in Cassandra caused p99 latency spikes under write bursts. ScyllaDB's C++ implementation offered the same data model with more predictable latency.
Before
Cassandra cluster with periodic GC-induced p99 spikes
After
ScyllaDB cluster: same data model, C++ runtime, lower p99 variance
Outcome
Eliminated JVM GC pauses. p99 latency became more consistent under write bursts. Same operational model since ScyllaDB is wire-compatible with Cassandra.
Lessons
- Runtime implementation choice (JVM vs C++) matters at scale: not just the data model
- Wire-compatible migrations reduce risk significantly
Key Lessons
Data model must match access pattern: not just store data
MongoDB's document model stored messages fine but required a full sorted index in RAM to serve channel history queries. Cassandra's partition model made channel+time the native access path.
Applicable when: You have a dominant access pattern that does not align with your current data model
GC pauses are a real operational problem at high write throughput
Cassandra's JVM runtime caused p99 spikes under write bursts: predictable in theory, painful in production. ScyllaDB's C++ runtime eliminated the problem without requiring a data model change.
Applicable when: You are running JVM-based databases under sustained high-frequency write workloads
Dual-write migration is the safe path for critical data stores
Discord wrote to both MongoDB and Cassandra simultaneously during the migration window, enabling incremental validation and zero-downtime cutover.
Applicable when: Migrating a write-heavy data store with no tolerance for data loss
Technologies
Patterns
Failure Modes Encountered
Related Scenarios
Sources
2 sources are pending verification and have been hidden until a followable citation is available.