GitHub MySQL at Scale
GitHub scaled MySQL by investing in tooling, building gh-ost for zero-downtime schema changes and adopting Orchestrator (created at Outbrain) for automated HA, rather than migrating away from a technology the team understood deeply.
GitHub's transactional data (issues, pull requests, comments, user accounts) runs on MySQL with read replicas, ProxySQL, and application-level routing. The primary scaling work was tooling: gh-ost to perform online schema changes on tables with hundreds of millions of rows without locking, and Orchestrator to automate MySQL failover without human intervention. The lesson is that deep operational knowledge of a technology compounds faster than migration overhead when the core bottlenecks can be addressed with tooling.
Scale at Decision Point
Users
~28M registered users at time of gh-ost development (~2016)
Data Volume
Tables with hundreds of millions of rows requiring frequent schema changes
Request Rate
Not disclosed publicly; sustained read-heavy load with write bursts on PR/issue activity
MySQL primaries with multiple read replicas per shard; ProxySQL for connection management
Architecture Evolution
Initial Architecture
MySQL primary-replica setup with application-level read routing. Schema changes required table locks, causing multi-minute outages on large tables. Failover was manual, requiring on-call engineer intervention.
- ALTER TABLE on large tables causes multi-minute lock: unacceptable for production
- Manual failover process: on-call engineers must intervene within minutes
- No automated topology awareness: application config must be manually updated after failover
Evolved Architecture
MySQL with gh-ost for online schema changes (shadow table plus CDC-style migration), Orchestrator for automated topology management and failover, ProxySQL for connection routing and read/write splitting.
- MySQL single-primary write path remains the ceiling: no horizontal write scaling
- gh-ost migrations add replication lag during large table changes
- ProxySQL adds a network hop to every query; must be sized correctly for connection throughput
Key Transitions
Trigger
ALTER TABLE on tables with 100M+ rows locked the table for minutes. Deploying schema changes required maintenance windows: incompatible with GitHub's continuous deployment model.
Before
Blocking ALTER TABLE: maintenance windows required for large schema changes
After
gh-ost online schema change via shadow table plus binlog replay: no table lock
Outcome
Schema changes on large tables became routine operations deployable during business hours without maintenance windows. gh-ost was open-sourced and adopted widely across the industry.
Lessons
- Invest in tooling for your existing stack before migrating to a new one
- Online schema change tools are mandatory at scale: not optional
Trigger
Manual MySQL failover required on-call engineers to intervene and update application config within minutes: not sustainable as GitHub scaled.
Before
Manual failover: engineer reconfigures primary endpoint post-failure
After
Orchestrator topology management: automated detection and failover orchestration
Outcome
MySQL failover became automated. Mean time to recovery reduced from minutes (human intervention) to seconds (automated detection and cutover). GitHub adopted Orchestrator (created by Shlomi Noach at Outbrain in 2014, later developed at Booking.com) in 2016 and became its upstream maintainer.
Lessons
- Automated failover is a prerequisite for high availability: manual failover does not scale
- Topology-aware tooling pays for itself in the first major incident it handles automatically
Key Lessons
Invest in operational tooling before migrating to a new database
GitHub built and open-sourced gh-ost, and adopted and maintained Orchestrator (created at Outbrain), rather than migrating to a different database. Deep operational expertise in MySQL compounded faster than a multi-year migration would have.
Applicable when: You are considering a database migration to solve operational pain that tooling could address
Online schema changes are not optional at scale
Once tables reach 100M+ rows, blocking ALTER TABLE is operationally unacceptable. Any team planning to operate at this scale needs an online schema change strategy before reaching it, not after.
Applicable when: Your relational tables are approaching 50M+ rows with active schema evolution
Automated failover must be built before you need it
Orchestrator was built proactively. Manual failover processes always fail under pressure: the engineer is woken up at 3am and must make topology decisions under stress with incomplete information.
Applicable when: You are running stateful primary-replica databases in production without automated failover
Technologies
Patterns
Failure Modes Encountered
Related Scenarios
Sources
- Engineering Bloggh-ost: GitHub's online schema migration tool for MySQL (2016)
- Engineering Blogorchestrator at GitHub (adopted from Outbrain) (2016)