Skip to content

Developer tools

A maxed-out data center behind GitHub’s August 17 outage

3 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Coding agents →
Image: github.blog

GitHub’s August 17 outage ran for almost eight hours, and no code change caused it. CTO Vlad Fedorov writes in the postmortem that GitHub’s infrastructure ran out of capacity and the failure cascaded from there, as The New Stack reports. It was GitHub’s second major outage in August.

The postmortem also gives the load figures. GitHub now handles 2.9 billion commits, 130 million merged pull requests and 24 million new repositories a month. In April it was already struggling at 1.4 billion commits a month. The New Stack attributes the jump in large part to the spread of coding agents.

Fedorov explains how the outage unfolded: “Our investigation found that the outage began when traffic reached a new peak, and a critical infrastructure component in our Central US data center failed to scale with it.” The capacity pressure then spread, “causing authentication failures and disrupting multiple GitHub services.” Almost every request passes through authentication, so once it degrades, services that had nothing to do with the original bottleneck go down too.

Retries do a lot of the spreading. When one service slows down, the services that call it time out and send the request again. Each retry adds load to a component that is already overloaded, and when several layers of callers all retry, a short slowdown keeps feeding itself. For now, to avoid a repeat of what triggered the recent outages, GitHub is applying consistent retry limits and retry budgets (a cap on how much of a service’s traffic can be retries) and tweaking timeouts across service-to-service interactions “to prevent retry storms and cascading load,” Fedorov writes.

GitHub has been adding hardware. Fedorov says the company brought 3 million CPU cores and 120 petabytes of high-speed storage online this year, and that its own data centers are full: “We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.” Azure now carries 58% of GitHub’s platform load, up from 12% in May, and serves half of all Git operations. Even with 3 million new cores, one traffic peak was enough to take the platform down.

Fedorov also says hardware isn’t the whole problem: “As the pace and complexity of change increased, our existing operational practices did not keep up.” He says teams and resources have been moved to availability work, including stronger testing, safer rollouts, better observability and more effective alerting, and that “this work is not complete.” Over the longer term, GitHub is isolating critical systems and removing the shared dependencies between them, work Fedorov says is meant to make outages less likely and smaller when they happen.

GitHub’s later August availability report adds detail. Its repair items cover capacity monitoring and management, retry policies that had made the impact of incidents worse, and the resiliency of core services. GitHub moved a 24-table authentication-core cohort off mysql1, its oldest shared database, which took about one million queries per second off that database’s replicas. Separate query cleanup removed another 120,000 queries per second.

For GitHub Actions, routing changes moved 33% of jobs from a constrained production cluster onto spare capacity. That cut peak cache CPU utilization from 98% to 80% and, by GitHub’s estimate, bought about three months of headroom. GitHub calls it “a near-term containment measure, not the finish line.” On August 11, GitHub ran a production MySQL primary from Azure for the first time. Later, on August 27, it did the same with two more primaries.

Competitors are moving in. Entire, the startup founded by former GitHub CEO Thomas Dohmke, is betting on a distributed system built for agent-driven development workloads. Cursor’s Git forge, Origin, rolled out in beta on August 17, the same day GitHub went down.

ClaudeClaude Code

v2.1.238

Plugin marketplaces in Claude Code 2.1.238 can mint their own auth: headersHelper runs a command that issues a short-lived token for catalog fetches, and on a catalog entry it runs only when you install or update.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.