惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

博客园_首页
IT之家
IT之家
博客园 - Franky
Stack Overflow Blog
Stack Overflow Blog
宝玉的分享
宝玉的分享
Recent Announcements
Recent Announcements
Engineering at Meta
Engineering at Meta
S
SegmentFault 最新的问题
V
Visual Studio Blog
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
Last Week in AI
Last Week in AI
H
Help Net Security
V
V2EX
H
Hackread – Cybersecurity News, Data Breaches, AI and More
量子位
博客园 - 叶小钗
J
Java Code Geeks
博客园 - 【当耐特】
月光博客
月光博客
爱范儿
爱范儿
人人都是产品经理
人人都是产品经理
酷 壳 – CoolShell
酷 壳 – CoolShell
小众软件
小众软件

StatusCake

What the Namecheap Outage Revealed About DNS Recovery - StatusCake Monzo's Stand-In Held Up on Wednesday. Some Customers Still Had a Bad Day. - StatusCake Automate StatusCake Monitoring with viaSocket - StatusCake Website Monitoring Checklist: What to Track Beyond Uptime Turn StatusCake into a verified alerting and escalation flow with Hermes Beyond Uptime: Building a Self-Healing OpenClaw Observability Stack When AWS us-east-1 Fails, Much of the Internet Fails With It In the Age of AI, Operational Memory Matters Most During Incidents AI Didn’t Kill the SDLC. It Made It Harder to See When Code Becomes Cheap: The New Reliability Constraint in Software Engineering Buy vs Build in the Age of AI (Part 3) Buy vs Build in the Age of AI (Part 2) Buy vs Build in the Age of AI (Part 1) Alerting Is a Socio-Technical System Designing Alerts for Action A Notification List Is Not a Team Alert Noise Isn’t an Accident — It’s a Design Decision - StatusCake The Incident Checklist: Reducing Cognitive Load When It Matters Most When Things Go Wrong, Systems Should Help Humans — Not Fight Them
What Broke GitHub on August 17 and How Retries Made the I...
James Barnes · 2026-08-19 · via StatusCake

GitHub’s incident on 17 August 2026 ran from 13:28 to 21:15 UTC, seven hours and forty-seven minutes. At peak, web and API traffic saw error rates of around 20%, while archive and raw-content downloads reached roughly 50%. SAML and OIDC authentication, SCIM and Team Sync were affected alongside Git operations, Actions, Pages, Issues, Pull Requests and Copilot.

GitHub has now published its root cause analysis, and the most interesting part isn’t what failed but what happened after.

The initial failure was contained. The retries made it worse.

GitHub says the immediate cause was network saturation on load balancers in its Central US datacenter during a new peak in traffic, which traced back to an Istio sidecar pod that reached its concurrency limit and failed to autoscale correctly. The autoscaling failed because its scaling policy was watching the capacity of the host service, rather than the sidecar itself. In short, the thing meant to add capacity wasn’t watching the part that had actually run out of it.

That initial failure continued to cascade until four HAProxy nodes exhausted their flow limits, degrading GitHub’s gateway authentication path and causing the widespread authentication latency and failures that made up most of the incident’s visible impact; however the capacity problem was only half the story. GitHub says it was worsened by optimistic retry logic that overloaded its own internal load balancers, and pausing HAProxy on the affected nodes is what produced what GitHub calls an “immediate broad recovery.” The system was already struggling to process requests, so instead of retries helping it catch up, it actually added more work to infrastructure that already couldn’t keep up with what it had.

The same failure pattern appeared again during recovery

GitHub moved some of the affected traffic from Central US to Northern Virginia while the original network problem was being investigated, and at first that worked. Then delayed responses from a single internal endpoint triggered a separate, latent retry bug in VS Code, which amplified traffic to the Copilot Token Service by roughly 10x and delayed its recovery specifically. Whilst normal traffic to that service runs at 7,000–9,000 requests per second; during the incident it reached 70,000–100,000, as a failed token operation generated multiple extra requests that themselves entered a retry loop. GitHub stabilized it the same way it stabilized the first failure: reduce the retry logic, block the responses that were triggering more retries, then ramp traffic back up gradually rather than all at once.

Two unrelated systems, two separate retry mechanisms, the same failure pattern, on the same day.

Scraping traffic made recovery harder too

GitHub also says a number of scraping attacks against codeload endpoints complicated its recovery. Whatever the intent behind that traffic, its effect on an already constrained system was the same as the retry storms: more load arriving at the exact point the system was least able to absorb it. Retry storms and unexpected external traffic are different problems that expose the same weakness, that systems built for normal conditions don’t necessarily behave the same way once capacity is already under pressure.

What this is actually about

None of this is unique to GitHub. Any distributed system that retries failed requests can make a bad situation worse if those retries arrive faster than the recovering service can absorb them, and GitHub’s own follow-up work reflects exactly that: correcting the autoscaling policy, auditing Istio concurrency and scaling limits, reviewing retry and backoff behaviour across gateways and clients, fixing the specific VS Code behaviour that amplified Copilot traffic, and improving load-balancer capacity monitoring and regional failover safeguards.

There’s a wider lesson here for anyone depending on a service like this one. Partial degradation is much harder to spot than a clean outage. A service can keep responding while error rates climb, one endpoint can recover while another stays degraded, authentication can fail intermittently while everything else looks healthy. That’s why monitoring a dependency is worth building around the specific behaviour your own systems actually rely on, rather than a single up-or-down signal, whether that’s the response an API is supposed to return, the health of an authentication path, or response times where latency is what actually matters to you.

External monitoring won’t tell you that an Istio sidecar has hit its concurrency limit, or that a retry storm is building inside a vendor’s own infrastructure. That needs the vendor’s own telemetry, and GitHub’s report is a good example of what that looks like once it’s published. What external monitoring can tell you is whether the dependency is behaving the way you expect it to, from outside that infrastructure, while the vendor is still working out why. Both views matter during an incident. They’re just answering different questions.