UPDATE 7/16/2025 - Cloudflare posted a detailed post-mortem analysis about their incident here.
TL;DR: A misconfiguration in Cloudflare’s internal systems accidentally linked the 1.1.1.1 IP prefixes to a non-production service topology. When a test location was added, it triggered a global withdrawal of those prefixes from Cloudflare’s data centers - eventually causing the outage.
The hijack detected was not the cause of the outage, but only a latent unrelated issue that surfaced due to the routes being withdrawn.
Catchpoint applauds Cloudflare’s commitment to transparency - sharing detailed insights into this incident not only reinforces trust but also helps the entire Internet community learn and improve.
“It’s always DNS.”
That’s the running joke in IT. When websites won’t load and apps grind to a halt, DNS—the internet’s address book—is often the first to get blamed. That’s because DNS translates human-friendly names like google.com into IP addresses that computers use to route traffic.
Millions of users rely on public resolvers like Cloudflare’s 1.1.1.1, Google’s 8.8.8.8, or Quad9 because they’re usually faster and more reliable than what their ISPs provide.
So, when 1.1.1.1 went dark on July 14, 2025, millions were cut off from the web. Reddit lit up, ISPs got blamed, and users rebooted routers to no avail. But this time? It wasn’t DNS —at least, not directly.
At 17:50 EDT on July 14, Catchpoint’s Internet Sonar detected a sharp and sudden spike in DNS query failures targeting Cloudflare’s public resolver, 1.1.1.1.
From every angle, it looked like a DNS outage: query timeouts, and toward the end of the incident, ServFail errors.
But the underlying cause wasn’t a malfunctioning DNS server—it was a BGP route hijack.
Behind the scenes, the disruption was not caused by DNS failures themselves—but by internet routes being tampered with. During the outage time, we witnessed a mass withdrawal of 1.1.1.0/24 - a range that belongs to Cloudflare - and the announcement of a hijacked route to Tata Communications India (AS4755). This unauthorized announcement propagated through Tata’s global backbone (AS6453), and rapidly spread to many networks worldwide.
This traceroute visualization tells the full story. Each hop is grouped by its Autonomous System Number (ASN) to show how traffic flowed—or failed to.
Cloudflare’s legitimate origin ASN (AS13335) is visible where valid routes remained, but many traceroutes never made it there. A significant number of our tests failed with “no route” errors—suggesting the legitimate path was withdrawn—while others exposed the propagation of a hijacked prefix via unauthorized AS paths.
While Cloudflare had a valid ROA for 1.1.1.0/24, the hijacked route originated by AS4755 was clearly marked as RPKI Invalid.
The chart here shows an example of a bunch of such invalid announcements, captured by Catchpoint’s platform—in this case, thanks to RouteViews collectors—from peers in the United States. Despite being invalid according to the ROAs, the announcements were still accepted and propagated by major networks, including AS6453.
During this period, many agents withdrew the valid route via Cloudflare’s ASN (AS13335). Some of those nodes lost reachability entirely, while others adopted the hijacked route pointing toward AS4755. Several ISPs like Lumen and AT&T had no routes at all for 1.1.1.0/24—suggesting they honored RPKI filtering and dropped the invalid announcements from Tata. However, it remains unclear why there were no other paths for 1.1.1.0/24. Under normal circumstances, we would expect to see Cloudflare’s own announcements reach these networks.
Whether it's better for a resolver to go dark or get silently rerouted through an unauthorized network is debatable—but in this case, many networks took the latter path, with consequences visible to millions of users.
Interestingly, RPKI was not able to save the day in this scenario. Cloudflare indeed was careful enough to create a proper ROA to state that the prefix hosting their DNS service can be originated only by Cloudflare AS number (AS13335), as can be seen here. Tata itself is known to apply filtering based on RPKI, as Cloudflare itself states on the Is BGP safe yet? website. However, it looks like Tata did not apply the RPKI filtering checks on routes originated by AS numbers of their own group, and the hijack/misconfiguration slipped out in the wild. Or as Murphy would say: “If anything simply cannot go wrong, it will anyway”.
Why this caused issues:
The effect for end users was immediate and confusing:
Cloudflare’s DNS outage is a lesson in how fragile the Internet’s routing infrastructure still is—and how easily it can be disrupted when best practices aren’t followed. Here’s what we learned.
Mission-critical services depend on external systems like DNS, BGP, and transit networks. DNS especially is a single point of failure—if it breaks, nothing works, regardless of how healthy your backend is. So, if it feels like “everything on the Internet is down,” there's a good chance it’s DNS. Public resolvers like 1.1.1.1 are foundational to connectivity—and when they become unreachable, even healthy services appear broken. A good monitoring strategy includes monitoring the entire Internet Stack, including protocols BGP and DNS.
After all, your customers won’t know it’s Tata or Cloudflare’s fault. They’ll blame you.
Read: The Need for Speed – Why you should give your network’s DNS performance a closer look
The route hijack that broke access to Cloudflare’s 1.1.1.1 was invalid. It should have been rejected. Networks that properly implemented filters based on RPKI-based origin validation did just that—and their users remained unaffected. No matter if there’s no other way to get to the destination—if the destination is clearly hijacked, you should never take that road. Filtering invalid routes is no longer optional; it's table stakes for a resilient Internet
Outages unfold fast. To get ahead of them, organizations need proactive Internet Performance Monitoring (IPM) powered by AI tools like Internet Sonar and Internet Stack Map. Together, they enable you to swiftly answer the question: “Is it me, or something else?” In the case of the Cloudflare outage, Sonar flagged the issue 23 minutes before Cloudflare posted on their status page—a window that’s invaluable when service continuity and SLAs are on the line.
By the time these platforms light up, the damage is already done. Social chatter is reactive, fragmented, and often misleading. Use data-driven monitoring tools to spot outages before your users do—not after they’ve started complaining.
Read: Is your cloud provider telling you everything, everything, all at once?
And if you're an end user? Always specify a backup DNS resolver from a different provider. For example, if your primary is Cloudflare (1.1.1.1), set Google (8.8.8.8) as your secondary. It’s a simple step that can keep you online even when one DNS service hits trouble.
The Cloudflare 1.1.1.1 outage wasn’t caused by DNS infrastructure failure—but by a breakdown in the trust-based fabric of Internet routing. A single misconfigured BGP announcement from one network, improperly validated and widely propagated, was enough to make a critical service unreachable for millions.
This incident is a reminder that resilience isn’t just about uptime—it’s about visibility, validation, and vigilance across the entire Internet Stack. Whether you operate a global network or rely on one, protecting users from outages like this means doing the hard work: enforcing RPKI, monitoring beyond your perimeter, and responding to failures with precision—not guesswork. There’s still more to understand about how this incident unfolded. We look forward to a detailed RCA from Cloudflare to help the broader community learn from it
Because sometimes, it’s “always DNS” …until it isn’t.
UPDATE 7/16/2025 - Cloudflare posted a detailed post-mortem analysis about their incident here.
TL;DR: A misconfiguration in Cloudflare’s internal systems accidentally linked the 1.1.1.1 IP prefixes to a non-production service topology. When a test location was added, it triggered a global withdrawal of those prefixes from Cloudflare’s data centers - eventually causing the outage.
The hijack detected was not the cause of the outage, but only a latent unrelated issue that surfaced due to the routes being withdrawn.
Catchpoint applauds Cloudflare’s commitment to transparency - sharing detailed insights into this incident not only reinforces trust but also helps the entire Internet community learn and improve.
“It’s always DNS.”
That’s the running joke in IT. When websites won’t load and apps grind to a halt, DNS—the internet’s address book—is often the first to get blamed. That’s because DNS translates human-friendly names like google.com into IP addresses that computers use to route traffic.
Millions of users rely on public resolvers like Cloudflare’s 1.1.1.1, Google’s 8.8.8.8, or Quad9 because they’re usually faster and more reliable than what their ISPs provide.
So, when 1.1.1.1 went dark on July 14, 2025, millions were cut off from the web. Reddit lit up, ISPs got blamed, and users rebooted routers to no avail. But this time? It wasn’t DNS —at least, not directly.
At 17:50 EDT on July 14, Catchpoint’s Internet Sonar detected a sharp and sudden spike in DNS query failures targeting Cloudflare’s public resolver, 1.1.1.1.
From every angle, it looked like a DNS outage: query timeouts, and toward the end of the incident, ServFail errors.
But the underlying cause wasn’t a malfunctioning DNS server—it was a BGP route hijack.
Behind the scenes, the disruption was not caused by DNS failures themselves—but by internet routes being tampered with. During the outage time, we witnessed a mass withdrawal of 1.1.1.0/24 - a range that belongs to Cloudflare - and the announcement of a hijacked route to Tata Communications India (AS4755). This unauthorized announcement propagated through Tata’s global backbone (AS6453), and rapidly spread to many networks worldwide.
This traceroute visualization tells the full story. Each hop is grouped by its Autonomous System Number (ASN) to show how traffic flowed—or failed to.
Cloudflare’s legitimate origin ASN (AS13335) is visible where valid routes remained, but many traceroutes never made it there. A significant number of our tests failed with “no route” errors—suggesting the legitimate path was withdrawn—while others exposed the propagation of a hijacked prefix via unauthorized AS paths.
While Cloudflare had a valid ROA for 1.1.1.0/24, the hijacked route originated by AS4755 was clearly marked as RPKI Invalid.
The chart here shows an example of a bunch of such invalid announcements, captured by Catchpoint’s platform—in this case, thanks to RouteViews collectors—from peers in the United States. Despite being invalid according to the ROAs, the announcements were still accepted and propagated by major networks, including AS6453.
During this period, many agents withdrew the valid route via Cloudflare’s ASN (AS13335). Some of those nodes lost reachability entirely, while others adopted the hijacked route pointing toward AS4755. Several ISPs like Lumen and AT&T had no routes at all for 1.1.1.0/24—suggesting they honored RPKI filtering and dropped the invalid announcements from Tata. However, it remains unclear why there were no other paths for 1.1.1.0/24. Under normal circumstances, we would expect to see Cloudflare’s own announcements reach these networks.
Whether it's better for a resolver to go dark or get silently rerouted through an unauthorized network is debatable—but in this case, many networks took the latter path, with consequences visible to millions of users.
Interestingly, RPKI was not able to save the day in this scenario. Cloudflare indeed was careful enough to create a proper ROA to state that the prefix hosting their DNS service can be originated only by Cloudflare AS number (AS13335), as can be seen here. Tata itself is known to apply filtering based on RPKI, as Cloudflare itself states on the Is BGP safe yet? website. However, it looks like Tata did not apply the RPKI filtering checks on routes originated by AS numbers of their own group, and the hijack/misconfiguration slipped out in the wild. Or as Murphy would say: “If anything simply cannot go wrong, it will anyway”.
Why this caused issues:
The effect for end users was immediate and confusing:
Cloudflare’s DNS outage is a lesson in how fragile the Internet’s routing infrastructure still is—and how easily it can be disrupted when best practices aren’t followed. Here’s what we learned.
Mission-critical services depend on external systems like DNS, BGP, and transit networks. DNS especially is a single point of failure—if it breaks, nothing works, regardless of how healthy your backend is. So, if it feels like “everything on the Internet is down,” there's a good chance it’s DNS. Public resolvers like 1.1.1.1 are foundational to connectivity—and when they become unreachable, even healthy services appear broken. A good monitoring strategy includes monitoring the entire Internet Stack, including protocols BGP and DNS.
After all, your customers won’t know it’s Tata or Cloudflare’s fault. They’ll blame you.
Read: The Need for Speed – Why you should give your network’s DNS performance a closer look
The route hijack that broke access to Cloudflare’s 1.1.1.1 was invalid. It should have been rejected. Networks that properly implemented filters based on RPKI-based origin validation did just that—and their users remained unaffected. No matter if there’s no other way to get to the destination—if the destination is clearly hijacked, you should never take that road. Filtering invalid routes is no longer optional; it's table stakes for a resilient Internet
Outages unfold fast. To get ahead of them, organizations need proactive Internet Performance Monitoring (IPM) powered by AI tools like Internet Sonar and Internet Stack Map. Together, they enable you to swiftly answer the question: “Is it me, or something else?” In the case of the Cloudflare outage, Sonar flagged the issue 23 minutes before Cloudflare posted on their status page—a window that’s invaluable when service continuity and SLAs are on the line.
By the time these platforms light up, the damage is already done. Social chatter is reactive, fragmented, and often misleading. Use data-driven monitoring tools to spot outages before your users do—not after they’ve started complaining.
Read: Is your cloud provider telling you everything, everything, all at once?
And if you're an end user? Always specify a backup DNS resolver from a different provider. For example, if your primary is Cloudflare (1.1.1.1), set Google (8.8.8.8) as your secondary. It’s a simple step that can keep you online even when one DNS service hits trouble.
The Cloudflare 1.1.1.1 outage wasn’t caused by DNS infrastructure failure—but by a breakdown in the trust-based fabric of Internet routing. A single misconfigured BGP announcement from one network, improperly validated and widely propagated, was enough to make a critical service unreachable for millions.
This incident is a reminder that resilience isn’t just about uptime—it’s about visibility, validation, and vigilance across the entire Internet Stack. Whether you operate a global network or rely on one, protecting users from outages like this means doing the hard work: enforcing RPKI, monitoring beyond your perimeter, and responding to failures with precision—not guesswork. There’s still more to understand about how this incident unfolded. We look forward to a detailed RCA from Cloudflare to help the broader community learn from it
Because sometimes, it’s “always DNS” …until it isn’t.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。